WO2021042490A1

WO2021042490A1 - Offshore current detection method based on binocular camera

Info

Publication number: WO2021042490A1
Application number: PCT/CN2019/115513
Authority: WO
Inventors: 王胜利; 李晓婉; 付庆军; 王思又
Original assignee: 山东科技大学
Priority date: 2019-09-02
Filing date: 2019-11-05
Publication date: 2021-03-11
Also published as: CN110675449B; CN110675449A

Abstract

An offshore current detection method based on a binocular camera, wherein same belongs to the field of computer vision and the technical field of safety assurance. The method comprises the following steps: firstly, training a convolutional neural network for recognizing an offshore current; secondly, during an actual detection process, recognizing, by means of the trained convolutional neural network, whether an offshore current is present in a sea wave image collected by a binocular camera; and finally, locating the recognized offshore current. According to the method, three-dimensional information of an offshore current can be acquired on the basis of binocular stereoscopic vision, and relatively accurate offshore current location information can be acquired in a timely manner; and the method is simple and highly operable, and relevant staff can mark the location of a sand beach on the basis of the three-dimensional information of the offshore current to warn tourists, thereby reducing the occurrence of drowning accidents caused by an offshore current.

Description

An Offshore Current Detection Method Based on Binocular Camera

Technical field

The invention belongs to the computer vision field and the technical field of safety assurance, and specifically relates to an offshore flow detection method based on a binocular camera.

Background technique

The flow rate of offshore stream is mostly 0.3-1 meters per second, and the fastest can reach 3 meters per second. Its length can reach 30-100 meters or even longer. The flow direction is almost perpendicular to the shoreline, which can quickly drag strong swimmers in. Deep water, causing drowning. Offshore currents have become another marine disaster that has caused harm to people's coastal tourism after storm surges and waves. About 90% of seaside drowning is caused by offshore currents. Offshore currents have brought a lot of problems to the attractive maintenance of coastal tourism, beach management, and accident dispute resolution, which seriously affects the healthy development of coastal tourism economy. At present, my country’s technical assessment and safety management of offshore stream disasters have just started, and relevant investigations and assessments, risk assessments, refined forecasts, safety management and public science warnings are extremely lacking; there are also blind spots in the public’s understanding of offshore streams. Misunderstandings, cognitive errors and lack of vigilance have caused a large number of drowning incidents in many hot tourist areas, increased rescue work, and difficulty in managing coastal tourism safety. Therefore, there is a great need for an efficient and simple offshore flow detection method. At present, the traditional detection method for offshore currents is to place buoys or current meters near the shore.

Summary of the invention

technical problem

The solution to the problem

Technical solutions

In view of the above-mentioned technical problems in the prior art, the present invention proposes a binocular camera-based offshore flow detection method, which is reasonable in design, overcomes the shortcomings of the prior art, and has good effects.

In order to achieve the above objectives, the present invention adopts the following technical solutions:

A method for detecting offshore currents based on binocular cameras, including the following steps:

Step 1: Collect the image set for training the convolutional neural network, and train the convolutional neural network;

Step 2: Use the image recognition algorithm of the trained convolutional neural network to recognize the offshore currents in the collected images; judge whether the offshore currents exist;

If: the judgment result is that the offshore current exists, find the characteristic points of the offshore current, and extract the characteristic points of the offshore current; the characteristic points include the midpoint and the left and right edge points of the offshore current;

Or the judgment result is that there is no offshore current, then re-acquire images for processing;

Step 3: Use the binocular positioning algorithm to locate the identified feature points of the offshore current, and use the location of the offshore current feature points to identify the location of the offshore current;

Obtain the three-dimensional coordinates of offshore streams based on binocular stereo vision technology;

Step 4: Based on the unknown location information of the feature points of the offshore stream, divide the dangerous area and feed it back to the relevant staff, who will remind visitors by marking.

Preferably, in step 2, the binocular camera includes left and right cameras. Since the data collected by the left and right cameras are basically the same, a single-sided camera image is used for offshore flow identification.

Preferably, the specific steps for training the convolutional neural network in step 1 are as follows:

Step 1.1: image data preprocessing; specifically including the following steps:

Step 1.1.1: Process the training data collected in Step 1; convert the collected training data into a data format that TensorFlow can recognize;

Step 1.1.2: Add label according to the image, put the image and label in the array, and convert the array into a format that Tensorflow can recognize;

Step 1.1.3: Standardize the image including cropping and supplementation;

Step 1.2: Build a convolutional neural network model based on the Tensorflow framework;

The classic convolutional neural network LeNet-5 model is used; the model is divided into 7 layers: convolutional layer-pooling layer-convolutional layer-pooling layer-fully connected layer-fully connected layer-fully connected output layer; among them, The convolution layer extracts the preliminary offshore current characteristics, the pooling layer extracts the main characteristics of the offshore current, and the fully connected layer summarizes the characteristics of each part;

Step 1.3: Use the built convolutional neural network model to train the neural network that recognizes offshore flows.

Preferably, step 3 specifically includes the following steps:

Step 3.1: Calibrate the left and right cameras;

Use Zhang Zhengyou calibration method to calibrate the camera to obtain the internal and external parameters of the left and right cameras;

Step 3.2: Correct the left and right camera images;

Use the internal and external parameters obtained by the calibrated left and right cameras to perform distortion correction and stereo correction on the image;

Step 3.3: Perform stereo matching on the image;

Use the SGBM algorithm to perform stereo matching on the image, and finally obtain the disparity map;

Step 3.4: Obtain the three-dimensional coordinate information of the offshore current;

Use the disparity map and the internal parameters of the left and right cameras to obtain the depth image, and obtain the three-dimensional coordinates of the feature points of the offshore current according to the camera model and the parameters obtained by the calibration camera;

Step 3.5: Regard the position information of the midpoint of the offshore current as the position of the offshore current, and divide the dangerous area at the left and right edge points of the obtained offshore current, that is, in the area of the offshore current according to the nature of the offshore current, according to the actual situation Carry out the expansion and get the dangerous area.

Preferably, in step 3.3, SGBM is a semi-global block matching algorithm, and the SGBM algorithm specifically includes the following steps:

S1: image preprocessing;

Use the horizontal Sobel operator to process the image, map the pixels to obtain a new image, and preprocess the gradient information of the original image;

S2: Cost calculation;

The cost calculation is divided into two steps: one is the gradient cost calculation of the gradient information of the image obtained after preprocessing through the sampling-based method; the second is the SAD cost calculation of the original image through the sampling-based method;

S3: Dynamic programming;

Accumulate energy in each direction according to the idea of dynamic programming, and then add the matching cost in each direction to get the total matching cost;

S4: Post-processing;

The post-processing part requires uniqueness detection, sub-pixel interpolation and left-right consistency detection.

The beneficial effects of the invention

Beneficial effect

The beneficial technical effects brought by the present invention:

The invention uses binocular cameras to collect images, uses image recognition algorithms based on convolutional neural networks to identify offshore flows, and uses binocular positioning principles to locate offshore flows, providing a new method for offshore flow detection; The method of detecting offshore flows based on binocular cameras can obtain more accurate position information of offshore flows in a timely manner. The method is simple, the labor cost is low, and the operability is strong. The relevant staff can use this method to obtain offshore flows. The location information of the stream marks the location of the beach, reminding tourists to reduce the occurrence of drowning incidents caused by offshore currents, and provides a practical method for coastal tourism management.

Brief description of the drawings

Description of the drawings

Figure 1 is a flow chart of an offshore flow detection method based on binocular cameras;

Figure 2 is a flow chart of the binocular positioning algorithm;

Figure 3 is a flowchart of an image recognition algorithm;

Figure 4 is a schematic diagram of the offshore current model.

Invention embodiment

Embodiments of the present invention

The present invention will be further described in detail below in conjunction with the drawings and specific implementations:

As shown in Figure 1, an offshore current detection method based on binocular cameras includes the following steps:

Step 1: Collect the image set for training the convolutional neural network, and train the convolutional neural network.

Among them, the training data is to provide the basis for subsequent image recognition. It can be a nearshore wave collected by a binocular camera. The image can have offshore currents or no offshore currents. The nearshore wave image data that exists on the network, including offshore currents and not including offshore currents, can also be computer-generated nearshore wave image data of an ideal model of offshore currents.

The specific steps for training the convolutional neural network in step 1 are as follows:

Step 1.1: image data preprocessing; specifically including the following steps:

Step 1.1.1: Process the training data collected in Step 1, and convert the collected training data into a data format that TensorFlow can recognize;

Step 1.1.3: Standardize the image including cropping and supplementation;

Step 2: After the training is completed, perform experimental operations. The collection of actual ocean waves is called actual data. The actual data is the image collected during the actual offshore current observation. Use the image recognition algorithm of the trained CNN convolutional neural network (the process is shown in Figure 2) to identify the offshore currents in the collected images; and determine whether the offshore currents exist;

If: the judgment result is that offshore currents exist, find the characteristic points of offshore currents; the characteristic points include the midpoint and left and right edge points of offshore currents;

Binocular cameras include left and right cameras. Since the data collected by the left and right cameras are basically the same, a single-sided camera image is used for offshore flow identification.

Step 3: Use the binocular positioning algorithm (the process is shown in Figure 3) to locate the identified feature points of the offshore current, and use the location of the offshore current feature points to identify the location of the offshore current;

Specifically, it includes the following steps:

Step 3.1: Calibrate the left and right cameras;

The camera is calibrated by Zhang Zhengyou calibration method to obtain the internal and external parameters of the left and right cameras; Zhang Zhengyou calibration method is a camera calibration method based on a moving plane template. This method is based on a method between the traditional camera calibration method and the camera self-calibration method . Overcoming the shortcomings of the two and combining the advantages of the two. The specific steps include calculating the homography matrix, calculating the internal parameter matrix, calculating the external parameter matrix, and calculating the distortion parameter.

Step 3.2: Correct the left and right camera images;

Step 3.3: Perform stereo matching on the image;

Use the SGBM algorithm to perform stereo matching on the image to obtain the disparity map, and finally obtain the actual three-dimensional coordinate information of the point in the picture. SGBM is a semi-global block matching algorithm, which has the characteristics of good parallax effect and fast speed; the steps of the SGBM algorithm are as follows:

S1: image preprocessing;

S2: Cost calculation;

S3: Dynamic programming;

S4: Post-processing;

Step 3.4: Obtain the three-dimensional coordinates of the offshore current;

Step 3.5: Regard the position information of the midpoint of the offshore current as the position of the offshore current, and divide the dangerous area at the left and right edge points of the obtained offshore current, that is, appropriately expand the area of the offshore current according to the nature of the offshore current, and get Dangerous area, for example: add 5m to the left from the left edge point, and 5m to the right from the right edge point.

Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the essential scope of the present invention shall also belong to the present invention. The scope of protection of the invention.

Claims

A method for detecting offshore currents based on binocular cameras, which is characterized in that it comprises the following steps:

Step 1: Collect the image set for training the convolutional neural network, and train the convolutional neural network;

Step 2: Use the image recognition algorithm of the convolutional neural network to identify the offshore currents in the collected images; judge whether the offshore currents exist;

If: the judgment result is that offshore currents exist, find the characteristic points of offshore currents, and extract the characteristic points of offshore currents; the characteristic points include the midpoint and left and right edge points of the offshore current;

Or the judgment result is that there is no offshore current, then re-acquire images for processing;

Step 3: Use the binocular positioning algorithm to locate the identified feature points of the offshore current, and use the location of the offshore current feature points to identify the location of the offshore current;

Obtain the three-dimensional coordinates of offshore streams based on binocular stereo vision technology;

Step 4: Based on the unknown location information of the feature points of the offshore stream, divide the dangerous area and feed it back to the relevant staff, who will remind visitors by marking.
The method for detecting offshore flows based on binocular cameras according to claim 1, characterized in that: in step 2, the binocular cameras include left and right cameras. Since the data collected by the left and right cameras are basically the same, a single-sided camera is used. Image for offshore flow identification.
The method for detecting offshore flows based on binocular cameras according to claim 1, wherein the specific steps of training the convolutional neural network in step 1 are as follows:

Step 1.1: image data preprocessing; specifically including the following steps:

Step 1.1.1: Process the training data collected in Step 1, and use a set of images collected by a single-side camera for processing; convert the collected training data into a data format that TensorFlow can recognize;

Step 1.1.2: Add label according to the image, put the image and label in the array, and convert the array into a format that Tensorflow can recognize;

Step 1.1.3: Standardize the image including cropping and supplementation;

Step 1.2: Build a convolutional neural network model based on the Tensorflow framework;

The classic convolutional neural network LeNet-5 model is used; the model is divided into 7 layers: convolutional layer-pooling layer-convolutional layer-pooling layer-fully connected layer-fully connected layer-fully connected output layer; among them, The convolution layer extracts the preliminary offshore current characteristics, the pooling layer extracts the main characteristics of the offshore current, and the fully connected layer summarizes the characteristics of each part;

Step 1.3: Use the built convolutional neural network model to train the neural network that recognizes offshore flows.
The method for detecting offshore currents based on binocular cameras according to claim 1, characterized in that: in step 3, it specifically includes the following steps:

Step 3.1: Calibrate the left and right cameras;

Use Zhang Zhengyou calibration method to calibrate the camera to obtain the internal and external parameters of the left and right cameras;

Step 3.2: Correct the left and right camera images;

Use the internal and external parameters obtained by the calibrated left and right cameras to perform distortion correction and stereo correction on the image;

Step 3.3: Perform stereo matching on the image;

Use the SGBM algorithm to perform stereo matching on the image, and finally obtain the disparity map;

Step 3.4: Obtain the three-dimensional coordinate information of the offshore current;

Use the disparity map and the internal parameters of the left and right cameras to obtain the depth image, and obtain the three-dimensional coordinates of the feature points of the offshore current according to the camera model and the parameters obtained by the calibration camera;

Step 3.5: Regard the position information of the midpoint of the offshore current as the position of the offshore current, and divide the dangerous area at the left and right edge points of the obtained offshore current, that is, in the area of the offshore current according to the nature of the offshore current, according to the actual situation Carry out the expansion and get the dangerous area.
The offshore flow detection method based on binocular cameras according to claim 4, characterized in that: in step 3.3, SGBM is a semi-global block matching algorithm, and the SGBM algorithm specifically includes the following steps:

S1: image preprocessing;

Use the horizontal Sobel operator to process the image, map the pixels to obtain a new image, and preprocess the gradient information of the original image;

S2: Cost calculation;

The cost calculation is divided into two steps: one is the gradient cost calculation of the gradient information of the image obtained after preprocessing through the sampling-based method; the second is the SAD cost calculation of the original image through the sampling-based method;

S3: Dynamic programming;

Accumulate energy in each direction according to the idea of dynamic programming, and then add the matching cost in each direction to get the total matching cost;

S4: Post-processing;

The post-processing part requires uniqueness detection, sub-pixel interpolation and left-right consistency detection.