TWI766237B - Object at sea distance measuring system - Google Patents

Abstract

An object at sea distance measuring system distinguish a common object by a plurality of image capture devices obtaining a plurality of sea images with a common object, an image processor and a convolutional neural network algorithm. The object at sea distinction system calculates a distance between the common object and each image capture device.

Description

Maritime object ranging system

The present invention relates to a marine object ranging system that uses a convolutional neural network to identify common objects in sea level images and calculates the distance between the common objects and each image capturing device.

The unmanned ship marine monitoring system has become a very important part of today's development. The unmanned ship maritime monitoring system can be used for military purposes to strengthen the monitoring and management of the sea area or investigate illegal acts; the unmanned ship maritime monitoring system can also be used for civilian purposes to facilitate the management of port transportation or maritime rescue.

However, unmanned ships may encounter obstacles such as collisions of underwater animals, typhoons or large drifting objects when sailing on the sea surface. The current marine monitoring system for unmanned ships can only capture images of the sea level, but cannot identify multiple objects on the sea in real time. What is needed is a method and system for identifying multiple objects at sea.

At present, the existing image recognition is to use a camera to capture, with multiple frames to scan each object in the image captured by the camera one by one, and each frame frames the objects in the image and inputs them to a convolutional neural network CNN) for classification. However, since the size of the object is unpredictable, the size of the frame needs to be changed each time, so it is necessary to scan multiple times to select an appropriate frame to frame the objects of different sizes, which results in a waste of computing resources and cannot immediately know the object. Image recognition cannot meet the needs of unmanned ships to identify objects at sea.

In view of the foregoing, the inventors of the present invention have considered and designed a ranging system for objects at sea, in order to improve the deficiencies of the prior art, thereby enhancing the implementation and utilization in the industry.

In view of the above-mentioned conventional problems, an object of the present invention is to provide a marine object ranging system to solve the problems faced in the conventional technology.

Based on the above objective, the present invention provides a ranging system for marine objects, which includes a plurality of image capturing devices, an image processor and a memory device. A plurality of image capturing devices capture a plurality of sea level images, each sea level image has a common object, and each sea level image has a different viewing angle. The image processor is electrically connected to a plurality of image capturing devices, the image processor divides each sea level image into a plurality of blocks, and assigns each block to a corresponding category. The memory device is connected to the image processor, the memory device stores the convolutional neural network road algorithm, and the convolutional neural network road algorithm is run by the image processor to produce a plurality of frames according to each block to which the common object belongs, and calculate each frame accordingly. Corresponding to the trust probability, the image processor identifies the common object according to the trust probability of each frame corresponding to the common object and each category. Wherein, the image processor knows the distance between the common object and each image capturing device according to the different viewing angles of the plurality of sea level images.

Preferably, the convolutional neural network road algorithm is run by the image processor to calculate the corresponding trust of each frame to which the common object belongs according to the probability that each frame contains the common object and the estimated value of the overlapping area between each frame and the corresponding actual frame. chance.

Preferably, each frame has the center coordinates of the common object and the length and width of the common object.

Preferably, when the image processor determines that the trust probability that the frame corresponds to the common object is 0, there is no common object in the frame.

Preferably, each sea level image has a plurality of common objects, and the number of the common objects and the number of the plurality of blocks are different.

Preferably, the convolutional neural network road algorithm has a complex number of convolution layers, a complex number of max pooling layers and a fully connected layer.

Preferably, the plurality of image capture devices are divided into a left image capture device and a right image capture device, and the plurality of sea level images are divided into left sea level images and right sea level images.

Preferably, the image processor converts the coordinates of the common object from a two-dimensional coordinate system to a three-dimensional coordinate system according to the position of the common object in the left sea level image and the right sea level image, the common object has three-dimensional coordinates, and the image processor according to the three-dimensional coordinates and the positions of the left image capture device and the right image capture device, to know the distance between the common object and the left image capture device and the distance between the common object and the right image capture device.

Preferably, the positions of the common object in the left sea level image and the right sea level image are different, and the difference between the positions of the common object in the left sea level image and the right sea level image is parallax.

Preferably, the image processor calculates the disparity between the left sea level image and the right sea level image using a disparity stream network.

Continuing from the above, the marine object ranging system of the present invention, through the configuration of the convolutional neural network road algorithm and the segmentation of the sea level image, can identify each common object in the sea level image at one time, thereby speeding up the speed of identifying the objects, and calculating each object. The distance between the common object and each image capture device.

The advantages, features, and technical means of achieving the present invention will be more easily understood by being described in more detail with reference to the exemplary embodiments and the accompanying drawings, and the present invention may be implemented in different forms, so it should not be construed as being limited to what is described herein. Rather, the embodiments are provided so that this disclosure will be thorough, complete and complete to convey the scope of the invention to those of ordinary skill in the art, and the invention will only be appended Defined by the scope of the patent application.

It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections You should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer and/or section from another element, component, region, layer and/or section. Thus, "first element", "first feature", "first region", "first layer" and/or "first portion" discussed below may be referred to as "second element", "second feature" , "Second Area", "Second Layer" and/or "Second Section" without departing from the spirit and teachings of the present invention.

Additionally, the terms "comprising" and/or "comprising" refer to the presence of stated features, regions, integers, steps, operations, elements and/or components, but do not exclude one or more other features, regions, integers, steps, operations , elements, components and/or the presence or addition of combinations thereof.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be construed as having definitions consistent with their meanings in the context of the related art and the present invention, and will not be construed as idealized or overly formal meaning, unless expressly defined as such herein.

Please refer to FIG. 1 , which is a block diagram of the marine object ranging system of the present invention. As shown in FIG. 1 , the marine object ranging system of the present invention includes a plurality of image capturing devices 10 , an image processor 20 and a memory device 30 . A plurality of image capturing devices 10 capture a plurality of sea level images SLM, each sea level image SLM has a common object CO, and each sea level image SLM has a different viewing angle. The image processor 20 is electrically connected to the plurality of image capturing devices 10 . The image processor 20 divides each sea level image SLM into a plurality of blocks B, and assigns each block B to a corresponding category C. The memory device 30 is connected to the image processor 20, and the memory device 30 stores the convolutional neural network road algorithm CNN. The convolutional neural network road algorithm CNN is run by the image processor 20 to manufacture a plurality of borders to which the common object CO belongs according to each block B. BB, and calculates the corresponding trust probability CS of each frame BB, and the image processor 20 identifies the common object CO according to the trust probability CS of each frame BB corresponding to the common object CO and each category C. The image processor 20 knows the distance between the common object CO and each image capture device 10 according to different viewing angles of the plurality of sea level images SLM; for example, a sea level image SLM is the view angle of the image capture device 10 . Shot at 60 degrees, another sea level image SLM is shot by the image capture device 10 from a viewing angle of -60 degrees. The image processor 20 compares the common object CO of the two sea level images SLM to obtain the parallax, and the image processor 20 obtains the parallax according to the parallax. The distance between the common object CO and each image capturing device 10 is estimated.

Among them, the convolutional neural network road algorithm CNN is run by the image processor 20 to create a plurality of borders BB to which the common object CO belongs according to the probability that each block B contains the common object CO, and then the convolutional neural network road algorithm CNN is imaged The processor 20 operates to calculate the corresponding trust probability CS of each border BB to which the common object CO belongs, according to the probability that each border BB contains the common object CO and the estimated value of the overlapping area between each border BB and the corresponding actual frame. The image processor 20 Identify the common object CO according to the trust probability CS of each frame BB corresponding to the common object CO and the category C to which each frame BB belongs; in addition, when the image processor 20 determines that the trust probability of the frame BB corresponding to the common object CO is 0, Border BB does not have a common object CO. Through the configuration of the convolutional neural network road algorithm CNN and the segmentation of the sea level image SLM, the common object O of the sea level image SLM is identified, and the speed of identifying the object O is accelerated.

It should be noted that, because not every block B in the sea level image SLM has a common object CO, a plurality of frames BB frame a single common object CO at different positions and directions, and each frame BB has a common object CO in the area. The center coordinates of the block B and the length and width of the common object CO relative to the sea level image SLM; furthermore, the estimated overlap area is the area of the intersection of the bounding box BB and the corresponding actual box divided by the union area.

It should be mentioned that each sea level image SLM has multiple common objects CO, and the number of common objects CO is different from the number of multiple blocks B. The identification of a single common object CO has been described in the preceding paragraph, and the other common objects CO Recognition is the same mechanism and will not be repeated here. In addition, the convolutional neural network road algorithm CNN is run by the image processor 20 to produce a plurality of frames BB to which each common object CO belongs according to the probability that each block B contains a plurality of corresponding common objects CO, and then the convolutional neural network The roadmap algorithm CNN is run by the image processor 20 to obtain a plurality of corresponding frames BB according to the probability that each frame BB contains a plurality of corresponding common objects CO and the estimated value of the overlapping area between each frame BB and the corresponding actual frame. The trust probability CS of the common object CO, the image processor 20 identifies a plurality of common objects CO according to the trust probability CS of each frame BB corresponding to the common object CO and the category C to which each frame BB belongs; in short, the convolutional neural network roadshow The algorithm CNN is used to simultaneously identify multiple common objects CO in the sea level image SLM and match the corresponding category C to speed up the recognition speed of the entire common object CO.

Continuing, the convolutional neural network road algorithm CNN continuously accelerates the recognition of common objects CO through machine learning, and can use the more important image features (such as shape, color or texture) in the common object CO as It can be used as the basis for CO identification of common objects, thereby saving the screening of artificial image features; the convolutional neural network roadmap algorithm CNN is continuously trained through the input of sea level image SLM for many times, so that the convolutional neural network roadmap algorithm CNN is continuously trained. CNN filters out suitable image features, weights, and classification criteria.

Please refer to FIG. 2 to FIG. 5, which are the structure diagram of the first aspect of the convolutional neural network algorithm of the present invention, the schematic diagram of the convolutional layer of the present invention, the schematic diagram of the maximum pooling layer of the present invention, and the schematic diagram of the present invention. Schematic diagram of the fully connected layer. As shown in Fig. 2 to Fig. 5, the convolutional neural network road algorithm CNN of the present invention includes a plurality of convolutional layers CL, a plurality of maximum pooling layers MPL and at least one fully connected layer FCL. Each maximum pooling layer MPL is set Between the convolutional layers CL of each layer, the fully connected layer FCL is connected to the last operation layer of the convolutional layers of the plurality of layers CL and connects a plurality of neurons NE.

As shown in Figure 3 and in conjunction with Figure 2, each convolution layer CL is connected to each other, the matrix corresponding to the sea level image SLM is the pixel matrix PM, and the convolution layer CL uses the filter layer F1 matrix and the pixel matrix PM corresponding to the pixel matrix PM. The elements are multiplied correspondingly to calculate the feature matrix EM of the sea level image SLM. As shown in Figure 4, and with Figures 2 to 3, the feature matrix EM is multiplied by the corresponding elements of the filter layer F2 matrix of the corresponding maximum pooling layer MPL to reduce the dimension of the feature matrix EM, and then The dimensionality-reduced feature matrix EM is obtained, thereby reducing the computational complexity of the convolutional neural network road algorithm CNN. It should be noted that the descriptions of the above-mentioned Figures 2 and 3 are only for the single-layer convolutional layer CL and the single-layer maximum pooling layer MPL, and the other convolutional layers CL and maximum pooling layers MPL are the same, and are not described here. Repeat the narration.

Through the operation of the multi-layer convolutional layer CL and the multi-layer maximum pooling layer MPL, the important image features of the pixel matrix PM (such as the color or shape of the common object CO) are screened, and the convolutional neural network algorithm CNN The convolutional layer CL and maximum pooling The number of layers of the layer MPL is adjusted according to actual needs, and the number of layers of the convolutional layer CL and the maximum pooling layer MPL is not limited here.

In fact, usually the feature matrix EM after the matrix operation of the filter layer F1 of the convolution layer CL will be multiplied by an activation function, and the activation function will retain the obvious important image features of the pixel matrix PM, and then remove it. Or reduce no obvious image features.

As shown in Figure 5 and with Figure 2, finally the feature matrix EM of the multi-layer convolution layer CL and the maximum pooling layer MPL is flattened into the feature map FM through the fully connected layer FCL, and the values in the feature map FM are The operation of the corresponding weight W is output to a plurality of neurons NE corresponding to each block B, and each neuron NE includes the probability of corresponding multiple common objects O according to the feature map FM after the operation and each block B, respectively, Create a plurality of bounding boxes BB to which each common object CO belongs, and then each neuron NE operates to estimate the overlapping area between each bounding box BB and the corresponding actual box according to the probability that each bounding box BB contains a corresponding plurality of common objects CO, respectively, Obtaining the trust probability CS of the plurality of common objects CO corresponding to each frame BB, the image processor 20 identifies the plurality of common objects CO according to the trust probability CS of each frame BB corresponding to the common object CO and the category C to which each frame BB belongs, So as to achieve the purpose of identifying the common objects CO of the sea level image SLM.

It should be mentioned that, in one embodiment, a plurality of image capture devices 10 are installed in the unmanned ship, and the image processor 20 is installed in a remote computer; therefore, the plurality of image capture devices 10 and the image processor 20 Corresponding wireless transceivers are set up for all of them to transmit sea level image SLM wirelessly. In another embodiment, a plurality of image capture devices 10 are installed in the unmanned ship, the image processor 20 is installed in a remote computer, and has a cloud server, the cloud server, the image capture device 10 and the remote computer Computer network connection; a plurality of image capture devices 10 transmit the sea level image SLM to the cloud server, and the remote computer is connected with the cloud server to obtain the sea level image SLM. The foregoing configuration is only an example, and of course other preferred configurations are also possible, and are not limited to the scope of the present invention.

Please refer to FIG. 6 , which is a flowchart of the marine object ranging system of the present invention. As shown in FIG. 6 , together with FIGS. 1 to 5 , the method for identifying marine objects of the present invention is described as follows: (1) Step S11 : use a plurality of image capturing devices 10 to capture sea level images SLM, each sea level image The SLM has a common object CO, and the number of common objects CO can be multiple. In order to simplify the description, the following steps only use a single common object CO to describe the identification process, and the other common objects CO are also the same identification process. (2) Step S12: the image processor 20 divides each sea level image SLM into a plurality of blocks B, and the image processor 20 assigns each block B to a corresponding category C. (3) Step S13: using the convolutional neural network road algorithm CNN of the memory device 30, the convolutional neural network road algorithm CNN is run by the image processor 20 to manufacture the common object CO according to the probability that each block B includes the common object CO The plurality of bounding boxes BB belong to, and then the convolutional neural network road algorithm CNN is run by the image processor 20 to calculate according to the probability that each bounding box BB contains the common object CO and the estimated value of the overlapping area between each bounding box BB and the corresponding actual box. The corresponding trust probability CS of each frame BB to which the common object CO belongs; in addition, when the image processor 20 determines that the trust probability of the frame BB corresponding to the common object CO is 0, there is no common object CO in the frame BB. (4) Step S14: The image processor 20 identifies the common object CO according to the trust probability CS of each frame BB corresponding to the common object CO and the category C to which each frame BB belongs.

個區塊B，影像處理器20以不同的顏色將各區塊B分類至相應的類別C。卷積神經網路演算法CNN被影像處理器20運行以根據各區塊B含括相應的多個共同物件CO的機率，製造各共同物件CO所屬的複數個邊框BB，接著卷積神經網路演算法CNN被影像處理器20運行以根據各邊框BB分別含括相應的多個共同物件CO的機率和各邊框BB和相應的實際框之重疊區域估計值，取得各邊框BB所對應的多個共同物件CO之信任機率CS(邊框的粗細即表示信任機率CS的高低)，而各邊框BB之信任機率CS乃根據下列公式取得，公式如下：

此公式為根據Redmon, J., Divvala, S.,Girshick, R., & Farhadi, A.於IEEE的2016期刊所發表的You Only Look Once: Unified, Real-Time Object Detection之論文，另外，本發明所述之卷積神經網路演算法也根據此論文的架構來應用於海上物件測距系統；其中，CS為共同物件CO於類別C標準答案之物件機率值，

) 指的是共同物件CO為類別C的機率值

) 所指的區塊B是否存有共同物件CO的機率值，通常以1 或 0表示，1代表區塊B有共同物件CO，0代表區塊B無共同物件CO，

所指的是正確答案的框(實際框)與偵測物件出來的框(邊框BB)所含蓋範圍

。最後，影像處理器20根據各類別C和各邊框BB之信任機率CS，辨識各共同物件CO(如第7圖的紫色邊框、黃色邊框以及綠色邊框內的共同物件CO)。Please refer to FIG. 7 , which is a flow chart of the operation of the convolutional neural network road algorithm of the present invention. As shown in FIG. 7 , and in conjunction with FIGS. 1 to 6 , the image processor 20 divides the image into

block B, the image processor 20 classifies each block B into a corresponding category C with different colors. The convolutional neural network road algorithm CNN is run by the image processor 20 to create a plurality of borders BB to which each common object CO belongs according to the probability that each block B contains the corresponding plurality of common objects CO, and then the convolutional neural network road algorithm The CNN is run by the image processor 20 to obtain a plurality of common objects corresponding to each frame BB according to the probability that each frame BB contains the corresponding plurality of common objects CO and the estimated value of the overlapping area between each frame BB and the corresponding actual frame The trust probability CS of CO (the thickness of the frame indicates the level of trust probability CS), and the trust probability CS of each frame BB is obtained according to the following formula, the formula is as follows:

This formula is based on the paper You Only Look Once: Unified, Real-Time Object Detection published by Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. in the 2016 journal of IEEE. In addition, this paper The convolutional neural network road algorithm described in the invention is also applied to the maritime object ranging system according to the structure of this paper; wherein CS is the object probability value of the common object CO in the standard answer of category C,

) refers to the probability value of the common object CO being class C

) refers to the probability value of whether the block B has a common object CO, usually expressed as 1 or 0, 1 means that the block B has a common object CO, 0 means that the block B has no common object CO,

It refers to the coverage of the correct answer box (actual box) and the box (border BB) from the detected object

. Finally, the image processor 20 identifies each common object CO (such as the common object CO in the purple border, the yellow border, and the green border in FIG. 7 ) according to the trust probability CS of each category C and each frame BB.

、

、

、

及

之矩陣)、4層最大池化層MPL(

)以及2層全連結層FCL，其中各卷積層CL還包括減少層RL以減少特徵矩陣EM的參數；像素矩陣PM經過24層卷積層CL、4層最大池化層MPL以及2層全連結層FCL之運算及影像處理器20的識別後，而取得

之張量來表示如第8圖所示的輸出偵測結果。另外，前述卷積神經網路演算法僅為例舉，本發明之卷積神經網路演算法CNN之卷積層CL、最大池化層MPL、全連結層FCL以及減少層RL之數目乃能根據實際需求而調整，而未侷限於本發明所列舉的範圍。Please refer to FIG. 8 , which is a structural diagram of the second aspect of the convolutional neural network road algorithm of the present invention. As shown in Fig. 8, actually, the convolutional neural network road algorithm CNN of the present invention includes 24 convolutional layers CL (

,

,

,

and

matrix), 4-layer max pooling layer MPL (

) and 2 fully connected layers FCL, wherein each convolutional layer CL also includes a reduction layer RL to reduce the parameters of the feature matrix EM; the pixel matrix PM passes through 24 convolutional layers CL, 4 maximum pooling layers MPL and 2 fully connected layers After the calculation of the FCL and the recognition of the image processor 20, the

tensor to represent the output detection result as shown in Figure 8. In addition, the aforementioned convolutional neural network road algorithm is only an example, the number of convolutional layer CL, maximum pooling layer MPL, fully connected layer FCL and reduction layer RL of the convolutional neural network road algorithm CNN of the present invention can be based on actual needs. and adjustment, but not limited to the scope of the present invention.

其中，y表示各邊框BB的資料，線性整流激活函數ReLU將各邊框BB中過小的資料壓縮成0，其可能造成過小的資料無法別識別且不可恢復；帶洩漏線性整流激活函數LReLU將各邊框BB的資料乘上一個很小的權重α(0 >

1)，而非直接把過小的資料壓縮成0。透過線性整流激活函數ReLU和帶洩漏線性整流激活函數LReLU的運算，壓縮各共同物件O所屬的複數個邊框BB的資料。Please refer to FIG. 9, which is a function diagram of the linear activation function and the leakage linear rectification activation function of the present invention. According to Fig. 7 and Fig. 8, with Fig. 5, the pixel matrix PM outputs the feature map FM to a plurality of neurons after the operation of the multi-layer convolution layer CL, the multi-layer max pooling layer MPL, and the multi-layer fully connected layer FCL. NE, each neuron NE manufactures a plurality of borders BB to which each common object CO belongs according to the calculated feature map FM and the probability that each block B contains a plurality of corresponding common objects O, and then each neuron NE runs with According to the probability that each frame BB contains a plurality of corresponding common objects CO and the estimated value of the overlapping area between each frame BB and the corresponding actual frame, the trust probability CS of the plurality of common objects CO corresponding to each frame BB is obtained. The element NE then uses the linear rectification activation function ReLU and the leaky linear rectification activation function LReLU as shown in Fig. 9 to compress the data amount of the multiple frames BB to which each common object CO belongs; finally, with the recognition of the image processor 20, the output is as follows The detection result shown in FIG. 8 is output, and then each common object CO is identified. Among them, the formulas of the linear rectification activation function ReLU and the leakage linear rectification activation function LReLU are as follows:

Among them, y represents the data of each frame BB, and the linear rectification activation function ReLU compresses the data that is too small in each frame BB to 0, which may cause the data that is too small to be identified and irrecoverable; the linear rectification activation function with leakage LReLU compresses the data of each frame. The data of BB is multiplied by a small weight α (0 >

1), rather than directly compressing too small data into 0. Through the operation of the linear rectification activation function ReLU and the leakage linear rectification activation function LReLU, the data of the plurality of frames BB to which each common object O belongs is compressed.

Please refer to FIG. 1 again, the plurality of image capture devices 10 are divided into a left image capture device and a right image capture device, and the plurality of sea level images SLM are divided into left sea level images and right sea level images; The capture device is equivalent to the left eye of a person and has a left imaging plane, the right image capture device is equivalent to the right eye of a person and has a right imaging plane, and the left image capture device has a corresponding right image capture device (ie, the left The image capture device and the right image capture device are set accordingly), the viewing angles of the sea level images SLM captured by each image capture device 10 are different, resulting in the location of the common object CO captured by each image capture device 10 The locations are also different.

Continuing, the image processor 20 converts the coordinates of the common object CO from the two-dimensional coordinate system to the three-dimensional coordinate system according to the positions of the common object CO in the left sea level image and the right sea level image, so that the common object CO has three-dimensional coordinates, and Because the positions of the common object CO in the left sea level image and the right sea level image are different, the position difference of the common object CO in the left sea level image and the right sea level image is parallax. The position and parallax of the right image capturing device are used to obtain the distance between the common object CO and the left image capturing device and the distance between the common object CO and the right image capturing device.

For example, the method for obtaining the parallax: the image processor 10 uses a parallax flow network (Disflownet) to estimate the parallax between the left sea level image and the right sea level image. In detail, the parallax flow network receives the left sea level image and the right sea level image and superimposes them, and then the parallax flow network uses the multi-layer convolution layer CL of the convolutional neural network road algorithm CNN it has (this convolutional neural network road algorithm). As shown in Figure 2 or Figure 8) and the deconvolution calculation, multiple common objects CO can be obtained, and then the parallax can be calculated according to the positions of the multiple common objects CO in the left sea level image and the right sea level image. The flow network is trained repeatedly for many times, so that the disparity flow network can calculate the disparity faster and faster. Specifically, the parallax flow network overlays the left sea level image and the right sea level image, and the first and second convolutional layers CL receive the left sea level image and the right sea level image and obtain features with multiple common objects CO and their image features feature map, and then use the image map and feature map calculated by the first layer of convolutional layer CL to perform deconvolution and input the result to the third layer of convolutional layer CL. At this time, the third layer of convolutional layer CL also receives the first The image map after the operation of the second-layer convolutional layer CL, the third-layer convolutional layer CL outputs another feature map accordingly, and then the other convolutional layers CL also calculate the feature map according to the aforementioned mechanism, and finally obtain multiple common objects CO (at this time The number of common objects CO is different from the common objects CO extracted from the first and second convolutional layers CL) and its image characteristics. The disparity flow network calculates the positions of the multiple common objects CO in the left sea level image and the right sea level image. out of parallax.

It should be noted that the above-mentioned left image capture device and right image capture device are only for classification, and the left sea-level image, right sea-level image, left imaging plane and right imaging plane are also classifications, and they are not quantitative. limits.

其中，

和

分別為各影像擷取裝置10原點至海平面影像SLM中心在x方向和y方向的偏移差(其中，影像擷取裝置10的原點為影像擷取裝置10的正中央位置，海平面影像SLM中心為影像擷取裝置10所拍攝的海平面影像SLM的正中央)，

和

為各影像擷取裝置10於x方向和y方向的焦距，

為成像方向(成像方向為將左海平面影像和右海平面影像呈現水平所拍攝到的海平面影像，由於左影像擷取裝置以及右影像擷取裝置所拍攝到海平面影像SLM可能會有角度導致交叉重疊，最理想狀態是左影像擷取裝置以及右影像擷取裝置是一致成水平拍攝)與海平面影像SLM的歪斜係數(skew factor)，

為海平面影像SLM的座標，

為實際空間的座標，

為齊次轉換矩陣，R為旋轉矩陣(rotation matrix)，t 為平移的向量(translation vector)。Please refer to FIG. 10, which is a schematic diagram of parallax of the marine object ranging system of the present invention. First, since the sea level image SLM captured by the image capture device 10 is a planar image, a conversion formula between the planar coordinates in the planar image and the actual space coordinates (stereocoordinates) is required to coordinate the objects of the sea level image SLM. The aforementioned conversion formula as follows:

in,

and

are the offset differences between the origin of each image capture device 10 and the center of the sea level image SLM in the x direction and the y direction (wherein, the origin of the image capture device 10 is the center of the image capture device 10, and the sea level The center of the image SLM is the center of the sea level image SLM captured by the image capture device 10),

and

are the focal lengths of each image capturing device 10 in the x-direction and the y-direction,

is the imaging direction (the imaging direction is the sea level image captured by presenting the left sea level image and the right sea level image horizontally, because the sea level image SLM captured by the left image capture device and the right image capture device may be angled. Cross-overlap, the ideal state is that the left image capture device and the right image capture device are uniformly shot horizontally) and the skew factor of the sea level image SLM,

are the coordinates of the sea level image SLM,

are the coordinates of the actual space,

is a homogeneous transformation matrix, R is a rotation matrix, and t is a translation vector.

As shown in Fig. 10, the sea level image SLM is captured by the left image capture device and the right image capture device (corresponding to the left imaging plane and the right imaging plane), and the sea level image SLM is converted by the conversion formula of plane coordinates and stereo coordinates Each of the common objects CO is stereo-coordinated, and corresponding to the left sea level image and the right sea level image to find the common object CO and its corresponding image features, and obtain the corresponding three-dimensional coordinates P(X, Y, Z of the image features of the common object O) ), according to the aforementioned three-dimensional coordinates (X, Y, _Z ), the three-dimensional coordinates _OL and OR of the left image capture device and the right image capture device, and the sea level images of the common object CO on the left imaging plane and the right imaging plane The relative coordinates P _L and P _R of the SLM, thereby obtaining the disparity value Z. The aforementioned parallax value acquisition is only discussed with respect to the image features of a single common object CO, and other common objects CO can also be based on the corresponding stereo coordinates of the left image capturing device and the right image capturing device and the corresponding image features of the common object O. and its relative coordinates on the left imaging plane and the right imaging plane to obtain the disparity value, which is similar here and will not be repeated here.

其中，

為屬於左影像擷取裝置和共同物件O於左成像平面之海平面影像SLM之相對座標P_L 之距離，

為右影像擷取裝置和共同物件O於右成像平面之海平面影像SLM之相對座標P_R 之距離，

為屬於左影像擷取裝置和共同物件CO之相對距離，

為屬於右影像擷取裝置和共同物件CO之相對距離。透過前述公式，能取得視差值Z、左影像擷取裝置和共同物件CO之相對距離以及右影像擷取裝置和共同物件CO之相對距離。Please refer to FIG. 11 , which is a schematic diagram of the parallax conversion distance of the marine object ranging system of the present invention. As shown in Fig. 11, in conjunction with Fig. 10, the corresponding three-dimensional coordinates P(X, Y, Z) of the image features of the common object CO, the three-dimensional coordinates O _L and O of the left image capture device and the right image capture device _R , the focal length f of the left image capture device and the right image capture device and its relative distance B, the relative coordinates _PL and P _R of the sea level image SLM of the common object CO on the left imaging plane and the right imaging plane, according to the similar triangle The type theorem can deduce the following formula:

in,

is the distance between the relative coordinates _PL of the sea level image SLM belonging to the left image capture device and the common object O on the left imaging plane,

is the distance between the right image capturing device and the relative coordinate _PR of the sea level image SLM of the common object O in the right imaging plane,

is the relative distance belonging to the left image capture device and the common object CO,

is the relative distance belonging to the right image capture device and the common object CO. Through the aforementioned formula, the disparity value Z, the relative distance between the left image capturing device and the common object CO, and the relative distance between the right image capturing device and the common object CO can be obtained.

As mentioned above, the method and system for identifying objects at sea of the present invention can identify multiple common objects CO of the sea level image SLM at one time through the configuration of the convolutional neural network road algorithm CNN and the segmentation of the sea level image SLM, thereby speeding up the identification The speed of the common object CO is calculated, and the distance between the common object CO and each image capturing device 10 is calculated. To sum up, the method and system for identifying objects on the sea of the present invention have the above advantages, and can instantly identify the common objects CO of the sea level image SLM and estimate the distances between the common objects CO and the image capturing devices 10 .

The above description is exemplary only, not limiting. Any equivalent modifications or changes that do not depart from the spirit and scope of the present invention shall be included in the appended patent application scope.

:左影像擷取裝置和共同物件之相對距離

:右影像擷取裝置和共同物件之相對距離 C:類別 CL:卷積層 CNN:卷積神經網路演算法 CS:信任機率 CO:共同物件 EM:特徵矩陣 f:影像擷取裝置的焦距 FCL:全連結層 FM:特徵圖 F1,F2:濾波層 MPL:最大池化層 NE:神經元 O_L :左影像擷取裝置10的立體座標 O_R :右影像擷取裝置10的立體座標 PM:像素矩陣 P(X,Y,Z):共同物件之立體座標 P_L :共同物件於左成像平面之海平面影像之相對座標 P_R :共同物件於右成像平面之海平面影像之相對座標 RL:減少層 SLM:海平面影像 S11~S15:步驟 W:權重

:左影像擷取裝置和共同物件於左成像平面之海平面影像之相對座標之距離

:右影像擷取裝置和共同物件於右成像平面之海平面影像之相對座標之距離 Z:視差值10: image capture device 20: image processor 30: memory device B: block BB: border

: The relative distance between the left image capture device and the common object

: Relative distance between right image capture device and common object C: Category CL: Convolutional layer CNN: Convolutional neural network road algorithm CS: Trust probability CO: Common object EM: Feature matrix f: Focal length of image capture device FCL: Full Connection layer FM: Feature map F1, F2: Filter layer MPL: Maximum pooling layer NE: Neuron _OL : Stereoscopic coordinates of the left image capture device 10 _OR : Stereoscopic coordinates of the right image capture device 10 PM: Pixel matrix P(X,Y,Z): the three-dimensional coordinates of the common object _PL : the relative coordinates of the common object in the sea level image of the left imaging plane P _R : the relative coordinates of the common object in the sea level image of the right imaging plane RL: reduction layer SLM: Sea Level Image S11~S15: Step W: Weight

: The distance between the relative coordinates of the sea level image of the left image capture device and the common object on the left imaging plane

: the distance between the relative coordinates of the sea level image of the right image capture device and the common object on the right imaging plane Z: parallax value

Fig. 1 is a block diagram of the marine object ranging system of the present invention. FIG. 2 is a structural diagram of the first aspect of the convolutional neural network algorithm of the present invention. Figure 3 is a schematic diagram of a convolutional layer of the present invention. FIG. 4 is a schematic diagram of the maximum pooling layer of the present invention. FIG. 5 is a schematic diagram of the fully connected layer of the present invention. FIG. 6 is a flow chart of the method for identifying objects at sea according to the present invention. FIG. 7 is a flow chart of the operation of the convolutional neural network algorithm of the present invention. FIG. 8 is a structural diagram of the second aspect of the convolutional neural network algorithm of the present invention. Fig. 9 is a function diagram of the linear activation function of the present invention and the linear rectification activation function with leakage. FIG. 10 is a schematic diagram of parallax of the marine object ranging system of the present invention. FIG. 11 is a schematic diagram of the parallax conversion distance of the marine object ranging system of the present invention.

10: Image capture device

20: Image processor

30: Memory Device

B: block

BB: border

C: Category

CNN: Convolutional Neural Network Road Algorithm

CS: Probability of Trust

CO: Common Object

Claims (9)

A marine object ranging system, comprising: a plurality of image capture devices to capture a plurality of sea level images, each of the plurality of sea level images has a common object, and each of the plurality of sea level images has different viewing angles; an image processor, electrically connected to the plurality of image capturing devices, the image processor divides each sea level image into a plurality of blocks, and assigns each of the blocks to a corresponding category; and a memory device is connected to the image processing the memory device stores a convolutional neural network road algorithm, the convolutional neural network road algorithm is run by the image processor to produce the plurality of bounding boxes to which the common object belongs according to each of the blocks, and according to the The confidence score corresponding to each of the borders is calculated, and the image processor identifies the common object according to the trust probability and each category of each of the borders corresponding to the common object; wherein, the image processor identifies the common object according to the The distances between the common object and each of the plurality of image capture devices are known from different viewing angles of a plurality of sea level images; wherein the convolutional neural network road algorithm is run by the image processor to include the frame respectively according to the frame. The probability of the common object and the estimated value (intersection over union) of each of the borders and a corresponding ground truth box are used to calculate the corresponding trust probability of each of the borders to which the common object belongs. The maritime object distance measuring system as described in claim 1, wherein each of the frames has the center coordinate of the common object and the length and width of the common object. According to the maritime object ranging system described in item 1 of the claimed scope, when the image processor determines that the trust probability that the frame corresponds to the common object is 0, the common object does not exist in the frame. The maritime object ranging system as described in claim 1, wherein each of the plurality of sea level images has a plurality of common objects, and the number of the common objects is different from the number of the plurality of blocks. The maritime object ranging system according to the first claim, wherein the convolutional neural network road algorithm has multiple layers of convolution layers, multiple layers of max pooling layers and a fully connected layer. According to the maritime object ranging system described in item 1 of the scope of application, the plurality of image capture devices are divided into a left image capture device and a right image capture device, and the plurality of sea level images are divided into a left sea level image and a right image capture device. A right sea level image. According to the maritime object ranging system described in item 6 of the scope of application, the image processor converts the coordinates of the common object from the two-dimensional coordinate system according to the positions of the common object in the left sea level image and the right sea level image is a three-dimensional coordinate system, the common object has a three-dimensional coordinate, and the image processor knows the common object and the left image capturing device according to the three-dimensional coordinates and the positions of the left image capturing device and the right image capturing device and the distance between the common object and the right image capture device. The maritime object ranging system as described in item 6 of the scope of application, wherein the positions of the common object in the left sea level image and the right sea level image are different, and the common object in the left sea level image and the right sea level image The positional difference of the images is a parallax. According to the maritime object ranging system described in item 8 of the patent application scope, the image processor uses a parallax flow network to estimate the parallax of the left sea level image and the right sea level image.

Images