CN116883861A - Method and system for identifying large and medium-sized ship activities in ports for on-orbit applications of micro-satellites - Google Patents

Abstract

The invention discloses a method and a system for identifying the activities of large and medium-sized ships in a port for the on-orbit application of microsatellites, wherein the method comprises the steps of S1, acquiring remote sensing images in a port area; s2, carrying out on-orbit preprocessing on remote sensing images in a port area; s3, acquiring remote sensing images of the wharf and the near-shore area; s4, geometric fine correction of remote sensing images of the wharf and the near-shore area; s5, slicing the remote sensing image of the wharf and the near-shore area after geometric fine correction; s6, primarily detecting and identifying ship targets; s7, final detection and identification of ship targets; s8, comparing the multi-time phase detection recognition results; s9, confirming the change condition of the ship; s10, the ship activities are identified again. The advantages are that: the method can ensure accurate detection and identification of the ship target of interest, and can realize identification of the ship target activity of interest by processing and comparing multi-temporal remote sensing images obtained when the satellite revisits for a plurality of times.

Description

Methods for identifying large and medium-sized ships in ports for on-orbit applications of micro-satellites and system

Technical field

The present invention relates to the technical field of intelligent remote sensing interpretation, and in particular to a method and system for identifying large and medium-sized ship activities in ports for on-orbit applications of micro-satellites.

Background technique

Using satellite remote sensing to identify the activities of large and medium-sized ships in the port, such as entry, departure, and offshore navigation, is an important means to grasp the operating status of the port and conduct analysis of ship behavior.

Currently, Earth observation and remote sensing satellite systems are developing in the direction of miniaturization and clustering. The continuous increase in the number of satellites and the iterative optimization of constellation configurations have led to the continuous increase in the frequency of coverage of important ground targets by the satellite system, the continuous shortening of revisit cycles, and the continuous improvement of the satellite system's spatial and temporal resolution capabilities and system mission capabilities. However, limited by the current mission model of "on-orbit observation - ground processing - on-orbit execution" of remote sensing satellites, there are still obvious shortcomings in remote sensing information processing and application, making it difficult to maximize the advantages of the new generation of earth observation satellite systems. . On the one hand, more satellite image data needs to be transmitted to the ground for processing, and the pressure on satellite-to-ground data transmission is increasing; on the other hand, intelligent remote sensing interpretation technology is not mature enough, and remote sensing image processing and application are not intelligent enough. , the work requiring manual participation continues to increase, and there is a trend of shortage of professional image interpretation capabilities; on the other hand, the remote sensing information acquisition, analysis and application processes are lengthy, and the information acquisition and application efficiency is low.

In recent years, the rise of artificial intelligence has led to rapid development and technological innovation in fields such as computer vision and natural language processing. Combining artificial intelligence with artificial experience, making full use of the combined advantages of high-speed machine performance and advanced human cognition, we can carry out intelligent remote sensing image interpretation and information compilation on-orbit, and develop autonomous intelligence on-orbit that does not rely on ground stations. Intelligent satellite systems with high processing capabilities can significantly reduce the amount of satellite-to-ground data transmission, significantly reduce personnel workload, improve the intelligence level of satellite systems, and meet the needs of fast processing and high-time application tasks.

In summary, based on artificial intelligence methods, developing on-orbit intelligent remote sensing interpretation and information acquisition technology, and improving the intelligence and timeliness of remote sensing information processing and application are urgent needs and practical to adapt to the development of a new generation of earth observation satellite systems. Require.

At present, the processing and application of remote sensing images are mostly carried out on the ground, and a large amount of remote sensing image data needs to be transmitted to the ground through satellite-to-ground digital transmission links. However, the satellite-to-ground data transmission link window is very limited. For a single ground station, there may be only a few minutes of data transmission time for each orbital lap, and remote sensing image data cannot be downloaded in a timely manner. As the Earth observation system develops towards the construction of large-scale constellations, this model not only brings huge pressure on satellite-to-ground data transmission, but also severely limits the application of satellite data in tasks with high time-sensitive requirements. At the same time, the specific processing methods of remote sensing image data, such as ship target interpretation and ship activity identification in the harbor, still rely on manual interpretation by professional interpretation personnel. Professional interpretation personnel are stuck in a large number of repetitive basic interpretation tasks, which are inefficient and ineffective. Based on this, for the task of obtaining information on the activities of large and medium-sized ships in the harbor, it is necessary to provide a method and system for identifying ship activities in the harbor for micro-satellite on-orbit applications.

Contents of the invention

The purpose of the present invention is to provide a method and system for identifying large and medium-sized ship activities in ports for on-orbit applications of micro-satellites, thereby solving the aforementioned problems existing in the existing technology.

In order to achieve the above objects, the technical solutions adopted by the present invention are as follows:

A method for identifying large and medium-sized ship activities in ports for on-orbit applications of micro-satellites, including the following steps:

S1. Acquisition of remote sensing images of the port area:

The satellite completes remote sensing imaging of the port area and obtains remote sensing images of the port area based on the longitude and latitude geographical coordinates of the target port, the satellite's own attitude, and the width of the imaging load;

S2. On-orbit preprocessing of remote sensing images of the port area:

The satellite performs radiation correction and geometric correction on the remote sensing images of the port area, and obtains the corrected remote sensing images of the port area;

S3. Acquisition of remote sensing images of docks and nearshore areas:

Based on the geographical longitude and latitude information of the minimum envelope of the port's terminal and near-shore area pre-stored on the satellite, the remote sensing image of the terminal and near-shore area is cropped from the corrected remote sensing image of the port area;

S4. Geometric precision correction of remote sensing images of docks and nearshore areas:

According to the reference datum image of the port and nearshore area pre-stored on the satellite, perform geometric precision correction on the remote sensing image of the wharf and nearshore area with respect to the reference datum image, and obtain the geometrically finely corrected remote sensing image of the wharf and nearshore area;

S5. Slices of remote sensing images of docks and nearshore areas after geometric correction:

Based on the layout of port terminal facilities, design a remote sensing image cropping scheme, and slice the geometrically finely corrected remote sensing images of the terminal and nearshore areas based on the remote sensing image cropping scheme;

S6. Preliminary detection and identification of ship targets:

Send the sliced images to the ship target detection and recognition model in sequence to obtain preliminary ship target detection and recognition results;

S7. Final detection and identification of ship targets:

The preliminary detection and recognition results of each ship target are combined to obtain the position of the preliminary detection and recognition results of each ship target in the geometrically finely corrected remote sensing images of the wharf and offshore areas, and the repeated preliminary detection and identification of overlapping areas of the sliced images are obtained The results are deduplicated; based on the pre-stored water and land area segmentation information on the satellite, the erroneous preliminary detection and identification results of ship targets are eliminated, and the final detection and identification results of ship targets are obtained;

S8. Comparison of multi-temporal detection and recognition results:

Combined with the geographical location of the dock, compare the final detection and recognition results of this ship target with the final detection and recognition results of the ship target obtained during the last satellite revisit, and obtain the ship target activity recognition results;

S9. Confirmation of ship changes:

Determine whether there are changes in the dock ship based on the ship target activity recognition results. If so, mark the changes in the remote sensing images of the dock and nearshore areas, and save the images;

S10. Re-identification of ship activities:

The next time the satellite revisits the port position, S1 to S9 will be executed again.

Preferably, step S4 specifically includes the following content:

S41. Extract SIFT corner points from the remote sensing images of the dock and near-shore area and the reference images of the dock and near-shore area respectively;

S43. Use the KNN method to match the SIFT corner points of the two images;

S44. Use the RANSAC method to estimate the projection transformation model parameters of the dock and nearshore area images with respect to the dock and nearshore area reference benchmark images, and use the projection transformation model to perform projection transformation on the dock and nearshore area remote sensing images;

S45. Use the geographical location information of the reference base image of the dock and nearshore area to correct the geographical location information corresponding to each pixel in the remote sensing image of the dock and nearshore area after projection transformation, achieve geometric precision correction, and obtain the precisely corrected dock and nearshore area. Regional remote sensing images.

Preferably, the remote sensing image cropping scheme includes the following principles:

(1) The size of each slice image is the same;

(2) The crop size of each slice image is between 960×960 pixels and 1280×1280 pixels, and should be a multiple of 32;

(3) The sliced image should completely cover each dock, and the land area of the dock should be selected as the dividing line. The dividing line should not cross the ship berth range on the dock to ensure that the berth area is not divided;

(4) There is a certain overlap area in adjacent slice images, and the width of the overlap area is not less than the width of the horizontal envelope of nearby resident ships.

Preferably, the ship target detection and recognition model is a deep learning network structure and model parameter file obtained after pre-training on the ground, which can realize intelligent detection and fine identification of ship targets of interest;

Construct a data set of ships, which includes ship sample images, the position of ship targets in the images, and ship model category information; divide the data set into a training set and a test set; construct a data set based on YOlOv5 and annular smooth labels Combined with the ship target rotation detection and recognition network, the training set and test set are used to train and test the ship target rotation detection and recognition network to obtain the ship target detection and recognition model.

Preferably, the ship target detection and recognition model adopts the K-means clustering method and combines the scale attributes of all ship target objects to design different candidate envelope boxes on the feature maps of the three types of output layers; the specific design process. for,

S61. Obtain the dimensional data of all N-type civilian ship samples and the dimensional data of all M-type large ship samples;

S62. Visualize the obtained dimensional data of the civilian ship samples and large ship samples as two-dimensional data points;

S63. Use the K-means method to cluster the size data of various types of civilian ship samples into 3 clusters to obtain 3*N civilian ship sample points. The size data of various types of large ship samples will be deduplicated. Each type Only 3 sample size data are retained for large ships, and 3*M large ship sample points are obtained;

S64. Merge 3*N civilian ship sample points and 3*M large ship sample points;

S65. Use the K-means method to cluster the above merged sample point data into 12 clusters, that is, generate 12 types of candidate envelope boxes; according to the root mean square value of the length and width of the candidate envelope boxes, the generated 12 types The candidate envelope boxes are sorted from small to large; the four candidate envelope boxes with the smallest size are used for the output layer of the scale 128×128; the four candidate envelope boxes with the middle size are used for the output layer of the scale 64×64; The four largest candidate envelope boxes are used for the output layer with a scale of 32×32.

Preferably, in step S7, the preliminary detection and recognition results of each ship target are merged, and the position of each ship target's preliminary detection and recognition result in the geometrically finely corrected remote sensing image of the dock and near-shore area is obtained. Specifically,

A1. When the range of the sliced image is completely within the remote sensing image of the wharf and near-shore area after geometric fine correction, then the position coordinates of the rectangular envelope box in the preliminary detection result of the ship target are added to the origin of the slice image after geometric fine correction. By using the coordinates in the remote sensing images of the wharf and near-shore area, we can obtain the coordinates of the preliminary detection and recognition results of the ship target in the geometrically corrected remote sensing images of the wharf and near-shore area;

A2. When the range of the sliced image is close to the edge of the geometrically finely corrected remote sensing image of the pier and nearshore area, and part of the sliced image exceeds the boundary of the geometrically finely corrected remote sensing image of the pier and nearshore area, the areas outside the area are first removed. Part of it is completed as a black image, and then the coordinates of the origin of the sliced image are updated with respect to the origin of the geometrically finely corrected remote sensing image of the wharf and offshore area, and then the position coordinates of the rectangular envelope box in the preliminary detection results of the ship target are added to the slice The coordinates of the origin of the image in the geometrically finely corrected remote sensing images of the dock and coastal areas can be used to obtain the coordinates of the initial detection and identification results of the ship target in the geometrically finely corrected remote sensing images of the docks and nearshore areas.

Preferably, in step S7, the deduplication process on the duplicate preliminary detection and identification results existing in the overlapping area of the sliced images specifically includes the following content:

B1. Import all preliminary detection and identification results of ship targets;

B2. Read an unverified preliminary ship target detection and identification result A;

B3. Match the initial detection and recognition result A of the ship target with the remaining preliminary detection and recognition results of the ship target one by one, and calculate the intersection and union ratio of the overlapping envelope rectangles of the two results. When the intersection and union ratio is less than the preset threshold, No processing will be done, and the preliminary detection and identification result A of the ship target will continue to be matched and compared with the remaining preliminary detection and identification results of other ship targets; if the intersection ratio is greater than or equal to the preset threshold, one of them will be retained according to the judgment rules Preliminary detection and identification results of ship targets;

B4. If the preliminary detection and identification result A of the ship target has not been deleted, return to B3; if the preliminary detection and identification result A of the ship target is deleted, determine whether all the remaining preliminary detection and identification results of the ship target have been corrected. After the verification is completed, if there are still results that have not been verified, return to step B2; if all results have been verified, output the preliminary identification results of ship detection after deduplication processing.

Preferably, the determination rule in step B3 is,

If the areas of the enveloping rectangles of the two preliminary detection and recognition results of ship targets are different, the preliminary detection and recognition results of the ship target with the larger area are retained, and the preliminary detection and recognition results of the other ship target that overlap with it are deleted; if the two If the envelope rectangular areas of the preliminary detection and recognition results of ship targets are the same, the preliminary detection and recognition results of ship targets with high confidence are retained, and the preliminary detection and recognition results of another ship target that overlap with them are deleted.

Preferably, in step S7, the preliminary detection and identification results of eliminating erroneous ship targets include the following:

C1. Through manual design, complete the binary distinction processing of land and water ranges to form a land and water segmentation map; perform binarization processing on the envelope rectangular frame range of the preliminary detection and identification results of each ship target after deduplication, Form a ship target envelope segmentation diagram;

C2. Express the land and water segmentation map and the target envelope segmentation map of each ship as a matrix;

C3. Dot-multiply the matrix corresponding to each ship's target envelope segmentation map with the matrix corresponding to the land and water segmentation map, and then sum the elements of the result matrix. If the summation result is larger than the predetermined size of the ship's target envelope segmentation map, Assuming a percentage, it is determined that the preliminary detection and recognition result of the ship target after deduplication is an erroneous detection result, and the erroneous detection result is deleted. Otherwise, the preliminary detection and recognition result of the ship target after deduplication is retained.

The purpose of the present invention is also to provide a port large and medium-sized ship activity identification system for micro-satellite in-orbit applications, including:

High-speed data acquisition and storage interface unit: used to provide high-speed interface support for the collection and reading of original remote sensing image data, and to provide high-speed interface support for data reading, writing, and storage between the embedded artificial intelligence processing unit and memory and solid-state drives;

Embedded artificial intelligence processing unit: composed of a CPU and a GPU, running a Linux kernel operating system on it, used to implement the method described in any one of claims 1 to 9;

Memory: used to cache the read original remote sensing image data;

Solid state drive: used to store the basic files required during the operation of the method and the related images and results generated after the method is run;

Power control unit: used to adjust the voltage of the external power supply provided by the satellite platform to provide the required power supply voltage and current for each unit in the system.

The beneficial effects of the present invention are: 1. Compared with traditional image change detection methods, this method can cope with the complex background of the nearshore port environment, diverse lighting conditions (at different times during the day), and the possible angles of ship targets appearing in remote sensing images. It is arbitrary and has the characteristics and difficulties of multiple ship targets possibly docking intensively. It can ensure accurate detection and identification of ship targets of interest by processing and comparing multi-temporal remote sensing images obtained by satellites during multiple revisits. Yes, to achieve the identification of target activities of ships of interest. 2. Compared with traditional remote sensing data downlink, ground processing, uplink control and other cumbersome data processing and application processes, this method can significantly reduce the amount of remote sensing image data transmission and the time required to complete this type of task. 3. The system has simple composition and low cost. It can provide a modular solution for the on-orbit interpretation application of port ships by micro-satellites. The system has good portability and compatibility. 4. While improving the timeliness and intelligence of remote sensing missions, it also significantly reduces the pressure on satellite-to-ground data transmission, achieving a significant improvement in the comprehensive benefits of remote sensing missions.

Description of the drawings

Figure 1 is a flow chart of a ship activity recognition method in an embodiment of the present invention;

Figure 2 is a comparison diagram of original remote sensing images, pre-processed images, cropped images and geometrically corrected images in the embodiment of the present invention;

Figure 3 is a geometric fine correction operation process of using a reference datum image to perform newly acquired images in an embodiment of the present invention;

Figure 4 is a diagram of a personalized cutting case in an embodiment of the present invention;

Figure 5 is a structural diagram of the ship target detection and recognition model in the embodiment of the present invention;

Figure 6 is a flow chart of a detection and recognition network candidate frame design method in an embodiment of the present invention;

Figure 7 is an image diagram of the dock and offshore area restored from the sliced image in the embodiment of the present invention;

Figure 8 is a processing flow of ship recognition results in the overlapping area of detection and recognition in the embodiment of the present invention;

Figure 9 is a water and land segmentation diagram of the pier and nearshore area images after fine correction in the embodiment of the present invention;

Figure 10 is a structural diagram of the ship activity recognition system in the embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

In the embodiment, a method and system for identifying large and medium-sized ship activities in ports for on-orbit applications of micro-satellites are provided. Based on the characteristics of relatively fixed ship berths and ship types at the port terminal, a lightweight deep learning target is designed. The detection and identification method realizes the detection and fine identification of ship targets in visible light remote sensing images, and gives full play to the energy-efficient edge computing power of remote sensing low-orbit small satellites to independently carry out identification tasks of large and medium-sized ship activities in the port in orbit, improving the traditional task mode. The disadvantages of large demand for digital transmission and low timeliness of information generation.

Embodiment 1

In this embodiment, by combining fine-grained target detection and recognition with prior geographical knowledge, an efficient and deployable method for fine interpretation of ship targets in harbor and intelligent identification of their activities is provided, which can be deployed on low-orbit remote sensing small satellites, as shown in Figure 1 As shown, it specifically includes the following steps:

1. Obtaining remote sensing images of the port area:

The satellite completes the remote sensing imaging of the port area and obtains the remote sensing image of the port area based on the longitude and latitude geographical coordinates of the target port (i.e., the prior guidance information for satellite imaging and photography), the attitude of the satellite, and the width of the imaging load, etc., as shown in Figure 2(a) Show. The imaging area only ensures coverage of the port area and controls the amount of image data as much as possible.

In this embodiment, the rough geographical coordinates of the center of the satellite imaging area can be calculated based on the satellite orbit positioning data and satellite attitude. Therefore, first, based on the satellite orbit positioning information, the satellite attitude is controlled so that the rough position of the satellite imaging center is in the center of the port terminal. Subsequently, remote sensing imaging was performed on the port area and remote sensing data were obtained. Since the satellite has a large width and the imaging center is close to the center of the pier, it can obtain a complete remote sensing image of the port area.

2. On-orbit preprocessing of remote sensing images of the port area:

The satellite performs radiation correction and geometric correction on the remote sensing image of the port area, and obtains the corrected remote sensing image of the port area, as shown in Figure 2(b).

3. Acquisition of remote sensing images of docks and nearshore areas:

According to the geographical longitude and latitude information of the minimum envelope of the port's terminal and near-shore area pre-stored on the satellite, the remote sensing image of the terminal and near-shore area is cropped from the corrected remote sensing image of the port area, as shown in Figure 2(c). The cropped image includes the port terminal and a small part of the offshore land and water areas.

4. Geometric precision correction of remote sensing images in docks and nearshore areas:

According to the reference datum image of the port and nearshore area pre-stored on the satellite, the remote sensing images of the wharf and nearshore area are geometrically finely corrected (i.e., relative correction) with respect to the reference datum image, and the geometrically finely corrected wharf and nearshore area are obtained. The remote sensing image of the shore area is shown in Figure 2(d).

As shown in Figure 3, specifically, the geometric precision correction includes the following contents,

1. Extract SIFT corner points from the remote sensing images of the dock and near-shore area and the reference benchmark images of the dock and near-shore area respectively;

3. Use the KNN method (K neighbor algorithm) to match the SIFT corner points of the two images;

4. Use the RANSAC method (Random Consistent Sampling Algorithm) to estimate the projection transformation model parameters of the dock and nearshore area images with respect to the dock and nearshore area reference benchmark images, and use this projection transformation model to project the remote sensing images of the dock and nearshore area. ;

5. Use the geographical location information of the reference benchmark image of the dock and nearshore area to correct the geographical location information corresponding to each pixel in the remote sensing image of the dock and nearshore area after projection transformation, achieve geometric precision correction, and obtain the precisely corrected dock and nearshore area. Regional remote sensing images.

5. Slices of remote sensing images of docks and offshore areas after geometric correction:

Due to the large size of remote sensing images in docks and offshore areas, a personalized remote sensing image cropping scheme is designed based on the layout of port terminal facilities to support ship target detection and recognition in sliced images with larger sizes and more isolated cropping ranges. ; Based on the remote sensing image cropping scheme, the geometrically finely corrected remote sensing images of the dock and offshore areas are sliced.

As shown in the personalized cropping case shown in Figure 4, on the one hand, considering that the scope of port terminals and near-shore areas is generally large, and the size of their remote sensing images is large, if the entire image is sent to the deep learning detection and recognition network, the calculation cost will be very high. The time will increase exponentially and the accuracy cannot be guaranteed; on the other hand, considering the identification of ship target activities, the key areas of focus are docks and offshore waters, and there is no need to waste computing power and time on some waters and land areas. Detection and identification; therefore, a personalized cutting plan is designed to cut out multiple slices centered on the dock from remote sensing images of docks and coastal waters, and detect and identify ship targets on these slice images. Personalized tailoring requires designers to be familiar with port terminals and ship berths. The personalized remote sensing image cropping plan includes the following principles:

(1) The size of each slice image is the same;

(2) The size of each slice image must be appropriate. This method is mainly suitable for visible light remote sensing images with a resolution of 0.8 meters to 0.5 meters. The crop size of each slice image is between 960×960 pixels and 1280×1280 pixels, and should be is a multiple of 32;

(3) The main principle of the sliced image is to completely cover each dock. Try to select the land area of the dock as the dividing line. The dividing line does not cross the ship berth range on the dock to ensure that the berth area is not divided;

(4) There is a certain overlap area in adjacent slice images, and the width of the overlap area is not less than the width of the horizontal envelope of nearby resident ships.

There is no clipping area design or ship target detection in land areas and waters far away from the dock. This can reduce the computational burden as much as possible while ensuring the detection and identification of ships in the dock and adjacent waters.

6. Preliminary detection and identification of ship targets:

The sliced images are sequentially sent to the ship target detection and recognition model to obtain preliminary ship target detection and recognition results. The preliminary detection and recognition results are given by rotating the rectangular envelope frame to achieve the minimum envelope of the ship target area.

The ship target detection and recognition model is a deep learning network structure and model parameter file obtained after pre-training on the ground, which can realize intelligent detection and fine identification of ship targets of interest;

Construct a data set of ships, which includes ship sample images, the position of the ship target in the image (given as the four vertex coordinates of the rotated rectangular envelope box), and ship model category information; divide the data set For the training set and test set; build a ship target rotation detection and recognition network based on YOlOv5 combined with Circular Smooth Label (CSL), use the training set and test set to train and test the ship target rotation detection and recognition network to obtain the ship target Detection and recognition models.

As shown in Figure 5, the ship target detection and recognition model is improved based on the YOLO_v5 network. Compared with the YOLO_v5 network, the deep learning network for large and medium-sized ship target detection and recognition mainly has the following improvements: (1) The scale of the network input image is 1024×1024×3; (2) Circular Smooth Label (CSL) is used The rotating target detection idea adds the prediction of the target rotation angle, and uses a rotating rectangular frame to achieve the envelope of the ship target; at the same time, refer to the CSL rotating target detection idea to calculate the network prediction loss value, and the rotating target non-maximum The value suppression algorithm (Non-MaximumSuppression, NMS) has also been adaptively improved. (3) The prediction output layer scale is improved to 128×128, 64×64, and 32×32. (4) Use the K-means clustering method and combine the scale attributes of all ship target objects to design different candidate envelope boxes on the feature maps of the three types of output layers, and design 4 types of candidate envelopes on each layer of feature maps. There are 12 types of boxes in total; the larger the scale of the feature map, the smaller the size of the corresponding candidate envelope box; vice versa. Each pixel of each output layer feature map corresponds to 4 prediction result parameters (one prediction result for each of the 4 candidate envelope boxes), and each prediction result contains K+6 parameters (K category parameters, 1 confidence parameters, 1 rotation angle parameter, 4 position coordinate parameters). Therefore, the dimensions of the output data of the three types of output layers are [4*(K+6)]×128×128, [4*(K+6)]×64×64, [4*(K+6)]× 32×32.

As shown in Figure 6, in this embodiment, the ship target detection and recognition model uses the K-means clustering method to design different candidate envelope frames. The specific design process is as follows:

1. Obtain the dimensional data of all N-type civilian ship samples and the dimensional data of all M-type large ship samples;

2. Visualize the obtained dimensional data (length and width) of civilian ship samples and large ship samples as two-dimensional data points;

3. Civilian ships have the characteristics of a large number of various types of ships, various sizes of ships (such as bulk carriers can be large or small), and an unbalanced number of various types of ships. In order to avoid the candidate box size being biased towards quantity, There are many civil ships with many size types, resulting in the candidate frame size not being representative and universally adaptable. Therefore, the size data of various types of civil ship samples are clustered into 3 clusters using the K-means method, and 3*N civil ship samples are obtained. point;

Large ships have the characteristics of relatively fixed sizes of various types and unbalanced numbers of various types of ships. In order to avoid the candidate box size being biased towards large numbers of large ships, resulting in the candidate box size not being representative and universal, so samples of various types of large ships are The size data will be deduplicated, and only 3 sample size data will be retained for each type of large ship, resulting in 3*M large ship sample points.

Among them, for a certain large ship with a fixed size, the corresponding three sample size data are all the fixed size; for a certain large ship with slight differences in size, the corresponding three sample size data are all the arithmetic average of different sizes.

4. Merge 3*N civilian ship sample points and 3*M large ship sample points;

5. Use the K-means method to cluster the above-mentioned merged sample point data into 12 clusters, that is, generate 12 types of candidate envelope boxes; according to the root mean square value of the length and width of the candidate envelope boxes, the generated 12 types The candidate envelope boxes are sorted from small to large; the four candidate envelope boxes with the smallest size are used for the output layer of the scale 128×128; the four candidate envelope boxes with the middle size are used for the output layer of the scale 64×64; The four largest candidate envelope boxes are used for the output layer with a scale of 32×32.

Through the above candidate frame design method, it can be ensured that the designed candidate frame is universal and can cover all ship targets to be detected and identified, thereby preventing the designed candidate envelope frame from being more inclined to ships with a larger number of samples. target, so that the ship target detection and recognition model can have a relatively balanced and strong prediction ability for all types of ships.

7. Final detection and identification of ship targets:

The preliminary detection and recognition results of each ship target are combined to obtain the position of the preliminary detection and recognition results of each ship target in the geometrically finely corrected remote sensing images of the wharf and offshore areas, and the repeated preliminary detection and identification of overlapping areas of the sliced images are obtained The results are deduplicated; based on the pre-stored water and land area segmentation information on the satellite, the erroneous preliminary detection and identification results of ship targets (that is, the preliminary detection and identification results of ship targets on land) are eliminated, and the final detection of ship targets is obtained. Recognition results.

In this embodiment, referring to Figure 7, O _C1 , O _C2 , and O _C3 are the origins of the single detection area image (slice image), and _OR is the origin of the finely corrected pier and nearshore area images.

The preliminary detection and recognition results of each ship target are combined to obtain the position of the preliminary detection and recognition results of each ship target in the geometrically finely corrected remote sensing images of the dock and coastal areas. The specific positions are as follows:

A1. When the range of the sliced image is completely within the remote sensing image of the wharf and near-shore area after geometric fine correction, then the position coordinates of the rectangular envelope box in the preliminary detection result of the ship target are added to the origin of the slice image after geometric fine correction. By using the coordinates in the remote sensing images of the dock and near-shore area, we can obtain the coordinates of the preliminary detection and identification results of the ship target in the geometrically corrected remote sensing images of the dock and near-shore area.

For example, for a sliced image with O _C3 as the origin, the position coordinates of the rectangular envelope box in the ship detection and recognition results are added to the coordinates of O _C3 in O _R X _R Y _R , and the ship detection and recognition results can be obtained after fine correction The coordinates in the image of the pier and nearshore area.

A2. When the range of the sliced image is close to the edge of the geometrically finely corrected remote sensing image of the pier and nearshore area, and part of the sliced image exceeds the boundary of the geometrically finely corrected remote sensing image of the pier and nearshore area, the areas outside the area are first removed. Partially complete the image as a black image, and then update the coordinates of the origin of the sliced image (such as O _C2 and O _C1 ) with respect to the origin of the geometrically corrected remote sensing image of the dock and offshore area (i.e. _OR ), and the subsequent ship detection and recognition results The coordinate calculation in the geometrically finely corrected dock and nearshore area images is the same as A1 (that is, the position coordinates of the rectangular envelope box in the preliminary ship target detection results are added to the origin of the sliced image in the geometrically finely corrected dock and coastal area images). The coordinates in the remote sensing images of the near-shore area can be used to obtain the coordinates of the preliminary detection and recognition results of the ship target in the geometrically corrected remote sensing images of the dock and near-shore area).

In this embodiment, according to the personalized cropping area design scheme, there is a certain overlapping area between adjacent cropped images, so the ship recognition results in the overlapping area need to be deduplicated. As shown in Figure 8, the deduplication process for the duplicate preliminary detection and identification results that exist in the overlapping area of the sliced images specifically includes the following contents:

B1. Import all preliminary ship target detection and recognition results (the four vertex coordinates of the envelope box). These results are the output of each sliced image predicted by the ship target detection and recognition model;

B2. Read an unverified preliminary ship target detection and identification result A;

B3. Match the preliminary detection and identification results A of the ship target with the remaining preliminary detection and identification results of the ship target one by one, and calculate the intersection and union ratio of the overlap of the envelope rectangular frames of the two results. When the intersection and union ratio (IOU) is less than the preset When the threshold (can be set according to the actual situation, set to 0.3 in this embodiment), no processing is performed, and the preliminary detection and identification result A of the ship target is continued to be matched and compared with the remaining preliminary detection and identification results of other ship targets. ; If the intersection ratio is greater than or equal to the preset threshold, one of the ship target preliminary detection and identification results will be retained according to the judgment rules.

The judgment rule is: if the areas of the enveloping rectangles of the two preliminary detection and recognition results of ship targets are different, the preliminary detection and recognition result of the ship target with the larger area is retained, and the preliminary detection and recognition result of the other ship target that overlaps with it is deleted. ; If the envelope rectangular areas of the two preliminary detection and recognition results of ship targets are the same, the preliminary detection and recognition result of the ship target with high confidence is retained, and the preliminary detection and recognition result of the other ship target that overlaps with it is deleted.

B4. If the preliminary detection and identification result A of the ship target has not been deleted, return to B3; if the preliminary detection and identification result A of the ship target is deleted, determine whether all the remaining preliminary detection and identification results of the ship target have been corrected. After the verification is completed, if there are still results that have not been verified, return to step B2; if all results have been verified, output the preliminary identification results of ship detection after deduplication processing.

In this embodiment, the preliminary detection and identification results of eliminating erroneous ship targets include the following:

C1. Through manual design, complete the binary distinction processing of land and water ranges to form a land and water segmentation map, as shown in Figure 9(b). The pixels corresponding to the land range are marked as "1" and are displayed in white in the figure. ; The pixels corresponding to the water area are marked as "0" and are displayed as black in the figure; the envelope rectangular frame range of the initial detection and recognition results of each ship target after deduplication is binarized to form the ship target envelope segmentation As shown in Figure 9(c), the pixels corresponding to the ship envelope range are marked as "1" and are displayed in white in the image, while the remaining parts are marked as "0" and are displayed in black in the image.

C2. Express the land and water segmentation map and the target envelope segmentation map of each ship as a matrix;

C3. Dot-multiply the matrix corresponding to each ship's target envelope segmentation map with the matrix corresponding to the land and water segmentation map, and then sum the elements of the result matrix. If the summation result is larger than the predetermined size of the ship's target envelope segmentation map, Assuming a percentage (which can be set according to actual needs, and is set to 60% in this embodiment), it is determined that the initial detection and identification result of the ship target after deduplication is an erroneous detection result, and the erroneous detection result is deleted, otherwise the deduplication is retained. The subsequent preliminary detection and recognition results of ship targets.

Thus, possible misidentified ship targets on land can be eliminated.

8. Comparison of multi-temporal detection and recognition results:

Combined with the geographical location of the dock, the final detection and identification results of this ship target are compared with the final detection and identification results of the ship target obtained during the last satellite revisit, and the locations of the ships of interest on the coastal sea and at each dock are given. Activities on the dock, such as ships leaving the dock, ships entering the dock, and ships operating in the waters near the dock.

9. Confirmation of ship changes:

Based on the ship target activity recognition results, it is determined whether there are changes in the dock ships. If so, the changes are marked in the remote sensing images of the dock and nearshore areas, and the images are saved.

10. Re-identification of ship activities:

The next time the satellite revisits the port location, perform steps one to nine again.

Embodiment 2

In this embodiment, a port large and medium-sized ship activity identification system for on-orbit applications of micro-satellites is provided, as shown in Figure 10, including a power control unit and an embedded artificial intelligence processing unit (including a CPU and a GPU processor) , high-speed data acquisition and storage interface unit (FPGA), memory (RAM) and solid-state drive (SSD), raw remote sensing image data read-in bus, external control bus and external data bus, etc.

(1) High-speed data acquisition and storage interface unit (FPGA), on the one hand, it can provide high-speed interface support for the acquisition and reading of original remote sensing image data, on the other hand, it can provide high-speed interface support for CPU or GPU and storage devices (memory RAM, solid state drive SSD) Provides high-speed interface support for data reading, writing, and storage between devices.

It can provide a high-speed bus for reading the original remote sensing image data, and it can also provide an internal high-speed bus for the embedded artificial intelligence processing unit to read and write data with the memory RAM or solid-state drive SSD.

(2) Solid-state drive (SSD) is used to store basic files such as the operating system required for program running, as well as images generated after the program is run, extracted ship target slice images, etc., mainly including: used for ship target detection and recognition Deep learning network structure and model parameter files; used to carry out newly acquired port remote sensing image radiation correction and geometric correction related parameter files, reference benchmark images required for geometric fine correction operations, etc.; operating system, software library, etc. required for program operation ;Remote sensing images of port terminals and nearshore areas after preprocessing (radiation correction, geometric correction, geometric fine correction); after the program is run, the result data such as cropped ship target slice images.

(3) The memory RAM is used to cache the read original remote sensing image data. The data is stored in the memory RAM by the imaging load after passing through the data reading bus and the high-speed data acquisition and storage interface unit.

(4) The embedded artificial intelligence computing unit is mainly composed of CPU and GPU, which runs the Linux kernel operating system, as well as image preprocessing (radiation correction, geometric correction, geometric fine correction), and ship target intelligent detection and recognition related applications. It provides powerful program control management, data flow processing, image processing and parallel computing functions; through the high-speed data acquisition and storage interface unit, it can read data in the memory RAM and read and write data in the solid state drive SSD; provides a control bus and data bus, which can interact with the satellite platform for mission control, data transmission and other operations.

It is responsible for the calculation and processing of intelligent algorithms related to remote sensing image preprocessing, intelligent detection and identification of ship targets, and identification of ship target activities. It also provides a control bus and data bus for the interaction between this system and the satellite platform.

(5) The power control unit can provide the required power supply voltage and current for each unit in the system by adjusting the voltage of the external power supply provided by the satellite platform.

In this embodiment, data is exchanged through an internal bus between the system's embedded artificial intelligence processing unit and the high-speed data collection and storage interface unit, and between the high-speed data collection and storage interface unit and the memory RAM and solid-state hard drive. This system can be used as a subsystem of the satellite and is compatible with various remote sensing satellite platforms, with good portability and strong compatibility.

By adopting the above technical solutions disclosed in the present invention, the following beneficial effects are obtained:

The present invention provides a method and system for identifying large and medium-sized ship activities in ports for micro-type on-orbit applications. Compared with traditional image change detection methods, this method can cope with the complex background of the near-shore port environment and diverse lighting conditions (different during the day). time), the angle at which ship targets appear in remote sensing images may be arbitrary, and multiple ship targets may be docked densely. This can ensure accurate detection and identification of ship targets of interest. By successively analyzing satellites Multi-temporal remote sensing images obtained during multiple revisits are processed and compared to identify the target activities of ships of interest. Compared with traditional remote sensing data downlink, ground processing, uplink control and other cumbersome data processing and application processes, this method can significantly reduce the amount of remote sensing image data transmission and the time required to complete the task. The system has a simple composition and low cost. It can provide a modular solution for micro-satellites to implement on-orbit interpretation applications for port ships. The system has good portability and compatibility. While improving the timeliness and intelligence of remote sensing missions, it also significantly reduces the pressure on satellite-to-ground data transmission and significantly improves the comprehensive benefits of remote sensing missions.

The above are only preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications can also be made. The scope of protection of the present invention should be considered.

Claims (10)

1. A method for identifying the activities of large and medium-sized ships in a port for on-orbit application of microsatellites is characterized by comprising the following steps: comprises the following steps of the method,

s1, acquiring remote sensing images of port areas:

the satellite completes remote sensing imaging of the port area according to longitude and latitude geographic coordinates of the target port, the self-posture of the satellite and the imaging load width, and remote sensing images of the port area are obtained;

s2, carrying out on-orbit preprocessing on remote sensing images in port areas:

performing radiation correction and geometric correction on the remote sensing image of the port area on the satellite to obtain a corrected remote sensing image of the port area;

s3, acquiring remote sensing images of wharfs and near-shore areas:

cutting out remote sensing images of the wharf and the near-shore area from the corrected remote sensing images of the harbour area according to geographic longitude and latitude information of minimum envelopes of the wharf and the near-shore area of the harbour prestored on the satellite;

s4, geometric fine correction of remote sensing images of wharfs and near-shore areas:

According to the pre-stored reference images of the wharf and the near-shore area of the harbour on the satellite, performing geometric fine correction on the reference images on the remote sensing images of the wharf and the near-shore area to obtain remote sensing images of the wharf and the near-shore area after the geometric fine correction;

s5, slicing the remote sensing image of the wharf and the near-shore area after geometric fine correction:

according to the port and dock facility layout, a remote sensing image cutting scheme is designed, and the remote sensing images of the dock and the near-shore area after geometric fine correction are sliced based on the remote sensing image cutting scheme;

s6, preliminary detection and identification of ship targets:

sequentially sending the slice images into a ship target detection and identification model to obtain a ship target preliminary detection and identification result;

s7, final detection and identification of ship targets:

combining the primary detection and identification results of the ship targets to obtain the positions of the primary detection and identification results of the ship targets in remote sensing images of wharfs and near-shore areas after geometric fine correction, and performing duplicate removal processing on repeated primary detection and identification results in overlapping areas of the slice images; removing the preliminary detection and identification result of the ship target with errors according to the pre-stored land area segmentation information on the satellite, and obtaining the final detection and identification result of the ship target;

S8, comparing the multi-time phase detection recognition results:

comparing the final detection and identification result of the ship target with the final detection and identification result of the ship target obtained in the last satellite revisit by combining with the geographic position of the wharf to obtain the activity identification result of the ship target;

s9, confirming ship change conditions:

determining whether the wharf ship changes according to the ship target activity recognition result, if so, marking the change condition in remote sensing images of the wharf and the near-shore area, and storing the images;

s10, identifying ship activities again:

the next time the satellite revisits to the port location, S1 to S9 are performed again.

2. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 1, which is characterized in that: step S4 specifically includes the following,

s41, respectively extracting SIFT angular points in remote sensing images of the wharf and the near-shore area and reference images of the wharf and the near-shore area;

s43, matching SIFT corner points of the two images by using a KNN method;

s44, estimating projection transformation model parameters of the dock and the near-shore area images relative to the dock and near-shore area reference images by using a RANSAC method, and performing projection transformation on the dock and near-shore area remote sensing images by using the projection transformation model;

S45, correcting geographical position information corresponding to each pixel in the remote sensing images of the dock and the near-shore area after projection transformation by using geographical position information of the reference images of the dock and the near-shore area, so as to realize geometric fine correction and obtain the remote sensing images of the dock and the near-shore area after fine correction.

3. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 1, which is characterized in that: the remote sensing image cutting scheme comprises the following principles:

(1) The sizes of the slice images are the same;

(2) The cutting size of each slice image is between 960×960 pixels and 1280×1280 pixels, and should be a multiple of 32;

(3) The sliced image should cover each wharf completely, the land area of the wharf is selected as a boundary, the boundary does not span the berth range of the ship on the wharf, and the berth area is ensured not to be divided;

(4) The adjacent slice images have a certain overlapping area, and the width of the overlapping area is not lower than the width of the horizontal envelope line of the nearby resident ship.

4. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 1, which is characterized in that: the ship target detection and identification model is a deep learning network structure and model parameter file obtained after ground pre-training, and can realize intelligent detection and fine identification of the ship target of interest;

Constructing a data set of a ship, wherein the data set comprises a ship sample image, the position of a ship target in the image and ship model type information; dividing the data set into a training set and a testing set; and constructing a ship target rotation detection and identification network based on the combination of YOlOv5 and the annular smooth tag, and training and testing the ship target rotation detection and identification network by utilizing a training set and a testing set to obtain a ship target detection and identification model.

5. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 4, which is characterized in that: the ship target detection and identification model adopts a K-means clustering method, combines the scale attributes of all ship target objects, and designs different candidate envelope frames on the characteristic diagrams of the three output layers; the specific design process is that,

s61, acquiring size data of all N types of civil ship samples and size data of all M types of large-scale ship samples;

s62, visually representing the acquired size data of the civil ship sample and the large-scale ship sample as two-dimensional data points;

s63, clustering size data of various civil ship samples into 3 clusters by using a K-means method to obtain 3*N civil ship sample points; carrying out de-duplication treatment on the size data of various large-scale ship samples, wherein each large-scale ship only retains 3 sample size data, and 3*M large-scale ship sample points are obtained;

S64, combining 3*N civil ship sample points and 3*M large ship sample points;

s65, clustering the combined sample point data into 12 clusters by using a K-means method, namely generating 12 candidate envelope frames; according to the root mean square value of the length and the width of the candidate envelope frames, sequencing 12 generated candidate envelope frames from small to large; the 4 candidate envelope frames with the smallest size are used in the output layer with the corresponding size of 128 multiplied by 128; 4 candidate envelope frames with middle sizes are used in an output layer with the corresponding size of 64 multiplied by 64; the 4 largest candidate envelope frames are used for the output layer of the corresponding scale 32×32.

6. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 1, which is characterized in that: in step S7, combining the primary detection and identification results of each ship target, and obtaining the positions of the primary detection and identification results of each ship target in the remote sensing images of the wharf and the near-shore area after the geometric fine correction is specifically,

a1, when the slice image range is completely in the remote sensing images of the wharf and the near-shore area after the geometric fine correction, adding the position coordinates of the rectangular envelope frame in the primary detection result of the ship target to the coordinates of the origin of the slice image in the remote sensing images of the wharf and the near-shore area after the geometric fine correction, and obtaining the coordinates of the primary detection and identification result of the ship target in the remote sensing images of the wharf and the near-shore area after the geometric fine correction;

A2, when the range of the slice image is close to the edge of the geometrically refined wharf and near-shore area remote sensing image, and the partial area of the slice image exceeds the boundary of the geometrically refined wharf and near-shore area remote sensing image, the part exceeding the area is fully filled with a black image, then the coordinates of the origin of the slice image with respect to the origin of the geometrically refined wharf and near-shore area remote sensing image are updated, and then the coordinates of the rectangular envelope frame in the ship target primary detection result plus the coordinates of the origin of the slice image in the geometrically refined wharf and near-shore area remote sensing image are added, so that the coordinates of the ship target primary detection recognition result in the geometrically refined wharf and near-shore area remote sensing image can be obtained.

7. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 1, which is characterized in that: in step S7, the duplicate removal processing for the repeated preliminary detection and identification result of the existence of the overlapping region of the slice images specifically includes the following,

b1, importing all ship targets into a primary detection and identification result;

b2, reading a ship target preliminary detection and identification result A which is not verified;

B3, matching the ship target preliminary detection recognition result A with the rest ship target preliminary detection recognition results one by one, calculating the overlapping ratio of the two result enveloping rectangular frames, and when the overlapping ratio is smaller than a preset threshold value, not processing, and continuously matching and comparing the ship target preliminary detection recognition result A with the rest ship target preliminary detection recognition results; if the cross ratio is greater than or equal to a preset threshold value, retaining a primary detection and identification result of one ship target according to a judgment rule;

b4, if the primary detection and identification result A of the ship target is not deleted, returning to the B3; if the ship target preliminary detection recognition result A is deleted, judging whether all the rest ship target preliminary detection recognition results are checked, and if the ship target preliminary detection recognition results are not checked, returning to the step B2; and if all the results are verified, outputting a ship detection primary identification result after the duplication removal treatment.

8. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 1, which is characterized in that: the decision rule in step B3 is that,

if the areas of the enveloping rectangular frames of the primary detection and identification results of the two ship targets are different, retaining the primary detection and identification results of the ship targets with large areas, and deleting the primary detection and identification results of the other ship target overlapped with the primary detection and identification results of the ship targets; if the enveloping rectangular areas of the primary detection and identification results of the two ship targets are the same, the primary detection and identification results of the ship targets with high confidence are reserved, and the primary detection and identification results of the other ship targets overlapped with the primary detection and identification results of the ship targets are deleted.

9. The method for identifying the activities of large and medium-sized ships in ports for on-orbit application of microsatellites according to claim 1, which is characterized in that: in step S7, the primary detection and identification result of the ship target with errors is removed, which comprises the following contents,

c1, finishing binarization distinguishing treatment of land and water area ranges through manual design to form a land and water area segmentation map; performing binarization processing on the envelope rectangular frame range of the preliminary detection and identification result of each ship target after the duplication removal to form a ship target envelope segmentation map;

c2, representing land and water area segmentation maps and each ship target envelope segmentation map as a matrix;

and C3, multiplying the matrix corresponding to each ship target envelope segmentation map with the matrix corresponding to the land and water area segmentation map, summing the elements of the result matrix, judging that the ship target preliminary detection and identification result after the duplication removal is an error detection result if the sum result is larger than the preset percentage of the ship target envelope segmentation map, deleting the error detection result, and otherwise, reserving the ship target preliminary detection and identification result after the duplication removal.

10. A port large and medium-sized ship activity recognition system for microsatellite on-orbit application is characterized in that: comprising the steps of (a) a step of,

High-speed data acquisition and storage interface unit: the system is used for providing high-speed interface support for acquisition and reading of original remote sensing image data and providing high-speed interface support for data reading and writing and storage between the embedded artificial intelligent processing unit and the memory and the solid state disk;

embedded artificial intelligence processing unit: consisting of a CPU and a GPU on which a Linux kernel operating system runs for implementing the method according to any of claims 1 to 9;

memory: the method comprises the steps of caching read-in original remote sensing image data;

solid state disk: the method is used for storing basic files required in the running process of the method and related images and results generated after the method is run;

a power supply control unit: the power supply system is used for carrying out voltage adjustment on an external power supply provided by a satellite platform end and providing required power supply voltage and current for each unit in the system.