TW202414247A - Real time vessel identity and image matching system and operating method thereof - Google Patents

Abstract

The present invention discloses a real time vessel identity and image matching system and operating method thereof. Specifically, the real time vessel identity and image matching system comprises a processing module, a storage module, at least one camera and a water surface object information receiving module. The processing module may match the camera location information, image recognition information which are provided by the camera with the water surface information which is provided by the water surface object information receiving module, generating a water surface object information tag. Therefore, the processing module is able to label the water surface object information tag beside the water surface object which is displayed.

Description

Ship identity image real-time recognition superposition system and its operation method

The present invention relates to a ship identity image real-time recognition and superposition system and its operation method, and in particular to a ship identity image real-time recognition and superposition system and its operation method which can integrate water object information and water object image information, and perform real-time annotation and updating.

Generally speaking, within the visual range of ports and the coastline, there are often a large number of water objects, such as ships and other water vehicles, which frequently sail and enter and exit. Therefore, in order to effectively control the activities of water objects such as ships, most of them are done through the signal of the Automatic Identification System (AIS).

However, currently only aquatic objects with a displacement of 20 tons or more are required to install an Automatic Identification System (AIS). Therefore, for aquatic objects that are not required to install an Automatic Identification System (AIS), there is a disadvantage that identity recognition is difficult.

Therefore, even if the water objects are equipped with Automatic Identification System (AIS), when there are multiple water objects in the port, it still depends on manual search and identification of the type of ship, which is time-consuming and inaccurate.

In order to solve the problems of the prior art, the present invention aims to provide a ship identity image real-time recognition and superposition system and its operation method. Specifically, the ship identity image real-time recognition and superposition system of the present invention comprises a processing module, a storage module, at least one camera and a water object information receiving module.

The storage module is connected to the processing module, and the storage module includes at least one water object data set. The at least one camera is connected to the processing module, and the at least one camera includes at least one camera positioning information.

Furthermore, the at least one camera captures at least one surveillance picture, and the processing module performs an object detection according to at least one aquatic object in the at least one surveillance picture, and then uses the at least one aquatic object data set to distinguish at least one image category of the at least one aquatic object through an object recognition.

The aquatic object information receiving module is connected to the processing module. The aquatic object information receiving module receives a plurality of aquatic object information within a period of time and transmits the plurality of aquatic object information to the processing module.

The processing module matches the at least one camera positioning information, the at least one image category and the plurality of water object information to generate at least one water object information tag, and the processing module marks the at least one water object information tag on the at least one water object in the at least one surveillance image.

At the same time, the operation method of the ship identity image real-time recognition and superposition system of the present invention includes the following steps. First, step (A) is to provide the ship identity image real-time recognition and superposition system as described above. Then, step (B) is to enable the water object information receiving module to receive the plurality of water object information transmitted by the at least one water object within the time segment, and transmit the plurality of water object information to the processing module.

In step (C), the processing module selects the at least one camera whose field of view includes the at least one aquatic object according to the plurality of aquatic object information, and captures the at least one surveillance image. Then, in step (D), the processing module performs the object detection according to the at least one aquatic object in the at least one surveillance image, and then uses the at least one aquatic object data set to distinguish the at least one image category of the at least one aquatic object by the object recognition.

Finally, in step (E), the processing module matches the at least one camera positioning information, the at least one image category and the plurality of water object information to generate at least one water object information tag, and the processing module labels the at least one water object information tag on the at least one water object in the at least one surveillance image.

The above brief description of the present invention is intended to provide a basic explanation of several aspects and technical features of the present invention. The invention brief description is not a detailed description of the present invention, so its purpose is not to specifically list the key or important components of the present invention, nor is it used to define the scope of the present invention. It is only to present several concepts of the present invention in a concise manner.

In order to understand the technical features and practical effects of the present invention and to implement it according to the contents of the specification, a preferred embodiment as shown in the drawings is further described in detail as follows:

The "connection" mentioned in this embodiment does not actually have an absolute limitation in space and position. Specifically, the word "connection" should be reasonably understood as any physical connection that can realize the function. The physical connection includes but is not limited to mechanical connection, electrical connection, wired connection or wireless connection, and the present invention is not limited to it.

First, please refer to FIG. 1, which is a structural diagram of an embodiment of the ship identity image real-time recognition and superposition system of the present invention. As shown in FIG. 1, the ship identity image real-time recognition and superposition system 10 described in this embodiment includes a processing module 100, a storage module 200, at least one camera (300a, 300b) and a water object information receiving module 400. First, in this embodiment, the processing module 100 is connected to the camera 300a and the camera 300b that can be set in two different ways.

Specifically, the camera 300a and the camera 300b of this embodiment are cameras for receiving visible light. Of course, if necessary, the camera 300a and the camera 300b can both use invisible light sensing technology such as thermal imaging to achieve the effect of assisting recognition when there is insufficient visible light, such as at night or on rainy days.

Under the condition of sufficient visible light, the number of cameras 300a and 300b of the present embodiment can be set to multiple according to the geographical location of different ports, and the present invention is not limited. Specifically, the difference between the camera 300a and the camera 300b of the present embodiment lies in the location of the installation. Among them, the camera 300a is generally installed on the shore; while the camera 300b is installed on the water vehicle. Therefore, the camera 300b of the present embodiment is carried by the water vehicle V. The connection between the camera 300b and the processing module 100 can actually be realized by wireless communication.

Since the positions of camera 300a and camera 300b will affect their field of view, each camera 300a or camera 300b includes camera positioning information. For example, camera 300a is a camera installed on the shore, so its actual position and rotatable field of view are fixed. Therefore, the camera positioning information of camera 300a does not need to be calibrated, and can be a fixed Global Positioning System (GPS) coordinate, and the position is confirmed in conjunction with the angle that has been rotated.

In contrast, since the camera 300b is carried by the water vehicle V, the camera positioning information of the camera 300b needs to be further corrected by the navigation instrument equipment VC on the water vehicle V. Specifically, the navigation instrument equipment VC can provide information such as the global positioning system (GPS) coordinates and heading of the water vehicle V as the camera navigation correction signal; combined with the current known rotation angle of the field of view of the camera 300b, the camera positioning information of the camera 300b can be corrected by the camera navigation correction signal and then returned to the processing module 100.

Of course, if the camera 300b has a built-in component with a global positioning system (GPS), an inertial measurement unit (IMU) or a combination thereof, it can also be calibrated without the aid of the avionics device VC. Of course, in this embodiment, the camera 300b can also choose to compare with the positioning information of the avionics device VC to more accurately calibrate the camera positioning information of the camera 300b.

Therefore, the camera 300a and the camera 300b of this embodiment can adopt a camera with at least three-dimensional camera movement capability. The at least three-dimensional camera movement capability can be a three-dimensional image control capability such as left and right (Pan), up and down (Tile), and zoom. Furthermore, for the rotation required for the above-mentioned camera movement, the camera 300a and the camera 300b used in this embodiment rotate at an angle of 120 degrees per second as the best. In some embodiments, the camera 300a and the camera 300b themselves can be built-in with invisible light sensing functions such as thermal imaging, and the present invention is not limited thereto.

Accordingly, the processing module 100 of the present embodiment can clearly know the most likely Global Positioning System (GPS) coordinates of each aquatic object within the field of view of the camera 300a or the camera 300b through the camera positioning information from the camera 300a or the camera 300b.

In addition, the surveillance images captured by the camera 300a and the camera 300b of this embodiment can be output and displayed by connecting to a terminal device with a screen or a display, and the present invention is not limited thereto. Therefore, the processing module 100 of this embodiment can perform object detection based on the water objects that appear in the surveillance images of the camera 300a and the camera 300b (i.e., enter the field of view of the camera 300a and the camera 300b). In this embodiment, the water objects can include but are not limited to any water objects that can sway on the water with the water body. Specifically, the water objects described in this embodiment can be water vehicles, such as ships, etc., and the present invention is not limited thereto.

Furthermore, the processing module 100 of this embodiment can further perform object recognition on the aquatic object through the monitoring images of the camera 300a and the camera 300b. Specifically, this embodiment performs object recognition on the aquatic object to confirm the image category of the aquatic object.

The object recognition is also achieved by the processing module 100 running the image recognition artificial intelligence model. Specifically, the storage module 200 of this embodiment can store the image recognition artificial intelligence model that is provided for the processing module 100 to run and detect or recognize the water objects appearing in the surveillance image of the camera 300a or the camera 300b.

Of course, the image recognition artificial intelligence model can also be accessed by the processing module 100 through the network to access the image recognition artificial intelligence model on the cloud server; even more, when the processing module 100 is set in the form of a single-chip microcomputer in the camera 300a or the camera 300b, the image recognition artificial intelligence model can also be run in the camera 300a or the camera 300b by itself, and the present invention is not limited to this. Specifically, in this embodiment, the image recognition artificial intelligence model provided by the processing module 100 to run is yolov3-tiny. The image recognition artificial intelligence model is trained using at least one aquatic object data set (Data Set) stored in the storage module 200, and is implemented only after confirming that the detection and recognition accuracy rates are both above 85%.

Therefore, the image recognition artificial intelligence model trained by the present embodiment and run by the processing module 100 can match the hardware specifications of the camera 300a and the camera 300b, and can achieve a detection and recognition accuracy of more than 85% for water objects within 1.5 kilometers of the camera 300a and the camera 300b without optical or digital amplification. In addition, when the water object is more than 1.5 kilometers away from the camera 300a and the camera 300b, the processing module 100 can control the camera 300a and the camera 300b to track the water object within a range of 0.5 to 6 kilometers by optical or digital amplification to assist in recognition.

Therefore, the processing module 100 of the present embodiment can utilize at least one aquatic object data set (Data Set) stored in the storage module 200 to distinguish the image category of the at least one aquatic object by object recognition (Object recognition). In the present embodiment, the image category may include but is not limited to container ships, tankers, fishing boats, yachts or warships. The image category that can be identified in the present embodiment is determined by the image recognition artificial intelligence model run by the processing module 100. More specifically, it is determined by the results of the aforementioned image recognition artificial intelligence model after being trained with the data content of the aquatic object data set (Data Set) stored in the storage module 200.

In other words, if the image recognition artificial intelligence model executed by the processing module 100 of the present embodiment can be used to define the ship classification according to, but not limited to, the ship classification in the Automatic Identification System (AIS). Therefore, the water object data set (Data Set) stored in the storage module 200 should store at least but not limited to manually labeled Wing In Grnd, airfoil, hydrofoil, patrol vessel, local vessel, fishing boat, tugboat, ferry, dredger, cruise ship, naval ship, container ship, sailing vessel, bulk carrier, pleasure craft, tanker, hovercraft, submarine, search and rescue vessel, port tender, pollution control vessel, hospital ship, etc. ship), special vessel, pilot vessel, and distant vessels.

On the other hand, the storage module 200 connected to the processing module 100 of this embodiment can be a component such as a solid state drive (SSD) to store at least one water object data set. The at least one water object data set can be a water object data set that has been manually labeled and provides the processing module 100 with the location and category of the water object for detection and recognition. Therefore, the processing module 100 of this embodiment can actually include a central processing unit (CPU), a graphics processing unit (GPU), a single-chip microcomputer or a combination thereof for running image recognition artificial intelligence.

The water object information receiving module 400 of this embodiment is also connected to the processing module 100. The water object information receiving module 400 includes an automatic identification system (AIS) receiving station. The automatic identification system (AIS) receiving station can be a building or equipment with a vessel traffic service system (VTS-Vessel Traffic Service), and the present invention is not limited thereto.

Specifically, the receiving station for receiving the Automatic Identification System (AIS) signal in the water object information receiving module 400 of this embodiment can be set to multiple, and the present invention is not limited thereto. Moreover, each Automatic Identification System (AIS) receiving station should be able to receive the Automatic Identification System (AIS) signal from at least one water object within at least 20 nautical miles.

Therefore, the aquatic object information receiving module 400 of this embodiment receives a plurality of aquatic object information within a time segment, and transmits the plurality of aquatic object information to the processing module 100. In this embodiment, the time segment adopted by the aquatic object information receiving module 400 is 60 seconds; data exceeding 60 seconds is discarded. The plurality of aquatic object information referred to in this embodiment is the Automatic Identification System (AIS) signal.

Specifically, the Automatic Identification System (AIS) signal includes Global Positioning System (GPS) coordinates, ship type, Maritime Mobile Service Identifier (MMSI), ship name, ship speed, heading or a combination thereof. Basically, the aforementioned water object data set is established in conjunction with the ship type. Therefore, the types of vessels received by the water object information receiving module 400 in this embodiment should at least include but not be limited to Wing In Grnd (airfoil), hydrofoil, patrol vessel, local vessel, fishing boat, tugboat, ferry, dredger, cruise ship, naval ship, container ship, sailing vessel, bulk carrier, pleasure craft, tanker, hovercraft, submarine, search and rescue vessel, port tender, pollution control vessel, hospital ship, special vessel, pilot vessel or a combination thereof.

In view of the fact that the Automatic Identification System (AIS) may update its ship type classification, when the Automatic Identification System (AIS) updates the relevant ship type, the water object data set (Data Set) of this embodiment can also be added by the user to improve the recognition capability of the processing module 100.

Accordingly, the processing module 100 of this embodiment can match the camera positioning information from the camera 300a and the camera 300b, the image category of the water object in the surveillance screen of the camera 300a and the camera 300b identified by the processing module 100, and the multiple water object information from the water object information receiving module 400.

In this embodiment, since the water object information includes information such as the ship type and the Maritime Mobile Service Identifier (MMSI), in conjunction with the camera positioning information obtained by the processing module 100 from the camera 300a and the camera 300b and the image category of the water object identified by the processing module 100, the processing module 100 can generate a water object information tag, and the processing module 100 can annotate the water object information tag after matching, and then annotate it on the water object in the monitoring image that best matches the matching result.

Specifically, the processing module 100 of this embodiment matches the water objects in the monitoring image and labels the water object information labels that are updated in real time. The first step is to compare the part of the global positioning system (GPS) coordinates in the multiple water object information with the camera positioning information of the camera 300a or the camera 300b. After the camera 300a or camera 300b whose field of view includes the global positioning system (GPS) coordinates of the target aquatic object is selected, the global positioning system (GPS) coordinates of the target aquatic object are compared with the camera positioning information of the camera 300a or camera 300b to form the image coordinates in the monitoring screen of the camera 300a or camera 300b.

Accordingly, the image coordinates in the camera 300a or the camera 300b are used to further compare the water objects in the surveillance image with the ship type, Maritime Mobile Service Identifier (MMSI), ship name, ship speed, heading or a combination thereof in the water object information, and the water object information tag is annotated on the target water object.

Next, please refer to FIG. 1 and FIG. 2 simultaneously. FIG. 2 is a flow chart of the operation method of the ship identity image real-time recognition and superposition system of the present invention. As shown in FIG. 2, step (A) of this embodiment is to first provide the ship identity image real-time recognition and superposition system 10 as described in the previous embodiment (i.e., FIG. 1). Then step (B) is to make the water object information receiving module 400 receive the multiple water object information transmitted by at least one water object within a time segment (60 seconds in this embodiment), and transmit the multiple water object information to the processing module 100.

In view of the fact that the update frequency of the aquatic object information receiving module 400 of this embodiment is not as real-time as the camera 300a and the camera 300b, the step (C) is that the processing module 100 selects at least one camera (300a or 300b) whose field of view includes the at least one aquatic object according to the plurality of aquatic object information, and captures at least one surveillance image. Specifically, since the camera 300a or the camera 300b of this embodiment includes the camera positioning information, the processing module 100 can determine the camera positioning information and then select the most suitable camera 300a or camera 300b.

In step (D) of the present embodiment, the processing module 100 performs the object detection according to at least one aquatic object in the surveillance image of the camera 300a or the camera 300b, and then utilizes at least one aquatic object data set in the storage module 200 to distinguish at least one image category of the at least one aquatic object by the object recognition.

Specifically, the object recognition is also achieved by the processing module 100 running the aforementioned image recognition artificial intelligence model. The object category can be, but is not limited to, a container ship, a tanker, a fishing boat, a yacht, or a warship. The vessel category that can be recognized in this embodiment is determined by the image recognition artificial intelligence model run by the processing module 100. More specifically, it is determined by the results of the aforementioned image recognition artificial intelligence model after being trained with the data content of the water object data set stored in the storage module 200.

Of course, when the processing module 100 of the present embodiment identifies the type of the aquatic object, if it is determined that the aquatic object is not sufficiently resolved in the monitoring image or is obscured by an obstruction, the processing module 100 will activate a detailed object recognition method. Specifically, the detailed object recognition method is that the processing module 100 first controls the corresponding camera 300a or 300b to enlarge the aquatic object in the monitoring image by an amplification means. The amplification means includes but is not limited to optical amplification, digital amplification or a combination thereof to increase the display area of the aquatic object.

Next, the processing module 100 runs the aforementioned image recognition artificial intelligence model according to the water object data set (Data Set) stored in the storage module 200, and performs the object recognition (Object recognition) on at least one identifiable characteristic object on the water object after being magnified by the magnification means. In this embodiment, the at least one characteristic object includes but is not limited to the bow, stern, chimney, boom, lifeboat, flag, color, radar, gun, identification plate or a combination thereof. In contrast, when there are characteristic objects that need to be recognized in particular, it should be understood that the water object data set (Data Set) stored in the storage module 200 also has corresponding labeled related images and is provided to the image recognition artificial intelligence model for training, and the present invention is not limited thereto. Finally, the processing module 100 can identify the image type of the aquatic object according to the at least one identified characteristic object category, and then enter step (E).

In the above-mentioned steps (A) to (D) of this embodiment, as long as any detection or recognition step encounters that the visible light brightness of the monitoring image is lower than a recognition brightness threshold, the processing module 100 can activate the built-in invisible light sensing function of the camera 300a or 300b.

Specifically, the brightness threshold can be determined by a photo sensor connected to the processing module 100. The photo sensor can measure the lumen (lm) value of the ambient visible light to determine whether it is cloudy, or various weather sensors (such as a barometer, a hygrometer, a suspended particle sensor or a thermometer, etc.) connected to the processing module 100 can be used to determine whether the weather conditions may block or hinder the camera 300a or 300b from receiving visible light, and the present invention is not limited thereto.

At the same time, in this embodiment, when the processing module 100 mainly processes the camera 300b installed on the water vehicle V, the water vehicle V carrying the camera 300b further feeds back the camera instrument correction signal through its instrument equipment VC to the processing module 100. Accordingly, the processing module 100 can correct the camera positioning information of the camera 300b according to the camera instrument correction signal.

Finally, in step (E), the processing module 100 matches the camera positioning information from the camera 300a and the camera 300b, the image category of the water object in the monitoring screen of the camera 300a and the camera 300b identified by the processing module 100, and the plurality of water object information from the water object information receiving module 400. In addition, the generated water object information label is annotated on the corresponding matched water object in the monitoring screen.

However, the above is only a preferred embodiment of the present invention, and should not be used to limit the scope of implementation of the present invention. That is, simple changes and modifications made according to the scope of the patent application and the description of the present invention are still within the scope of the present invention.

10: Ship identity image real-time recognition superposition system 100: Processing module 200: Storage module 300a: Camera 300b: Camera 400: Water object information receiving module V: Water vehicle VC: Navigation equipment (A)~(E): Steps

FIG. 1 is a structural diagram of an embodiment of the ship identity image real-time recognition and superposition system of the present invention.

FIG. 2 is a flow chart of the operation method of the ship identity image real-time recognition and superposition system of the present invention.

(A)~(E): Steps

Claims (15)

A ship identity image real-time recognition and superposition system, comprising: a processing module; a storage module connected to the processing module, the storage module comprising at least one water object data set (Data Set); at least one camera connected to the processing module, the at least one camera comprising at least one camera positioning information, the at least one camera photographing at least one surveillance picture, and the processing module performs an object detection (Object detection) based on at least one water object in the at least one surveillance picture, and then uses the at least one water object data set (Data Set) to distinguish at least one image category of the at least one water object by an object recognition (Object recognition); and A water object information receiving module is connected to the processing module, and the water object information receiving module receives a plurality of water object information within a period of time and transmits the plurality of water object information to the processing module; Wherein, the processing module matches the at least one camera positioning information, the at least one image category and the plurality of water object information to generate at least one water object information tag, and the processing module marks the at least one water object information tag on the at least one water object in the at least one monitoring screen. A real-time recognition and superposition system for ship identity images as described in claim 1, wherein the processing module includes a central processing unit (CPU). A real-time recognition and superposition system for ship identity images as described in claim 2, wherein the processing module further includes a graphics processing unit (GPU). A real-time recognition and superposition system for ship identity images as described in claim 1, wherein the at least one aquatic object is a water vehicle. A vessel tracking and monitoring system as described in claim 1, wherein the plurality of water object information is an Automatic Identification System (AIS) signal, and the plurality of water object information includes Global Positioning System (GPS) coordinates, vessel type and Maritime Mobile Service Identifier (MMSI). A vessel tracking and monitoring system as described in claim 5, wherein the plurality of water object information further includes ship name, ship speed, heading or a combination thereof. The vessel tracking and monitoring system as described in claim 5, wherein the vessel type comprises a wing in Grnd (airfoil), a hydrofoil, a patrol vessel, a local vessel, a fishing boat, a tugboat, a ferry, a dredger, a cruise ship, a naval ship, a container ship, a sailing vessel, a bulk carrier, a pleasure craft, a tanker, a hovercraft, a submarine, a search and rescue vessel, a port tender, a pollution control vessel, a hospital ship, a special vessel, a pilot vessel, or a combination thereof. A ship tracking and monitoring system as described in claim 1, wherein the at least one camera is installed on the water vehicle, the shore, or a combination thereof. As described in claim 8, when the at least one camera is installed on the water vehicle end, the at least one water vehicle carrying the at least one camera further feeds back at least one camera instrument correction signal to the processing module, and the processing module further corrects the at least one camera positioning information based on the at least one camera instrument correction signal. A method for operating a ship identity image real-time recognition and superposition system, comprising: (A) providing the ship identity image real-time recognition and superposition system as described in claim 1; (B) the water object information receiving module receives the plurality of water object information transmitted by the at least one water object within the time segment, and transmits the plurality of water object information to the processing module; (C) the processing module selects the at least one camera whose field of view includes the at least one water object according to the plurality of water object information, and captures the at least one surveillance image; (D) the processing module performs the object detection (Object detection) according to the at least one water object in the at least one surveillance image, and then uses the at least one water object data set (Data Set) to identify the object (Object recognition) to distinguish at least one image category of the at least one aquatic object; and (E) the processing module matches the at least one camera positioning information, the at least one image category and the plurality of aquatic object information to generate at least one aquatic object information tag, and the processing module marks the at least one aquatic object information tag on the at least one aquatic object in the at least one surveillance image. The operating method of the ship identity image real-time recognition and superposition system as described in claim 10, wherein the plurality of water object information is an Automatic Identification System (AIS) signal, and the plurality of water object information includes Global Positioning System (GPS) coordinates, ship type and Maritime Mobile Service Identification Code (MMSI). The operating method of the ship identity image real-time recognition and superposition system as described in claim 11, wherein the multiple water object information further includes the ship name, ship speed, heading or a combination thereof. The method for operating the real-time recognition and superposition system of ship identity images as described in claim 12, wherein the ship types include wing in Grnd (airfoil), hydrofoil, patrol vessel, local vessel, fishing boat, tugboat, ferry, dredger, cruise ship, naval ship, container ship, sailing vessel, bulk carrier, pleasure craft, tanker, hovercraft, submarine, search and rescue vessel, port tender, pollution control vessel, hospital ship, special vessel, pilot boat, etc. Vessel) or a combination thereof. The operating method of the ship identity image real-time recognition and superposition system as described in claim 11, wherein the at least one camera is installed on the water vehicle end, the shore end or a combination thereof. The operating method of the ship identity image real-time recognition and superposition system as described in claim 14, wherein when the at least one camera is installed at the end of the water vehicle, the at least one water vehicle carrying the at least one camera further feeds back at least one camera instrument correction signal to the processing module, and the processing module further corrects the at least one camera positioning information based on the at least one camera instrument correction signal.

Images