KR102054153B1 - Artificial intelligence automatic identification system by fusion of deep run based submarine sonar data and periscope image data - Google Patents

Abstract

The present invention relates to an artificial intelligence automatic identification system by fusion with deep learning-based submarine sonar sound wave data and periscope image data and, more specifically, to an artificial intelligence automatic identification system by fusion with deep learning-based submarine sonar sound wave data and periscope image data, capable of performing rapid and accurate targeting as target identification information is extracted by learning based on the deep learning in periscope image analysis and sonar sound wave analysis performed depending on a skill level of an existing sonar listener. To achieve the purpose of the present invention, the artificial intelligence automatic identification system by fusion with deep learning-based submarine sonar sound wave data and periscope image data includes: a sonar sensor identifying the kind of a target by receiving noise or reflection sound wave data in which a sound wave is radiated and returned by being reflected; a periscope generating the image data by photographing an object on the water; a platform server extracting the sound wave on the target by analysis of the image data and the sound wave data by receiving the image data and the sound wave data from the sonar sensor and the periscope and determining a first target model by comparing the sound wave extracted with the learned sound wave model data, wherein the platform server also determines a final target model by receiving confirmation information of the sonar listener and confirming the first target model predetermined and generates the identification information on the target by matching the image data of the telescope and the sound wave data of the final target model; and a client terminal connected to communicate with the platform server by data and receiving the target information and a result of the image data and the sound wave data to an administrator by receiving the target information and the result of the image data and the sound wave data identified by the platform server, wherein the client terminal displays warning data in accordance with event occurrence in accordance with the setting.

Description

Artificial intelligence automatic identification system by fusion of deep run based submarine sonar data and periscope image data}

The present invention relates to an artificial intelligence automatic identification system through the convergence of deep learning submarine sonar sound wave data and periscope image data. More specifically, the sonar sound wave analysis and periscope image analysis performed in dependence on the proficiency of a conventional sonar listener are performed. The present invention relates to an artificial intelligence automatic identification system through the convergence of deep learning-based submarine sonar or sound wave data and periscope image data, which can be performed quickly and accurately as target identification information is extracted through learning based learning.

SONAR is an abbreviation of Sound Navigation and Ranging. It is a device that finds the direction and distance of a target by sound waves. It is also called a sound detection device or sound detector.

In the air, electromagnetic waves are transmitted faster and farther than sound waves, so it is possible to detect air, ground and sea targets using radar, and the corresponding underwater equipment is sonar.

The sound wave applied to the sonar is a pressure wave that is about 1,500m per second and has a good propagation in water. Therefore, it is used as the only means of detecting active and passive targets in water until now, and various types of sonars have been developed and operated according to the purpose and use.

The sonar of these submarines is the most important means of identifying enemies, taking into account the nature of their navigation in the water. In addition, the sonar of the submarine may be passive sonar, intercept sonar, passive ranging sonar, flank array sonar, or ship in consideration of detection and operational characteristics. It can be classified into Towed Array Sonar, Active Sonar, and Mine Avoidance Sonar.

These sonars can be used to derive the horizontal orientation and distance to the target from the submarine. The target's orientation is detected through a water bath, a waterproof station, a side array station, or an array of towing units, and the distance of the target is controlled by the PRS installed in the longitudinal direction of the submarine or by TMA (Target Motion Analysis). Can be identified.

Submarines operate three types of navigational operations: water navigation, underwater navigation and snorkel navigation. Sea navigation refers to general voyages, such as existing ships and merchant ships, and operates when sailing with low-depth voyages and coastal navigation standards.

Underwater voyage means that when a submarine reaches a submarine navigable area after the sea voyage, the ship voyages under water for special purposes for submarine operations, training, and security. It can be divided into depth, operation depth, and test depth. The final snorkel is a special purpose voyage, including the submarine's snorkel navigation for the operation of a diesel engine after submarine navigation, communication, object identification and information, and the entire procedure for injury (sea navigation). .

The purpose of snorkel navigation is that diesel submarines are powered by batteries mounted underneath the submarine to power the combat system and engine system during underwater navigation. Since the main power of the diesel submarine is the battery, periodic battery charging after a certain period of time is performed by operating the diesel engine.

At this time, the external air to be introduced during the operation of the diesel engine is supplied through the snorkel mast. The purpose of the correspondence communication is to raise the communication master to send information to the upper part of the submarine in order to report the information from the upper part of the navigation or special information during the underwater guard.

The purpose of object identification and information acquisition is to use the sonar auditor (sound exploration) to listen to information through real-time listening and analysis through periscope through sound waves coming in through SONAR, which acts as the eyes and ears of the submarine during underwater navigation. Periscope sails for confirmation and filming.

In order to perform the last injury (sea navigation), the submarine performs the operation at the normal operation depth, changes the depth to the safety depth, listens to the information identified through SONAR, and analyzes the injury in the zone of injury. A visual periscope is used to identify whether there is a sea object in order to prevent a collision.

The most dangerous moment of the voyage is when the submarine rises from the depth of safety to the periscope. Inadequate sonar listening and analysis results in tactical exposure of the submarine resulting from errors in the analysis and judgment of target types (ships, fishing vessels, warships), bearings, and distances. If there is a moving target, all officers are prepared to enter emergency preparedness in case of periphery, since the situation is very dangerous to the crew safety, such as the submarine break through due to collision.

If it is determined that the submarine's operational tactics and safety have been secured after checking the type of target (merchandise / fishing boat / ship), bearing, and distance based on the identified information listened and judged by the primary sonar auditor (sound probe). The captain decides to enter the periscope and orders a change in depth. After entering the periscope, the captain should see the situation on the water surface within 10 seconds through the periscope, and if there is no problem after judging that the submarine's confidentiality and stability can be maintained, the second visual confirmation through the periscope will confirm the periscope Perform the purpose.

If the first sonar auditorium (sound probe) is different from the situation identified through the listening judgment after entering the periscope, it is necessary to prevent the tom and collision depending on the type of target (ship, fishing vessel, warship), bearing, and distance. Make an emergency dive. As such, periscope navigation is essential for the operational purpose of the submarine and for the operation of the ship. These essential periscopes vary depending on the situation, but enter the periscope depths more than five to six times to achieve object identification and information within two to three basic periods within 24 hours, to achieve that special purpose.

In order to perform this important special purpose, the submarine must be identified and judged accurately by the information received through the sonar, which acts as the ear in the primary water, and the visual information coming through the periscope, which acts as the secondary eye. The advantage of confidentiality can ensure successful operations and safe navigation of the submarine.

However, the mandatory, tactical importance, and the most vulnerable periscope for safety are still analyzed only by human experience and judgment through sonar audit. Because of this, the dependence of the experienced sonar auditor (sound exploration) is very high.

In addition, the submarines have different tactical efficiencies because of their different experiences and proficiency. Although sonar auditing (sound exploration) has high experience and proficiency, the accuracy of the analyzed results also differs depending on the experience of noisy noise or the impact of sound waves or ambient noise.

Korean Patent Publication No. 10-2015-0059191

The present invention has been made in order to solve the above problems, the sonar sound wave analysis and periscope image analysis performed in accordance with the existing sonar auditor's proficiency extract the target identification information through deep learning based fast and accurate target specification The purpose is to provide an artificial intelligence automatic identification system through the convergence of deep learning-based submarine sonar sound wave data and periscope image data that can be performed.

The present invention has the following features to achieve the above object.

The present invention includes a sonar sensor for transmitting the sound wave and receiving the reflected sound wave data returned or reflected and identifying the type of target by receiving noise; A periscope for photographing objects in the water and generating image data; Receives sound wave data and image data from the sonar sensor and periscope, and analyzes the sound wave data and image data to identify the primary target model through sound wave extraction for the target and comparison with the learned sound wave model data. A platform server configured to receive the information, identify a specific primary target model, identify a final target model, and generate identification information of a corresponding target by matching sound wave data of the final target model with image data of a periscope; A client terminal connected to communicate with the platform server to receive sound wave data and image data identified from the platform server and provide target information to the administrator, and display the alarm data according to an event occurrence according to a setting; Include.

The sonar sensor includes a sound wave data receiver for receiving sound wave data, and the periscope includes an image data receiver for receiving captured image data.

In addition, the platform server is a data storage unit for receiving and storing the sound wave data and the image data from the sound wave data receiver and the periscope image data receiver of the sonar sensor, and the sound wave data stored in the data storage unit to call the sound wave to the target A sound wave analysis unit for identifying the final target model by identifying the primary target model by inputting the confirmation information of the sonar auditor, by identifying the primary target model through the comparison with the extracted acoustic wave model data based on extraction and deep learning. And an image analyzer which separates a scene from the image data by calling the image data stored in the data storage unit, and generates a representative shot, an object, an action verb, a key frame and visual knowledge, and a final target model of the sound wave analyzer. Match the visual knowledge of the image analyzer to generate identification information of the target and store it in the database in the platform server It said matching includes identification unit for transmitting the sound wave data and the result of the image data, the final target or a target model identification information for identifying to the client terminal.

In addition, the sound wave analysis unit extracts the object sound wave for the target by calling the sound wave data stored in the data storage unit to generate a target sound wave data, and the target sound wave data extracted from the target sound wave extraction unit and Deep learning-based feature-specific sound wave data is compared with the trained sound wave model data, the sound wave learning model unit for extracting the sound wave model data having a consistency above the set value, and the target information according to the extracted sound wave model data primary target A target model diagnosis unit for specifying the model, and a target model confirmation unit for receiving the confirmation information of the sonar auditor for the specified primary target model to specify the final target model.

The image analyzer may further include: separating a scene by calling image data stored in the data storage unit, extracting a representative shot from each of the separated scenes, and a frame constituting the representative shot. Extracting an object from the object; extracting an action verb based on the mutual relationship between the extracted objects; and keyframes a frame that is determined to have the best relationship with the object on which the action verb is extracted. Analyzing is performed including the step of selecting and generating the visual knowledge based on the selected keyframe.

In addition, the client terminal receives the results of the sound wave data identified from the matching identification unit of the platform server and the sound wave data display unit for displaying along with the sound wave data of the target model having a matching degree above the set value, and identification from the matching identification unit of the platform server Image data display unit for displaying the result of the image data and the image data of the target model having a matching degree above the set value and the target information display unit for receiving the identification information of the final target model or target from the matching identifier of the platform server and display it And an alarm display unit for generating and displaying alarm data according to a setting when the target approaches a certain distance from the submarine or changes a certain angle direction.

In addition, the matching identification unit generates a warning message to receive additional learning or target reanalysis command information according to the situation when new information that does not match the matching condition or is acquired when the final target model of the sound wave analysis unit and the visual analysis unit of the image analysis unit is acquired. Let's do it.

According to the present invention, the sonar sound wave data and the periscope image data are learned by utilizing deep learning based learning, compared to the existing technology of performing target identification depending on the proficiency of the primary sonar auditory sound (sound exploration) and the secondary periscope navigator. As a result, the automatic identification and analysis of the target can be performed more quickly and accurately, thereby improving the performance of the combat system as well as ensuring safe navigation, and it can be used in the training of the submarine crew simulation.

1 is a block diagram showing the internal configuration of an artificial intelligence automatic identification system according to an embodiment of the present invention.
2 is a diagram illustrating a sound wave data and image data acquisition process according to an embodiment of the present invention.
3 is a diagram illustrating sound wave data and image data acquired according to an embodiment of the present invention.
4 is a diagram illustrating an analysis process of an image analyzer according to an exemplary embodiment of the present invention.
5 is a conceptual diagram illustrating a structure of a scene.
6 is a view showing an output screen of the client terminal according to an embodiment of the present invention.

In order to explain the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, the following describes exemplary embodiments of the present invention and looks at it with reference.

First, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention, and singular forms may include plural forms unless the context clearly indicates otherwise. Also in this application, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, one or more other It is to be understood that the present invention does not exclude the possibility of the presence or the addition of features, numbers, steps, operations, components, parts, or a combination thereof.

In describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

1 is a block diagram showing the internal configuration of an artificial intelligence automatic identification system according to an embodiment of the present invention, Figure 2 is a view showing a sound wave data and image data acquisition process according to an embodiment of the present invention, Figure 3 Is a diagram showing sound wave data and image data acquired according to an embodiment of the present invention.

Referring to the drawings, the artificial intelligence automatic identification system 100 according to an embodiment of the present invention transmits sound waves to receive the reflected sound wave data returned or reflected or sonar sensor 110 to identify the type of the target by receiving noise And a periscope 120 for generating image data by photographing an object of the water, and receiving sound wave data and image data from the sonar sensor 110 and periscope 120 and analyzing the sound wave data and image data to a target. Specify the primary target model through sound wave extraction, and compared with the sound wave model data learned, and confirm the specific primary target model by receiving the confirmation information of the sonar auditor to specify the final target model, and the final target model of the A platform server 130 for matching the sound wave data with the image data of the periscope to generate identification information of the target, and connecting the platform server 130 to enable data communication. It is composed of a client terminal 140 to display the warning data based on the event data in accordance with the sound waves and by receiving the results, and the target information of the video data providing them to the administrator sets identified from the server platform 130.

Here, the sonar sensor 110 is provided to identify the distance, position, type, etc. of the target through the received sound wave data, such sonar sensor 110 is largely passive sonar, active sonar, side array, water, or the like Sonar or tow sonar and the like may be composed of one or a combination.

Passive sonar can identify the type of target by receiving noise from engines or propellers from a distance when the submarine is sailing, as if the submarine sailed quietly at low speed in the core waters or on the street, listening to the sound of the stomach with a stethoscope. The sonar only listens to all the sounds coming from underwater. The manual sonar analyzes whether it is a warship, a merchant ship or a fishing boat, how many propulsion shafts, how many screw blades, course and speed.

The active sonar is a signal returned from the target by the sound waves emitted from the water vessel can detect the enemy submarine or mine, and can know the direction and distance of the target. As if yelling "yay" at the top of the mountain, it returns to echo, the submarine transmits sound waves and hits the target and then returns. However, when a submarine transmits sound waves, its location is immediately exposed, and it is rarely used except for instantaneous transmission to find the exact distance just before underwater obstacle check or torpedo attack in safe waters.

The side array sonar is a sonar system equipped with two long passive arrays on the left and right sides of the submarine to detect sound of a low frequency band, and the waterproof sonar analyzes the sonar sound waves of the opposing submarine to detect the submarine. You can check the type and find out which trap is equipped with the sonar.

The geostationary sonar is a sonar that has three passive arrays arranged on the side of the submarine and can measure the target distance by the time difference of the sound received from the target, and the towing sonar tows the passive array so as not to be disturbed by the submarine's own noise. Low frequency sonar.

The submarine itself is quietly built, so the propulsion noise is small and difficult to detect. However, low frequency noise from various pumps and machines can be transmitted over long distances, making the side array sonar (FAS) or towing sonar (TAS) the most useful sonar system for detecting quiet submarines.

The tow son is at least a few hundred meters including cable, so on small submarines it's a Clip on System, and the submarines hang on the submarine at departure, but on larger submarines the Winding System is used.

The sonar sensor 110 formed of any one or more of these various sonars includes a sound wave data receiver 111 for receiving sound wave data, and the sound wave data receiver 111 is capable of data communication with the platform server 130. And to transmit the received sound wave data.

Meanwhile, the periscope 120 is provided to photograph a specific area or a specific object in the water when the submarine sails the periscope, and the image data photographed by the periscope 120 is received by the image data receiver 121, which is Is connected to the platform server 130 so as to enable data communication is configured to transmit the received image data.

In addition, the platform server 130 receives sound wave data and image data from the sonar sensor 110 and the periscope 120 to generate identification information of the target through analysis of the sound wave data and the image data, the platform The server 130 is a data storage unit 131 for receiving and storing the sound wave data and image data from the sound wave data receiver 111 of the sonar sensor 110 and the image data receiver 121 of the periscope 120, respectively; Call the sound wave data stored in the data storage unit 131 to specify the primary target model through the sound wave extraction, deep learning-based learning sound wave model data for the target, receiving the confirmation information of the sonar auditor A sound wave analysis unit 132 for identifying the final target model to identify the final target model and the image data stored in the data storage unit 131 to call a scene from the image data. The image analysis unit 133 generates a representative shot, an object, an action verb, a key frame, and a visual knowledge, and a final target model of the sound wave analysis unit 132 and the visual knowledge of the image analysis unit 133. Generating the identification information of the target by matching and storing in the database 130a in the platform server 130 and transmitting the result of the sound wave data and image data identified by the client terminal 140, the final target model or identification information of the target Matching identification unit 134 is made.

Here, the sound wave analyzing unit 132 calls the sound wave data stored in the data storage unit 131, extracts the object sound wave for the target, and generates a target sound wave data to generate the target sound wave data, and the target sound wave. Acoustic wave learning model unit 132b for extracting sound wave model data having a matching degree above a predetermined value by comparing the target sound wave data extracted from the extractor 132a and the target sound wave data for each target based on deep learning with the learned sound wave model data. And a target model diagnosis unit 132c for specifying target information according to the extracted sound wave model data as a primary target model, and receiving confirmation information of the sonar acheiver for the specified primary target model. It consists of a target model identification part 132d which specifies.

The target sound wave extracting unit 132a extracts the object sound wave that is determined as an object from the collected sound wave data as illustrated in FIG. 3, generates a target extraction sound wave data, and transmits the extracted sound wave data to the sound wave learning model unit 132b. The model unit 132b extracts sound wave model data having a matching degree or more than a set value of the received target extraction sound wave data.

Here, the acoustic wave model data is a model data trained based on deep learning, and it is found by comparing the acoustic wave model data having the highest coincidence or having a higher coincidence or higher than the set value among the sound wave model data stored for various existing objects in consideration of the environment variables. Transfer to model diagnosis unit 132c.

Of course, depending on the situation, there is no extracted sound wave model data, or two or more sound wave model data may be extracted. In this case, the target model diagnosis unit may use the sound wave model data determined to have high agreement among two or more sound wave model data depending on the setting. The two or more sound wave model data transmitted to or extracted from 132c may be delivered to the target model diagnosis unit 132c.

Accordingly, the target model diagnosis unit 132c specifies the object information of the received acoustic wave model data, that is, the target information as the primary target model, and then transfers it to the target model confirmation unit 132d, and the sonar model confirmation unit 132d transmits the sonar. It provides the listener (sound exploration) with the target extraction sound wave data, the sound wave model data extracted as having high concordance, and the primary target model to receive the confirmation information of the sonar auditor.

That is, the sonar auditor examines the data and the model to see if the target model diagnosis unit 132c has his or her opinions that are the same as the first target model, or if there are other specific target models that are judged to be identical. Confirmation information about the input is received through a separate input means.

Of course, if the opinion of the sonar auditor is different, it can be configured to perform a separate comparison, judgment, analysis process.

As such, the target model diagnosis unit 132c identifies the specific primary target model as the final target model through the confirmation information of the sonar auditor, thereby simultaneously meeting the promptness and accuracy through two decision processes.

In the conventional case, it was necessary to take a long time to understand the sound wave data of a large amount of objects depending on the skill of the sonar listener and to find one specific target model from the sound wave data acquired from the sonar sensor. In the present invention, the target model diagnosis unit 132c provides the primary target model to the sonar hearing cooker in a short time, and provides the environment to simply compare the collected acoustic wave data with the acoustic wave model data of the primary target model. The time required for this is also short.

Therefore, it is possible to ensure the accuracy through the two comparison, analysis, and judgment processes as well as the speed of the judgment process, and to maintain the uniformity of the overall result value without depending on the skill of the sonar auditor.

On the other hand, the image analyzer 133 is provided to analyze the image data received from the periscope during periscope, the analysis process of the image analyzer 133 by calling the image data stored in the data storage unit 131 Separating the scene (S100), extracting a representative shot from each of the separated scenes (S110), extracting an object from a frame constituting the representative shot (S120); And extracting an action verb based on the extracted relations between the objects (S130), and selecting a keyframe as a keyframe that is determined to have the best correlation with the object on which the behavior verbs are extracted. In operation S140, a visual knowledge is generated based on the selected keyframe.

4 is a diagram illustrating an analysis process of an image analyzer according to an exemplary embodiment of the present invention, and FIG. 5 is a conceptual diagram illustrating a structure of a scene.

First, the terms scene, shot, representative shot, key frame, and primitive visual knowledge will be briefly described with reference to FIG. 5.

The input of the visual knowledge providing apparatus for constructing the visual knowledge targets a digital image sequence (for example, a moving image, a digital motion picture, etc.). Digital image sequences are those in which digital images exist continuously per unit time. Typically, the density of a digital image sequence is expressed in terms of a few frames per second (eg, 10 frames per second, 20 frames per second).

For convenience of understanding and explanation, an example is described as a movie, which is a typical digital image sequence. A scene is the smallest scene unit in a movie scenario, where a scene in a movie refers to a series of events that occur at the same time zone in one place, and if any of the places or times change, it becomes a different scene. A scene is composed of one or more shots, and one shot may be a scene. Shot is the lowest unit when dividing a scene into cuts, and generally, shots are uninterrupted shooting or recording that begins when the director says "action" and ends when he says "cut".

These shots combine to form a scene, and a series of scenes combine to create a sequence. In other words, a typical digital image sequence is a scene-specific event, and a particular event consists of one or more shots. When a shot including an event representing a scene is defined as a representative shot in one or more shots constituting the scene, the representative shot includes a frame including an event representing a scene. More specifically, when one frame including an event representing the scene is selected in this representative shot, the frame may be a key frame.

A special example of the digital image sequence is a CCTV image similar to the periscope image data of the present invention. CCTV images are continuously stored, so it is difficult to distinguish between scenes and shots. Therefore, in case of digital image sequence recorded continuously such as CCTV image, it is classified into normal situation (the situation where no special event occurs) or abnormal condition (the situation where special event is expected) based on specific threshold information set by the administrator. Each abnormal situation is treated as one scene. In the case of a CCTV image, a separate scene becomes one shot (representative shot), and one frame representing an abnormal event is extracted from the shot and used as a key frame.

Visual knowledge expresses the information inherent in the scene in the form of a standardized metadata. It extracts representative shots from scenes that divide digital image sequences, such as CCTV images, based on events, and extracts objects from representative shots using traditional vision techniques. It extracts (syntactic information of the scene), infers event information (semantic information of the scene) based on the correlation between the extracted objects, and expresses these two information in the form of standardized metadata.

Accordingly, the image analysis unit 133 according to the present invention extracts the image data received from the data storage unit 131 to separate the scene (Scene) (S100).

At this time, the image analyzer 133 monitors the amount of change in the image data and tags the time when the amount of change exceeds the threshold as the start of the scene, and the time when the amount of change falls below the threshold as the end point of the scene. The interval image is separated and the separated image is saved as an analysis scene.

Such a threshold may be set by an administrator, and it is preferable to set the threshold high when the amount of change is large and set the threshold low when the amount of change is small according to the object.

Then, the image analyzer 133 extracts a representative shot from each of the separated scenes (S110), for example, by moving the interval region along the separated scene using a sliding window technique, entropy in the interval region. , And a section having the highest entropy can be extracted as a representative shot.

When the representative shot is extracted, the image analyzer 133 extracts an object from a frame constituting the representative shot (S120), for example, an object using a conventional vision technique for the image frame constituting the extracted representative shot. Extract and recognize and calculate the coordinates on the image of the object.

Accordingly, when object extraction is completed, an action verb is extracted based on the relationship between the extracted objects (S130).

For example, the behavior for the event of interest is defined in a formal format such as ontology, and a composition rule of unit actions constituting the defined behavior may be built in advance. The image analyzer 133 satisfies the pre-composite composition rule based on the temporal and / or spatial change relations of the extracted objects (for example, the moving speed, the moving direction, and the combination of the objects forming the mutual relationship). Can infer multiple behavioral verbs

Typically, a plurality of behavior verbs may be inferred from various relationship information between objects, and ranking may be assigned to the inferred behavior verbs. Behavior definition and composition rules model behavior and examine the characteristics of the movements of body parts, the various actions of one person, the actions that occur between two objects, and the group actions that occur in groups of multiple objects. It can be built as a formal ontology based on that.

Accordingly, the image analyzing unit 133 selects a frame that is determined to be the best representation of the object on which the behavior verb is extracted and the correlation (S140), for example, a plurality of constituent representative shots. An image frame that has the highest ranking behavior verb in the image frame may be selected as a keyframe.

Through this, visual knowledge is generated based on the selected key frame (S150). For example, an image of a selected keyframe, a visual word for the area of the minimum bounding rectangle (MBR) of the objects contained in the keyframe, an object property and an on-screen position of the objects, an inferred behavior verb, and an action verb. By synthesizing the syntactic rules used in the extraction of the syntactic and semantic elements, visual knowledge can be generated in a formal format such as XML. In addition, the visual knowledge may include basic information of the image source used in the keyframe selection process, for example, the type of periscope camera, the position of the periscope camera, the management ID of the periscope camera, the path where the raw image is stored, and the name of the raw image. , A start and end frame number of a scene, a start and end frame number of a representative shot, a frame number of a key frame, and the like.

The generated visual knowledge is transmitted to the matching identifier 134, which will be described later, wherein the matching identifier 134 is the final target model of the sound wave analyzer 132 and the visual knowledge of the image analyzer 133. It is provided to receive the matching to generate identification information of the target by matching them.

In addition, the matching identification unit 134 stores corresponding sound wave data, sound wave model data, final target model, visual knowledge, and identification information of the target in the database 130a in the platform server 130, and stores identification information of the target. It stores information such as the type, specification, direction, distance, date and time, and location detected in order.

In addition, the identification information of the final target is transmitted to the client terminal 140 together with information such as sound wave data, sound wave model data and the final target model, visual knowledge of the target.

On the other hand, the matching identifier 134 further learns according to the situation when new information that does not match the matching condition or is not acquired when the final target model of the sound wave analyzing unit 132 and the visual knowledge matching of the image analyzer 133 are acquired. Alternatively, a warning message may be generated to receive target reanalysis command information.

According to the warning message, the administrator or the sonar auditor, the navigator, etc. can input the reanalysis command information, so that the re-execution of the sound wave analysis unit 132 and the image analysis unit 133 described above is performed.

6 is a view showing an output screen of the client terminal according to an embodiment of the present invention.

Meanwhile, as described above, the client terminal 140 receives sound wave data, image data, final target model, visual knowledge, and target identification information, etc. identified from the matching identification unit 134 of the platform server 130 and transmits the same to the manager. The client terminal 140 receives the result of the sound wave data identified from the matching identification unit 134 of the platform server 130 and displays the sound wave data of the target model having a matching degree above the set value. The sound wave data display unit 141 and the image data display unit 142 for displaying the result of the image data identified from the matching identification unit 134 of the platform server 130 and the image data of the target model having a matching degree above the set value. The target information display unit 143 receives the final target model or identification information of the target from the matching identification unit 134 of the platform server 130, and displays the table. It comprises a warning display unit 144 to display the generated alert data in accordance with a submarine and a distance or an angle approaching direction to change the setting.

The client terminal 140 may be configured to transmit a control signal to receive the administrator's command information according to the setting and perform reanalysis to the platform server 130, and to receive and read the corresponding data by requesting the past history information. It may be.

As described above, the present invention has been described with reference to one embodiment shown in the drawings, but this is merely an example, and those skilled in the art may realize various modifications and other equivalent embodiments therefrom. I will understand.

Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

110: sonar sensor 111: sound wave data receiver
120: periscope 121: image data receiver
130: platform server 130a: database
131: data storage unit 132: sound wave analysis unit
132a: target sound wave extraction unit 132b: sound wave learning model unit
132c: target model diagnosis unit 132d: target model diagnosis unit
133: image analysis unit 134: matching identification unit
140: client terminal 141: sound wave data display unit
142: image data display unit 143: target information display unit
144: alarm display unit 100: AI automatic identification system

Claims (7)

A sonar sensor (110) for identifying the type of the target by receiving the sound wave data that is reflected by returning the sound wave and returning;
A periscope 120 for photographing an object of water and generating image data;
Receives sound wave data and image data from the sonar sensor 110 and periscope 120, and analyzes the sound wave data and image data to identify the primary target model through sound wave extraction for the target and comparison with the learned sound wave model data. A platform for identifying the final target model by receiving the confirmation information of the sonar auditor, identifying the final target model, and generating identification information of the target by matching the acoustic data of the final target model with the image data of the periscope. Server 130;
It is connected to the platform server 130 so as to communicate data, and receives the result and target information of the sound wave data and image data identified from the platform server 130 to provide it to the administrator and alarm data in accordance with the event occurrence according to the setting Displaying the client terminal 140; artificial intelligence automatic identification system through the deep learning based submarine sonar sound wave data and periscope image data fusion.
The method of claim 1,
The sonar sensor 110 includes a sound wave data receiver 111 for receiving sound wave data, and the periscope 120 includes a deep data learning unit 121 for receiving the captured image data. Artificial intelligence automatic identification system by combining submarine sonar sound wave data and periscope image data.
The method of claim 2,
The platform server 130 is
A data storage unit 131 for receiving and storing sound wave data and image data from the sound wave data receiver 111 of the sonar sensor 110 and the image data receiver 121 of the periscope 120, respectively;
Call the sound wave data stored in the data storage unit 131 to specify the primary target model through the sound wave extraction, deep learning-based learning sound wave model data for the target, receiving the confirmation information of the sonar auditor A sound wave analysis unit 132 for identifying the final target model by identifying the primary target model;
An image analyzer 133 for calling the image data stored in the data storage unit 131 to separate a scene from the image data, and generating a representative shot, an object, an action verb, a key frame and visual knowledge;
The final target model of the sound wave analyzer 132 and the visual knowledge of the image analyzer 133 are matched to generate identification information of the target, stored in the database 130a in the platform server 130, and the client terminal 140. Deep learning-based submarine sonar sound wave data and periscope image data, characterized in that it comprises a matching identification unit 134 for transmitting the result of the sound wave data and image data identified by the), the final target model or target identification information AI automatic identification system.
The method of claim 3,
The sound wave analyzer 132 is
A target sound extraction unit 132a for calling the sound data stored in the data storage unit 131 to extract object sound waves for a target to generate target extraction sound wave data;
A sound wave learning model unit for extracting the sound wave model data having a matching degree higher than the set value by comparing the target sound wave data extracted from the target sound wave extracting unit 132a and the target sound wave data based on deep learning with the learned sound wave model data With 132b,
A target model diagnosis unit 132c for specifying target information according to the extracted sound wave model data as a primary target model;
The deep learning-based submarine sonar sound wave data and periscope image data, characterized in that it comprises a target model confirmation unit 132d for receiving the confirmation information of the sonar auditor for the specified primary target model to specify the final target model. Artificial intelligence automatic identification system through convergence.
The method of claim 3,
The image analyzer 133 is
Separating a scene (Scene) by calling the image data stored in the data storage unit 131 (S100),
Extracting a representative shot from each of the separated scenes (S110);
Extracting an object from a frame constituting the representative shot (S120);
Extracting an action verb based on the mutual relationship between the extracted objects (S130);
Selecting as a key frame a frame that is determined to best express the relationship between the object on which the behavior verb is extracted (S140);
Deep learning-based submarine sonar or sound wave data and periscope image data, characterized in that the analysis comprises the step of generating the visual knowledge based on the selected key frame.
The method of claim 3,
The client terminal 140 is
A sound wave data display unit 141 which receives the sound wave data result identified from the matching identification unit 134 of the platform server 130 and displays the sound wave data of the target model having a matching degree higher than the set value;
An image data display unit 142 for displaying the image data of the target model having a degree of agreement with the result of the image data identified from the matching identification unit 134 of the platform server 130 and above the set value;
A target information display unit 143 which receives the final target model or identification information of the target from the matching identification unit 134 of the platform server 130 and displays it;
Deep learning-based submarine sonar sound wave data and periscope image data, characterized in that it comprises an alarm display unit 144 to generate and display the alarm data according to the setting when the target is a certain distance approach or change the angle direction with the submarine AI automatic identification system.
The method of claim 3,
The matching identification unit 134 may be further learning depending on the situation when new information is not matched or the matching condition is acquired when the final target model of the sound wave analysis unit 132 and the visual analysis of the image analysis unit 133 are acquired. Deep learning-based submarine sonar sound waves and periscope image data, characterized in that generating a warning message to receive the target re-analysis command information artificial intelligence automatic identification system.

Images