KR20230000568A - Underwater living thing monitoring device and method using Virtual Reality - Google Patents

Abstract

Disclosed are an underwater living thing monitoring device and method using a virtual reality (VR) image. According to the present invention, the underwater monitoring device using a VR image comprises: a position acquisition unit acquiring identification information and position information of each underwater living thing from a tracking device attached to an underwater robot; a control unit acquiring an underwater image photographed by a rotatably installed camera and a rotation angle of the camera; an analysis unit analyzing the image based on the position information and the control information, and extracting identification information corresponding to an individual shown on the underwater image; a management unit searching for the information on the underwater living thing corresponding to the identification information extracted from an underwater database; and an output unit generating a VR image showing the searched image information on the underwater image, and providing the VR image to a user terminal. Therefore, the accuracy and reliability of analysis can be improved.

Description

Underwater living thing monitoring device and method using Virtual Reality

The present invention relates to an apparatus and method for monitoring aquatic organisms using VR images, and more particularly, to a device for monitoring aquatic organisms using VR images, which utilizes VR (Virtual Reality) to enable an individual to realistically observe an aquatic environment. and how. This achievement was carried out with the support of “Tongmyong University University Innovation Research Complex Creation Project” among the university innovation research complex development projects of Busan Metropolitan City.

Recently, as the development of the ocean is accelerated in order to solve the depletion of land resources, population density, etc., underwater structures such as marine plants, bridges, and undersea tunnels are being built. Accordingly, the demand for technology for monitoring the environment of aquatic organisms living in the water is also increasing.

For such underwater monitoring, a method in which a diver submerged and directly photographed and directly inspected an image of an aquatic organism was conventionally used.

However, this conventional monitoring method for capturing underwater images not only carries the risk of human accidents, but also has limitations in image acquisition time and range because it is affected by conditions such as divers, weather, and the sea. In addition, since an expert in underwater environment analysis needs to analyze the underwater environment based on a two-dimensional image having a low sense of realism, there is a problem in that the accuracy of the analysis result is inevitably lowered.

Furthermore, in the conventional method of monitoring aquatic life, etc., since an aquatic environment analysis expert must analyze the aquatic environment based on a two-dimensional image with a low sense of realism, there is a problem in that the accuracy of the analysis result is inevitably reduced.

Therefore, in order to obtain information necessary for location recognition in an underwater environment, various studies have been conducted on monitoring of aquatic organisms using virtual reality, including searching for underwater objects using underwater image information.

The virtual reality (VR) is an interface between a human and a computer that creates a desired specific environment or situation with an information processing device such as a computer and makes the user feel as if they are interacting with the actual surrounding situation and environment. says In order for the user to feel a sense of reality through virtual reality, information acquired through the user's five senses, such as sight, hearing, and touch, must be provided. Therefore, a monitoring method applying the virtual reality (VR) is also gradually emerging.

Using virtual reality with these functions, it is possible to more accurately grasp the state of the aquatic environment that can monitor current aquatic organisms than the prior art in order to minimize direct and indirect effects caused by the aquatic environment, and analyze data. The development of a detection system or monitoring system and method that can be easily understood and efficiently utilized results has been required.

Korean Patent Publication No. 2012-0075899 Korean Patent Publication No. 2018-0137916 Korean Patent Publication No. 2020-0009852

Therefore, the present invention uses a video recording device such as a camera to provide a plurality of underwater images to a person who intends to monitor aquatic life with VR (virtual reality) images based on image information through a user terminal, thereby creating virtual reality. An object of the present invention is to provide a device and method for monitoring aquatic organisms using VR images that can further increase the sense of reality underwater by introducing the present invention.

In order to solve this problem, the present invention is an underwater monitoring device using VR images, by a location acquisition unit that obtains identification information and location information of each aquatic creature from a tracking device attached to an underwater robot and a rotatably installed camera. A control unit that obtains the captured underwater image and control information on the rotation angle of the camera, and an analysis unit that extracts identification information corresponding to an object appearing in the underwater image through image analysis based on the location information and the control information. A management unit that retrieves information about aquatic organisms corresponding to the extracted identification information from a database, and aquatic organism information that searches a species DB embedded in the aquatic database to generate analysis data specifying the species of the aquatic organisms. It is characterized in that it comprises a processor and an output unit for generating a virtual reality image representing the searched image information in the underwater image and providing the virtual reality image to a user terminal.

In addition, the underwater database includes a species DB in which species and genus information of aquatic organisms is stored, a map DB providing an underwater map corresponding to an area to be monitored, and underwater map information in which analysis data of specific species is mapped is stored. It is characterized in that it includes an underwater observation DB, weather data collected from an external weather server, and a weather DB in which underwater observation data are stored.

The control unit may transmit a control signal to the camera so that the camera rotates in a direction according to a user's manipulation.

And, the management unit is characterized in that the user himself additionally inputs the information on the aquatic organisms input from the user terminal or corrects the previously input information on the aquatic organisms.

Therefore, the apparatus and method for monitoring aquatic organisms using VR images of the present invention generate VR images obtained by using an image capturing device such as a camera and provide them directly to a user to monitor aquatic organisms, thereby monitoring aquatic organisms. It is possible to increase the sense of realism, and accordingly, in the case of analyzing the underwater environment based on the VR image, there is an effect of increasing the accuracy and reliability of the analysis.

1 is a schematic configuration diagram of an aquatic life monitoring device using virtual reality according to an embodiment of the present invention.
2 is a block diagram of an aquatic life monitoring device using virtual reality according to an embodiment of the present invention.
3 is a diagram illustrating an interface provided to a user terminal to remotely manipulate the direction of a camera according to an embodiment of the present invention.
4 is a diagram illustrating a virtual reality image provided to a user terminal according to an embodiment of the present invention.
5 is a block diagram showing the relationship and components between an aquatic database and an aquatic organism information processing unit;
6 is a flowchart illustrating a procedure for analyzing aquatic organisms using an aquatic database and an aquatic organism information processing unit;
7 is a block diagram of a failure diagnosis system;
8 is a process flow diagram illustrating the operation of the fault diagnosis system;

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings.

In addition, in the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

In addition, since the terms used in this application are only used to describe specific embodiments, it is not intended to limit the present invention, and it is clear in advance that a singular expression also means a plurality of expressions unless the context clearly indicates otherwise. want to leave

First, a description of virtual reality (VR), which is a term appearing in this specification, will be given. The virtual reality (VR) is a user interface technology that uses a special device to simulate what is possible inside a software program of a computer through human senses, such as sight and hearing, as if it were reality. The virtual environment or situation created in this way stimulates the user's five senses and provides a spatial and temporal experience similar to the real one, allowing the user to freely move in and out of the boundary between reality and imagination. In addition, the user can not only be immersed in virtual reality, but also interact with things implemented in virtual reality, such as manipulating or giving commands using real devices.

1 is a schematic configuration diagram of an underwater monitoring device using virtual reality according to an embodiment of the present invention, FIG. 2 is a block diagram of an underwater monitoring device using virtual reality according to an embodiment of the present invention, and FIG. 3 is It is a diagram showing an interface provided to a user terminal to remotely manipulate the direction of a camera according to an embodiment of the present invention, and FIG. 4 is a diagram showing a virtual reality image provided to a user terminal according to an embodiment of the present invention. 5 is a block diagram showing the relationship and components between the aquatic database and the aquatic organism information processing unit, and FIG. 6 is a flowchart illustrating a procedure for analyzing aquatic organisms using the aquatic database and aquatic organism information processing unit, and FIG. 7 is a block diagram of the fault diagnosis system, and FIG. 8 is a flowchart illustrating the operation of the fault diagnosis system.

1 is a schematic configuration diagram of an aquatic life monitoring device using virtual reality according to an embodiment of the present invention.

As shown, the tracking device 10 and the camera 20 are attached to the underwater robot 2 underwater, and the tracking device 10 communicating with GPS satellites (not shown) is coordinated, It has a function of receiving location information. The camera 20 for taking pictures of underwater may be equipped with a picture-taking app for obtaining image information on aquatic organisms.

In addition, the tracking device 10 may store unique identification information for identifying underwater objects found underwater with other underwater objects.

The tracking device 10 is connected to the location acquisition unit 110 (see FIG. 2), and transmits location information and identification information of each aquatic organism to the location acquisition unit 110 according to the request of the location acquisition unit 110. can

The camera 20 refers to a device that obtains a generally used image, enables photography of an underwater phenomenon or state in the water, and may be rotatably installed. Furthermore, the camera 20 is not limited to a camera and may be a sonar or a scanner. In addition, the camera 20 may be installed on an object (not shown) that moves underwater, and takes underwater pictures or videos in the form of a small underwater camera attached to the underwater robot 2.

In addition, when the camera 20 is facing a specific direction in the water, control information about the rotation angle of the camera 30 at this time may be stored in the camera 20 . The camera 20 is connected to the controller 120.

Referring to FIG. 1 again, an image of a certain area of underwater space through the camera 20 attached to the underwater robot 2 is obtained, transmitted to the controller 120, and through the aquatic creature information processing unit 50. When information on aquatic organisms living in the area is acquired and transmitted to the controller 120, the controller 120 transmits it to the user terminal 40 to display it, and applies a control signal to the output unit 150. do. The output unit 150 generates a virtual reality image representing the image information retrieved from the underwater image, and transmits the virtual reality image to the user terminal 40 as well.

2 is an overall block diagram of an aquatic life monitoring device 100 using virtual reality according to an embodiment of the present invention.

As shown, the virtual reality-based underwater monitoring device 100 according to the present invention includes a location acquisition unit 110, a control unit 120, an analysis unit 130, a management unit 140, and an output unit 150. ) may be included. In addition, the aquatic life monitoring device 100 using virtual reality (VR) of the present invention includes the tracking device 10 and the camera 20 of the underwater robot 2, the underwater database 30 and the user terminal 40 can link up

The underwater database 30 performs a function of collecting and storing information on each aquatic organism living in the water. Specifically, the information may include information such as the state of aquatic organisms, types of aquatic organisms, and characteristics of aquatic organisms. As shown, the underwater database 30 is also connected to the management unit 140.

The user terminal 40 refers to a device capable of transmitting and receiving various data, control signals, etc. via a network (not shown) according to a user's key manipulation, and displaying the processing result of data, and is a general PC, laptop ( laptop), tablet PC (Tablet PC), smart phone, personal digital assistant (PDA: Personal Digital Assistant), and any one of mobile communication terminals, but is not limited thereto.

In addition, the user terminal 40 is also connected to the output unit 150 of the present invention and also serves to display the virtual reality (VR) image received from the output unit 150.

Hereinafter, the main components of the aquatic life monitoring device 100 using virtual reality (VR) of the present invention will be described.

The location acquisition unit 110 of the present invention works in conjunction with the tracking device 10 attached to the underwater robot 2 in the water, and performs a function of obtaining underwater image information and location information from the tracking device 10. . For example, in the water, the tracking device 10 may store coordinate information received by communicating with an identification number (eg, 'A00010') and a GPS satellite (not shown).

As such, the location acquisition unit 110 of the present invention may receive ecological information and location information such as states of corresponding aquatic organisms from the tracking device 10 attached to the underwater robot 2 .

As such, it is attached to the underwater robot 2 and the like to transmit the underwater state to the outside, enabling real-time external monitoring.

Furthermore, the control unit 120 of the present invention may receive an underwater image captured by the camera 20 by connecting to and interlocking with the camera 20 .

In addition, as described above, the camera 20 is installed to be rotatable in the water, and at this time, control information on the rotation angle of the camera 20 may be stored, and the controller 120 controls the underwater image from the camera 20. And it receives control information about the rotation angle of the camera 20 and applies a control signal to the output unit 150 to be described later.

According to an embodiment of the present invention, the control unit 120 may transmit a control signal to the camera 20 for controlling the rotation direction so that the camera 20 faces an arbitrary direction, thereby transmitting an image to the user terminal 40. . In addition, as shown in FIG. 3, the user can change the moving direction and size of the image by touching one of the directions of up, down, left, and right on the video screen (not shown) displayed on the user terminal 40. can be manipulated

The controller 120 may transmit a control signal to the camera 20 to rotate the camera 20 in a direction according to a user's manipulation based on an input signal input by the user terminal 40 .

In another example, the controller 120 may transmit a control signal for controlling the direction of the camera 20 so that the camera 20 faces a specific direction.

As described above, the location acquisition unit 110 periodically obtains identification information and location information corresponding to each of the objects, such as a plurality of plants in the water.

Therefore, when the user selects a certain underwater plant through the user terminal 40, a control signal for controlling the rotation angle of the camera 20 may be transmitted based on the location information associated with the identification information of the object.

As such, the control unit 120 of the present invention directly adjusts the direction of the camera 20 through a user's manipulation, such as a touch input through an image displayed on the user terminal 40, or the identification information of a specific object of an aquatic plant and related thereto. Based on the location information, a control signal for controlling the camera 20 to illuminate a specific object may be transmitted.

The analysis unit 130 performs a function of identifying and analyzing aquatic organisms appearing in the underwater image captured by the camera 20 through image analysis.

More specifically, the analyzer 130 receives control information about the rotation angle of the camera 20 obtained by the control unit 120, and the object obtained by the control information and the location acquisition unit 110. Identification information corresponding to aquatic organisms present in the direction in which the camera 20 is facing may be extracted based on the location information of .

For example, when the camera 20 faces the 6 o'clock direction, an image taken in the 6 o'clock direction is received. Since the position of living things in the water can be confirmed by the position acquisition unit 110, identification information located in the 6 o'clock direction with respect to the camera 20 can be extracted. In this process, the analysis unit 130 generates identification information corresponding to an object appearing in the underwater image captured by the camera 20 based on position information for each aquatic creature and control information on the rotation angle of the camera 20. can be extracted. Preferably, the analysis unit 130 may perform various algorithms such as a mixed Gaussian model, Bayesian background learning, and ViBe for recognizing a specific object in an image.

In addition, the management unit 140 is connected to the underwater database 30, and if the underwater plants are extracted by the above-described analysis unit 130, the underwater plants are identified through the underwater database 30 using the identification information of the extracted underwater plants. It performs a function of searching for and receiving information on aquatic plants corresponding to the corresponding aquatic plants.

The information on the aquatic organisms may include basic information as well as phylogenetic tree information. Here, the aforementioned basic information may include information on at least one of the sex, state, type, and other characteristics of aquatic organisms.

In addition, the state of microorganisms inhabiting the water may be photographed and the image of the captured microorganisms may be displayed on the user terminal 40 or the like to observe the state of microorganisms in real time.

It is preferable that the management unit 140 of the present invention is configured to allow the user to additionally input information on aquatic organisms input from the user terminal 40 in conjunction with the aquatic database 30 or to correct previously input information on aquatic organisms. Do.

The output unit 150 of the present invention combines the search information retrieved by the management unit 140 with the underwater image captured by the camera 20 to generate a virtual reality (VR) image representing the information of the underwater image. It performs a function of providing a real image to the user terminal 40 .

Hereinafter, based on the configuration of the aquatic life monitoring device 100 using virtual reality of the present invention described above, an operation process will be described.

As shown in FIG. 2, a tracking device 10 is attached to the underwater robot 2 in the water. Accordingly, the location information and identification information about the underwater robot 2 can be obtained through the location acquisition unit 110. The location acquisition unit 110 receives location information and identification information from the tracking device 10, respectively.

Meanwhile, the camera 20 is installed to be rotatable, and the controller 120 receives an underwater image captured by the camera 20 and control information about a rotation angle of the camera 20 . If the camera 20 faces a certain direction, control information about the rotation angle of the camera 20 is stored in the camera 20, and the controller 120 controls the rotation angle of the camera 20. Receive control information about.

Next, the analysis unit 130 of the present invention can identify the underwater robot 2 shown in the underwater image based on the rotation angle of the camera 20 and the width of the underwater image obtained by the camera 20. . This is possible because the location acquisition unit 110 receives and stores location information and identification information from the tracking device 10 .

As a result, the analysis unit 130 can identify the underwater robot 2 shown in the underwater image taken by the camera 20 facing a specific direction from other objects in the water.

In the present invention, the controller 120 may transmit a control signal for controlling the rotation direction of the camera to the camera 20 according to a user's manipulation.

Specifically, referring to FIG. 3 , the user touches the movement buttons displayed on the left (a), right (b), lower (c), and upper (d) sides of the video screen displayed on the user terminal 40. You can manipulate the moving direction of the image.

At this time, the controller 120 may transmit a control signal to the camera 20 so that the camera 20 rotates in the direction according to the user's manipulation based on the input signal input by the user terminal 40 .

Hereinafter, a method for extracting a virtual reality (VR) image will be described.

When an arbitrary aquatic organism is selected and clicked through the display window (not shown) of the user terminal 40, an output window (not shown) of the system, type, scientific name, and biological characteristics of the aquatic organism is created, An information window (reference numerals are omitted) displaying the corresponding information of the selected aquatic creature in detail is automatically generated.

4 is a diagram showing a virtual reality image provided to the user terminal 40 . If a user selects a plant of a corresponding photo (reference numerals are omitted) in the user terminal 40, a data storage unit (not shown) formed in the aquatic database 30 includes data related to aquatic organisms 400. Meta data 402 capable of describing plants of the corresponding photo by classification is generated, and a data table T is created in which scientific names related to aquatic organisms including the meta data 402 are automatically generated. Accordingly, the data table T is displayed on the screen of the user terminal 40 .

For example, the aquatic life related data 400 may be included in a header portion of the aquatic life related data 400 so that the aquatic life related data 400 can be classified according to a common item included in the meta data 402 .

Items included in the meta data 402 as above may be configured in various ways, including habitat area information, etc., in addition to these common items included in the meta data 402 to provide additional information of the aquatic life-related data 400. It is used to enable accurate and readable information retrieval.

In addition, the analysis unit 130 extracts identification information corresponding to the aquatic organisms in the underwater image based on the control information on the rotation angle of the camera 20 and the location information of the respective aquatic organisms, and the management unit 140 extracts identification information corresponding to the aquatic organisms in the underwater database. (30) and information corresponding to the aquatic organisms extracted from the aquatic organism information processing unit 50 to be described later is received.

Also, the output unit 150 may display information on aquatic organisms in the underwater image and provide the information to the output window. In addition, identification information of aquatic organisms recognized by the analyzer 130 may be displayed on the information window, and information of aquatic organisms searched by the management unit 140 may be displayed.

As described above, according to the present invention, based on the rotation angle of the camera 20 and the position received from the GPS module (not shown) attached to the underwater robot 2, aquatic organisms are accurately identified in the underwater image, By providing users with virtual reality (VR) images that combine detailed information on the relevant aquatic organisms with underwater images, the aquatic organisms in the images can be identified from each other through these virtual reality images, and detailed information on the relevant organisms can be easily obtained. It has advantages that can be ascertained.

Hereinafter, referring to FIG. 5 , a description will be given of a method of obtaining information on aquatic organisms and monitoring them using the aquatic organism information processing unit 50 and the aquatic database 30 .

As shown, the aquatic creature information processing unit 50 includes a transceiver 51, an image information collection unit 52, a species determination unit 53, a map mapping unit 54, a weather link analysis unit 55, It may include a DB management unit 56, etc., and the underwater database 30 may form and include a species DB 31, a map DB 32, an underwater observation DB 33, and a weather DB 34. there is.

First, the transceiver 51 transmits underwater map information retrieved and provided from the underwater observation DB 33 under the control of the DB management unit 56, that is, underwater map information mapped with analysis data of aquatic species or aquatic species and It has a function of transmitting weather data and underwater map information mapped with linked analysis data obtained by linked analysis of underwater observation data to the user terminal 40 through the control unit 120 .

In addition, the image information collection unit 52 collects the aquatic creature image information received through the transceiver 51, transmits it to the species determination unit 53, and stores it in the underwater observation DB 33. can provide Here, the aquatic organisms may mean aquatic organisms, marine plants, red tides, and the like.

Next, when the aquatic creature image information is transmitted from the image information collection unit 52, the species determination unit 53 searches the species DB 31 in the aquatic database 30 (eg, scientific name search), thereby Functions such as generating analysis data specifying a species may be provided.

At this time, when the species of aquatic organisms is not specified, it may be determined at the genus level. Image analysis based on such a species DB may be performed, for example, using an image analysis algorithm well known in the art. .

To this end, the species DB 31 in the aquatic database 30 stores species and genus information about various aquatic organisms in an aquatic ecosystem including aquatic organisms.

In addition, the map mapping unit 54 aggregates the analysis data of the specified species and maps them to the target location of the underwater map provided from the map DB 32, thereby generating underwater map information to which the analysis data is mapped to the area to be monitored. An underwater map corresponding to may be provided, and underwater map information generated here may be stored in the underwater observation DB 33.

Here, the process of aggregating the analysis data of the specified species and mapping them to the underwater map may be performed, for example, at a predetermined period.

Next, the meteorological link analysis unit 55 combines the meteorological data provided from the meteorological DB 34, the underwater observation data, and the biological species specified through the species determination unit 53 for linkage analysis, thereby generating linked analysis data. And, it is possible to provide functions such as transmitting the generated linkage analysis data to the underwater observation DB 33 and the map mapping unit 54.

To this end, weather data collected from an external weather server (not shown) and underwater observation data may be stored in the weather DB 34 in the underwater database 30.

Here, as meteorological data, for example, temperature, humidity, precipitation, east-west wind strength, north-south wind strength, convective strength (up/down movement strength of air), surface layer temperature, depth temperature, salinity, pH, Chlorophyl-a and the like may be included.

Therefore, the map mapping unit 54 maps the linkage analysis data obtained by linkage analysis of the specified species, meteorological data, and marine observation data to target locations on the underwater map provided from the map DB 32, so that the linkage analysis data is Mapped underwater map information can be created.

In addition, the DB management unit 56 retrieves underwater map information from the underwater observation DB 33 at predetermined intervals, and when a request for transmission of an underwater map is received from the user terminal 40 through the control unit 120, the underwater observation Functions such as fetching the underwater map information from the DB 33 and transmitting it to the user terminal 40 may be provided.

Here, the underwater map information may be underwater map information mapped with analysis data of aquatic species or underwater map information mapped with linked analysis data obtained by linkage analysis of aquatic species and meteorological data.

In addition, the DB management unit 56 stores or updates map information provided periodically (preset constant period) or aperiodically from an external map server (not shown) in the map DB 32, and from an external weather server. It may provide a function of storing or updating collected meteorological data in the meteorological DB 34 by being provided periodically (preset constant period) or non-periodically.

Hereinafter, with reference to the drawings, a description will be given of an order and procedure for monitoring aquatic organisms by analyzing aquatic organism information using the aquatic database 30 and aquatic organism information processing unit 50 described above.

Referring to FIG. 6 , in the image information collection unit 52 of the aquatic organism information processing unit 50, aquatic organism image information transmitted from the user terminal 40 and received through the internal transmission/reception unit 51, for example, a picture of aquatic organisms Collects aquatic creature image information including , GPS coordinates, image capture time, etc. (step 102).

Then, in the species determination unit 53, when harmful organism image information is transmitted from the image information collection unit 52, the species DB 31 in the aquatic database 30 is searched (eg, scientific name search), thereby aquatic organisms. Analytical data specifying the species of is generated (step 104).

Here, when the species of aquatic life is not specified, it may be determined at the genus level.

Next, the map mapping unit 54 aggregates the analysis data of the specified species (step 106), and then maps them to the target location on the ocean map provided from the map DB 32, thereby mapping the analysis data to the mapped underwater map information. Generates (step 108).

Here, the process of aggregating the analysis data of the specified species and mapping them to the underwater map may be performed at a predetermined period, for example.

In addition, the DB management unit 56 may retrieve underwater map information to which analysis data is mapped from the underwater observation DB 33 at a predetermined period and transmit the underwater map information to a remote underwater management server (not shown) through a network. For reference, the aquatic management server refers to a server that is provided, for example, in the Ministry of Oceans and Fisheries, Korea Water Resources Corporation, etc. to provide functions such as monitoring and managing the aquatic ecological environment, and analysis data of aquatic species are mapped. Underwater map information or underwater map information mapped with linked analysis data obtained by linking and analyzing aquatic species, meteorological data, and underwater observation data may be provided.

On the other hand, the DB management unit 56 monitors whether or not a transmission request for an underwater map is received from the user terminal 40 through the transmission/reception unit 51 (step 110). As a result of monitoring here, the underwater map is transmitted. When the request is received, the underwater map information to which the analysis data is mapped is retrieved from the underwater observation DB 33 and transmitted to the user terminal 40 through the transceiver 51 and the network (step 112).

Hereinafter, a description will be given of a system for diagnosing and maintaining the aquatic life monitoring device 100 using VR images according to the present invention by using the Internet of Things (IoT), etc., with accompanying drawings.

Referring to FIG. 7 , when the monitoring device 100 is operated, the failure diagnosis system collects device status information (data collection), manages the device (data processing), detects and analyzes abnormal situations, and performs preliminary diagnosis. Optimize maintenance and repair through (data analysis).

Specifically, when such maintenance is performed, the system first detects the state of the device in a plurality of IoT sensors (I). For example, by detecting the states of the underwater robot 2, the camera 20, the user terminal 40, the aquatic creature information processing unit 50, the control unit 120, the output unit 150, etc., state information is stored in the controller ( 200) through integrated real-time transmission.

The controller 200 transfers the status information to the management information processing device 300, and the management information processing device 300 monitors the status of the monitoring device 100 and stores the data.

In addition, the management information processing device 300 performs a control operation corresponding to an abnormal state for each state, for example, with the controller 200, and the controller 200 adjusts the operation accordingly.

In addition, in this case, since the management information processing device 300 accumulates the aforementioned related information and generates a model related to diagnosis, the monitoring device 100 is maintained through preliminary diagnosis.

As shown in FIG. 7, the system according to an embodiment includes a plurality of controllers 200-1 to 200-n that collect various states from a plurality of IoT sensors I and a management information processing device 300. include

Additionally, the management information processing device 300 connects with an external linked service center to provide overall service, and includes a troubleshooting center management device 400-1 and the like.

In this case, as shown, the fault diagnosis system connects the plurality of controllers 200-1 to 200-n and the management information processing device 300 to each other through a self-network and interlocks each group. At this time, the private network uses a wireless communication network such as Wi-Fi or Ethernet.

The IoT sensor (I) is installed for each element of the surrounding environment, and detects state information from each component such as the underwater robot 2 and the camera 20.

The controllers 200-1 to 200-n deliver current status information to the pre-registered management information processing device 300 when status information is sensed by the IoT sensor I. Then, the controllers 200-1 to 200-n perform different adjustments for each device type under the control of the management information processing device 300.

The management information processing device 300 pre-registers information for controlling the operation of different devices according to the status of different device types based on manager setting information. Thus, the management information processing device 300 determines the current state from the registered result when the current state information is transmitted from the plurality of controllers 200-1 to 200-n and controls the operation. So, through this, when the monitoring system according to an embodiment operates, the monitoring device 100 is managed by collecting state information, processing data, detecting and analyzing an abnormal situation.

The IoT sensor (I) and the management information processing device 300 will be described in more detail as follows.

First, when the IoT sensor (I) is described, the state can be checked by detecting, for example, temperature and vibration state. In addition, the IoT sensor (I) monitors various operating functions by detecting the status of the underwater robot 2, the camera 20, the user terminal 40, and the like. Therefore, through this, the state and the state of the surrounding environment in the water are diagnosed and analyzed, and customized diagnosis is performed for each of the various surrounding environments.

Hereinafter, the management information processing device 300 will be described. In order to efficiently process whether or not the monitoring device 100 has an abnormality based on data, the management information processing device 300 from the following preprocessing configuration For example, data may be preprocessed for each device. Specifically, this preprocessing is performed as follows.

a) First, collect data in real time, including pressure, temperature, etc.

b) Then, operational operational data is extracted from the start to the end of a pre-specified operation.

c) Then, different facility codes and, for example, operation data collected for each operation start and end time of the monitoring device 100 are registered.

d) Next, based on the registered result, data is extracted for each different facility by a certain range.

e) Therefore, data is registered by operation operation code (including operation code, date, time, pressure, temperature, etc.) using the result extracted in this way.

Therefore, when performing maintenance monitoring, such a failure diagnosis system collects various data, extracts operational data in work units from the operation start time to the end time for each component, and pre-processes the data.

Therefore, in the case of such monitoring, whether or not the monitoring device 100 is abnormal according to data from the preprocessing configuration is efficiently processed, and through this, maintenance is more effectively monitored.

Hereinafter, the data structure collected in the pre-processing process will be described.

First, the data structure is a structure that collects data from the controllers 200-1 to 200-n in real time by equipment and by status (temperature, etc.) in the case of the above-mentioned monitoring, and is transmitted by code of the device. Collect data.

And, the data structure is an operating structure that automatically collects start and end times of performing tasks for each device or as an operator's input, and stores state information for each task.

Next, when the final diagnostic prediction model is created through this, it is a data structure combined by the finally extracted operating unit to be used. This exists as a file for each operating code, and in each file, data is combined from the date and time of operation start to the end.

Since maintenance is also performed through preliminary diagnosis according to data analysis, optimized maintenance is performed, and the management information processing device 300 performs the following operation.

First, when the current state information is transmitted, the management information processing device 300 compares the reference state attribute information of the monitoring device 100 of the present invention and the current state attribute information by type from a pre-registered time series diagnostic prediction model. .

As a result of the comparison, if the difference between the reference state attribute information and the current state attribute information is within a preset error range, it is diagnosed as normal, and if not, it is diagnosed as a failure. tell you if it's normal In this case, the time series diagnostic prediction model is implemented as follows according to an embodiment.

First of all, the time series diagnostic prediction model is a model that classifies and learns the time required for each of a number of different components and surrounding environment elements and status information for each operating time period when diagnosing whether the monitoring device 100 is out of order. define.

At this time, the learning model is as follows.

1) A data set representing the operation and environmental state characteristics for each of a number of different components and surrounding environment elements is extracted.

2) Next, the dataset extracted in this way is attributed by reflecting information of a plurality of different operating time zones and surrounding environments.

3) Then, on the basis of the attributable result, the required time and the attributes of the state information are determined by information maintaining the same state for a period representing a persistence characteristic for each learning model.

4) And normalize these determined results.

5) Therefore, based on these normalized results, the required time and state attribute information are set for each learning model, and information for predicting the required time and state attribute information for each of a number of different components and surrounding environment elements is generated. set as independent (standard state attribute information) and dependent (state) variables.

6) Then, the result set in this way is created as learning and training data.

7) Therefore, a time-series diagnostic prediction model based on deep learning is created from the generated data.

Additionally, the time series diagnostic prediction model will be further described as follows.

First, when various data for monitoring are transmitted, there are cases where a large number of data is not collected due to errors in communication, IoT devices, devices, etc., or abnormal values such as temperature occur, and the data file must be removed. .

So, after creating the basic dataset through this, additionally add attributes for the cumulative required time and status. In addition, if some data is not collected due to data disconnection, the corresponding data is removed. In this case, in order to use the time series-based deep learning learning model according to an embodiment, a serial data set must be created by setting a certain serial input length and step. So, next, valid properties for each model are determined, normal values are generated, and independent and dependent variables are determined. For example, in the required time model, cumulative amount and cumulative state may be valid dependent variables, and information indicating a failure state, that is, reference state attribute information may be set as an independent variable.

Then, in order to create a learning model, learning and training data are created from among the entire data. In general, 70% of the entire dataset is used as training data and 30% is used as training data to test the model after creating a model. Next, we create the learning model. At this stage, you decide which learning model to use. For example, it refers to a configuration that sets the maximum number of outputs by configuring an input layer and an output layer by configuring necessary layers based on deep learning. The generated model is evaluated, and if the error rate is satisfied, the model is simulated with new data. If the model update is not required, the learning model is saved and used as a prediction model.

8 is a flowchart illustrating the operation of the fault diagnosis system.

As shown, first, a plurality of IoT sensors (I) are installed for each device and each element of the surrounding environment to detect state information from each element when the monitoring device 100 is operated.

At this time, the controller 200 transmits the current status information to the pre-registered management information processing device 300 when status information is sensed by the plurality of IoT sensors I.

Next, by controlling the management information processing device 300, the operation is differently adjusted for each type.

In this case, the management information processing device 300 pre-registers information for controlling different operations according to different devices based on manager setting information. So, when the current state information is transmitted from the controller 200, the current state is determined from the registered result and the operation of the monitoring device 100 is differently integrated and controlled.

So, when the monitoring device 100 of the present invention is operated through this, state information is collected, data is processed, and an abnormal situation is sensed and analyzed to perform maintenance. For example, when a sudden increase in pressure, temperature, rapid end of operation, or non-collection of data occurs, an abnormality notification may be provided.

In addition, when maintenance is performed in this way, maintenance of the monitoring device 100 is also performed through preliminary diagnosis according to data analysis, so that optimized maintenance can be performed.

To this end, the failure diagnosis system performs the following operations.

First, when the status information of the current monitoring device 100 is transmitted, the management information processing device 300 obtains reference state attribute information and current state attribute of the device from a pre-registered time-series diagnostic prediction model (see the above). Compare information by device type.

As a result of the comparison, when the difference value between the reference state attribute information and the current state attribute information corresponds to a preset reference difference value, it is diagnosed as normal.

On the other hand, as a result of the comparison, if the difference value between the reference state attribute information and the current state attribute information does not correspond to the reference difference value, the monitoring device 100 is maintained because it is diagnosed as a failure.

As described above, the scope of the present invention is indicated by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present invention. should be interpreted as being

2: Underwater Robot
10: tracking device 20: camera
30: underwater database 40: user terminal
50: aquatic life information processing unit 100: monitoring device
Reference Numerals 110: location acquisition unit 120: control unit 130: analysis unit 140: management unit 150: output unit 200: controller
300: management information processing device
400: Data related to aquatic organisms 402: Meta data

Claims (4)

An underwater robot equipped with a camera for taking pictures underwater and a tracking device for receiving location information by communicating with GPS satellites;
a location acquisition unit receiving identification information and location information of each aquatic creature in the water from the tracking device;
a control unit that is connected to the camera, obtains control information about a rotation angle of the camera, and applies a control signal to an output unit;
an analysis unit that identifies the underwater robot appearing in the underwater image based on the rotation angle of the camera and the width of the underwater image acquired by the camera;
a management unit for retrieving information on aquatic plants corresponding to the corresponding aquatic plants from an aquatic database using the identification information of the aquatic plants extracted by the analysis unit;
an underwater database for collecting and storing information on each of the aquatic organisms;
an aquatic organism information processing unit that searches a species DB formed in the aquatic database and generates analysis data specifying the species of the aquatic organisms; and
and an output unit generating a virtual reality (VR) image representing information of the underwater image by combining the search information retrieved by the management unit with the underwater image, and providing the virtual reality (VR) image to a user terminal. A device for monitoring aquatic organisms using VR images.
According to claim 1,
The underwater database includes a species DB in which species and genus information of aquatic organisms is stored, a map DB providing an underwater map corresponding to an area to be monitored, and an underwater observation in which underwater map information in which analysis data of specified species are mapped is stored. A DB, an aquatic life monitoring device using VR images, characterized in that it includes a weather DB in which meteorological data collected from an external meteorological server are stored.
According to claim 1,
The control unit causes the camera to rotate in the direction according to the user's manipulation, and transmits a control signal for rotating the camera in the direction according to the user's manipulation based on an input signal input by the user terminal. An aquatic life monitoring device using VR images, characterized in that applied to.
According to claim 1,
The aquatic life monitoring device using VR images, characterized in that the management unit allows the user to additionally input the information on the aquatic organisms input from the user terminal or to correct the previously input information on the aquatic organisms.

Images