KR102584718B1 - Real-time video/voice based remote maintenance system for ship and overseas equipment using smart glasses - Google Patents

Abstract

The present invention reduces the transmission bandwidth through effective compression of images within ships and overseas facilities, performs satellite communication or network communication between countries, and remotely inspects and maintains ships and overseas facilities in real time by utilizing compressed images. This is about a real-time video/voice-based remote maintenance system for ships and overseas facilities using smart glasses.
The real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses of the present invention is installed in a remote support center that performs remote maintenance of ships or overseas facilities, and on the ships or overseas facilities, and is installed on the ships or overseas facilities. It consists of a video communication system that transmits and receives video, audio, and files for remote maintenance of overseas facilities with a remote support center through satellite communication or cross-border network communication.

Description

Real-time video/voice based remote maintenance system for ship and overseas equipment using smart glasses}

The present invention relates to a real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses, and specifically, to enable inspection and maintenance of ships and overseas facilities in real time from a distance, using video, audio, and files. to carry out communication.

In addition, the present invention relates to a real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses that can transmit images within ships and overseas facilities with a low communication bandwidth in satellite communication and cross-border networks based on image compression. will be.

In general, for the operation and management of ships, the remote support center of a shipping company on land and the remote support center of a ship equipment company frequently transmit and receive data using satellite communication. In particular, when ships send data such as monitoring messages to land, messages generated at regular intervals are transmitted to land.

Due to the advantages of such satellite communication, such as a wide coverage range and no need to install a base station, its use is increasing not only for ships at sea but also for data communication in areas very far from the city center.

Meanwhile, research has continued on a system that transmits the ship's video data to land and remotely controls the ship through real-time monitoring on land.

For example, Republic of Korea Patent Publication No. 10-2015-7024126 proposed a ship remote control system that monitors the surrounding images of an operating ship in real time from land and enables remote control.

However, in this case, a large bandwidth is required to transmit images of the ship's surroundings to land, so there is a problem that satellite communication costs for transmission are excessive. In addition, even in the case of facilities located overseas, when transmitting images from overseas facilities through a cross-border network, a large bandwidth is required and communication costs are high due to the nature of the cross-border network. Therefore, there is a problem of excessive network communication costs for transmission.

Republic of Korea Patent Publication No. 10-2015-7024126 (2014.05.01)

The purpose of the present invention is real-time video/audio-based remote maintenance of ships and overseas facilities using smart glasses that can perform satellite communication or cross-border network communication by reducing the transmission bandwidth through effective compression of images within ships and overseas facilities. To provide a system.

The present invention provides a real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses that can remotely inspect and maintain ships and overseas facilities in real time using compressed video. There is a purpose.

In addition, the present invention provides real-time video/voice-based remote maintenance of ships and overseas facilities using smart glasses, which can improve the operational stability of ships and overseas facilities by performing real-time remote maintenance and response to failures. There is another purpose in providing the system.

The real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses according to the present invention includes a remote support center that performs remote maintenance of ships or overseas facilities; and a video communication system installed on the ship or overseas facility, and transmitting and receiving video, audio, and files for remote maintenance of the ship or overseas facility with the remote support center through satellite communication or cross-border network communication. can do.

Here, the video communication system includes HoloLens, which provides mixed reality images to users; A wireless headset responsible for transmitting and receiving voice; A digital microscope that collects magnified images; A relay server that performs satellite communication or cross-border network communication with the remote support center; and a control PC that controls the HoloLens, the wireless headset, the digital microscope, and the relay server.

At this time, the control PC can compress the video with a lossy compression codec, and the lossy compression codec can apply HEVC.

In addition, the control PC can control network traffic of satellite communication or inter-country network communication, and can control at least one of the image capture speed or image quality so as not to exceed a preset network traffic.

In addition, the control PC can control the frames per second (FPS) of the video, and in the case of satellite communication or inter-country network communication, the number of frames per second can be controlled to a maximum of 15 or less, and the ship or For communication within overseas facilities, the number of frames per second can be controlled to at least 15 or more.

Additionally, the control PC can display the same mark on the screen of the control PC when the image displayed on the HoloLens is marked.

Additionally, the control PC can share documents with the HoloLens.

Meanwhile, the remote support center can perform security on the monitor screen within the remote support center when remotely controlling the video communication system. At this time, security on the monitor screen within the remote support center includes displaying a watermark, At least one of screen capture prevention or recording prevention can be applied.

In addition, the remote support center can perform security when transmitting files through the video communication system, and security when transmitting files through the video communication system is checked for stability using an anti-virus search engine before transmitting the files. You can send it after performing .

Meanwhile, the relay server includes a remote connection unit that remotely connects the remote support center and the video communication system by applying the HEVC codec; A screen transmission unit that compresses the screen in the control PC with HEVC codec and transmits it to the remote support center; A HoloLens video transmission unit that compresses the video of the HoloLens with the HEVC codec and transmits it to the remote support center; A digital microscope image transmission unit that compresses the enlarged image of the digital microscope with HEVC codec and transmits it to the remote support center; A video communication video transmission unit that compresses video for video communication with the HEVC codec and transmits it for video communication between the remote support center and the video communication system; a voice data transmission unit that performs voice communication between the remote support center and the video communication system; And it may include a file transmission unit that transmits files between the remote support center and the video communication system.

The real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses according to the present invention can perform satellite communication or cross-border network communication by reducing the transmission bandwidth through effective compression of images within ships and overseas facilities. There is an advantage.

The real-time video/voice-based remote maintenance system for ships and overseas facilities using smart glasses according to the present invention has the advantage of being able to remotely inspect and maintain ships and overseas facilities in real time using compressed video. there is.

In addition, the real-time video/voice-based remote maintenance system for ships and overseas facilities using smart glasses according to the present invention improves the operational stability of ships and overseas facilities by performing real-time remote maintenance and response to failures. There is an advantage to being able to do it.

Figure 1 is a configuration diagram showing a real-time video/voice-based remote maintenance system for ships and overseas facilities using smart glasses according to an embodiment of the present invention.
FIG. 2 is a diagram showing the video communication system of FIG. 1 in detail.
Figure 3 is a diagram showing the relay server of Figure 2 in more detail.

Hereinafter, specific embodiments for carrying out the present invention will be described with reference to the attached drawings.

In describing the present invention, terms such as first and second may be used to describe various components, but the components may not be limited by the terms. Terms are intended only to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the present invention.

When a component is mentioned as being connected or connected to another component, it can be understood that it may be directly connected to or connected to the other component, but other components may exist in between. .

The terms used herein are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions may include plural expressions, unless the context clearly indicates otherwise.

In this specification, terms such as include or have are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, including one or more other features or numbers, It can be understood that the existence or addition possibility of steps, operations, components, parts, or combinations thereof is not excluded in advance.

Additionally, the shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

Hereinafter, with reference to the attached drawings, a real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses according to the present invention will be described in detail.

Figure 1 is a configuration diagram showing a real-time video/voice-based remote maintenance system for ships and overseas facilities using smart glasses according to an embodiment of the present invention, and Figures 2 and 3 are detailed configurations for explaining Figure 1 in detail. It's a degree.

Hereinafter, a real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses according to an embodiment of the present invention will be described with reference to FIGS. 1 to 3.

First, referring to Figure 1, the real-time video/voice-based remote maintenance system for ships and overseas facilities using smart glasses according to the present invention includes a remote support center (100) that performs remote maintenance of ships and overseas facilities, and A video communication system (200) that is installed on ships and overseas facilities and transmits and receives video, voice, and files for remote maintenance of ships and overseas facilities with the remote support center (100) through a communication satellite or cross-border network (300). ).

The present invention performs real-time diagnosis, remote support, and video/voice communication for ships and overseas facilities through a video communication system (200) installed on ships and overseas facilities at a remote remote support center (100), It is possible to secure an uninterrupted operating environment for ships and overseas facilities through real-time remote inspection and maintenance of overseas facilities, and also improve the operational stability of ships and overseas facilities through immediate response to failures.

In general, technologies that can perform remote control in an environment with a network bandwidth of 100Mbps or more are widely used, but existing remote control technologies cannot be connected in a low network bandwidth environment of 1Mbps or less, such as satellite communication, or even when connected, they are difficult to connect. It has the disadvantage of being difficult to use because it does not perform screen control properly.

In particular, since the network traffic used in smart glasses such as HoloLens used in the present invention is 15-20 Mbps, it is difficult to use smart glasses independently in a low network bandwidth environment of 1 Mbps or less, such as satellite communication.

In order to overcome this problem, the present invention performs compression on images within a ship to be transmitted for remote maintenance, and applies HEVC (High Efficiency Video Coding) compression technology for effective image compression, while using low-bandwidth satellite communication. To ensure smooth operation, network traffic control and frames per second (FPS) control of compressed video are performed.

Meanwhile, the present invention can also be applied when transmitting video to overseas facilities using a cross-border network, that is, applying HEVC (High Efficiency Video Coding) compression technology to the video to be transmitted, while controlling network traffic and compressed video. By performing Frames per second (FPS) control to enable communication at low bandwidth, network communication costs can be greatly reduced.

Hereinafter, referring to FIG. 2, the method of using mixed reality using smart glasses in the present invention and the method of performing security when remotely controlling the video communication system 200 from the remote support center 100 are discussed. Explain in detail.

FIG. 2 is a diagram showing the video communication system 200 of FIG. 1 in detail.

As can be seen in Figure 2, the video communication system 200 of the present invention includes a HoloLens 210 that provides mixed reality images to the user, a wireless headset 250 in charge of voice transmission and reception, and an enlarged image collection. a digital microscope 240, a relay server 220 that performs satellite communication or inter-country network communication with the remote support center 100, a HoloLens 210, a wireless headset 250, a digital microscope 240, and It consists of a control PC (230) that controls the relay server (220).

Here, when transmitting video from the video communication system 200 to the remote support center 100, the control PC 230 can compress the video using the HEVC (High Efficiency Video Coding) codec and then transmit it.

The H.264 codec, which is currently being commercialized as video compression coding, has the disadvantage of having a large video data capacity because it is a lossless compression method. However, the HEVC codec, which is a lossy compression method, has the advantage of being optimal for satellite communication as its compression performance is 35% to 50% more effective than H.264.

Meanwhile, in the present invention, when transmitting video from the video communication system 200 to the remote support center 100, the control PC 230 controls network traffic to set the capture speed and quality of the video so as not to exceed the preset network traffic. It can be controlled according to the value.

Through this, the present invention can transmit images in real time even through satellite communications with limited transmission speeds or cross-border networks with low bandwidth. For example, this transmission rate control can be performed in units of 100 kbps.

In addition, the present invention can control the number of frames per second (FPS) of the video in the control PC 230. For example, in the case of satellite communication or cross-border networks, it can be controlled in units of 5, 10, and 15 FPS in the range of 15 FPS or less. In the local communication network within ships and overseas facilities, it can be controlled at 15, 30, and 60 FPS in the range of 15 FPS or higher.

Meanwhile, the HoloLens 210 is a smart glass that enables mixed reality, and video and audio are shared between the HoloLens 210 and the control PC 230 over a Wi-Fi network. possible.

In the present invention, if the image displayed on the HoloLens 210 is marked, the same is displayed on the screen of the control PC 230. Depending on the implementation, the control PC 230 may display 30 or more markings on the screen of the HoloLens 210.

Additionally, documents can be shared from the control PC 230 to the HoloLens 210 so that the user can refer to the document contents while working. In this case, at least three or more documents can be shared.

The wireless headset 250 can be used for voice communication in case the HoloLens, a smart glass, cannot be used due to a malfunction, etc. In particular, the engine room of a ship, power plant, etc. are very noisy, so the wireless headset ( 250), it is desirable to use an aviation headset.

Meanwhile, the present invention, when remotely controlling the video communication system 200 of a ship or overseas facility from the remote support center 100, displays a watermark on the monitor screen in the remote support center, prevents real-time capture, and prevents real-time recording, etc. Security can be strengthened through .

That is, when remotely controlling the screen of the control PC 230 of the video communication system 200 from the remote support center 100, displaying a watermark, preventing real-time capture, and preventing real-time recording on the monitor screen within the remote support center. Security can be improved by applying it.

In addition, when transferring files from the remote support center (100) to the video communication system (200) of ships and overseas facilities, security can be strengthened by performing a stability check using an anti-virus search engine before transmitting the files. For example, it can be transmitted after scanning with the Kaspersky anti-virus engine.

Meanwhile, according to the implementation, all images transmitted from the video communication system 200 of ships and overseas facilities are encrypted and transmitted with compression, and decryption is performed before display at the remote support center 100, thereby improving security. can be further improved.

The present invention as described above can reduce the transmission bandwidth through effective compression of images within ships and overseas facilities to perform satellite communication or network communication between countries, and can also remotely inspect ships and overseas facilities using compressed images. and maintenance can be performed in real time.

Next, Figure 3 is a diagram showing the relay server 220 of Figure 2 in more detail. In the present invention, the relay server 220 transmits various videos, voices, and files to the remote support center 100 under the control of the control PC 230.

As can be seen in Figure 3, the relay server 220 of the present invention includes a remote connection unit 221 that remotely connects the remote support center 100 and the video communication system 200 by applying the HEVC codec, and a control PC ( 230) The screen transmission unit 222 compresses my screen with the HEVC codec and transmits it to the remote support center 100, and the HoloLens compresses the video of the HoloLens 210 with the HEVC codec and transmits it to the remote support center 100. An image transmission unit 223, a digital microscope image transmission unit 224 that compresses the enlarged image of the digital microscope 240 with HEVC codec and transmits it to the remote support center 100, the remote support center 100 and the video communication system 200. A video communication video transmission unit 225 that compresses and transmits video for video communication with the HEVC codec for inter-video communication, and a voice data transmission unit 226 that performs voice communication between the remote support center 100 and the video communication system 200. , and a file transmission unit 227 that transmits files between the remote support center 100 and the video communication system 200.

Depending on the implementation, since the HEVC codec is a lossy compression method, it is possible to achieve more reliable transmission by not applying the HEVC codec compression in voice data and file transmission that is sensitive to data errors.

Meanwhile, when using INMARSAT/VSAT as the communication satellite 300, the bandwidth available for communication is 500kbps to 700kbps, so file transfer may be limited to 200kbps.

As described above, the real-time video/audio-based remote maintenance system for ships and overseas facilities using smart glasses according to the present invention reduces the transmission bandwidth through effective compression of images within ships and overseas facilities to perform satellite communication or cross-border network communication. inspection and maintenance of ships and overseas facilities can be performed remotely in real time using compressed video. In addition, real-time remote maintenance and failure response for ships and overseas facilities are possible, improving the operational stability of ships and overseas facilities.

What has been described above includes examples of one or more embodiments. Of course, for the purposes of describing the above-described embodiments, it is not possible to describe every possible combination of components or methods, and those skilled in the art will recognize that many additional combinations and permutations of the various embodiments are possible. Accordingly, the described embodiments are intended to include all alternatives, modifications and alterations that fall within the spirit and scope of the appended claims.

Claims (16)

Remote support center that performs remote maintenance of ships or overseas facilities; and
A video communication system installed on the ship or overseas facility and transmitting and receiving video, audio, and files for remote maintenance of the ship or overseas facility with the remote support center through satellite communication or cross-border network communication; ,
The video communication system is,
Smart glasses that provide mixed reality images to users;
A wireless headset responsible for transmitting and receiving voice;
A digital microscope that collects magnified images;
A relay server that performs satellite communication or cross-border network communication with the remote support center; and
It includes a control PC that controls the smart glasses, the wireless headset, the digital microscope, and the relay server,
The control PC is,
Compressing the video with a lossy compression codec applying HEVC, controlling network traffic of the satellite communication or cross-border network communication, and controlling at least one of the capture speed or quality of the video so as not to exceed a preset network traffic, and , Controls the frames per second (FPS) of the video, and in the case of satellite communication or inter-country network communication, controls the number of frames per second to a maximum of 15 or less, and in the case of communication within the ship or overseas facility, the The number of frames per second is controlled to at least 15, and if the image displayed on the smart glasses is marked, the same is displayed on the screen of the control PC and the document is shared with the smart glasses.
The remote support center is,
When remotely controlling the video communication system, security is performed on the monitor screen within the remote support center, and security on the monitor screen within the remote support center includes at least one of displaying a watermark, preventing screen capture, or preventing recording. is applied, and security is performed when transmitting files to the video communication system. Security is performed when transmitting files to the video communication system, and the files are transmitted after performing a stability check using an anti-virus search engine before transmitting the files. ,
The relay server is,
In remotely connecting the video communication system and the remote support center, a remote connection unit that remotely transmits data compressed with the HEVC codec and generated;
A screen transmission unit that compresses the screen in the control PC with the HEVC codec and transmits the compressed screen in the control PC to the remote support center;
A smart glasses video transmission unit that transmits a compressed screen in the smart glasses generated by compressing the mixed reality image of the smart glasses with the HEVC codec to the remote support center;
A digital microscope image transmission unit that transmits a compressed screen within the digital microscope generated by compressing the enlarged image of the digital microscope with the HEVC codec to the remote support center;
A video communication video transmission unit that transmits a compressed video for video communication generated by compressing the video for video communication with the HEVC codec for video communication between the remote support center and the video communication system;
a voice data transmission unit that performs voice communication between the remote support center and the video communication system; and
A remote maintenance system comprising a file transmission unit that transmits files between the remote support center and the video communication system. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete