KR102468685B1 - Workplace Safety Management Apparatus Based on Virtual Reality and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a virtual reality-based workplace safety management device and a driving method of the device. A communication interface unit that receives the captured image, and a virtual reality (VR) background image for the scene of the work site is created using the received captured image, and the object is imaged based on the deep learning result of the object extracted from the received captured image. And, it may include a control unit for generating and outputting a VR image of the work site by synthesizing the generated VR background image and the imaged object.

Description

Workplace Safety Management Apparatus Based on Virtual Reality and Driving Method Thereof

The present invention relates to a virtual reality-based safety management device at a work site and a method for driving the device, and relates to an object (e.g., object/person/equipment, etc.) for which a signal for location tracking is not detected at a work site such as tunnel construction. It relates to a virtual reality-based workplace safety management device that can easily monitor the workplace through virtual reality (VR) images even in the event of a signal error, and a method for driving the device.

In general, it will be said that the safety of each worker is required as the top priority in construction, civil engineering, and plant sites. However, until now, most of the safety of these workers has been conducted in advance, and in the case of being put into the actual work site, it is common for each individual to be in charge of safety by himself. In recent years, due to the development of smart devices and wearable devices, some systems have been proposed that enable the central control server to check the location or body information of workers.

A monitoring system in a work space such as a conventional tunnel is generally managed and supervised by a control center by monitoring images displayed through CCTV (Closed Circuit Television). In the case of monitoring the work site only with these CCTVs, it was difficult to check the exact location and information of the object in real time.

In order to compensate for this, conventionally, it is possible to check information on an object within a certain space using a beacon or a beacon scanner, but when an error occurs in an object without a beacon signal or a beacon signal, furthermore, variously changing real-time sites There are many cases where it is not possible to properly check the situation information inside.

Korea Patent Registration No. 10-2074684 (2020.02.03) Korea Patent Registration No. 10-1944823 (2019.01.28) Korean Registered Patent Publication No. 10-2014652 (2019.08.20) Korean Patent Publication No. 10-2020-0030784 (2020.03.23) Korean Patent Publication No. 10-2020-0031312 (2020.03.24)

An embodiment of the present invention is a virtual reality that can easily monitor a work site through a virtual reality (VR) image even in the event of an object or signal error in which a signal for position tracking is not detected at a work site such as tunnel construction, etc. Its purpose is to provide a work site safety management device based on the device and a driving method of the device.

A virtual reality-based workplace safety management device according to an embodiment of the present invention includes a communication interface unit that receives a photographed image of the work site from a camera at the work site, and a communication interface unit for receiving the captured image of the work site. A virtual reality (VR) background image for the appearance is generated, the object is imaged based on the deep learning result of the object extracted from the received photographed image, and the generated VR background image and the imaged object are synthesized to It includes a control unit that generates and outputs a VR image of the work site.

The control unit extracts unit frame images of the same time from each of the captured images received from a plurality of cameras installed at different locations in the work site, and stitches each of the extracted unit frame images to obtain the VR background. You can create video.

The control unit may obtain depth information of the extracted object by analyzing the received captured image, and may combine the imaged object with the VR background image by applying the obtained depth information.

The control unit may further use a distance value calculated by beacon information received in relation to the extracted object in order to acquire depth information of the extracted object.

The control unit may switch to the VR video control mode when an event in the work site or an error is detected in the object.

In addition, the driving method of the virtual reality-based workplace safety management device according to an embodiment of the present invention includes the step of receiving, by a communication interface unit, a photographed image of the workplace from a camera of the workplace, and the control unit, A virtual reality (VR) background image for the scene of the work site is created using the received captured image, the object is imaged based on the deep learning result of the object extracted from the received captured image, and the VR background image is created. and generating and outputting a VR video of the work site by synthesizing the video and the imaged object.

In the generating and outputting step, unit frame images of the same time are extracted from each of the captured images received from a plurality of cameras installed at different locations in the work site, and each of the extracted unit frame images is stitched to obtain the VR A background image can be created.

In the generating and outputting step, depth information of the extracted object may be acquired by analyzing the received captured image, and the imaged object may be synthesized with the VR background image by applying the acquired depth information.

The generating and outputting may further use a distance value calculated by beacon information received in relation to the extracted object in order to acquire depth information of the extracted object.

The generating and outputting step may switch to VR video control mode when an event or an error is detected in the object at the work site.

According to an embodiment of the present invention, when an error occurs in an object without a signal such as a beacon or a signal such as a beacon in a work site (or work space), the situation or situation information in the field in real time that is further converted into VR You will be able to quickly and accurately check through the video.

In addition, according to an embodiment of the present invention, if necessary, the work site can be monitored in real time with VR images, so that incidents and accidents can be dealt with more quickly.

1 is a diagram showing a virtual reality-based workplace safety management system according to an embodiment of the present invention;
Figure 2 is a block diagram illustrating the detailed structure of the construction site safety management device of Figure 1,
3 is a block diagram illustrating the structure of the VR image processing unit of FIG. 2;
Figure 4 is a flow chart showing the driving process of the virtual reality-based workplace safety management device according to an embodiment of the present invention, and
5 is a flowchart showing another driving process of a virtual reality-based workplace safety management device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram showing a virtual reality-based workplace safety management system according to an embodiment of the present invention.

As shown in FIG. 1, the virtual reality-based workplace safety management system 90 according to an embodiment of the present invention includes a user device 100, field devices 113, 115, and 117, a communication network 120 and ( It includes some or all of the construction site safety management device 130 (based on virtual reality).

Here, "including some or all" means that the system 90 is configured by omitting some components such as the scanner device 115 constituting the field devices 113, 115, and 117, or the construction site safety management device This means that some or all of the components constituting the communication network 120 can be integrated into a network device (eg, a wireless switching device, etc.) to help a sufficient understanding of the invention. described as inclusive.

The user device 100 is, for example, a beacon device equipped with a work cap worn by a worker or a supervisor at a construction site (or work site) such as a tunnel end, etc., or a terminal device such as a smartphone owned by a worker, and further detects the location of the worker It includes various devices such as a dedicated terminal equipped with a module that can be tracked (eg, short-distance communication module). Of course, it may also include a wearable device such as a domestic S company's Galaxy Gear worn by a worker on the wrist or the like. In the embodiment of the present invention, it is preferable to use a beacon device that transmits a beacon signal when considering aspects such as construction cost without threatening work safety, but it will not be particularly limited thereto. That is, if location tracking is possible in the field, various communication devices capable of short-distance wireless communication may be used.

The field devices 113, 115, and 117 include a camera 113, a scanner device 115, and a communication relay device 117, and may further include various other devices capable of detecting a construction site. Typically, a detection sensor such as a gas sensor for sensing the internal environment of a construction site or a mobile robot that moves within a construction site and monitors through filming and transmits a warning broadcast may be further included. Of course, the detection sensor may be mounted on the mobile robot. The camera 113 may include various types of cameras such as 3D cameras, CCTVs, and IP cameras. For example, as shown in FIG. 1 , a plurality of cameras may actually be installed at the end of the tunnel. In addition, a plurality of cameras are installed in different locations to facilitate the creation of VR images inside the curtain, take pictures of the interior, and provide the captured images to the construction site safety management device 130 through the communication relay device 117. For example, in the case of a 3D camera, a 3D captured image may be provided as a captured image, but when a plurality of 3D cameras are installed, each 3D camera may transmit each 3D captured image. Of course, each camera transmits information about the shooting time or identification information (ID) of the camera 113 together when providing the captured image. The 3D camera may generate and further provide depth information on objects in the photographed image by itself. However, since a 3D camera is expensive, it is preferable to apply a CCTV in an embodiment of the present invention, but it will not be particularly limited thereto.

The scanner device 115 communicates with the user device 100 in the vicinity. The scanner device 115, for example, as a beacon scanner, receives a beacon signal from a beacon device mounted on a working hat by a nearby worker and provides it to the communication relay device 117. Of course, when providing the received beacon signal, the beacon scanner 115 may transmit its own device identification information (ID), device identification information of the received beacon device, and information about signal reception strength. Corresponding information is provided to the construction site safety management device 130 via the communication relay device 117. Of course, the scanner device 115 according to an embodiment of the present invention may include various types of communication devices capable of short-distance communication if they can track the location of a worker or equipment by communicating with the user device 100 in addition to the beacon scanner. can A beacon can operate at a distance of up to 50m, and can be very useful at a construction site because it performs short-distance communication of less than 100m using wireless communication technologies such as infrared and RF. Scanner device 115 or a plurality of devices may be installed in the construction site, for example, the construction site safety management device 130 measures the distance based on which scanner device 115 strongly detects the signal of a specific beacon device location can be determined. Of course, scattering or reflection of the beacon signal may occur inside the construction site, and measurement may be performed in various ways, such as increasing accuracy by setting a plurality of scanner devices 115 as a group. For example, if the signal is the strongest in a specific group of scanner devices 115 and the signal is continuously received, the position can be calculated based on that.

Communication may not be smooth in a dead end such as tunnel construction. For example, GPS communication is impossible. Therefore, the communication relay device 117 according to an embodiment of the present invention is installed at the entrance of the curtain wall and can relay communication between the user device 100 outside the curtain wall and inside the curtain wall so that smooth communication is achieved. By adjusting the signal radiation direction of the antenna, signal transmission and reception to the inside can be done as much as possible. The communication repeater 117 may be an access point (AP) as well. Therefore, it may be included in the concept of the communication network 120 . However, in that the communication relay device 117 can be directly installed and used by a construction company, etc., it is intended to be distinguished from the communication network 120 installed by a key communication company.

The communication network 120 includes both wired and wireless communication networks. For example, as the communication network 120, a wired or wireless Internet network may be used or interlocked. Here, the wired network includes an Internet network such as a cable network or a public switched telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, EPC (Evolved Packet Core), LTE (Long Term Evolution), and Wibro networks. meaning to include Of course, the communication network 120 according to an embodiment of the present invention is not limited thereto, and may be used, for example, in a cloud computing network or a 5G network under a cloud computing environment. For example, when the communication network 120 is a wired communication network, an access point within the communication network 120 can access an exchange center of a telephone company, but in the case of a wireless communication network, it accesses an SGSN or GGSN (Gateway GPRS Support Node) operated by a communication company to transmit data. or data can be processed by accessing various repeaters such as Base Station Transmission (BTS), NodeB, e-NodeB, etc.

The communication network 120 may include an access point. The access point here includes a small base station such as a femto or pico base station that is often installed in a building. Femto or pico base stations are classified according to the maximum number of communication repeaters 117 that can be connected to small base stations. Of course, the access point may include a communication repeater 117 and a short-range communication module for performing short-range communication such as Zig-bee and Wi-Fi. The access point may use TCP/IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, short-distance communication may be performed in various standards such as Bluetooth, ZigBee, infrared, radio frequency (RF) such as ultra high frequency (UHF) and very high frequency (VHF), and ultra-wideband communication (UWB) in addition to Wi-Fi. Accordingly, the access point extracts the location of the data packet, designates the best communication path for the extracted location, and forwards the data packet to the next device, for example, the construction site safety management device 130, etc. along the designated communication path. . An access point may share several lines in a general network environment, and includes, for example, a router, a repeater, and a repeater.

The construction site safety management device 130 may include various devices for monitoring workers at the construction site. Representatively, it may include a server for processing data provided from a construction site, and may further include, for example, a computer body or a monitor on which a control agent performs control. Of course, it may include a VR device such as a controller's or manager's smartphone or head-mounted device (HMD). On the monitor, the interior of the construction site and the location of workers performing work inside may be displayed. Various additional information may be displayed. This monitoring screen may be named 'dashboard'. Of course, the monitoring screen may be configured in various forms and implemented on the screen. Normally, the construction site is monitored through CCTV, but it is difficult to check the location and information of an accurate object (eg, object/person, etc.) in real time through this. To compensate for this, information on an object within a certain space can be checked using a beacon and a beacon scanner. However, even in this case, there are many cases where an object whose signal is not detected or an error in signal detection occurs, as well as situation information in a real-time field that varies in various ways cannot be checked.

To this end, the construction site safety management device 130 according to an embodiment of the present invention provides a photographed image of the camera 113 provided from the field devices 113, 115, and 117 installed in the tunnel or a scanner device 115, etc. Based on scan data of the user device 100, for example, a beacon device, a VR image can be generated and switched to a VR image mode so that monitoring can be performed. For example, when it is determined that an incident or accident occurs at a construction site or an error has occurred in a beacon device, when an object without a beacon signal is detected, and also at the request of a control agent, it can be switched to VR video mode. . VR images are preferably 360-degree VR images. For example, a 3D image shown based on each point where the camera 113 is installed may be used. In addition, when a controller designates a specific location, a 3D image may be displayed on the screen based on the location. In this case, for example, identification of a beacon device in which a beacon signal is not detected can be more easily performed.

The construction site safety management device 130 uses a captured image obtained when the camera 113 is initially installed at the construction site to create a VR image, for example, to create and store a VR background image for the inside of the construction site. . Here, the background refers to a part excluding the object when viewed from the photographed image. Therefore, it can be seen as a part excluding the object from the inside of the construction site. Since the internal appearance of the construction site changes whenever time passes through blasting work, etc., the construction site safety management device 130 may generate a VR image as a background image in real time or at regular time intervals in consideration of this. For example, the construction site safety management device 130 may install a 3D camera to observe the interior of the construction site in 3D. However, in order to implement 360-degree VR, the construction site safety management device 130 stitches unit frame images of captured images received from a plurality of cameras at the construction site by time, more precisely, unit frame images of the same time each other. ) to generate real-time VR images. For example, with respect to time t1, a VR image may be generated by stitching unit frame images corresponding to the same time. For example, a unit frame image corresponding to time t1 (e.g. 1/60 sec) from a captured image received from camera 1 and a unit frame image corresponding to the same time t1 received from camera 2 provided at a different location on the construction site can be acquired and stitched together. Of course, a calibration operation for correcting an image may be performed in this process. Through this, it is possible to convert images taken from multiple viewpoints into a panoramic image from multiple viewpoints by splicing together, and the user can check the image by changing the viewpoint according to the user's intention. Since the generation of such a VR background image may be generated using streaming data transmitted in real time, the embodiment of the present invention will not be particularly limited to any one form. Of course, the VR background image may be made intermittently when there is an event, such as blasting work, in the construction site in order to reduce the amount of data calculation processing.

In addition, the construction site safety management device 130 according to an embodiment of the present invention determines the distance based on the captured images received from the plurality of cameras 113 and the communication signal between the user device 100 and the scanner device 115. can be measured Even in the case of the scanner device 115, it can be installed in various locations, and accurate distance measurement can be made through triangulation of the user device 100. Of course, the measurement distance can be calculated using the received signal strength. For example, when a 3D camera is used as the camera 113, a VR image may be generated using the depth (depth) information provided by the operator. The VR image corresponds to an image formed by overlapping a VR background image and an object. In the case of IP cameras or CCTVs other than 3D cameras, depth (or depth) information can be calculated through image data, that is, pixel analysis. For example, a distance can be measured using pixels in a unit frame image of a photographed image, but beacon information, that is, a distance calculated by a signal reception direction or received signal strength may be used as depth information. Here, the depth information may be positional information about where an object is located in a 3D space. Therefore, if the construction site safety management device 130 acquires two-dimensional information through pixel analysis, it may generate three-dimensional information by further using information such as beacons.

For example, when the VR video control mode is switched to the VR video control mode under designated conditions (e.g., when an incident or accident occurs) to monitor a construction site, the construction site safety management device 130 is a pre-generated or real-time VR The construction site VR image generated by overlapping (or synthesizing) objects (or object images, image files, etc.) in the construction site on the background image can be displayed on the monitor. Here, the mode may mean that when a specified condition is met (or when there is a selection), a specified operation is automatically operated or a series of preset operations are operated sequentially. In the process of generating a VR image of a construction site, the construction site safety management device 130 performs a deep learning operation to increase the accuracy of overlapping objects in relation to overlapping objects, and then finally confirmed by the deep learning result. It can be seen that the objects overlap. For example, when the user device 100 that has been detected in real time is suddenly not detected, a clear determination as to why the user device 100 is not detected can be made by switching to the VR video control mode. This will enable prompt action to be taken. In addition, in the space where there is blasting work, new objects related to the blasted debris are accurately detected through deep learning and implemented in VR images, thereby further increasing the accuracy of monitoring the inside of the construction site.

Figure 2 is a block diagram illustrating the detailed structure of the construction site safety management device of Figure 1.

As shown in FIG. 2, the construction site safety management device 130 according to an embodiment of the present invention includes a part of a communication interface unit 200, a control unit 210, a VR image processing unit 220, and a storage unit 230. or all inclusive.

Here, "including some or all" means that some components such as the storage unit 230 are omitted or some components such as the VR image processing unit 220 are integrated with other components such as the control unit 210. As meaning what can be configured, etc., it will be described as including all in order to help a sufficient understanding of the invention.

The communication interface unit 200 may communicate with the user device 100 via the communication network 120 of FIG. 1 and the communication relay device 117 . In the process of performing these communication operations, the communication interface unit 200 may perform operations such as modulation/demodulation, encoding/decoding, muxing/demuxing, scaling to convert resolution, and the like. In this regard, since it is obvious to those skilled in the art, further description will be omitted.

The communication interface unit 200 receives image data for a photographed image of the camera 113 and scan data provided by the scanner device 115 as monitoring data of a construction site, and provides the received image data to the control unit 210 . The scan data may be, for example, location tracking information for tracking the location of the user device 100, such as a beacon device or a smartphone.

In addition, the communication interface unit 200 may perform an operation to control the interior of the construction site with VR images when an event (eg, incident, accident) occurs in the construction site or there is a request from a control agent or supervisor. . To this end, VR image data may be transmitted so that the VR image is displayed on a control agent's computer or monitor, or on the electronic display board of the control center.

The control unit 210 may perform overall control operations of the communication interface unit 200, the VR image processing unit 220, and the storage unit 230 of FIG. 2 constituting the construction site safety management device 130 of FIG. 1. . Typically, the control unit 210 receives image data for a photographed image of a construction site received through the communication interface unit 200 and scan data for sensing the position of a worker or equipment, and temporarily stores it in the storage unit 230. After that, it can be called out if necessary and provided to the VR image processing unit 220.

For example, when the camera 113 is initially set at the construction site, the controller 210 may generate a VR background image of the construction site using images captured by each camera. In addition, the created VR background image can be updated whenever the interior of the construction site is changed in real time or periodically. To this end, of course, it can operate in conjunction with the VR image processing unit 220. In addition, the controller 210 may output a VR image generated by overlapping objects on a previously created VR background image when switching to a VR image control mode of a construction site.

The VR image processing unit 220 generates a VR image so that the interior of the construction site can be monitored in real time VR. Here, the VR image is composed of a VR background image excluding objects in the interior of the construction site, an object overlapping the background image, or object images. Of course, all of the objects here can be observed, even those whose location is not tracked by a beacon or the like. For example, in the case of a conventional CCTV, it may be difficult to check the exact location or information of objects in real time. Typically, it may be a problem of resolution. In addition, even when an IP camera, 3D camera, etc. is used, it is inevitable to observe the inside of the construction site limited to the location where it is installed, apart from the high installation cost.

However, the VR image processing unit 220 according to an embodiment of the present invention provides a VR image based on the location of the construction site by using captured images obtained from a plurality of cameras. 360-degree VR video becomes possible. For example, unit frame images corresponding to time t1 may be obtained from images taken by cameras 1 and 2, respectively. Also, an interpolation image located in the middle may be generated using two unit frame images. In this way, a 360-degree VR image can be created by generating interpolated images for each angle and then stitching all the images.

In addition, the VR image processing unit 220 may set and store coordinate values in the generated VR image when the VR image is generated, so that the VR image can be controlled based on this. Of course, such coordinates, that is, the view to be controlled, are provided as two-dimensional coordinate values, but, for example, requested by a control agent as two-dimensional coordinate values, they are converted into three-dimensional coordinate values, and based on this, VR image control is performed you can make it Of course, in order to perform such an operation, 2D coordinate values and 3D coordinate values may be matched and stored in a look-up table (LUT), etc., and then, based on this, a VR image may be output.

Above all, the VR image processing unit 220 accurately distinguishes objects by deep learning operation in order to insert or overlap the exact object in the VR background image. For example, when it is assumed that a blasting operation is performed in a construction site, it may be difficult to accurately determine an object such as a stone newly created thereby. In particular, it may be more difficult to discriminate through a photographed image acquired through CCTV. Therefore, in this case, a deep learning operation for object classification may be performed for accurate determination, and the object finally classified by deep learning may be overlapped with a background image. Above all, depth information of an object is required for overlapping, but it has been described that it can be acquired in various forms. In the case of using a 3D camera, since the camera itself provides depth information, this can be used. If an IP camera or CCTV is used, pixel analysis is used, but scan data scanned by the scanner device 115 may be further used. For example, suppose that the beacon device of worker B is not detected between worker A and worker C in a construction site, or an error occurs. In this case, the distance of worker B can be inferred based on the distance between workers A and C. Accordingly, the corresponding distance value may be used as depth information.

For example, the case of controlling the construction site while watching VR images by designating camera 1 and the case of controlling the construction site while watching VR images by designating camera 2 are different from the viewing position, that is, the view. It can be seen that a VR image is generated and the image is output. To this end, the VR image processing unit 220 may already generate a VR image corresponding to each location, but it may be possible to generate and show it in real time. Since the method for generating a VR image has been sufficiently described above, it is intended to be replaced with the contents. However, when using multi-viewpoints, since the amount of data processing for VR video generation increases, the selectable viewpoints can be set as default, in other words, the selectable viewpoints can be reduced, and one of them can be selected and watched. have.

In addition, the VR image processing unit 220 may transmit VR images to a separate VR device implementing VR images in association with the controller 210 . Therefore, the VR image processing unit 220 outputs images taken at the existing construction site or other images to a dashboard such as an electric signboard, but the VR image may be controlled through a VR device such as a head-mounted device (HMD). have. A VR device may implement a VR image by having a panel such as a (micro) lenticular lens capable of realizing 3D.

The storage unit 230 may temporarily store various information or data processed under the control of the control unit 210, and may output them when requested by the control unit 210. The output data may be provided to the VR image processing unit 220 . Here, it is meaningless to distinguish the meaning of information or data, and it is common to use them interchangeably. Information may mean a control command such as a request (REQ) or a response (ACK), and data may mean image data of a photographed image or scan data of the scanner device 115 . In addition, when examining data packets, the header part includes additional information and the like, and the payload part includes pixel data and the like. Although it can be understood to this extent, the embodiments of the present invention will not specifically limit the concept of such terms.

Meanwhile, as another embodiment of the present invention, the control unit 210 of FIG. 2 includes a CPU and memory, and the CPU and memory may be configured as a single chip. The controller 210 may include a control circuit, an arithmetic unit (ALU), a command interpreter, a registry, and the like, and the memory may include RAM. The control circuit may perform a control operation, the calculation unit may perform calculation and processing of binary bit information, and the command interpretation unit may interpret input binary bit information related to the execution of any operation. Here, the interpretation may be that after matching data on the left and right sides of the lookup table (LUT), the control circuit searches for data on the right side based on the left side data and performs an operation. The registry is involved in software data storage. According to the above configuration, when the construction site safety management device 130 is initially driven, the CPU copies the program of the VR image processing unit 220, loads it into the memory, and then executes it to quickly increase the data processing speed.

FIG. 3 is a block diagram illustrating the structure of the VR image processing unit of FIG. 2 .

The VR image processing unit 220' of FIG. 3 is a modified example of the VR image processing unit 220 of FIG. And (background and object) VR image generator 330 includes some or all.

Here, "including some or all" means that some components such as the object image generator 320 are omitted, or the VR background generator 300 or the object image generator 320 is a VR video generator. 330, which can be integrated and configured, and will be described as including all of them to help a sufficient understanding of the invention.

The VR background generator 300, object deep learning unit 310, object image generator 320, and VR image generator 330 in FIG. 3 are hardware (H/W) modules and software (S/W) It may be configured by a module or a combination thereof.

The VR background generating unit 300 generates a VR background image for the interior of the construction site based on the captured image received from the camera 113. Here, as the VR background image, a 3D captured image received through a 3D camera may be used, and a captured image captured through an IP camera or CCTV may be used. Of course, in the case of 3D images, it may be easier to create VR images. A 360 VR background image can be obtained by generating a VR image for each angle using the received 3D captured image.

In addition, as mentioned above, the VR background generator 300 may create a VR background image by stitching unit frame images corresponding to the same time to the captured images of the cameras 113 installed in different locations. . For example, camera 2 corresponds to a position having a specific angle with respect to camera 1. Accordingly, a VR image for each angle can be generated by combining the unit frame image of camera 2 with the unit frame image of camera 1. Of course, by generating an interpolation image between two images and combining them, a smoother, that is, flexible VR image can be realized at many angle changes.

The object deep learning unit 310 may include, for example, a template-based Deep Convolution Neural Network (DCNN) classifier for object classification, and increase object discrimination accuracy through deep learning such as artificial intelligence (AI) for classification results. can For example, a stone can be typically detected when there is blasting at a construction site, but since various other objects can be detected, it will be possible to clearly classify them through deep learning and implement them in VR images.

The object image generating unit 320 may calculate depth information of an object from the received captured image or may use depth information provided from a 3D captured image. In addition, the object image generator 320 may extract an object in an image form from a unit frame image of the received captured image and manage it. Imaging may include operations such as graphic processing or correction of the extracted object. The object image generator 320 may provide the extracted object image to the background and object VR image generator 330 to be inserted into the VR background image according to the depth information.

The VR image generator 330 creates a VR image by overlapping an object image (or object image) on a VR background image. Overlap may mean that the layer of the VR background image and the layer of the object image are different from each other, as in normal augmented reality, and the object image may mean that the image is implemented by maintaining a transparent state in the remaining area except for the object. However, in the embodiment of the present invention, it will not be particularly limited to such a method. Through this, it is possible to obtain the same effect as controlling the internal space at a specific location on a construction site.

4 is a flowchart illustrating a driving process of a virtual reality-based workplace safety management device according to an embodiment of the present invention.

Referring to FIG. 4 together with FIG. 1 for convenience of explanation, the (virtual reality-based) construction site safety management device 130 of FIG. is received (S400).

In addition, the construction site safety management device 130 generates a background image of virtual reality (VR) for the appearance of the work site using the received captured image, and based on the deep learning result of the object extracted from the received captured image, the object is imaged, and a VR image of the work site is generated and output by synthesizing (or combining, overlaying) the (previously) created VR background image and the imaged object (S410).

Here, imaging an object may mean extracting an object from a unit frame image and performing GUI graphic processing on the extracted object. Alternatively, the pixel value of the extracted object may be analyzed and implemented in the VR background image.

In addition, the workplace safety management device 130 can perform various operations, and since other details have been sufficiently described above, they will be replaced with the contents.

5 is a flowchart showing another driving process of a virtual reality-based workplace safety management device according to an embodiment of the present invention.

Referring to FIG. 5 together with FIG. 1 for convenience of explanation, the work site safety management device 130 of FIG. 1 according to an embodiment of the present invention, for example, captures an image of a work site from a camera 113 of the work site and provides Receive (S500).

In addition, the workplace safety management device 130 generates a VR background image using the received captured image (S510). In order to generate a VR background image, the workplace safety management device 130 may create a VR background image by generating an image for each angle of the received captured image and combining them. Since this has been sufficiently explained above, we will replace it with the contents.

The workplace safety management device 130 extracts an object from the unit frame image of the received captured image, and performs learning on the extracted object to define the object (S520 and S530). The object may typically be a worker (eg, beacon information) or equipment (eg, equipment data), but may be, for example, a stone newly introduced during various construction work or newly discovered during blasting work. You can search the learning DB for object definition.

In addition, the workplace safety management device 130 may generate depth information in order to insert the above extracted object into the VR background image. For example, when a 3D camera is used, since depth information is provided by the camera, depth information can be used to apply to 360-degree VR images based on this. If it is difficult to calculate depth information from unit frame images of captured images, such as CCTVs or IP cameras, depth information may be generated using beacon information. For example, when a specific object that is not tracked is located between the location of worker A and worker B, for example, when a pixel analysis result is close to the location of worker A, depth information may be generated based on this. In this way, when generating depth information, beacon information may be used. Coordinate values for the object can be set in memory as 3-dimensional coordinates (x, y, z).

Furthermore, the workplace safety management device 130 images the object through pixel analysis of the extracted object, and creates a VR image by overlapping the created VR background image (S540, S550). When overlapping an object image on a VR background, it is possible to implement a 360-degree VR image for each angle by reflecting the above depth information. For example, even in the case of an object, it can be implemented in three dimensions through three-dimensional imaging.

In addition, the workplace safety management device 130 displays a VR image (S560). VR images can be implemented on the electronic display board of the control center, but it will also be possible to watch them through VR devices separately worn by control personnel or supervisors monitoring the construction site.

In addition to the above, since the details related to the safety management device 130 at the work site have been sufficiently described above, they will be replaced with the contents.

On the other hand, even if all the components constituting the embodiment of the present invention are described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all of the components may be selectively combined with one or more to operate. In addition, although all of the components may be implemented as a single independent piece of hardware, some or all of the components are selectively combined to perform some or all of the combined functions in one or a plurality of pieces of hardware. It may be implemented as a computer program having. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art. Such a computer program may implement an embodiment of the present invention by being stored in a computer-readable non-transitory computer readable media, read and executed by a computer.

Here, the non-transitory readable recording medium means a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is common in the art to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: user device 113, 115, 117: field installation device
120: communication network 130: construction site safety management device
200: communication interface unit 210: control unit
220: VR image processing unit 230: storage unit
300: VR background generation unit 310: object deep learning unit
320: object image generator 330: VR video generator

Claims (10)

a communication interface unit for receiving a photographed image of the work site from a camera of the work site; and
A virtual reality (VR) background image for the appearance of the work site is created using the received captured image, the object is imaged based on the deep learning result of the object extracted from the received captured image, and the generated VR A control unit for synthesizing a background image and the imaged object to generate and output a VR image of the work site;
The extracted objects include workers, equipment, and stones found during blasting in the work site,
The control unit graphics the extracted object or corrects it to form a three-dimensional image of the object,
The control unit is based on virtual reality for determining why the user device is not detected by switching to a VR video control mode using the generated VR image when the user device in the work site that has been detected in real time is not detected. Work site safety management device. According to claim 1,
The control unit extracts unit frame images of the same time from each of the captured images received from a plurality of cameras installed at different locations in the work site, and stitches each of the extracted unit frame images to obtain the VR background. A virtual reality-based workplace safety management device that generates images. According to claim 2,
The control unit analyzes the received captured image to obtain depth information of the extracted object, and applies the obtained depth information to synthesize the imaged object to the VR background image for virtual reality-based workplace safety. management device. According to claim 3,
The control unit further uses a distance value calculated by beacon information received in relation to the extracted object in order to obtain depth information of the extracted object. According to claim 3,
The control unit is a virtual reality-based workplace safety management device that switches to a VR video control mode when an event or an error is detected in the object at the workplace. Receiving, by a communication interface unit, a photographed image of the work site from a camera of the work site; and
The control unit generates a virtual reality (VR) background image for the appearance of the work site using the received captured image, images the object based on a deep learning result of an object extracted from the received captured image, Including; generating and outputting a VR image of the work site by synthesizing the generated VR background image and the imaged object,
The extracted objects include workers, equipment, and stones found during blasting in the work site,
The control unit graphics the extracted object or corrects it to form a three-dimensional image of the object,
The control unit is based on virtual reality for determining why the user device is not detected by switching to a VR video control mode using the generated VR image when the user device in the work site that has been detected in real time is not detected. Operation method of workplace safety management device of According to claim 6,
The step of generating and outputting,
Based on virtual reality for generating the VR background image by extracting unit frame images of the same time from each of the captured images received from a plurality of cameras installed at different locations in the work site and stitching each of the extracted unit frame images Operation method of workplace safety management device of According to claim 7,
The step of generating and outputting,
A method of driving a virtual reality-based workplace safety management device for obtaining depth information of the extracted object by analyzing the received captured image, and synthesizing the imaged object to the VR background image by applying the acquired depth information. According to claim 8,
The step of generating and outputting,
A driving method of a virtual reality-based workplace safety management device further using a distance value calculated by beacon information received in relation to the extracted object to obtain depth information of the extracted object. According to claim 8,
The step of generating and outputting,
A method of driving a virtual reality-based workplace safety management device that switches to a VR video control mode when an event or an error is detected in the object at the workplace.

Images