KR102627538B1 - System for providing 360 degree panoramic video based safety education contents generating service - Google Patents

Abstract

A 360-degree panoramic video-based safety education content creation service provision system is provided. A 360-degree panoramic camera is used to film the construction site to create 360-degree panoramic content, upload TBM (Tool Box Meeting) training data, and provide training at the construction site. When created in this virtual space, a supervisor terminal maps training data and messages after tagging a risk identifier at the location where risk factors exist in the virtual space, TBM training data uploaded from the supervisor terminal, and risk identifier tagged in the virtual space. , an upload unit that uploads 360-degree panoramic content captured with a 360-degree panoramic camera from worker terminals and supervisor terminals that output training data and messages mapped to risk identifiers, and creates a construction site map using the 360-degree panoramic content. A construction unit that constructs the construction site as a virtual space after creation; a storage unit that maps and stores the training data and messages entered from the supervisor's terminal to the location where the risk identifier is tagged when the supervisor's terminal tags a risk identifier in the virtual space; It includes a generation service providing server that includes a transmission unit that transmits TBM training data, a risk identifier tagged in virtual space, training data mapped to the risk identifier, and a message to the worker terminal.

Description

360 degree panoramic video-based safety education content generation service provision system {SYSTEM FOR PROVIDING 360 DEGREE PANORAMIC VIDEO BASED SAFETY EDUCATION CONTENTS GENERATING SERVICE}

The present invention relates to a system for providing safety education content creation services based on 360-degree panoramic images. The construction site is photographed with a 360-degree panoramic camera to build a construction site map, tagging hazard sign identifiers in the 360-degree panoramic content, and providing educational materials and It provides a system to provide safety training to workers by mapping messages.

Despite the rapid development of technology in the era of the 4th Industrial Revolution, industrial accidents continue to occur at industrial sites. Of course, companies try to form a corporate safety culture and prevent industrial accidents by improving workers' industrial safety awareness through occupational safety and health education, but even though most industrial accidents can be prevented by identifying the cause and establishing countermeasures, there is no response to them. Comparisons and the development of new technologies are weak. Currently, most industrial safety and health education for managers and supervisors in companies is operated in the form of company-owned training or consignment training through external companies. In order to provide efficient and effective supervisory occupational safety and health education, factors such as learners, instructors, teaching activities, learning and performance environments, etc. must closely interact, and improvement work is done to modify and supplement the educational program with a focus on learner responses. This is needed.

At this time, a method of conducting safety education using virtual reality or immersive content was researched and developed. In relation to this, the prior art, Korean Patent No. 10-2034549 (announced on November 8, 2019) and Korean Patent No. 10-2278900 In the issue (announced on July 20, 2021), the operation data of beginner pilots who operate a virtual reality-based experiential construction equipment safety education solution system was extracted and the extracted data was analyzed to determine whether safety accidents occurred due to the beginner's operating behavior patterns. An immersive virtual reality-based heavy equipment simulation training device is provided to derive the correlation between required elements, provide an operational curriculum in an experiential simulator, provide realistic physical training to heavy equipment trainees, and provide 3D video and sound. Media, a Leap Motion-based gesture recognition control function, and a haptic manipulation device that provides hyper-realistic five senses to the user are the components of the simulator, providing education training, self-training, national qualification practical training, and situational training as user services. Each of these is disclosed.

However, since both the former and the latter configurations are simulated in a place far away from the construction site called virtual reality, when the actual site is input, it is difficult to determine where the site is dangerous due to differences or mismatch between the site and the simulation space. There is no training at all in the simulation on what things to watch out for and what work to do at the deployed site. No matter how good you are at driving at a driving school, when you actually go out on the road, there are so many variables that you won't be able to drive properly. Accordingly, the construction site is filmed with a 360-degree panoramic camera, and the construction site itself is created as 360-degree panoramic content. Using this, the supervisor displays dangerous places or instructions, etc. and provides them as safety education content to create educational content. Research and development of a platform that can do this is required.

In one embodiment of the present invention, after filming a construction site using a 360-degree panoramic camera, 360-degree panoramic content is generated, the location and type of the risk identification marker are input from the supervisor's terminal, and tagging is displayed on the 360-degree panoramic content. When uploading messages and training data to a location where a hazard identification marker is tagged on a supervisor's terminal, the worker can view the 360-degree panoramic content on the terminal to determine where the hazard identification marker is displayed and what needs to be done there. , by intuitively understanding what safety rules to pay attention to before work, all simulations are performed virtually within content filmed at the actual site before entering the actual site, thereby zeroing out the error between reality and simulation, and It is possible to provide a 360-degree panoramic video-based safety education content creation service provision system that minimizes accidents that occur due to inadequate training. However, the technical challenge that this embodiment aims to achieve is not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-described technical problem, one embodiment of the present invention uses a 360-degree panoramic camera to photograph a construction site to create 360-degree panoramic content and upload TBM (Tool Box Meeting) training data. When the construction site is created in a virtual space, a risk identifier is tagged at the location where risk factors exist in the virtual space, and then the training data and messages are mapped to the supervisor terminal, the TBM training data uploaded from the supervisor terminal, and the virtual space. An upload unit that uploads 360-degree panoramic content captured with a 360-degree panoramic camera from worker terminals and supervisor terminals that output tagged risk identifiers, training data and messages mapped to risk identifiers, and a construction site map using 360-degree panoramic content. After creating a (Map), when tagging a risk identifier in the virtual space at the construction department or supervisor terminal that constructs the construction site as a virtual space, the training data and messages entered from the supervisor terminal are mapped and stored at the location where the risk identifier is tagged. It includes a storage unit that transmits TBM training data to the worker terminal, a risk identifier tagged in virtual space, and a transmission unit that transmits training data and messages mapped to the risk identifier.

According to one of the means for solving the problems of the present invention described above, after photographing a construction site using a 360-degree panoramic camera, it is possible to determine what content to create 360-degree panoramic content and what safety rules to pay attention to. The location and type of the hazard identification marker are input from the supervisor's terminal and tagged to be displayed in the 360-degree panoramic content. When messages and training data are uploaded to the location where the hazard identification marker is tagged on the supervisor's terminal, the worker's terminal displays the 360-degree panoramic content. By looking at the screen and figuring out where the risk identification marker is displayed, you have to work from there to intuitively understand it before working. Before entering the actual site, all simulations are done virtually within the content filmed at the actual site, so there is a gap between reality and simulation. Zero errors and minimize accidents due to lack of proper training while working.

1 is a diagram illustrating a system for providing a 360-degree panoramic image-based safety education content creation service according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a creation service providing server included in the system of FIG. 1.
Figures 3 and 4 are diagrams for explaining an embodiment in which a 360-degree panoramic image-based safety education content creation service is implemented according to an embodiment of the present invention.
Figure 5 is an operation flowchart illustrating a method of providing a 360-degree panoramic image-based safety education content creation service according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this does not mean excluding other components unless specifically stated to the contrary, but may further include other components, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The terms "about", "substantially", etc. used throughout the specification are used to mean at or close to that value when manufacturing and material tolerances inherent in the stated meaning are presented, and are used to enhance the understanding of the present invention. Precise or absolute figures are used to assist in preventing unscrupulous infringers from taking unfair advantage of stated disclosures. The term “step of” or “step of” as used throughout the specification of the present invention does not mean “step for.”

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware. Meanwhile, '~ part' is not limited to software or hardware, and '~ part' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, apparatus, or device may instead be performed on a server connected to the terminal, apparatus, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is identifying data of the terminal. It can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

1 is a diagram illustrating a system for providing a 360-degree panoramic image-based safety education content creation service according to an embodiment of the present invention. Referring to FIG. 1, the 360-degree panoramic video-based safety education content creation service provision system 1 includes at least one supervisor terminal 100, a creation service provision server 300, and at least one worker terminal 400. can do. However, since the 360-degree panoramic image-based safety education content creation service providing system 1 shown in FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1.

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1, at least one supervisor terminal 100 may be connected to the creation service providing server 300 through the network 200. In addition, the creation service providing server 300 may be connected to at least one supervisor terminal 100 and at least one worker terminal 400 through the network 200. Additionally, at least one worker terminal 400 may be connected to the creation service providing server 300 through the network 200.

Here, the network refers to a connection structure that allows information exchange between each node, such as a plurality of terminals and servers. Examples of such networks include a local area network (LAN) and a wide area network (WAN). Wide Area Network, Internet (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), and Wi-Fi. , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC ( It includes, but is not limited to, Near-Field Communication (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, and DMB (Digital Multimedia Broadcasting) network.

In the following, the term at least one is defined as a term including singular and plural, and even if the term at least one does not exist, each component may exist in singular or plural, and may mean singular or plural. This should be self-explanatory. In addition, whether each component is provided in singular or plural form may be changed depending on the embodiment.

At least one supervisor terminal 100 captures a risk identifier (risk identifier, It may be a terminal that selects the type of tag (risk-specific tag), tags it by specifying a location within the 360-degree panoramic content, and maps and stores training data and messages. In addition, the supervisor terminal 100 may be a terminal that monitors by mirroring whether the video is being viewed on the worker terminal 400, checks the progress of the construction site through 360-degree panoramic content, and reports accident status or It could be a supervisor's terminal that outputs worker status, weather, etc. to a dashboard.

Here, at least one supervisor terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one supervisor terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one supervisor terminal 100 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) ) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

The creation service providing server 300 may be a server that provides a 360-degree panoramic image-based safety education content creation service web page, app page, program, or application. In addition, when a 360-degree panoramic photo or video is received from the supervisor terminal 100, the creation service providing server 300 builds a construction site map to create a virtual space based on the 360-degree panoramic photo or video and creates 360-degree panoramic content. It may be a server that is created. The creation service providing server 300, when the supervisor terminal 100 specifies a location within the 360-degree panoramic content, tags a risk identifier, enters a message, or uploads training data, stores the risk identifier, message, and training data at that location. It may be a server that maps and stores the information and transmits it to the worker terminal 400 to conduct safety training. The generation service providing server 300 may be a server that provides a dashboard so that the supervisor terminal 100 can monitor the safety training status, accident status, weather status, etc. of the worker terminal 400.

Here, the creation service providing server 300 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc.

At least one worker terminal 400 may be a worker's terminal that uses a web page, app page, program, or application related to a 360-degree panoramic image-based safety education content creation service. In addition, at least one worker terminal 400 may be a terminal that outputs 360-degree panoramic content according to the foreign worker's language settings.

Here, at least one worker terminal 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one worker terminal 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one worker terminal 400 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) ) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

Figure 2 is a block diagram for explaining the creation service providing server included in the system of Figure 1, and Figures 3 and 4 are a 360-degree panoramic video-based safety education content generation service implemented according to an embodiment of the present invention. This drawing is for explaining one embodiment.

Referring to Figure 2, the creation service providing server 300 includes an upload unit 310, a construction unit 320, a storage unit 330, a transmission unit 340, a risk identification icon unit 350, and a PPE check unit. It may include (360), an on-site situation sharing unit (370), an education monitoring unit (380), and a 3D modeling unit (390).

The generation service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one supervisor terminal 100 and at least one worker terminal 400 provides 360-degree panoramic image-based information. When transmitting a safety education content creation service application, program, app page, web page, etc., at least one supervisor terminal 100 and at least one worker terminal 400 are provided with a 360-degree panoramic video-based safety education content creation service application. , you can install or open programs, app pages, web pages, etc. Additionally, a service program may be run on at least one supervisor terminal 100 and at least one worker terminal 400 using a script executed in a web browser. Here, a web browser is a program that allows the use of web (WWW: World Wide Web) services and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, etc. Additionally, an application refers to an application on a terminal and includes, for example, an app running on a mobile terminal (smartphone).

Referring to FIG. 2, the upload unit 310 may receive 360-degree panoramic content captured with a 360-degree panoramic camera from the supervisor terminal 100. The supervisor terminal 100 can capture a construction site using a 360-degree panoramic camera to create 360-degree panoramic content. As will be explained later, there is a part where the supervisor tags the risk identifier. If each object is searched and marked in the 360-degree panoramic content in advance, the supervisor can tag the risk identifier more conveniently later.

At this time, methods for detecting objects can be broadly divided into three types. There are ways to obtain a point cloud from a distance sensor, a way to use a vision sensor, and a way to fuse the distance sensor and vision sensor. Among these, the first method uses LiDAR to obtain point clouds to recognize surrounding objects. Unlike the vision method, it has the advantage of accurately measuring 3D distances without being affected by illumination, but it is difficult to recognize objects. There is a problem of requiring an expensive LiDAR sensor to obtain a high-density point cloud. The second method is 2D object detection, which can distinguish objects more accurately, but cannot obtain information about distance. The third method can take the advantages of two sensors through sensor fusion and compensate for the disadvantages of using a single sensor.

Since one embodiment of the present invention uses a 360-degree panoramic camera, 3D object detection in all directions of 360 degrees can be performed using a low-cost VLP-16 LiDAR sensor. Among the deep learning models SVGA-Net, PV-RCNN, F-ConvNet, and IPOD for 3D object detection, F-convNet, which is capable of camera and lidar sensor fusion, can be selected. The principle of the F-convNet model is to detect a 2D object through a camera, set the space behind the object as a frustum space, cluster the point cloud data within the frustum space, and then use a deep learning model to determine the object's width, height, and center point. Get information about To obtain vector values from images, a 2D detection deep learning model must be run. There are many deep learning models for 2D detection, but since the supervisor scans the scene with a 360-degree panoramic camera, the YOLO V4 model with the fastest calculation speed can be selected.

To detect objects in this way, point cloud data coming from an expensive 64-channel LiDAR sensor is required. However, in this embodiment of the present invention, objects are detected using a low-cost LiDAR sensor with 16 channels and a performance of one quarter of 64 channels. To do this, it is necessary to further supplement the point cloud distance information. For this purpose, a Self-Supervised Sparse-to-Dense model can be used. Information about RGB-D for each pixel can be obtained through the depth image obtained from Sparse to Dense. By implementing distance information in RGB-D in the form of point cloud data in a PCD format in a coordinate system, it is possible to supplement the data received from the existing 16-channel VLP-16 LiDAR sensor to achieve more expensive LiDAR performance.

The construction unit 320 can create a construction site map using 360-degree panoramic content and then construct the construction site as a virtual space. To create a construction site map, four major methods can be used, including a method using images that are 360-degree panoramic content. It can be divided into ① manual modeling method, ② BIM (Building Information Model) data utilization method, ③ LiDAR data utilization method, and ④ image data utilization method. First, ① the manual modeling method is a manual modeling method and is mainly performed by modeling experts. In the past, Google Earth used this method to build a mirror world by combining 3D building models modeled by users with the Earth's ellipsoid and satellite photos. This manual modeling method is frequently used in the field, but as the number of buildings increases, the cost and time required for modeling rapidly increases, so it is inefficient in situations that require a large number of buildings, such as city models.

② BIM data is a method of utilizing engineering design data collected during the construction process, and has the advantage of being able to obtain detailed geometric shapes and obtain results most similar to actual buildings. BIM data has a very high usability, but since not all buildings in the city have this BIM data, it can be seen as a disadvantage that data collection itself is often difficult or impossible. ③ As mentioned earlier, the method using LiDAR data can obtain a relatively high quality model, and this is because more precise sampling is possible compared to the method using image data, which will be mentioned later. However, the model may show incomplete results at low altitudes because it does not handle interference results from passers-by or vehicles.

④ Methods of utilizing image data are divided into methods using multiple images and methods utilizing a single image. The method using multiple images is mainly used to estimate the 3D position of the camera through feature point matching between images and to reconstruct 3D information of pixels based on this. These methods generally require multiple images of a single building due to requirements for accuracy and robustness of feature point matching. On the other hand, it is also possible to model a 3D building using a small amount of images. Considering that the 3D shape of a building is mainly composed of a set of planes, the 3D shape can be reconstructed without estimating detailed 3D information for each pixel. Make it possible.

When the supervisor terminal 100 tags a risk identifier in a virtual space, the storage unit 330 may map and store training data and messages input from the supervisor terminal 100 to the location where the risk identifier is tagged. The supervisor terminal 100 uploads TBM (Tool Box Meeting) training data, and when the construction site is created in a virtual space, tags a risk identifier at the location where risk factors exist in the virtual space and then sends the training data and message. It can be mapped. The risk identifier is provided in the form of an icon and output as augmented reality content on 360-degree panoramic content, so when linked with smart glasses such as Google Glass, it is not only output as training data, but also allows each worker to use any information at any location in the field. You need to be careful about the details, and this position can also provide guidance on what tasks should be performed and with what care.

The transmission unit 340 may transmit TBM training data, a risk identifier tagged in virtual space, and training data and messages mapped to the risk identifier to the worker terminal 400. The worker terminal 400 can output TBM training data uploaded from the supervisor terminal 100, a risk identifier tagged in virtual space, and training data and messages mapped to the risk identifier.

When tagging a risk identifier in the supervisor terminal 100, the risk identification icon management unit 350 selects one risk identification icon among at least one type of risk identification icon, and the risk identification icon is 360-degree panoramic content. It can be displayed and output as augmented content in a virtual space composed of.

When the PPE check unit 360 outputs TBM training data from the worker terminal 400 and outputs a checklist for self-checking whether at least one Personal Protective Equipment is worn, the worker terminal ( It can be output in a language according to the language settings of 400, and a response to the checklist can be received and stored from the worker terminal 400.

<Detection of personal protective equipment>

Personal Protective Equipment (PPE) detection uses computer vision and AI technology to check whether field workers are wearing protective equipment. PPE includes hard hats, safety vests, seat belts, safety shoes, and gloves. In Korea, employers are required to provide protective equipment and workers are required to wear protective equipment according to the rules on occupational safety and health standards and the Occupational Safety and Health Act. Field workers often work without wearing protective equipment that is frustrating and cumbersome, and field managers have limitations in monitoring all workers. Therefore, automatically checking whether workers are wearing protective equipment through PPE detection technology can protect the safety of workers, reduce the human and time effort of management, and increase work efficiency.

The core technology applied to PPE detection is object detection technology. Object detection is an automated technique that identifies the object of interest in an image by distinguishing it from the background. It allows the presence and location of the object of interest in the image to be determined by setting a bounding box. A deep learning algorithm may be applied for object detection, for example, the YOLO (You Only Look Once) algorithm may be used. YOLO shows fast processing speed and relatively high detection accuracy compared to other deep learning-based object detection algorithms. Because of this, YOLO is being used in a variety of real-time object detection applications, including PPE detection, but YOLO has a chronic problem. That is, it is a weak point in detecting small objects. In one embodiment of the present invention, YOLOv4, which applies various techniques such as Bag of Freebies and Bag of Specials, can be used to increase detection performance.

CCTV cameras used for construction site monitoring and PPE detection are installed at various site locations. In an outdoor environment, workers appear relatively small in the image output of a camera installed at a distance to secure a wide viewing angle. When YOLOv4-based PPE detection is performed on images from cameras installed in such locations, small protective equipment is often not detected. Therefore, in one embodiment of the present invention, when detecting PPE in a distant shooting video, an image pre-processing technique that can improve detection performance for small-sized PPE objects may be further used. In YOLOv4, the size of the image input for object detection is resized to be the same as the size of the input image set during learning. Therefore, even if a high-resolution image is provided as input, the original input image size is reduced and adjusted, which affects object detection.

Detection performance can be improved if an input image that maintains the quality of the original image is provided through the preprocessing process. Accordingly, when detecting an object in an image frame, only meaningful parts are extracted and provided as input. In other words, only the parts where motion occurs are extracted from the video frame, combined and reconstructed in a mosaic method, and provided as input to the YOLOv4 object detection model. By extracting only the region of interest or meaningful portion, the size of the input image can be reduced and the original resolution of the region subject to detection in the image can be maintained as much as possible. This can improve detection performance for small objects.

<Object tracking>

To apply a video-based location tracking method at a construction site, the 3D location of construction resources such as vehicles and personnel can be successfully tracked using only cameras. However, since there is no separate automated entity matching process, many It is difficult to track objects simultaneously. Due to the nature of the image-based location tracking method using stereo images, it is necessary to not only detect and track objects in the captured images, but also pair the same objects among the objects detected in the two images. If this process is not automated, it is virtually impossible to track the locations of multiple objects using images. Accordingly, in one embodiment of the present invention, an image-based location tracking method that can track multiple objects can be used by adding an object matching process based on epipolar geometry. In the entity matching process, the epipolar line that can be calculated from the fundamental matrix is used as entity matching information.

<Location tracking of multiple personnel>

A method for tracking the location of multiple construction personnel can complement the camera calibration process and object matching process. A separate object matching process can be added to track the positions of multiple objects, and camera calibration can be performed using GCP instead of checkerboard. For this purpose, the basic matrix is calculated from the first frame of each image. The calculated basis matrix is used not only in the camera calibration process but also in the object matching process. Objects are detected and tracked independently in each image. Tracked objects undergo an object matching process using the calculated basic matrix, and these results are used to finally calculate 3D position coordinates.

<Object matching>

Isopolar geometry can be applied for object matching. The epipolar line is calculated using the basis matrix from the object center point obtained during the object tracking process, and the distance between the calculated epipolar line and the center point of the tracked object is compared to determine whether the object matches. To calculate the basic matrix, feature points are extracted using the first frame of each image. There are various feature point extraction algorithms, but by comparing the most commonly used SIFT (Scale Invariant Feature Fransform) algorithm and SURF (Speeded Up Robust Features) algorithm, an algorithm that returns more feature points can be used. By comparing the feature points extracted from the left and right images, only the same corresponding points are used to calculate the basic matrix, and in this process, MSAC (M-estimate SAmple Consensus) can be used to determine the corresponding points.

<Calculation of 3D position coordinates>

In order to calculate 3D position coordinates from an image, corresponding points in the two images are required. When a point in real space and its corresponding point on the image plane are located, the coordinates of the object can be calculated from the corresponding point through the geometric correlation between the image plane where the corresponding point is located and the space where the object is located. In this process, a camera matrix is needed, and the relationship between the corresponding points and the camera matrix can be established. The camera matrix can be calculated from the basic matrix, but a sufficiently large number of corresponding points is required for accurate calculation. The process of calculating 3D position coordinates from corresponding points in the image involves calculating a basic matrix from matching pairs of feature points, assuming a camera matrix from the calculated basic matrix, and calculating the position coordinates of the GCP using the assumed camera matrix. , calculate the homography matrix by comparing the actual coordinates of the GCP and the calculated coordinates, correct the assumed camera matrix using the homography matrix, and calculate the position coordinates of the tracking object corresponding points using the corrected camera matrix.

The basis matrix is used to calculate the camera matrix. Assume a camera matrix from the basic matrix and correct it using GCP. The initial camera matrix is assumed from the relationship between the camera matrix and the basic matrix, and the camera matrix is re-corrected using the geometric relationship between the position coordinates of the scene reconstructed using the assumed camera matrix and the ground truth position coordinates (GCP). If a unique basic matrix is determined using multiple corresponding points in a stereo image, the 3D position of the camera and each corresponding point can be reconstructed using the basic matrix and the corresponding points. In order to reconstruct it in a form with errors removed as much as possible, correction of the camera matrix is necessary, and a homography matrix is used for this correction. The homography matrix can be calculated using actually measured 3D position coordinates and can be calculated using GCP.

If the corrected camera matrix and the corresponding points of the image for calculating 3D position coordinates are given, 3D position coordinates can be calculated through triangulation. Linear triangulation method is used to calculate 3D position coordinates. The triangulation method is a method of calculating the position coordinates of a point from corresponding points created when a point in actual 3D space is projected onto an image. Of course, in addition to the above-mentioned methods, each object can be tracked in various ways. Each object can be tracked simply using YOLO, or each object can be tracked using intelligent CCTV. Once objects have been identified and tracked in this way, based on industrial accident safety and health guidelines or government guidelines, it is necessary to define which states are at high or low risk of industrial accidents and which actions are dangerous actions that violate the law. If such conditions or actions are observed, risk notification can be provided immediately.

The site situation sharing unit 370 provides a dashboard that determines the weather at the location of the construction site set in the supervisor terminal 100, the training status of the worker terminal 400, and the accident status at the construction site. You can. Statistical data is collected in real time and visualized in tables, charts, or graphs, allowing users to intuitively understand what is happening in the field.

The education monitoring unit 380 can monitor whether a screen for safety education is being output on the screen of the worker terminal 400 by mirroring the screen of the worker terminal 400 to the screen of the supervisor terminal 100. there is.

The 3D modeling unit 390 can map and store the GPS location and number of floors in the 360-degree panoramic content captured by the supervisor terminal 100 and then create the construction site as a virtual space through 3D modeling.

Hereinafter, the operation process according to the configuration of the generation service providing server of FIG. 2 described above will be described in detail using FIGS. 3 and 4 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to FIG. 3A, an embodiment of the present invention proceeds with the configuration of FIG. 3C as shown in FIGS. 3D to 3F based on the plan of FIG. 3B with the goal of developing technology to secure the safety of workers at construction sites. It may have the same differentiation as Figures 3g and 3h. Looking at Figure 4a, the process according to an embodiment of the present invention consists of a total of 7 steps, the first step is site description, the second step is creating a construction site map, the third step is creating educational materials, and the fourth step is Training material registration, the 5th step is sending worker registration, the 6th step is TBM training, and the 7th step is settings and statistics management. In the first step, after installing the application for creating field modeling, the 360-degree panoramic camera and the supervisor terminal 100 are linked and photographs are taken while moving to each floor, and then the supervisor terminal 100 creates a 360-degree panorama of the photographed site. The photo is transmitted to the creation service providing server 300 that receives the photo. In the second step, referring to Figure 4b, a map can be created using the transmitted 360-degree panoramic photo data, a project management function can be implemented, and the progress rate can be confirmed by comparing field photos by date.

Referring to Figure 4c, in the third step, training materials on unit operations, risk factors, and improvement measures according to the process can be created and shared at the supervisor terminal 100, and referring to Figure 4d, in the fourth step, safety education materials are created. When registering, you can set the risk identifier appropriate for the risk type as a tag, create a list of content created by work site and process, and provide insufficient or additional work instructions at the work site and manage them as history. . Referring to Figure 4e, the fifth step is the worker registration sending step, where workers can be registered and managed, and text messages can be sent to the worker terminal 400 within the generated construction site map, including action details, time, text, and photos. You can record changes, etc. Referring to Figure 4f, the sixth step is TBM training, and the worker terminal 400 can determine whether the worker has properly prepared PPE and whether the worker is viewing 360-degree panoramic content through the supervisor terminal 100. Referring to Figures 4g and 4h, the 7th step provides a dashboard to manage weather, training status, accident status, etc. Self-diagnosis status, training completion status, etc. can also be checked and managed with statistics. . This is summarized in Table 1 below.

Step1 - Import 2D/360 omnidirectional images of the site using a camera
- Import the generated photo data into SW and map it to the VR skybox
- As a procedure for creating safety education content, a dedicated camera for 360 panoramic photo shooting is linked using a dedicated application for on-site shooting, and the 360 panoramic photo is taken and transmitted to the server to create the site as a virtual space. Step2 - Institutional managers (safety training supervisors, etc.) log in and manage the site by process and date based on the taken site photos to create a 360/2D map of the site's virtual space. Step3 - Automatically saves the contents of the T.B.M. training process, manages training documents by date, and uses the document printing function to quickly and easily create a daily safety inspection log Step4 - While moving through the created virtual space of the site, the safety management supervisor places hazard identification marks on locations where accidents occur frequently or is judged to be dangerous, and records educational materials and messages to ensure safety. Step5 - Safety training content completed for each process can be sent via text message to registered workers. Workers who receive the link address via text message access the link, go through the login process, and then go through the self-diagnosis of whether they are wearing uniforms and other self-diagnosis set by the supervisor. Later, safety training content created by the supervisor was used to provide training during the morning TBM training session. Step6 - Accessibility so that workers can conduct mobile training by accessing a link or QR code received via text message without having to install a separate application. Step7 - Import 2D/360 omnidirectional images of the site using a camera
- Import the generated photo data into SW and map it to the VR skybox.

Referring to FIGS. 4i to 4k, an application dedicated to managers (supervisors) including setting functions such as job name, work location, work process, and operator (worker) can be further provided. ① Enter member information in the application and request authentication from the control system. ② The authentication (membership) management module of the control system returns the authentication processing results. ③ Start creating safety education content in the application. ④ Take photos and videos and classify them with the media management module. ⑤ Enter the data, work processes, workers, etc. managed by the media management module into the safety education content creation and editing module. ⑥ Upload safety training contents and materials managed by the media management module to the control system and request a result report. ⑦ When the control system receives machine learning data from the media management module of the application, it transmits the machine learning source to the machine learning system. Tables 2 to 4 show authentication module (CA) -> authentication management module (SA) [REST API], authentication management module (SA) -> notification module (SN) [internal module call], authentication management module (SG) - > This explains each web module (SW) [internal module call].

id Request Response explanation CASA-0101 {
“agency_id”:””,
“agent_id”:””
} {
“error”:””,
“result”:”
} Request to check supervisor account status.
agent_id(email) is not a unique value in the entire DB, but only within the organization. In other words, if you are registered at institution A and then join institution B after leaving the company, the inquiry results must be returned to a supervisor who does not exist at institution B.

[Request]
agency_id: Agency ID to which the supervisor belongs
agent_id: ID to use for inquiry

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set.
result: If the supervisor account does not exist, set “” If the supervisor account exists, but it is inactive after leaving the company, “” If the supervisor account exists and is usable, set “”. CASA-0102
{
“error”:””,
“agencies”:
[
“name”:””,
“id”:””
]
} Request a list of disaster guidance organizations.

[Request]
N/A

[Response]
agencies: Arrangement of agency information, consisting of the name and ID of each agency CASA-0103 {
“agency_id”:””,
“agent_id”:””
} {
“error”:””
} Request a password reset.
Create a password reset page and send it by email.

[Request]
agency_id: Agency ID
agent_id: Supervisor ID

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set. CASA-0104 {
“agency_id”:””,
“agent_id”:””
} {
“error”:””
} Request registration as a supervisor.
The authentication management module that received the request requests the web module to create a page to proceed with supervisor registration and sends it to the appropriate email address.
The created page checks whether the entered password satisfies the password conditions (length, special characters, etc.) and whether repeatedly entered passwords are the same, and creates an account for the relevant supervisor.

[Request]
agency_id: Agency ID to which the supervisor belongs
agent_id: Supervisor ID to register

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set. CASA-0105 {
“session_id”:””,
“password”:””
} {
“error”:””,
“match”
} Request whether the password matches or not.
Used when trying to access the personal information page on My Page.

[Request]
session_id: Session ID issued upon login
password: password

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set.
match: set to true if the password is correct, false if it is not correct. CASA-0106 {
“session_id”:””,
…
} {
“error”:””,

} Request an update to your supervisor information.

[Request]
session_id: Session ID issued upon login
… : Contents to be updated (undetermined)

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set. CASA-0107 {
“agency_id”:””,
“id”:””,
“password”:””,
“uuid”:””
} {
“error”:””,
“session_id”:””
} Request to log in.

[Request]
agency_id: agency ID
id: supervisor ID
password: password
uuid: Device unique number

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set.
session_id: Session ID CASA-0108 {
“session_id”:””
} {
“error”:””
} Request logout.
Delete session-related data.

[Request]
session_id: Session ID

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set.

id Request Response explanation SASN-0201 message error Request notification of approval of supervisor registration.
Push notifications are sent to devices connected to the account.

[Request]
message: Approval message to be sent via Push Notification

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set. SASN-0202 message error Request notification of on-site assignment/reassignment of a specific supervisor.
Push notifications are sent to devices connected to the account.

[Request]
message: Assignment/reassignment message to be sent through Push Notification

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set.

id Request Response explanation SASW-0301 agency_id
email error Send a supervisor registration page (password setting).
Create a password setting page and send it to the target email address.
Once a password is set, the page is discarded and when refreshed or returned to, it connects to a page that displays a message that a password has been set.

[Request]
agency_id: Agency ID to which the supervisor belongs
email: Email address to send the password setting page to
[Response]
N/A SASW-0302 agency_id
email error Request to be sent a supervisor account password reset page.
Create a password reset page and send it to the target email.
After resetting the password, the page is discarded and when refreshed or returned, it connects to a page that displays a message that it has been reset.

[Request]
agency_id: Agency ID to which the supervisor belongs
email: Email address to send the password setting page to

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set. SASW-0303 email error Send the institution registration page (password setting).
Create a password setting page and send it to the target email address.
Once a password is set, the page is discarded and when refreshed or returned to, it connects to a page that displays a message that a password has been set.

[Request]
email: Email address to send the password setting page to

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set. SASW-0304 email error Request to be sent a password reset page.
Create an institutional account password reset page and send it to the target email.
After resetting the password, the page is discarded and when refreshed or returned, it connects to a page that displays a message that it has been reset.
[Request]
email: Email address to send the password setting page to

[Response]
error: If there is no error, an empty string is set, and if there is an error, the corresponding content is set.

In addition, safety education materials created at multiple construction sites can be used for safety education by sharing information between construction companies using SW and registering and loading education materials appropriate for the site. A DB can be built by re-editing and digitizing safety training materials held by the Occupational Safety and Health Agency, construction companies, etc., the server upload of the written safety training materials is managed, and a daily safety training log can be easily and quickly automatically created. Includes. The type of training is mainly produced and provided as safety education corresponding to serious disasters, and can be produced in-house to suit the site. It is produced as media material using safety education (Korea Occupational Safety and Health Agency, etc.) materials for five types of serious disasters and uploaded to the server. , it can include the ability to create and upload materials at construction sites using safety education content creation software and share education content between construction companies. T.B.M. training or VR training can provide functions that can be fully utilized in regular safety training and special safety training specified in the Occupational Safety and Health Act.

Matters that are not explained about the method of providing the 360-degree panoramic video-based safety education content creation service in FIGS. 2 to 4 are the same as those previously explained in FIG. 1 regarding the method of providing the 360-degree panoramic video-based safety education content creation service. Since it is the same or can be easily inferred from the explained content, the description below will be omitted.

FIG. 5 is a diagram illustrating a process in which data is transmitted and received between components included in the 360-degree panoramic image-based safety education content creation service provision system of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of the process of transmitting and receiving data between each component will be described with reference to FIG. 5, but the present application is not limited to this embodiment, and the process shown in FIG. 5 according to the various embodiments described above It is obvious to those skilled in the art that the process of transmitting and receiving data can be changed.

Referring to FIG. 5, the creation service providing server uploads 360-degree panoramic content captured with a 360-degree panoramic camera from the supervisor terminal (S5100).

In addition, the creation service providing server creates a construction site map using 360-degree panoramic content, builds the construction site as a virtual space (S5200), and tags risk identifiers in the virtual space on the supervisor's terminal. The training data and messages entered from the supervisor's terminal are mapped and stored in the location where the identifier is tagged (S5300).

In addition, the creation service providing server transmits TBM training data, a risk identifier tagged in virtual space, and training data and messages mapped to the risk identifier to the worker terminal (S5400).

The sequence between the above-described steps (S5100 to S5400) is only an example and is not limited thereto. That is, the order between the above-described steps (S5100 to S5400) may change, and some of the steps may be executed simultaneously or deleted.

Matters that are not explained regarding the method of providing the 360-degree panoramic video-based safety education content creation service in FIG. 5 are the same as those previously explained in FIGS. 1 to 4 regarding the method of providing the 360-degree panoramic video-based safety education content creation service. Since it is the same or can be easily inferred from the explained content, the description below will be omitted.

The method of providing a 360-degree panoramic image-based safety education content creation service according to an embodiment described with reference to FIG. 5 is in the form of a recording medium containing instructions executable by a computer, such as an application or program module executed by a computer. can also be implemented. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The method of providing a 360-degree panoramic image-based safety education content creation service according to an embodiment of the present invention described above includes an application installed by default in a terminal (this may include programs included in a platform or operating system that is installed by default in the terminal). ), and may also be executed by an application (i.e. program) installed by the user directly on the master terminal through an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the method of providing a 360-degree panoramic image-based safety education content creation service according to an embodiment of the present invention described above is implemented as an application (i.e., program) installed by default in the terminal or directly installed by the user, and is installed in the terminal, etc. It can be recorded on a computer-readable recording medium.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (6)

Using a 360-degree panoramic camera, the construction site is photographed to create 360-degree panoramic content, TBM (Tool Box Meeting) training data is uploaded, and when the construction site is created in a virtual space, risk factors in the virtual space are checked. A supervisor terminal that tags risk identifiers in existing locations and then maps training data and messages;
A worker terminal that outputs TBM training data uploaded from the supervisor terminal, a risk identifier tagged in the virtual space, and training data and messages mapped to the risk identifier; and
An upload unit that uploads 360-degree panoramic content captured with a 360-degree panoramic camera from the supervisor's terminal, and a construction unit that creates a construction site map using the 360-degree panoramic content and then constructs the construction site as a virtual space. , when the supervisor terminal tags a risk identifier in the virtual space, a storage unit that maps and stores training data and messages input from the supervisor terminal to the location where the risk identifier is tagged, the TBM training data to the worker terminal, A generation service providing server including a transmission unit that transmits a risk identifier tagged in the virtual space, education data mapped to the risk identifier, and a message,
The creation service providing server,
When tagging the risk identifier on the supervisor terminal, any one risk identification icon among at least one type of risk identification icon is selected, and the risk identification icon is displayed as augmented content in a virtual space consisting of the 360-degree panoramic content. A risk identification icon management unit that is displayed and printed;
When printing TBM training data from the worker terminal, when printing a checklist for self-checking whether at least one personal protective equipment is worn, print it in a language according to the language setting of the worker terminal. , a PPE check unit that receives and stores a response to the checklist from the worker terminal;
A site situation sharing unit that provides a dashboard that determines the weather at the location of the construction site set in the supervisor terminal, the training status of the worker terminal, and the accident status at the construction site;
An education monitoring unit that monitors whether a screen for safety education is displayed on the screen of the worker terminal by mirroring the screen of the worker terminal to the screen of the supervisor terminal; and
It further includes a 3D modeling unit that maps and stores the GPS location and number of floors to the 360-degree panoramic content captured by the supervisor terminal, and then generates the construction site as a virtual space through 3D modeling,
The generation service providing server sets the risk identifier appropriate for the risk type as a tag when registering safety education materials, and generates a content list for each work site and process. Safety education content based on 360-degree panoramic video. Creation service provision system. delete delete delete delete delete

Images