KR102622907B1 - Method for object recognition of constrction sites and apparatus using the same - Google Patents

Abstract

The present invention relates to an object recognition method at a construction site and an apparatus using the same, which includes the object recognition method implemented by a processor, comprising the steps of receiving image data from a manager device, and a process for a first process selected by the user. Receiving data, determining object data for the first process using an object recognition model learned to output object data using the image data as input, and based on the process data and the object data and determining an object recognition exception zone for the image data.

Description

Object recognition method at construction site and device using the same {METHOD FOR OBJECT RECOGNITION OF CONSTRCTION SITES AND APPARATUS USING THE SAME}

The present invention relates to an object recognition method at a construction site and a device using the same.

At construction sites, multiple CCTVs can be placed on site to check the safety of workers. At this time, worker safety detection technology may include object detection, recognition, tracking, alarm, etc. Here, the alarm function can be provided when workers violate safety rules, such as not wearing a safety vest or safety helmet.

However, 2D image data received through fixed CCTV is not recognized as a person when a worker is present between buildings, and is subject to various shapes, changes in surrounding noise depending on the season, and distortion depending on angle and distance. This may cause false detections and non-detection, resulting in a decrease in image recognition rate.

In addition, in the case of construction sites, the site changes continuously as construction progresses, and depending on the site, the floor height is high or various construction machines can be used, which can lead to misdetection of objects, or non-detection and errors if they are obscured by structures or temporary buildings. Detection may occur.

The technology behind the invention has been written to facilitate easier understanding of the invention. It should not be understood as an admission that matters described in the technology underlying the invention exist as prior art.

The inventors of the present invention have developed a system that receives image data from a construction site, detects objects in the received image data, and checks the safety of workers.

In particular, the inventors of the present invention have invented a method of setting areas in which many false detections of objects detected from image data of construction sites occur as critical sections or exception sections and excluding alarms.

Accordingly, the problem to be solved by the present invention is to receive image data from a construction site, receive process data for the first process, determine the object for the first process through an object recognition model, and It provides an object recognition method at a construction site that sets the area where it occurs as an exception area.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the problems described above, an object recognition method at a construction site according to an embodiment of the present invention is provided. The method includes receiving image data from a manager device, receiving process data for a first process selected by a user, and using an object recognition model learned to output object data using the image data as input. , determining object data for the first process and determining an object recognition exception zone for the image data based on the process data and the object data.

According to another feature of the present invention, the method may further include labeling at least one region of the image data.

According to another feature of the present invention, the step of determining at least one area of the image data as an important area may be further included.

According to another feature of the present invention. Determining an object recognition exception zone for the image data includes counting the number of malfunctions of the object recognition model for the zone of the image data, and when the number of malfunctions is greater than or equal to a threshold, the zone of the image data is classified as an object. It may include the step of determining a recognition exception zone.

According to another feature of the present invention, the first process may include at least one of rebar assembly, painting work, electrical equipment work, mechanical equipment work, material lifting work, and retaining wall form assembly.

According to another feature of the present invention, the object data may be at least one mask (MASK) including a worker, manager, hard hat, safety vest, excavator, or crane identified from the image data.

According to another feature of the present invention, the method may further include determining whether the worker or manager of the image data is wearing at least one of a safety helmet and a safety vest.

According to another feature of the present invention, if the worker or the manager does not wear the safety helmet and safety vest, the step of providing an alarm to the manager device may be further included.

According to another feature of the present invention, the object recognition model may have a MASK R-CNN structure.

In order to solve the problems described above, an object recognition device at a construction site according to an embodiment of the present invention is provided. The device includes a communication unit, a storage unit, and a processor operably connected to the communication unit and the storage unit, wherein the processor receives image data from a manager device and process data for a first process selected by the user. Receives, determines object data for the first process using an object recognition model learned to output object data using the image data as input, and generates the image data based on the process data and the object data. It is configured to determine the object recognition exception zone for.

According to another feature of the present invention, the processor may be further configured to perform labeling of at least one region of the image data.

According to another feature of the present invention, the processor may be further configured to determine at least one area for the image data as an important area.

According to another feature of the present invention, the processor counts the number of malfunctions of the object recognition model for the region of the image data, and when the number of malfunctions is greater than or equal to a threshold, the region of the image data is designated as an object recognition exception region. It can be further configured to decide.

According to another feature of the present invention, the first process may include at least one of rebar assembly, painting work, electrical equipment work, mechanical equipment work, material lifting work, and retaining wall form assembly.

According to another feature of the present invention, the object data may be at least one mask (MASK) including a worker, manager, hard hat, safety vest, excavator, or crane identified from the image data.

According to another feature of the present invention, the processor may be further configured to determine whether the worker or manager of the image data is wearing at least one of a hard hat and a safety vest.

According to another feature of the present invention, if the worker or the manager does not wear the safety helmet and safety vest, it may be further configured to provide an alarm to the manager device.

According to another feature of the present invention, the object recognition model may have a MASK R-CNN structure.

Specific details of other embodiments are included in the detailed description and drawings.

The present invention can detect workers' violations of safety rules by recognizing objects in image data through an object recognition model, thereby preventing accidents in advance.

In particular, the present invention can reduce the frequency of false detections and non-detection by setting areas with a high frequency of misdetection and non-detection among areas where alarms generated from object data are generated as object recognition exception zones, thereby reducing the frequency of false detections and non-detection. The accuracy of provision can be improved.

The effects according to the present invention are not limited to the details exemplified above, and further various effects are included within the present invention.

1 is a schematic diagram of an object recognition system at a construction site using a device for providing an object recognition system at a construction site according to an embodiment of the present invention.
Figure 2 is a block diagram showing the configuration of a manager device according to an embodiment of the present invention.
Figure 3 is a block diagram showing the configuration of an object recognition server at a construction site according to an embodiment of the present invention.
Figure 4 is a flowchart of an object recognition method at a construction site according to an embodiment of the present invention.
5 to 9 are exemplary diagrams of an object recognition method at a construction site according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms, but the present embodiments only serve to ensure that the disclosure of the present invention is complete and are within the scope of common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In this document, expressions such as “have,” “may have,” “includes,” or “may include” refer to the existence of the corresponding feature (e.g., a numerical value, function, operation, or component such as a part). , and does not rule out the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” includes (1) at least one A, (2) at least one B, Or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in this document can modify various components regardless of order and/or importance, and can refer to one component. It is only used to distinguish from other components and does not limit the components. For example, a first manager device and a second manager device may represent different manager devices regardless of order or importance. For example, the first component may be renamed as the second component without departing from the scope of rights described in this document, and similarly, the second component may also be renamed as the first component.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

As used in this document, the expression “configured to” depends on the situation, for example, “suitable for,” “having the capacity to.” ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

Terms used in this document are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this document, have an ideal or excessively formal meaning. It is not interpreted as In some cases, even terms defined in this document cannot be interpreted to exclude embodiments of this document.

Each feature of the various embodiments of the present invention can be partially or fully combined or combined with each other, and as can be fully understood by those skilled in the art, various technical interconnections and operations are possible, and each embodiment may be implemented independently of each other. It may be possible to conduct them together due to a related relationship.

For clarity of interpretation of this specification, terms used in this specification will be defined below.

1 is a schematic diagram of an object recognition system at a construction site using an object recognition device at a construction site according to an embodiment of the present invention.

Referring to FIG. 1, the object recognition system 1000 at a construction site includes a manager device 100 and a user interface providing server 200 (hereinafter referred to as the object recognition server 200 at a construction site).

The object recognition system 1000 at a construction site may receive image data and process data for the first process, and provide object data for the image data corresponding to the first process received through an object recognition model. At this time, the object recognition system 1000 at the construction site may provide an alarm for object data. here. The object recognition system 1000 at a construction site may set an area with a high frequency of false alarms due to non-detection and false detection of object data as an object recognition exception area. Meanwhile, the object recognition system 1000 at the construction site transmits image data and process data to the object recognition server 200 at the construction site, and the manager device 100 receives object data and alarms for the image data and the received It may be configured as an object recognition server 200 at a construction site that provides object data and alarms for the object data based on image data and process data.

As a result, managers can detect workers' violations of safety rules by recognizing objects in image data through an object recognition model, thereby preventing accidents in advance.

Specifically, the manager device 100 may be a device that requests an object recognition system at a construction site and receives the result. At this time, the manager device 100 is an electronic device capable of recording voice and outputting a screen, and may include a smartphone, tablet PC, PC, laptop, etc.

Figure 2 is a block diagram showing the configuration of a manager device according to an embodiment of the present invention.

Referring to FIG. 2 , the manager device 100 may include a memory interface 110, one or more processors 120, and a peripheral interface 130. Various components within the manager device 100 may be connected by one or more communication buses or signal lines.

The memory interface 110 is connected to the memory 150 and can transmit various data to the processor 120. Here, the memory 150 is a flash memory type, hard disk type, multimedia card micro type, card type memory (for example, SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, and cloud. , It may include at least one type of storage medium among the blockchain database.

In various embodiments, memory 150 includes operating system 151, communications module 152, graphical user interface module (GUI) 153, sensor processing module 154, telephony module 155, and application module 156. ) can be stored. Specifically, the operating system 151 may include instructions for processing basic system services and instructions for performing hardware tasks. The communication module 152 may communicate with at least one of one or more other devices, computers, and servers. The graphical user interface module (GUI) 153 can process a graphical user interface. Sensor processing module 154 may process sensor-related functions (eg, processing voice input received using one or more microphones 192). The phone module 155 can process phone-related functions. Application module 156 may perform various functions of a user application, such as electronic messaging, web browsing, media processing, navigation, imaging, and other processing functions. In addition, the manager device 100 may store one or more software applications 156-1 and 156-2 associated with one type of service in the memory 150. At this time, the application 156-1 may provide the manager device 100 with information about the object recognition system at the construction site. Here, the application 156-1 may provide an image data input interface or a process data input interface to the manager. Additionally, the application 156-1 may provide the user with object data matched to image data received from the object recognition server 200 at the construction site.

In various embodiments, memory 150 may store a digital assistant client module 157 (hereinafter referred to as DA client module), thereby storing various user data 158 and instructions for performing client-side functions of the digital assistant. (e.g. other data such as administrator data, image data records, etc.) can be stored.

Meanwhile, the DA client module 157 uses the user's voice input, text input, touch input, and/or gesture input through various user interfaces (e.g., I/O subsystem 140) provided in the manager device 100. It can be obtained.

Additionally, the DA client module 157 can output data in audiovisual and tactile forms. For example, the DA client module 157 may output data consisting of a combination of at least two or more of voice, sound, notification, text message, menu, graphics, video, animation, and vibration. In addition, the DA client module 157 can communicate with a digital assistant server (not shown) using the communication subsystem 180.

In various embodiments, the DA client module 157 may collect additional information about the surrounding environment of the administrator device 100 from various sensors, subsystems, and peripheral devices to construct a context associated with user input. . For example, the DA client module 157 may infer the user's intention by providing context information along with user input to the digital assistant server. Here, context information that may accompany the user input may include sensor information, for example, lighting, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, the context information may include the physical state of the manager device 100 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc.). As another example, context information may include information related to the software status of the manager device 100 (e.g., processes running on the manager device 100, installed programs, past and present network activity, background services, error logs, resource usage). etc.) may be included.

In various embodiments, the memory 150 may include added or deleted instructions, and further, the manager device 100 may also include additional components other than those shown in FIG. 2 or exclude some components.

The processor 120 can control the overall operation of the manager device 100 and executes various commands to implement an interface for providing object recognition results at a construction site by running an application or program stored in the memory 150. .

The processor 120 may correspond to a computing device such as a Central Processing Unit (CPU) or an Application Processor (AP). Additionally, the processor 120 may be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) in which various computing devices such as a neural processing unit (NPU) are integrated.

The peripheral interface 130 is connected to various sensors, subsystems, and peripheral devices and can provide data so that the manager device 100 can perform various functions. Here, what function the manager device 100 performs may be understood as being performed by the processor 120.

The peripheral interface 130 may receive data from the motion sensor 160, the light sensor (light sensor) 161, and the proximity sensor 162, through which the manager device 100 can determine orientation, light, and proximity. It can perform detection functions, etc. For another example, the peripheral interface 130 may receive data from other sensors 163 (positioning system-GPS receiver, temperature sensor, biometric sensor), through which the manager device 100 can use other sensors. Functions related to (163) can be performed.

In various embodiments, the manager device 100 may include a camera subsystem 170 connected to the peripheral interface 130 and an optical sensor 171 connected thereto, through which the manager device 100 can take photos and video. You can perform various shooting functions such as clip recording.

In various embodiments, the manager device 100 may include a communication subsystem 180 connected to the peripheral interface 130. The communication subsystem 180 consists of one or more wired/wireless networks and may include various communication ports, radio frequency transceivers, and optical transceivers.

In various embodiments, the manager device 100 includes an audio subsystem 190 connected to a peripheral interface 130, which includes one or more speakers 191 and one or more microphones 192. By including this, the manager device 100 can perform voice-activated functions, such as voice recognition, voice duplication, digital recording, and phone functions.

In various embodiments, the manager device 100 may include an I/O subsystem 140 connected to a peripheral interface 130. For example, the I/O subsystem 140 may control the touch screen 143 included in the manager device 100 through the touch screen controller 141. As an example, the touch screen controller 141 uses any one of a plurality of touch sensing technologies such as capacitive, resistive, infrared, surface acoustic wave technology, proximity sensor array, etc. to detect the user's touch and movement or touch. and cessation of movement can be detected. As another example, the I/O subsystem 140 may control other input/control devices 144 included in the manager device 100 through other input controller(s) 142. As an example, other input controller(s) 142 may control one or more buttons, rocker switches, thumb-wheels, infrared ports, USB ports, and pointer devices such as a stylus.

Figure 3 is a block diagram showing the configuration of an object recognition server at a construction site according to an embodiment of the present invention.

Referring to FIG. 3, the object recognition server 200 at a construction site may include a communication interface 210, a memory 220, an I/O interface 230, and a processor 240, and each configuration includes one or more They can communicate with each other through communication buses or signal lines.

The communication interface 210 can be connected to the manager device 100 through a wired/wireless communication network to exchange data. For example, the communication interface 210 may receive image data or process data for the first process from the manager device 100. At this time, the image data may be image data for the first process. Here, the communication interface 210 may provide an interface that provides object data for received image data and process data. Additionally, the communication interface 210 may provide alarms for object data.

Meanwhile, the communication interface 210 that enables transmission and reception of such data includes a communication pod 211 and a wireless circuit 212, where the wired communication port 211 is connected to one or more wired interfaces, for example, Ethernet, May include Universal Serial Bus (USB), Firewire, etc. Additionally, the wireless circuit 212 can transmit and receive data with an external device through RF signals or optical signals. Additionally, wireless communications may use at least one of a plurality of communication standards, protocols and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

The memory 220 may store various data used in the object recognition server 200 at a construction site. For example, the memory 220 may store image data, process data, manager data, image data records, etc.

In various embodiments, memory 220 may include volatile or non-volatile recording media capable of storing various data, instructions, and information. For example, the memory 220 may be a flash memory type, hard disk type, multimedia card micro type, card type memory (e.g. SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage storage. , cloud, or blockchain database may include at least one type of storage medium.

In various embodiments, memory 220 may store configuration of at least one of operating system 221, communication module 222, user interface module 223, and one or more applications 224.

Operating system 221 (e.g., embedded operating system such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) is a variety of software to control and manage common system tasks (e.g., memory management, storage device control, power management, etc.) It may contain components and drivers and may support communication between various hardware, firmware, and software components.

The communication module 223 may support communication with other devices through the communication interface 210. The communication module 220 may include various software components for processing data received by the wired communication port 211 or the wireless circuit 212 of the communication interface 210.

The user interface module 223 may receive a user's request or input from a keyboard, touch screen, microphone, etc. through the I/O interface 230 and provide a user interface on the display.

Applications 224 may include programs or modules configured to be executed by one or more processors 240 .

The I/O interface 230 may connect at least one of input/output devices (not shown) of the object recognition server 200 at the construction site, such as a display, keyboard, touch screen, and microphone, with the user interface module 223. The I/O interface 230 may receive user input (eg, voice input, keyboard input, touch input, etc.) together with the user interface module 223 and process commands according to the received input.

The processor 240 is connected to the communication interface 210, memory 220, and I/O interface 230 to control the overall operation of the object recognition server 200 at the construction site, and can control the overall operation of the object recognition server 200 at the construction site. Various commands can be executed to provide a system that provides screen-based help through an application or program.

The processor 240 may correspond to a computing device such as a Central Processing Unit (CPU) or an Application Processor (AP). Additionally, the processor 240 may be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) in which various computing devices are integrated. Alternatively, the processor 240 may include a module for calculating an artificial neural network model, such as a Neural Processing Unit (NPU).

In various embodiments, processor 240 may receive image data from a manager device. At this time, the processor 240 may transmit an image data upload interface to the manager device 100. Here, the image data may include CCTV data for the construction site. Additionally, the processor 240 may perform labeling on regions of image data. As another example, the processor 240 may receive labeled image data.

The processor 240 may receive process data for the first process selected by the user. At this time, the first process may include rebar assembly, painting work, electrical equipment work, mechanical equipment work, material lifting work, and retaining wall form assembly. Here, the process data may be data on the number of accidents in the first process, the number of times a safety helmet is not worn, the number of times a safety vest is not worn, etc.

The processor 240 may determine object data for the first process using an object recognition model learned to output object data by inputting image data. At this time, the object recognition model may have a MASK R-CNN structure. Here, the object data may be at least one mask (MASK) including a worker, manager, hard hat, safety vest, excavator, or crane identified from the image data. Meanwhile, the processor 240 can determine whether workers or managers should wear safety helmets and safety vests based on object data. At this time, the processor 240 may provide an alarm to the manager device 100 when it is determined that the worker or manager is not wearing a safety helmet or safety vest.

The processor 240 may determine an object recognition exception zone for image data based on process data and object data. At this time, the processor 240 may count the number of malfunctions of the object recognition model for the area of the image data. Here, the processor 240 may determine the corresponding area of the image data as the object recognition exception area when the number of malfunctions is greater than the threshold.

Additionally, processor 240 may determine important areas for image data. At this time, the processor 240 may determine an area in which the number of times workers and managers do not wear safety helmets and safety vests is more than a threshold value as an important area.

In the above, it has been described that outputting the object of the image data, determining the alarm for the object and the object recognition exception zone, etc. are performed through the object recognition server 200 at the construction site, but is not limited to this, and is not limited to this, and is not limited to the manager device (100) ), all the steps described above may be performed.

Figure 4 is a flow chart of an object recognition method at a construction site according to an embodiment of the present invention, and Figures 5 to 9 are illustrative diagrams of an object recognition method at a construction site according to various embodiments of the present invention.

First, referring to FIG. 4, the object recognition server 200 at the construction site receives image data from the manager device (S410). At this time, the manager device's object recognition server 200 at the construction site may transmit an image data upload interface to the manager device 100. Here, the image data may include CCTV data for the construction site. Additionally, the object recognition server 200 at the construction site may perform labeling of areas of image data. As another example, the object recognition server 200 at a construction site may receive labeled image data.

More specifically, referring to FIG. 5 , the object recognition server 200 at a construction site may perform labeling of a region of image data. At this time, the object recognition server 200 at the construction site may perform labeling based on the components of the construction site corresponding to the image data. For example, referring to (a) of FIG. 5, the object recognition server 200 at a construction site may receive image data about the construction site. At this time, referring to (b) of FIG. 5, the object recognition server 200 at the construction site may label the area 510 where workers can walk based on whether or not the worker can walk. Here, the object recognition server 200 at the construction site may perform labeling based on components among areas where workers cannot walk. For example, the object recognition server 200 at a construction site may label image data about the construction site as Walkable_area (510), Soil_slope (520), rock_mound (530), rock_slope (540), etc.

Returning to FIG. 4, the object recognition server 200 at the construction site receives process data for the first process selected by the user (S420). At this time, the first process may include rebar assembly, painting work, electrical equipment work, mechanical equipment work, material lifting work, and retaining wall form assembly. Here, the first process may be a process at a construction site for the received image data. Additionally, the process data may be data on the number of accidents in the first process, the number of times a safety helmet is not worn, the number of times a safety vest is not worn, etc.

More specifically, referring to FIG. 6 , the object recognition server 200 at a construction site may receive process data for the first process. At this time, the object recognition server 200 at the construction site may receive process data for one or more of the following tasks: rebar assembly, painting work, electrical equipment work, mechanical equipment work, material lifting work, and retaining wall form assembly. Here, the first process may be a process at a construction site for image data received from the manager device 100.

Returning to FIG. 4, the object recognition server 200 at the construction site determines object data for the first process using an object recognition model learned to output object data by inputting image data (S430). At this time, the object recognition model may have a MASK R-CNN structure. Here, the object data may be at least one mask (MASK) including a worker, manager, hard hat, safety vest, excavator, or crane identified from the image data. Meanwhile, the object recognition server 200 at the construction site can determine whether workers or managers should wear safety helmets and safety vests based on object data. At this time, the object recognition server 200 at the construction site may provide an alarm to the manager device 100 when it is determined that the worker or manager is not wearing a hard hat or safety vest.

More specifically, referring to FIG. 7A, the object recognition model may have a MASK R-CNN structure. At this time, the object recognition server 200 at the construction site may set the input data of the object recognition model to image data. Here, image data set as input data may be image data for the first process. At this time, the object recognition server 200 at the construction site can resize the size of the input image data through interpolation. For example, the object recognition server 200 at a construction site can resize the input image data to a size of 1024 * 1024 through bilinear interpolation. Meanwhile, the object recognition server 200 at the construction site can adjust the resized image data to an input size of 1024 * 1024. At this time, the object recognition server 200 at the construction site can adjust the size of the resized image data to input it to the backbone network. Here, the object recognition server 200 at the construction site can generate feature maps (C1, C2, C3, C4, C5) of each layer (stage) through ResNet. At this time, the object recognition server 200 at the construction site can generate P2, P3, P4, P5, and P6 feature maps based on the feature map through FPN (Feature Pyramid Network). Additionally, the object recognition server 200 at the construction site can generate classification and bbox regression output on the feature map through RPN (Region Proposal Network). At this time, the object recognition server 200 at the construction site may generate an anchor box based on bbox regression through projection. Here, the object recognition server 200 at the construction site can adjust the size of the anchor box through Region of Interest (RoI) align. At this time, RoI may be a region of interest within a video or image. Meanwhile, the object recognition model may include a classification branch, bounding box regression branch, mask branch 710, etc. At this time, the object recognition server 200 at the construction site can predict the class label through the classification branch. Here, the object recognition server 200 at the construction site can predict the bounding box offset through the bounding box regression branch. Additionally, the object recognition server 200 at the construction site can predict the segmentation mask for each RoI through the mask branch 710. At this time, the object recognition server 200 at the construction site may input an image of a fixed size into the mask branch 710 through RoI align. Here, the object recognition server 200 at the construction site can output a binary mask for each class for each RoI through the mask branch 710. Additionally, the object recognition server 200 at the construction site can output a feature map of size K^2m through the mask branch 710. At this time, m may be the size of the feature map, and K may be the number of classes. Here, the object recognition server 200 at the construction site can recognize each object in the image data through an object recognition model.

Meanwhile, the object recognition server 200 at the construction site can apply multi-task loss to each RoI. At this time, the loss function can be determined by the following formula.

At this time, is Classification loss, is bounding box loss, is binary cross entropy loss. At this time, Can be determined through the sigmoid function for each cell of the K^2m-sized feature map output from the mask branch.

For example, referring to FIG. 7B, the object recognition server 200 at a construction site may output a binary mask through a mask branch. At this time, the object recognition server 200 at the construction site may output binary mask 1 (711), binary mask 2 (712), binary mask 3 (713), etc. Here, the object recognition server 200 at the construction site can identify worker 1 through binary mask1 (711), worker 2 through binary mask2 (712), and worker 3 through binary mask3 (713). It can be identified through . Because of this, the object recognition server 200 at the construction site can treat objects of the same class as different labels.

Referring to FIG. 8, the object recognition server 200 at a construction site may output object data through an object recognition model. At this time, the object data may be at least one mask (MASK) including a worker, a manager, a safety helmet 810, a safety vest 820, an excavator, or a crane. Here, the object recognition server 200 at the construction site can determine whether workers and managers are wearing safety helmets 810 and safety vests 820. At this time, the object recognition server 200 at the construction site may generate an alarm based on whether workers and managers are wearing hard hats and safety vests. Here, the object recognition server 200 at the construction site may provide an alarm to the manager device 100.

Returning to FIG. 4, the object recognition server 200 at the construction site determines an object recognition exception zone for image data based on process data and object data (S440). At this time, the object recognition server 200 at the construction site may count the number of malfunctions of the object recognition model for the area of the image data. Here, the object recognition server 200 at the construction site may determine the corresponding area of the image data as the object recognition exception area when the number of malfunctions is greater than the threshold.

Additionally, the object recognition server 200 at the construction site can determine important areas for image data. At this time, the object recognition server 200 at the construction site may determine an area in which the number of times workers and managers do not wear safety helmets and safety vests is greater than a threshold value as an important area. Here, the number of times workers and managers do not wear safety helmets and safety vests can be received from process data.

In more detail, referring to FIG. 9, the object recognition server 200 at a construction site may provide an object recognition interface. At this time, the object recognition server 200 at the construction site may receive object data through an object recognition model. Here, the object recognition server 200 at the construction site may receive a worker 910 not wearing a safety helmet and a worker 920 wearing a safety helmet. Additionally, the object recognition server 200 at the construction site can set an area with many workers not wearing safety helmets as an important area. As another example, the object recognition server 200 at a construction site may set an area with many records of workers and managers not wearing safety helmets and safety vests received from process data as an important area. As another example, the object recognition server 200 at a construction site may set an area where accidents occur frequently as an important area.

The object recognition server 200 at the construction site may set an object recognition exception zone 930 based on the number of malfunctions. At this time, the object recognition server 200 at the construction site may set the area where the number of malfunctions is greater than the threshold as the object recognition exception area 930. For example, the object recognition server 200 at a construction site may generate an alarm in an area where safety helmets are not required to be worn. At this time, the object recognition server 200 at the construction site may set an area where safety helmets are not required to be worn as the object recognition exception zone 930 based on the number of alarm malfunctions.

Although embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and may be modified and implemented in various ways without departing from the technical spirit of the present invention. there is. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

100: Administrator device
110: memory interface 120: processor
130: peripheral interface 140: I/O subsystem
141: Touch screen controller 142: Other input controller
143: touch screen
144: Other input control devices
150: Memory 151: Operating system
152: Communication module 153: GUI module
154: sensor processing module 155: phone module
156: Applications
156-1, 156-2: Application
157: Digital assistant client module
158: User data
160: motion sensor 161: light sensor
162: Proximity sensor 163: Other sensors
170: Camera subsystem 171: Optical sensor
180: Communication subsystem
190: Audio subsystem
191: Speaker 192: Microphone
200: Object recognition server at construction site
210: communication interface
211: wired communication port 212: wireless circuit
220: memory 221: operating system
222: communication module 223: user interface module
224: Application 230: I/O interface
240: processor

Claims (18)

An object recognition method implemented by a processor, comprising:
Receiving image data and process data for the first process selected by the user from the manager device;
performing labeling on at least one region of the image data;
determining object data for the first process using an object recognition model learned to output object data using the image data as input;
Determining whether to wear safety helmets and safety vests for workers and managers based on the object data;
determining at least one area of the image data as an important area when the number of times the workers and managers do not wear the safety helmet and safety vest is greater than or equal to a first threshold;
using the process data for the first process, counting the number of false detections of the object for at least one of the regions of the image data; and
When the number of false detections is greater than or equal to a second threshold, determining at least one zone of the image data as an object recognition exception zone; comprising,
Object recognition method at construction site. delete delete delete According to paragraph 1,
The first process is,
An object recognition method at a construction site, including at least one of rebar assembly, painting work, electrical equipment work, mechanical equipment work, material lifting work, and retaining wall formwork. According to clause 5,
The object data is,
At least one MASK including the worker, the manager, the hard hat, the safety vest, the excavator or the crane identified from the image data,
Object recognition method at construction site. delete According to clause 6,
If the worker or the manager does not wear the safety helmet and the safety vest, providing an alarm to the manager device; further comprising,
Object recognition method at construction site. According to clause 8,
The object recognition model has the structure of MASK R-CNN,
Object recognition method at construction site. Ministry of Communications;
storage unit; and
Comprising a processor operably connected to the communication unit and the storage unit,
The processor,
Receive image data from the manager device and process data for the first process selected by the user - the process data for the first process includes the number of times workers and managers do not wear safety helmets and safety vests -
Perform labeling on at least one region of the image data,
Determining object data for the first process using an object recognition model learned to output object data using the image data as input,
Determining whether to wear the safety helmet or safety vest for the worker or the manager from the object data,
If the number of times the worker and the manager have not worn the safety helmet and the safety vest is greater than or equal to a first threshold, determining at least one area of the image data as an important area;
Using the process data for the first process, count the number of false detections of the object for at least one region of the image data,
configured to determine at least one area of the image data as an object recognition exception area when the number of false detections is greater than or equal to a second threshold,
Object recognition device at construction site. delete delete delete According to clause 10,
The first process is,
An object recognition device at a construction site, including at least one of rebar assembly, painting work, electrical equipment work, mechanical equipment work, material lifting work, and retaining wall formwork. According to clause 14,
The object data is,
At least one MASK including the worker, the manager, the hard hat, the safety vest, the excavator or the crane identified from the image data,
Object recognition device at construction site. delete According to clause 15,
Further configured to provide an alarm to the manager device when the worker or the manager does not wear the hard hat and the safety vest,
Object recognition device at construction site. According to clause 17,
The object recognition model has the structure of MASK R-CNN,
Object recognition device at construction site.