KR102153486B1 - Multi-AVM Camera System and method for obtaining Construction Machine and Operator Position - Google Patents

Abstract

Provided are a multi-AVM camera system for measuring locations of construction machines and workers and a method for measuring locations of construction machines and workers thereof. The multi-AVM camera system for measuring locations of construction machines and workers comprises: a plurality of fixed AVM devices installed on different fixed locations of a construction site, recognizing an object by analyzing fixed photographed images obtained by photographing the construction site, estimating location information of the recognized object, and transmitting attribute information (hereinafter, referred to as fixed attribute information) including the estimated location information to a central server; at least one mobile AVM device installed in a construction machine to be able to move, recognizing an object by analyzing a mobile photographed image obtained by photographing the construction site, estimating location information of the recognized object, and transmitting attribute information (hereinafter, referred to as mobile attribute information) including the estimated location information to the central server; and a central server for correcting the location information of the object based on multiple views by analyzing a plurality of fixed attribute information and at least one mobile type attribute information, which are received.

Description

Multi-AVM Camera System and method for obtaining Construction Machine and Operator Position, and a multi-AVM Camera System and Method for Measuring Construction Machine and Worker Position

The present invention relates to a multi-AVM camera system for measuring the position of a construction machine and a worker, and a method for measuring the location of the construction machine and the worker, and more particularly, by using fixed and mobile AVM cameras, It relates to a construction machine capable of measuring and correcting the position and a multi-AVM camera system for measuring the position of a worker, and a method of measuring the position of the construction machine and the worker thereof.

As a safety accident prevention technology, safety technology for construction machinery is constantly being researched, and sensor technology is a key field among them.

When looking at the technology applied in the field of construction machinery, a technology using a laser sensor is applied to prevent safety accidents while performing work using an excavator, and the sensor is used to determine whether obstacles exist around the excavator and control to prevent accidents. Perform.

In addition, there is a technology that generates an alarm for an object close to a construction machine using an ultrasonic sensor.

However, such existing technology is one-dimensional sensors that only play a role of determining whether an object exists or not, but does not provide accurate information about the object.

Therefore, when using an existing technology, it is difficult to distinguish between a worker and a construction machine, and even a case of misrecognizing a work object may decrease the efficiency of work.

In order to compensate for these shortcomings, there is a trend of increasing demand to use a high-dimensional sensor capable of recognizing surrounding objects, and representatively, there is an image sensor.

For example, based on the AVM (Around View Monitor) camera technology, a camera and an ultrasonic sensor are combined to complement the object recognition function, which is a disadvantage of the ultrasonic sensor, with a camera, and four AVM cameras are installed on the excavator to determine the proximity of the worker. The technology was studied. However, this technology also does not present an algorithm for determining whether an operator exists in the vicinity through an image, and an excavator driver must visually determine whether an operator exists through a monitor.

Domestic Patent Registration No. 10-1916467

In order to solve the above-described problems, the present invention provides information on the distance from the construction machine to the target object, as well as information on the types of objects existing around the construction machine and location information in the image through the camera image. It is to present a multi-AVM camera system for measuring the position of construction machinery and workers that can also grasp direction information, and a method of measuring the position of construction machinery and workers thereof.

The problem of the present invention is not limited to those mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above-described technical problem, according to an embodiment of the present invention, a multiple AVM camera system for measuring a construction machine and a worker's position is installed at different fixed positions on a construction site, and photographing a construction site. Recognizing the object by analyzing the fixed-type photographed images, and transmitting attribute information (hereinafter referred to as'fixed-type attribute information') including the estimated position information by estimating the position information of the recognized object to the following central server. Fixed AVM devices; Attribute information that is installed to be movable on a construction machine, recognizes an object by analyzing a movable photographed image of a construction site, and estimates the position information of the recognized object (hereinafter referred to as ``movable attribute At least one mobile AVM device that transmits'information') to the following central server; And a central server that analyzes the received plurality of fixed type attribute information and at least one mobile type attribute information and corrects the location information of the object based on a multi-view.

The fixed attribute information includes location information and type of an object recognized in the fixed photographed image, a fixed photographed image including the object, a photographing time, and location information on which a fixed AVM device is installed, and the movable attribute information includes the It includes the location information and type of the object recognized in the mobile photographing image, the mobile photographing image including the object, the photographing time, and the location information of the construction machine.

The central server includes: a server communication unit configured to receive a plurality of fixed type attribute information and at least one mobile type attribute information from the plurality of fixed AVM devices and at least one mobile AVM device; An AVM calibration unit for acquiring a relationship parameter between the fixed AVM devices by performing calibration and calibration of the plurality of fixed AVM devices; A first position correction unit correcting position information of an object recognized by the plurality of fixed AVM devices by using the obtained relationship parameter and the plurality of fixed attribute information; And a second position correction unit for finally correcting the position information of the object by comparing the position information of the object corrected by the first position correction unit with the position information of the object estimated by the at least one mobile AVM device. do.

The first position correction unit may determine an error range of a camera measurement estimated to be located at the object from position information and captured images among the plurality of fixed attribute information for each of the plurality of fixed AVM devices. Judgment unit; And an overlap checker configured to determine an overlapping area of the plurality of camera measurement error ranges determined by the error range determination unit as corrected position information of the object.

The second position correcting unit checks the overlapping area of the overlapping area checked by the overlapping check unit and the positional information of the object included in the movable property information, and corrects it as the final positional information of the object.

Each of the plurality of fixed AVM devices includes an AVM camera unit including a plurality of fixed AVM cameras each outputting a fixed image; An operation for recognizing temporary objects by analyzing the plurality of stationary photographed images, matching the recognized temporary objects to determine the object, measuring location information, and generating fixed property information including location information of the measured object Control unit; And a communication unit for transmitting the generated fixed attribute information to a central server.

The at least one mobile AVM device may include a GPS unit for receiving location information of the construction machine; An AVM camera unit including a plurality of mobile AVM cameras each outputting a mobile photographed image; An operation for recognizing temporary objects by analyzing the plurality of mobile photographed images, matching the recognized temporary objects to determine the object, measuring position information, and generating mobile attribute information including position information of the measured object Control unit; And a communication unit for transmitting the generated mobile attribute information to a central server.

The calculation control unit converts the measured location information of the object into world coordinates in association with the location information of the construction machine received from the GPS unit, and determines the converted world coordinates as the location information of the object.

On the other hand, according to another embodiment of the present invention, the method for measuring the position of a construction machine and a worker of a multi-AVM camera system includes (A) a plurality of fixed AVM devices are installed at different fixed positions of a construction site to photograph a construction site. Recognizing an object by analyzing the fixed-type photographed images, and transmitting attribute information (hereinafter referred to as'fixed-type attribute information') including the estimated position information by estimating the position information of the recognized object to the following central server ; (B) At least one movable AVM device is installed to be movable on a construction machine and recognizes an object by analyzing a movable photographed image taken of a construction site, and estimates the location information of the recognized object to obtain the estimated location information. Transmitting the included attribute information (hereinafter, referred to as'portable attribute information') to the following central server; And (C) analyzing, by the central server, a plurality of fixed property information and at least one mobile property information received from steps (A) and (B), and correcting the location information of the object based on multiple views. Includes;

The step (C) may include: (C1) the central server performing calibration of the plurality of fixed AVM devices to obtain a relationship parameter between the fixed AVM devices; (C2) the central server correcting, by the central server, location information of objects recognized by the plurality of fixed AVM devices, using the obtained relationship parameter and the plurality of received fixed attribute information; And (C3) comparing, by the central server, the location information of the object corrected in step (C2) with the location information of the object estimated in step (B), and finally correcting the location information of the object. Include.

In the step (C2), the central server determines a camera measurement error range estimated to be located from the location information and captured images among the plurality of fixed property information to each of the plurality of fixed AVM devices. Is determined, and the overlapped region of the determined measurement error ranges of the plurality of cameras is checked and determined as corrected position information of the object.

In the step (C3), the central server checks the overlapping area of the overlapping area checked in the step (C2) and the location information of the object included in the movable property information, and corrects it as the final location information of the object.

In the step (A), each of the plurality of fixed AVM devices, (A1) recognizes temporary objects by analyzing fixed image images captured by a plurality of fixed AVM cameras, matches the recognized temporary objects to determine the object. Measuring the location information, and generating fixed property information including location information of the measured object; And (A2) transmitting the generated fixed type attribute information to a central server.

The (B) step, the at least one mobile AVM device, (B1) receiving the location information of the construction machine; (B2) Recognize temporary objects by analyzing mobile photographed images captured by a plurality of mobile AVM cameras, match the recognized temporary objects to determine the object, measure location information, and include location information of the measured object Generating mobile attribute information to be performed; And (B3) transmitting the generated mobile attribute information to a central server.

According to the present invention, by collecting the location information of the object (worker or other equipment) and the construction machine acquired from a construction machine-mounted AVM device or a construction site fixed AVM device, and by estimating the exact location information of the object from the collected location information Dangerous situations that may occur at construction sites can be prevented in advance.

In addition, according to the present invention, it is possible to prevent a safety accident that occurred due to the absence of an information sharing channel at a construction site, and contribute to a reduction in the accident rate.

The effects of the present invention are not limited to those mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic diagram of a multiple AVM (Around View Monitor) camera system for measuring a construction machine and a worker's position according to an embodiment of the present invention;
FIG. 2 is a block diagram showing a multiple AVM camera system for measuring the location of the construction machine and worker shown in FIG. 1;
3 is a block diagram showing a first position correction unit according to an embodiment of the present invention;
4 is an exemplary diagram for explaining position information correction;
5 is a flow chart illustrating a method for measuring a construction machine and a worker's position of a multi-AVM camera system according to an embodiment of the present invention;
6 is a flow chart showing in detail step S500, and,
7 is a detailed flowchart illustrating step S540.

The above objects, other objects, features, and advantages of the present invention will be easily understood through the following preferred embodiments related to the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content may be thorough and complete, and the spirit of the present invention may be sufficiently conveyed to those skilled in the art.

In the present specification, when terms such as first and second are used to describe components, these components should not be limited by these terms. These terms are only used to distinguish one component from another component. The embodiments described and illustrated herein also include complementary embodiments thereof.

In addition, if an element, component, device, or system is stated to include a program or a component made of software, the element, component, device, or system is executed by the program or software, even if there is no explicit mention. Or, it should be understood to include hardware (eg, memory, CPU, etc.) or other programs or software (eg, a driver required to drive an operating system or hardware) required to operate.

In addition, it should be understood that an element (or component) may be implemented in software, hardware, or any form of software and hardware, unless otherwise specified in the implementation of any element (or component).

In addition, terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements.

Hereinafter, the present invention will be described in detail with reference to the drawings. In describing the specific embodiments below, a number of specific contents have been prepared to explain the invention in more detail and to aid understanding. However, readers who have knowledge in this field to the extent that they can understand the present invention can recognize that it can be used without these various specific contents.

In some cases, it is mentioned in advance that parts that are commonly known in describing the invention and are not largely related to the invention are not described in order to avoid confusion in the description of the invention without any reason.

Each configuration of the multi-AVM camera system shown in FIGS. 2 and 3 indicates that each configuration may be separated functionally and logically, and it means that each configuration is divided into a separate physical device or written in a separate code. It may be easily inferred by an average expert in the technical field of the present invention.

1 is a schematic diagram of a multiple AVM (Around View Monitor) camera system for measuring a construction machine and a worker's position according to an embodiment of the present invention.

Referring to Figure 1, an embodiment of the present invention is to measure and correct the position of a worker or other construction machine using a mobile AVM camera device mounted on a construction machine and a fixed AVM camera device installed in the construction site. It is possible to perform a calibration operation between cameras and a multi-view-based position estimation and correction operation. Accordingly, by measuring and sharing the location of workers or other construction machinery on the construction site, it is possible to increase the prevention of safety accidents and to determine the danger situation in advance.

FIG. 2 is a block diagram illustrating a multi-AVM camera system for measuring the location of a construction machine and a worker shown in FIG. 1.

Referring to FIG. 2, a multi-AVM camera system for measuring a construction machine and a worker's position according to an embodiment of the present invention includes a first fixed AVM device 100, a second fixed AVM device 200, and a mobile AVM device 300. And a central server 400. In FIG. 2, two fixed AVM devices 100 and 200 and one mobile AVM device 300 are shown, but the number is not limited thereto, and two or more units may be provided.

The first fixed AVM device 100 and the second fixed AVM device 200 are installed at different fixed positions on a construction site.

The first fixed AVM device 100 is installed at a first location of the construction site to photograph the construction site, and analyzes the captured image (hereinafter referred to as'first fixed type photographing image') to determine the object in the first fixed type photographed image. Recognizes and estimates the location information of the recognized object and transmits property information (hereinafter referred to as “first fixed property information”) including the estimated location information to the central server 400.

Accordingly, the first fixed AVM device 100 may include a first AVM camera unit 110, a first storage unit 120, a first operation control unit 130, and a first communication unit 140.

The first AVM camera unit 110 includes a plurality of fixed AVM cameras each outputting a fixed image. In an embodiment of the present invention, the first AVM camera unit 110 includes a first AVM camera 112 and a second AVM camera 114.

Before the first fixed AVM device 100 photographs the construction site, the first operation control unit 230 to be described later calibrates the camera in advance for the relationship parameter between the first AVM camera 112 and the second AVM camera 114 (Camera Calibration) can be performed.

The world that people see with their own eyes is three-dimensional, but when photographed with a camera, it turns into a two-dimensional image. At this time, where the three-dimensional dots are formed on the image is determined geometrically, it is determined by the position and direction of the camera at the time the image is taken. However, the actual image is greatly affected by the mechanical parts inside the camera, such as the lens used, the distance between the lens and the image sensor, and the angle between the lens and the image sensor. Therefore, when obtaining a position where the 3D points are projected onto an image or, conversely, restoring a 3D spatial coordinate from an image coordinate, accurate calculation is possible only by removing these internal factors. The process of obtaining the parameter values of these internal factors is called camera calibration.

Relationship parameters obtained by camera calibration include an Intrinsic parameter and an Extrinsic parameter. The internal parameters include focal length, image center point, and asymmetry coefficient information. The external parameter includes information on the coordinate system of each camera in the world coordinate system. At this time, when all camera coordinate systems are expressed in one world coordinate system, calibration is complete. There are two methods of performing camera calibration: a method using a check board and a method using natural terrain (Field Calibration).

The first and second AVM cameras 112 and 114 capture a construction site (for example, the same location) designated to be photographed by the first fixed AVM device 100 in real time, and respectively, the first fixed type photographed image and the second fixed type. You can output the captured image.

In order to implement and/or provide an operation, function, etc. provided by the first fixed AVM device 100, the first storage unit 120 includes instructions or data related to the components 110 to 140, one or more programs, and / Or you can store software, etc.

In addition, the first storage unit 120 may store the first fixed-type photographed image and the second fixed-type photographed image captured by the first AVM camera 112 and the second AVM camera 114 together with a photographing time.

Also, the first storage unit 120 may store location information of the first fixed AVM device 100 input by an administrator. The location information of the first fixed AVM device 100 may be based on the world coordinate system.

As a device or module capable of performing artificial intelligence, the first operation control unit 130 may perform an algorithm for recognizing and matching a plurality of types of construction machines or people set in advance by analyzing fixed-type photographed images in real time. . The first operation control unit 130 analyzes a plurality of fixed-type photographed images to recognize temporary objects, matches the recognized temporary objects to determine the object, and measures location information, and a fixed type including location information of the measured object. You can create attribute information.

In detail, the first operation control unit 130 analyzes the first fixed-type photographed image, extracts an object of a preset type from the first fixed-type photographed image, determines the type of the object, and recognizes it as a first temporary object. Similarly, by analyzing the second fixed-type photographed image, the second temporary object may be recognized from the second fixed-type photographed image. The first and second temporary objects may be construction machinery or construction personnel at a construction site.

The first operation control unit 130 includes a first condition in which the time points at which the first and second temporary objects are photographed are the same or within an error range, and the similarity between the shape of the first temporary object and the second temporary object exceeds the standard. If the second condition and at least one of the third conditions in which the difference between the absolute position of the first temporary object in the first fixed type image and the absolute position of the second temporary object in the second fixed type image is within the error range are met, The first temporary object and the second temporary object may be determined as the same object.

The first operation control unit 130 compares the recognized first temporary object with the second temporary object and determines that it is the same object, matches the first temporary object with the second temporary object, and determines the object as an object, using a triangulation technique. It is possible to estimate the physical distance and direction from the first fixed AVM device 100 to the object.

The first operation control unit 130 may measure the location information of the object using the location information of the first fixed AVM device 100 stored in the first storage unit 120 and the estimated distance and direction to the object. Alternatively, the first operation control unit 130 may determine the estimated distance and direction to the object as the location information of the object.

The first operation control unit 130 captures the location information and type of the object recognized and matched in the first and second fixed-type photographed images, and the first and second fixed-type photographed images including the first and second temporary objects. First fixed type attribute information including time and location information on which the first fixed type AVM device 100 is installed may be generated.

In addition, the first operation control unit 130 may extract feature points from the first fixed-type photographed image and the second fixed-type photographed image. In this case, the first operation control unit 230 may generate first fixed type attribute information further including feature points.

The first communication unit 140 receives the location information of the first fixed AVM device 100 and the location information of the nearby workers from the devices of the nearby workers (not shown, for example, portable terminals) and transmits them to the central server 400. I can. In addition, the first communication unit 140 transmits the first fixed type attribute information (eg, information related to nearby workers) generated by the first operation control unit 130 to the central server 400 through a wired or wireless network.

The second fixed AVM device 200 is installed at a second location of the construction site to photograph the construction site, analyzes the captured image (hereinafter, referred to as'second fixed type photographing image'), and analyzes the object in the first fixed type photographed image. Recognizes and estimates the location information of the recognized object and transmits property information (hereinafter referred to as “second fixed property information”) including the estimated location information to the central server 400. The first fixed AVM device 100 and the second fixed AVM device 200 may be installed to photograph the same point in a construction site.

To this end, the second fixed AVM device 200 may include a second AVM camera unit 210, a second storage unit 220, a second operation control unit 230, and a second communication unit 240. The operations of the second AVM camera unit 210, the second storage unit 220, the second operation control unit 230, and the second communication unit 240 of the second fixed AVM device 200 are performed by the first fixed AVM device 100. Since the first AVM camera unit 110, the first storage unit 120, the first operation control unit 130, and the first communication unit 140 is almost the same, a detailed description will be omitted.

However, the second AVM camera unit 210 includes a third AVM camera 212 that outputs a third fixed-type photographed image and a fourth AVM camera 214 that outputs a fourth fixed-type photographed image.

The second operation control unit 230 analyzes the third and fourth fixed-type photographed images to recognize the third and fourth temporary objects, matches the recognized third and fourth temporary objects, and determines the object as an object. Measure

In addition, the second operation control unit 230 includes the location information and type of the object recognized and matched in the third and fourth fixed type photographed images, and third and fourth fixed type photographed images including the third and fourth temporary objects. It is possible to generate second fixed type attribute information including information on a photographing time and location of the second fixed AVM device 100. Optionally, the second operation control unit 230 may extract feature points from the third and fourth fixed type captured images, and generate second fixed type attribute information further including the feature points.

The mobile AVM device 300 is installed to be movable on a construction machine such as an excavator, and recognizes an object by analyzing an image photographed of a construction site (hereinafter referred to as a'portable photographing image'), and location information of the recognized object. Attribute information including the estimated location information (hereinafter, referred to as'portable attribute information') may be transmitted to the central server 400 by estimating.

To this end, the mobile AVM device 300 includes a third AVM camera unit 310, a third storage unit 320, a location information receiving unit 330, a third operation control unit 340, and a third communication unit 350. . The operation of the third AVM camera unit 310, the third storage unit 320, the third operation control unit 340 and the third communication unit 350 is performed by the first AVM camera unit 110 of the first fixed AVM device 100, Since the first storage unit 120, the first operation control unit 130, and the first communication unit 140 are almost the same, a detailed description will be omitted.

The third AVM camera unit 310 includes a fifth AVM camera 312 and a sixth AVM camera 314 respectively outputting a moving image.

The location information receiving unit 330 is a device that receives the location of a construction machine equipped with the mobile AVM device 300 and may be a GPS module. As construction machinery continuously moves during work or when moving, the location of construction machinery also changes. Accordingly, the mobile AVM device 300 may further include a location information receiving unit 330 in order to represent the location of the mobile AVM device 300 in a world coordinate system.

The third operation control unit 340 analyzes the first and second movable photographed images to recognize the fifth and sixth temporary objects, matches the recognized fifth and sixth temporary objects, and determines the object as an object. Is measured, and mobile property information including the location information of the measured object may be generated.

The movable attribute information is the location information and type of the object recognized and matched in the first and second movable photographed images, the first and second movable photographed images and the photographing time, a mobile AVM device received from the location information receiving unit 330 (300) or includes location information of construction machinery.

The third operation control unit 340 may match the fifth temporary object and the sixth temporary object to determine an object, and estimate the physical distance and direction from the mobile AVM device 300 to the object using a triangulation technique. . In addition, the third operation control unit 340 converts the location information of the object into a world coordinate system at the construction site by linking the location information of the construction machine received from the location information receiving unit 330 with the distance and direction to the estimated object, The coordinates of the transformed world coordinate system can be used as the location information of the object. Accordingly, the shape of the location information of the object measured by the first and second fixed AVM devices 100 and 200 and the location information of the object measured by the mobile AVM device 300 may be matched with the world coordinate system.

The third communication unit 350 transmits the mobile attribute information generated by the third operation control unit 340 to the central server 400.

The central server 400 analyzes first and second fixed type attribute information received from the first and second fixed type AVM devices 100 and 200 and at least one mobile type attribute information received from the mobile AVM device 300 Thus, the location information of the object can be corrected based on multiple points.

To this end, the central server 400 includes a server communication unit 410, a server storage unit 420, an AVM calibration unit 430, a first position correction unit 440, a second position correction unit 450, and a sharing unit ( 460).

The server communication unit 410 communicates with the first fixed AVM device 100, the second fixed AVM device 200, and the mobile AVM device 300 to provide first and second fixed attribute information and at least one mobile attribute information. Can receive.

In addition, the server communication unit 410 may transmit a control command generated from the central server 400 to the first fixed AVM device 100, the second fixed AVM device 200, or the mobile AVM device 300.

The server storage unit 420 is a command or data related to the components 410 to 460, one or more programs and/or software, in order to implement and/or provide operations, functions, etc. provided by the central server 400, The operating system can be saved. The program stored in the server storage unit 420 may include a calibration program for AVM calibration and a location calibration program for correcting multi-view-based location information of an object.

In addition, the server storage unit 420 may store the received first and second fixed attribute information and at least one mobile attribute information.

The AVM calibration unit 430 performs calibration of a plurality of fixed AVM devices, for example, the first and second fixed AVM devices 100 and 200 to perform the calibration of the first and second fixed AVM devices 100 and 200. ) Can be obtained. The AVM calibration unit 430 may perform calibration between at least one of the first and second AVM cameras 112 and 114 and at least one of the third and fourth AVM cameras 212 and 214.

The AVM revising unit 430 uses the Scale Invariant Feature Transform (SIFT)/Speeded Up Robust Features (SURF) algorithm or the Harris Corner algorithm from the captured images included in the first and second fixed attribute information. A feature (or nature feature) is extracted, and the extracted feature is applied to a field calibration method to perform calibration, and a relationship parameter between the first and second fixed AVM devices 100 and 200 may be obtained from the calibration result. . That is, the AVM calibration unit 430 may obtain a relationship parameter between at least two fixed AVM cameras installed at the construction site from the camera calibration.

Features are points that appear mainly near the boundary line due to shadows in the image obtained from the camera, and may include points that have an unusual shape. In general, a feature acquired from natural terrain objects shown in a photographed image is called a nature feature.

In an embodiment of the present invention, a case in which the central server 400 extracts a feature from a captured image is described, but the first and second operation control units 130 of the first and second fixed AVM devices 100 and 200 are described. , 230) may extract features from the first and second fixed-type photographed images, respectively, and transmit them to the central server 400. In this case, the AVM verification unit 430 may acquire a relationship parameter using the received feature.

Relationship parameters include Intrinsic and Extrinsic parameters, and are extracted through camera calibration. The relationship parameters are described in detail above.

In an embodiment of the present invention, an operation of estimating a relationship parameter by performing a calibration by the AVM calibration unit 430 will be described as follows.

Two first and second fixed AVM devices 100 and 200 are installed at different locations. One of the first fixed AVM device 100 (for example, the first AVM camera 112) and one of the second fixed AVM device 200 (for example, the third AVM camera 212) are the same construction site The area will be shot from a different angle. This is because both the first AVM camera 112 and the third AVM camera 212 can view 360 degrees.

At this time, the AVM inspection unit 430 is the first AVM camera 112 and the third AVM camera 212 are viewed from different angles of natural terrain (construction site area, that is, the first fixed type photographed image and the third fixed type Feature is extracted from the photographed image). In addition, the AVM calibration unit 430 may obtain a relationship parameter by performing a matching algorithm between features and performing calibration between cameras.

However, the scale information cannot be known through camera calibration. This is because all camera coordinate systems are converted to world coordinate systems, but actual physical scale information is not reflected. Accordingly, the AVM verification unit 430 may further reflect physical 3D scale information through matching between the location information in which the first and second fixed AVM devices 100 and 200 are installed and the relationship parameter.

In order to reflect the physical 3D scale information, and to fit the real world into the world coordinate system, the administrator uses a user interface device (not shown) to install the first and second fixed AVM devices 100 and 200. Physical 3D scale information can be input through matching between location information and relationship parameters. When the actual distance is measured through other measuring equipment and reflected as scale information, the relationship parameter between the fixed AVM devices 100 and 200 is in the physical 3D coordinate system reflecting the real world as it is. For example, if the distance between the first and second fixed AVM devices 100 and 200 is 10 meters, 10 meters is input and reflected as physical 3D scale information.

Accordingly, the AVM verification unit 430 may obtain a relationship parameter including scale information, and the real world coordinate system can be adjusted to the world coordinate system.

When an object is recognized from a captured image, only 2D coordinates of the image coordinate system (pixel information), that is, (x, y) are acquired. If a relationship parameter including scale information is defined as in the present invention, the position of the object in the world coordinate system In other words, it is possible to know the 3D coordinates of (X, Y, Z), and through this, it is possible to accurately and objectively know the location information of an object such as a vehicle or a worker.

The first position correction unit 440 uses the relationship parameter obtained from the AVM verification unit 430 and the first and second fixed type attribute information, and is recognized by the first and second fixed type AVM devices 100 and 200. You can correct the location information of the object.

3 is a block diagram illustrating a first position correction unit 440 according to an embodiment of the present invention, and FIG. 4 is an exemplary diagram for explaining position information correction.

Referring to FIG. 3, the first position correction unit 440 may include an error range determination unit 442 and an overlap check unit 444.

The error range determination unit 442 determines the camera measurement error range estimated to be located from the location information and the captured images among the first and second fixed type property information, and the first and second fixed type AVM devices 100, 200) You can judge for each.

First, if the type of the object included in the first fixed property information and the type of the object included in the second fixed property information are the same, and the difference between the location information is also within the error range, the error range determination unit 442 is the same object. It can be judged as.

In addition, the error range determination unit 442 converts the location information of the object having 2D coordinates into a 3D world coordinate system using the location information of the object and the relationship parameter included in the first fixed property information, and the second fixed property information The location information of the object can be converted into a 3D world coordinate system using the location information of the included object and the relationship parameter.

In this case, the error range determination unit 442 converts the location information of the object from the first fixed property information received in real time into a 3D coordinate system, so that the first camera measurement error range as shown in FIG. 4 may be extracted. In addition, since the error range determination unit 442 converts the location information of the object from the second fixed type attribute information received in real time into a 3D coordinate system, it is possible to extract the measurement error range of the second camera as shown in FIG. 4.

The overlap confirmation unit 444 may determine an overlapping area of the measurement error ranges of the first and second cameras determined by the error range determination unit 442 as corrected position information of the object to be measured.

Alternatively, since the overlap confirmation unit 444 knows the location information of the measurement error ranges of the first and second cameras determined by the error range determination unit 442, the average of the location information is calculated and corrected as the final location information of the object. May be.

The second position correction unit 450 compares the position information of the object corrected by the first position correction unit 440 with the position information of the object estimated by the at least one mobile AVM device 300 to finalize the position information of the object. Can be corrected with

In detail, the second position correction unit 450 may determine that the object for which the position information is corrected by the first position correction unit 440 and the object included in the movable attribute information are the same. In the second position correction unit 450, the type of the object for which the location information is corrected by the error range determination unit 442 and the type of the object included in the mobile attribute information are the same, and the difference between the photographed time point is within the error range, If the difference between the captured world coordinates is within the error range, it can be determined as the same object.

If the object is the same as each other, the second position correction unit 450 checks the overlapping area of the overlapping area checked by the overlapping check unit 444 and the positional information of the object included in the movable property information, and determines the checked overlapping area of the object. It can be corrected as final location information. At this time, the second position correction unit 450 collects the position information of the object from a plurality of mobile attribute information received from the mobile AVM device 300 in real time and converts it into a 3D coordinate system, so the third camera is measured instead of a single point. The error range may be extracted, and the overlapped area may be finally confirmed by comparing the extracted third camera measurement error range and the overlapping area checked by the overlapping confirmation unit 444.

If the first position correction unit 440 determines that the object for which the position information is corrected and the object included in the movable attribute information are not the same, the second position correction unit 450 performs the first position correction unit 440 The position information corrected in can be determined as the final position information of the object.

The sharing unit 460 maps the location information of the object corrected by the first position correction unit 440 or the second position correction unit 450 to a digital construction site map to enable monitoring.

In addition, the sharing unit 460 performs a work risk situation analysis and judgment algorithm based on real-time location information reflected on the construction site map, and transmits the execution result to the construction machine driver or nearby workers to provide an alarm function for preventing safety accidents. Can provide.

In addition, the sharing unit 460 may provide guidance information for safe work of a construction machine in consideration of location information of workers around the construction machine. For example, when the construction machine is a crane, the sharing unit 460 makes it possible for the operator to prohibit access to the lower part of the crane lifted object so that the crane can be safely lifted.

FIG. 5 is a flowchart illustrating a method of measuring the position of a construction machine and a worker in a multi-AVM camera system according to an embodiment of the present invention, FIG. 6 is a flowchart illustrating step S500 in detail, and FIG. 7 is a flowchart illustrating step S540 in detail.

The first and second fixed AVM devices 100 and 200, the mobile AVM device 300, and the central server 400 illustrated in FIG. 5 have been described with reference to FIGS. 1 to 4, and thus detailed descriptions thereof will be omitted.

Referring to FIG. 5, the first and second fixed AVM devices 100 and 200 are installed at different fixed positions on the construction site to analyze fixed images taken of the construction site to recognize and recognize an object. The first and second fixed property information including the estimated location information may be transmitted to the central server 400 by estimating the location information of the object in operation (S500 to S530).

At least one movable AVM device 300 is installed to be movable on a construction machine and recognizes an object by analyzing a movable photographed image taken of a construction site, and includes location information estimated by estimating the location information of the recognized object. It is possible to transmit the mobile attribute information to the central server 400 (S540, S550).

The central server 400 stores first and second fixed attribute information and at least one mobile attribute information received from steps S510, S530, and S550 (S560).

The central server 400 performs calibration between the first and second fixed AVM devices 100 and 200 to obtain a relationship parameter between the first and second fixed AVM devices 100 and 200, and obtains scale information from the administrator. Acquire and reflect (S570).

The central server 400 may correct the location information of the object recognized by the first and second fixed AVM devices 100 and 200 using the obtained relationship parameter and the first and second fixed attribute information ( S580). In step S580, the central server 400 determines the first and second camera measurement error ranges from the location information and captured images among the first and second fixed property information, and measures the determined first and second cameras. The overlapping area of the error ranges can be checked and corrected as the location information of the object.

The central server 400 may finally correct the location information of the object by comparing the location information of the object estimated by the mobile AVM device 300 with the location information of the object corrected in step S580 (S590).

In step S590, the central server 400 transmits the position information of the final corrected object to the monitoring device of the construction machine equipped with the mobile AVM device 300, the driver, or the neighboring worker to share the location information and notify the degree of risk. (S595).

On the other hand, the method for measuring the location of construction machines and workers of the multi-AVM camera system according to the present invention may be provided by being included in a recording medium that can be read through a computer by tangibly implementing a program of instructions for implementing it. Technicians can easily understand.

That is, the construction machine and worker position measurement method of the multi-AVM camera system according to the present invention is implemented in the form of a program that can be performed through various computer means, and can be recorded on a computer-readable recording medium, and the computer-readable recording The medium may include program instructions, data files, data structures, and the like alone or in combination. The computer-readable recording medium includes magnetic media such as a hard disk, optical media such as CD-ROM and DVD, and ROM, RAM, flash memory, and USB memory. Hardware devices specially configured to store and execute program instructions are included.

Accordingly, the present invention provides a program stored in a computer-readable recording medium executed on a computer controlling the multi-AVM camera system in order to implement a method for measuring the position of a construction machine and a worker in a multi-AVM camera system.

On the other hand, although it has been described and illustrated in connection with a preferred embodiment for exemplifying the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, and deviates from the scope of the technical idea. It will be appreciated by those skilled in the art that a number of changes and modifications can be made to the present invention. Accordingly, all such appropriate changes, modifications, and equivalents should be considered to be within the scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical idea of the attached registration claims.

100: first fixed AVM device 110: first AVM camera unit
120: first storage unit 130: first operation control unit
140: first communication unit 200: second fixed AVM device
210: second AVM camera unit 220: second storage unit
230: second operation control unit 240: second communication unit
300: mobile AVM device 310: third AVM camera unit
320: third storage unit 330: location information receiving unit
340: third operation control unit 350: third communication unit
400: central server 410: server communication unit
420: server storage unit 430: AVM inspection department
440: first position correction unit 450: second position correction unit
460: share part

Claims (15)

Attribute information (hereinafter, referred to as hereinafter, which is installed at different fixed positions of a construction site) and includes positional information estimated by analyzing fixed-type photographed images of the construction site to recognize an object, and estimating the position information of the recognized object. A plurality of fixed AVM devices that transmit'fixed attribute information') to the following central server;
Attribute information that is installed to be movable on a construction machine, recognizes an object by analyzing a movable photographed image of a construction site, and estimates the position information of the recognized object (hereinafter referred to as ``movable attribute At least one mobile AVM device that transmits'information') to the following central server; And
A central server for analyzing the received plurality of fixed attribute information and at least one mobile attribute information and correcting the position information of the object based on a multi-view; multiplex for measuring the position of a construction machine and a worker, comprising: AVM camera system. The method of claim 1,
The fixed type attribute information,
Including location information and type of the object recognized in the fixed-type photographed image, a fixed-type photographed image including the object, a photographing time, and location information of a fixed AVM device,
The mobile attribute information,
A multi-AVM camera system for measuring the location of a construction machine and a worker, characterized in that it includes location information and type of the object recognized in the mobile photographed image, a mobile photographed image including the object, a photographing time, and location information of the construction machine. . The method of claim 1,
The central server,
A server communication unit for receiving a plurality of fixed type attribute information and at least one mobile type attribute information from the plurality of fixed AVM devices and at least one mobile AVM device;
An AVM calibration unit for acquiring a relationship parameter between the fixed AVM devices by performing calibration and calibration of the plurality of fixed AVM devices;
A first position correction unit correcting position information of an object recognized by the plurality of fixed AVM devices by using the obtained relationship parameter and the plurality of fixed attribute information; And
Comprising: a second position correction unit for finally correcting the position information of the object by comparing the position information of the object corrected by the first position correction unit and the position information of the object estimated by the at least one mobile AVM device; Multi-AVM camera system for measuring the position of construction machinery and workers, characterized in that. The method of claim 3,
The first position correction unit,
An error range determination unit for determining, for each of the plurality of fixed AVM devices, a camera measurement error range in which the object is estimated to be located from position information and captured images among the plurality of fixed attribute information; And
A multi-AVM for measuring the position of a construction machine and a worker, comprising: an overlap confirmation unit that checks an overlapping area of a plurality of camera measurement error ranges determined by the error range determination unit and determines the corrected position information of the object. Camera system. The method of claim 4,
The second position correction unit,
Multiple AVM for measuring the position of a construction machine and a worker, characterized in that the overlapping area checked by the overlap check unit and the overlapping area of the location information of the object included in the movable property information are corrected as the final location information of the object Camera system. The method of claim 1,
Each of the plurality of fixed AVM devices,
An AVM camera unit including a plurality of fixed AVM cameras each outputting a fixed image;
An operation for recognizing temporary objects by analyzing the plurality of stationary photographed images, matching the recognized temporary objects to determine the object, measuring location information, and generating fixed property information including location information of the measured object Control unit; And
A communication unit for transmitting the generated fixed attribute information to a central server; a multi-AVM camera system for measuring the location of construction machinery and workers, comprising a. The method of claim 1,
The at least one mobile AVM device,
A GPS unit receiving location information of the construction machine;
An AVM camera unit including a plurality of mobile AVM cameras each outputting a mobile photographed image;
An operation for recognizing temporary objects by analyzing the plurality of mobile photographed images, matching the recognized temporary objects to determine the object, measuring position information, and generating mobile attribute information including position information of the measured object Control unit; And
A communication unit for transmitting the generated mobile attribute information to a central server; a multi-AVM camera system for measuring the position of a construction machine and a worker, comprising a. The method of claim 7,
The operation control unit,
A construction machine and worker position measurement, characterized in that the measured location information of the object is converted into world coordinates in connection with the location information of the construction machine received from the GPS unit, and the converted world coordinates are determined as the location information of the object. AVM camera system for the camera. (A) A number of fixed AVM devices are installed at different fixed locations on the construction site to recognize the object by analyzing the fixed-type photographed images taken of the construction site, and the estimated position by estimating the location information of the recognized object. Transmitting attribute information including information (hereinafter, referred to as'fixed attribute information') to the following central server;
(B) At least one movable AVM device is installed to be movable on a construction machine and recognizes an object by analyzing a movable photographed image taken of a construction site, and estimates the location information of the recognized object to obtain the estimated location information. Transmitting the included attribute information (hereinafter, referred to as'portable attribute information') to the following central server; And
(C) analyzing, by the central server, a plurality of fixed property information and at least one mobile property information received from steps (A) and (B), and correcting the location information of the object based on multiple views; Construction machinery and worker position measurement method of a multi-AVM camera system comprising a. The method of claim 9,
The fixed type attribute information,
Including location information and type of the object recognized in the fixed-type photographed image, a fixed-type photographed image including the object, a photographing time, and location information of a fixed AVM device,
The mobile attribute information,
A method for measuring a location of a construction machine and a worker in a multi-AVM camera system, comprising: location information and type of the object recognized in the mobile photographing image, a mobile photographing image including the object, a photographing time, and location information of a construction machine. . The method of claim 9,
The (C) step,
(C1) obtaining, by the central server, a relationship parameter between the fixed AVM devices by performing calibration and calibration of the plurality of fixed AVM devices;
(C2) the central server correcting, by the central server, the location information of the object recognized by the plurality of fixed AVM devices by using the obtained relationship parameter and the received plurality of fixed attribute information; And
(C3) comparing, by the central server, the location information of the object corrected in step (C2) with the location information of the object estimated in step (B) to finally correct the location information of the object; Construction machinery and worker position measurement method of a multi-AVM camera system, characterized in that. The method of claim 10,
The (C2) step,
The central server determines, for each of the plurality of fixed AVM devices, a camera measurement error range estimated to be located from the location information and the captured images among the plurality of fixed property information, and the determined A construction machine and worker position measurement method of a multi-AVM camera system, characterized in that the overlapping area of the measurement error ranges of a plurality of cameras is identified and determined as corrected position information of the object. The method of claim 12,
The (C3) step,
The central server, a multi-AVM camera system, characterized in that the overlapping area checked in step (C2) and the overlapping area of the location information of the object included in the mobile property information are corrected as final location information of the object. Construction machinery and worker position measurement method. The method of claim 9,
In the step (A), each of the plurality of fixed AVM devices,
(A1) Recognizing temporary objects by analyzing fixed-type photographed images captured by a plurality of fixed AVM cameras, matching the recognized temporary objects to determine the object, measuring location information, and including location information of the measured object Generating fixed attribute information; And
(A2) transmitting the generated fixed attribute information to a central server; construction machine and worker position measurement method of a multi-AVM camera system comprising a. The method of claim 9,
In the step (B), the at least one mobile AVM device,
(B1) receiving location information of the construction machine;
(B2) Recognizing temporary objects by analyzing mobile photographed images captured by a plurality of mobile AVM cameras, matching the recognized temporary objects to determine the object, measuring location information, and including location information of the measured object Generating mobile attribute information to be performed; And
(B3) transmitting the generated mobile attribute information to a central server; construction machine and worker position measurement method of a multi-AVM camera system comprising a.

Images