WO2023095939A1

WO2023095939A1 - Safety enhancement method for industrial equipment, and device thereof

Info

Publication number: WO2023095939A1
Application number: PCT/KR2021/017343
Authority: WO
Inventors: 심용수; 심상우
Original assignee: 심용수
Priority date: 2021-11-24
Filing date: 2021-11-24
Publication date: 2023-06-01

Abstract

Disclosed are a safety enhancement method for industrial equipment, and a device thereof. The safety enhancement device uses one or more imaging devices installed at a blind spot of a piece of industrial equipment to generate a virtual object for one or more objects present in the blind spot, and displays the virtual object in augmented reality or virtual reality.

Description

Industrial Equipment Safety Reinforcement Method and Device

An embodiment of the present invention relates to a method and apparatus for enhancing the safety of various industrial equipment such as trucks, heavy equipment, and construction equipment, and more particularly, to prevent safety accidents that may occur in blind spots of industrial equipment. It relates to a method and device for enhancing safety.

Industrial equipment such as trucks, heavy equipment, or construction equipment has blind spots that workers cannot see directly with the naked eye. Safety accidents can occur when workers operate industrial equipment without recognizing people or objects located in blind spots. To prevent this, a camera or a proximity sensor is installed in a blind spot of a truck or heavy equipment. However, if the blind spot is wide, it is difficult to cover all the blind spots with one camera, and if the angle of view of the camera is widened, the captured image is distorted, making it difficult for the operator to determine the exact distance. exist. In the case of a sensor, it is possible to know that a nearby object exists, but it is difficult to know exactly where the location of the object is, and it is difficult to determine the type of the nearby object.

The technical problem to be achieved by the embodiment of the present invention is to create an object in a blind spot as a virtual object and provide it in virtual reality or augmented reality, so that workers can easily identify objects located around industrial equipment and enhance safety. It is to provide a method for enhancing the safety of equipment and its device.

An example of a method for enhancing the safety of industrial equipment according to an embodiment of the present invention for achieving the above technical problem is using at least one or more photographing devices installed in the blind spot of the industrial equipment to at least one or more existing in the blind spot. creating a virtual object for the object; and displaying the virtual object in augmented reality or virtual reality.

In order to achieve the above technical problem, an example of a safety enhancement device according to an embodiment of the present invention uses at least one photographing device installed in a blind spot of industrial equipment to detect at least one or more objects present in a blind spot. a virtual object creation unit that creates virtual objects; and a display unit displaying the virtual object in augmented reality or virtual reality.

According to an embodiment of the present invention, a worker can accurately and easily identify unknown objects located around industrial equipment through virtualized objects, thereby preventing safety accidents. As another example, it is possible to monitor everything from the moment an object enters the dangerous zone of industrial equipment to the moment it leaves the safe zone, thereby helping to prevent safety accidents.

1 is a diagram showing an example in which a photographing device for generating a virtual object is installed in industrial equipment according to an embodiment of the present invention;

2 is a diagram showing an example of a safety accident prevention method for industrial equipment according to an embodiment of the present invention;

3 is a view showing an example in which objects around industrial equipment are created as virtual objects and displayed in a virtual space according to an embodiment of the present invention;

4 is a diagram showing an example of an overall system for enhancing the safety of industrial equipment according to an embodiment of the present invention;

5 is a diagram showing an example of a safety reinforcement method using object recognition according to an embodiment of the present invention;

6 is a diagram showing another example of a safety reinforcement method using object recognition according to an embodiment of the present invention;

7 is a flowchart showing an example of a method for enhancing safety of industrial equipment according to an embodiment of the present invention;

8 is a view showing the configuration of an example of a safety reinforcement device according to an embodiment of the present invention;

9 is a diagram showing an example of a schematic configuration of a system for creating objects in a virtual space according to an embodiment of the present invention;

10 is a view showing an example of a photographing method of a photographing device according to an embodiment of the present invention;

11 is a flowchart illustrating an example of a method for generating an object in a virtual space according to an embodiment of the present invention;

12 is a diagram showing an example of a method for distinguishing a background and an object of a video frame and a measurement frame according to an embodiment of the present invention;

13 is a diagram showing an example of a lattice space according to an embodiment of the present invention;

14 is a diagram showing an example of displaying objects classified in a video frame in a grid space according to an embodiment of the present invention;

15 is a diagram showing an example of a method of correcting a depth value of a pixel of an object according to an embodiment of the present invention;

16 is a diagram showing an example of a method for generating a 3D virtual object according to an embodiment of the present invention;

17 is a diagram showing an example of extracting a point cloud for generating a 3D virtual object according to an embodiment of the present invention;

18 is a diagram showing an example of creating a 3D virtual object according to an embodiment of the present invention, and

19 is a diagram showing the configuration of an example of a virtual object generating device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, a method and device for enhancing safety of industrial equipment according to an embodiment of the present invention will be described in detail.

1 is a diagram showing an example in which a photographing device for generating a virtual object is installed in industrial equipment according to an embodiment of the present invention.

Referring to FIG. 1 , the truck 100 includes at least one

photographing device

110 , 112 , and 114 for photographing nearby objects. For example, the photographing

devices

110 , 112 , and 114 may be present in the rear and front side portions of the truck, which are blind spots that the driver cannot directly see with the naked eye. As an example, the photographing

devices

110 , 112 , and 114 may be arranged to photograph the entire periphery of industrial equipment as well as blind spots.

Although the present embodiment presents the truck 100 as an example of industrial equipment for better understanding, this is only one example and the industrial equipment may be various such as heavy equipment or construction equipment. For example, various devices used in construction, transportation, factories, etc., such as forklifts, cranes, and excavators, may correspond to the industrial equipment of the present embodiment. However, hereinafter, for convenience of description, the truck 100 will be mainly described as an example of industrial equipment.

The photographing

devices

110 , 112 , and 114 include a depth camera for obtaining photographic data for generating a virtual object, not a general camera attached to a vehicle or the like. As an example, a virtual object can be created with data photographed using a depth camera, but in this case, there is a problem in that the shape of the virtual object is also distorted due to lens distortion of the camera. As a method of solving this problem, the photographing

devices

110 , 112 , and 114 may include a LiDAR (Light Detection and Ranging) together with a depth camera. Although the present embodiment does not exclude the photographing

devices

110 , 112 , and 114 composed of only the depth camera, for convenience of description, a case in which the photographing

devices

9110 , 112 , and 114 include a lidar together with a depth camera will be described.

By using a depth camera with a wide angle of view, a small number of photographing

devices

110 , 112 , and 114 can take pictures of surrounding areas including blind spots of industrial equipment. However, when a depth camera with a wide angle of view is used, there is a problem in that lens distortion occurs more than that of a depth camera with a narrow angle of view. In the case of including lidar together, it is possible to use a depth camera with severe lens distortion because a virtual object is created by correcting the distortion of the depth camera with lidar. A detailed method of generating an accurate virtual object (for example, a digital twin) for an object by using a depth camera and LIDAR together will be reviewed again below in FIG. 9 .

2 is a diagram showing an example of a method for preventing safety accidents of industrial equipment according to an embodiment of the present invention.

Referring to FIG. 2A , an object 210 may exist around the truck 200 . Although this embodiment shows a person as an example of the object 210, the object 210 may be various objects or animals that may collide with the movement of the truck, such as objects or facilities. Not all cases in which the object 210 exists around the truck 200 are safety accident risk situations. For example, when the truck 200 goes straight on the road, vehicles located in other lanes or people on the sidewalk are not at risk of a safety accident. Simply informing the driver that the presence of the object 210 around the truck 200 as a safety accident risk situation may rather interfere with driving.

Referring to FIG. 2B , the safety enhancement device determines whether a surrounding object 210 is in a dangerous situation according to the moving direction of the truck 200 . For example, when the truck 200 turns right, it may be determined that an object 210 located within a certain area 220 on the right side of the truck 200 is a safety accident risk factor. To this end, the safety reinforcement device may predefine a risk area during movement of the industrial equipment according to the type of the industrial equipment. For example, if the industrial equipment is a truck, the danger area when the truck goes straight is a certain area in front of the truck's traveling direction, and when the truck turns right or left, the danger area can be predefined as certain areas on the right and left sides of the truck. . In the case of an excavator, all within a certain radius around the excavator can be defined as a risk area. Since the movement characteristics of industrial equipment are all different, the safety reinforcement device can register and store risk areas according to movement of each industrial equipment in advance.

The safety enhancement device receives movement information (for example, angle information of a truck steering device, etc.) of industrial equipment directly from each industrial device, or through a GPS or direction sensor installed in industrial equipment, a filming device, or a safety enhancement device. The direction of movement of industrial equipment can be grasped. As another example, the safety reinforcement device may analyze the image frame captured by the photographing device to determine the moving direction or speed of the industrial equipment. In addition to this, various conventional methods for determining the moving direction or speed of industrial equipment may be applied to this embodiment, and are not limited to any one example.

3 is a diagram illustrating an example in which objects around industrial equipment are created as virtual objects and displayed in a virtual space according to an embodiment of the present invention.

Referring to FIGS. 2 and 3 together, the safety enhancement device determines that an object 210 exists in a predefined danger area 220 when the truck 200 turns right, and determines it as a safety accident risk situation, thereby determining the object in the real space. Creates a virtual object for (210) and displays it in augmented reality or virtual reality to notify the driver. Since the safety enhancement device does not display objects around the truck as a two-dimensional image, but displays them as a three-dimensional image in real time as shown in FIG. can be checked with For example, the safety enhancement device can display truck images and virtual objects at various angles desired by the driver, such as a head-up display (HUD), augmented reality (AR) glasses, and various display devices.

4 is a diagram showing an example of an overall system for enhancing the safety of industrial equipment according to an embodiment of the present invention.

Referring to FIG. 4 , the system for enhancing safety of industrial equipment includes a photographing device 400 , a safety enhancement device 410 and a display device 420 . As described in FIG. 1 , the photographing device 400 is a

device

110 , 112 , and 114 for photographing the surroundings including blind spots of industrial equipment, and may include a depth camera and lidar. Depending on the type of industrial equipment, the photographing device may be located in various places of the industrial equipment.

The safety enhancement device 410 uses the photographing data of the photographing device 400 to create virtual objects for objects located around industrial equipment. And, the display device 420 displays the virtual object in augmented reality or virtual reality. The display device 420 may be various according to embodiments such as a HUD, AR glasses, or a general display device.

5 is a diagram showing an example of a method for enhancing safety using object recognition according to an embodiment of the present invention.

Referring to FIG. 5, the safety reinforcement device 410 generates an alarm through sight or hearing so that the user can know when an object exists in the risk area of industrial equipment, and creates and augments a surrounding object as a virtual object. displayed in real or virtual reality.

If an object exists in the dangerous area of industrial equipment, there is a risk of safety accident. However, if the object is a tree branch blown by the wind or garbage, it is an object that has nothing to do with safety accidents. If the safety enhancement device 410 generates an alarm or the like to the user whenever an object or the like completely unrelated to the safety accident is located around the industrial equipment, this may rather interfere with the work.

As an example, the safety enhancement device 410 may recognize the type of object located in the danger area to determine whether there is an actual risk of safety accident. An artificial intelligence model 500 may be used to determine the type of object. For example, the safety reinforcement device 410 inputs a virtual object 510 (ie, a 3D image) generated based on an object located in a danger area to the artificial intelligence model 500 to recognize the object type (520) can do. The artificial intelligence model 500 is a model learned to recognize object types based on 3D images. Since the artificial intelligence model itself for object recognition is already a well-known technology, a detailed description thereof will be omitted.

The safety reinforcement device 410 determines whether the object type identified through the artificial intelligence model 500 using a risk factor object list or a non-risk factor list is an object corresponding to a risk factor or an object that does not correspond to a risk factor. can figure it out For example, if leaves or garbage are defined in the object list that is not a risk factor, and the type of virtual object identified through the artificial intelligence model 500 is a leaf, the safety reinforcement device 410 determines that the object is a safety accident. An alarm may not be generated for a user or the like by determining that the element is not an element.

6 is a diagram showing another example of a method for enhancing safety using object recognition according to an embodiment of the present invention.

Referring to FIG. 6 , the safety enhancement device 410 may recognize an object suddenly entering industrial equipment and notify the user. For example, while driving a truck, a rider on a kickboard may suddenly enter in front of the truck or a person may suddenly enter within a work radius of the forklift truck. At this time, since it may be difficult for operators of industrial equipment to respond to the sudden appearance of objects, it is necessary to identify and inform them in advance.

To this end, the safety enhancement device 410 may set the measurement range of the lidar of the photographing device and the measurement range of the depth camera to be different from each other. That is, the radius of the first area 620 measured by the lidar around the industrial equipment 600 may be larger than the second area 610 captured by the depth camera. In this embodiment,

circular areas

610 and 620 are shown around the industrial equipment 600 for better understanding. However, if the industrial equipment is the truck 100 as shown in FIG. 1, the first area 610 and the second area 620 may be a triangular area. That is, the shapes of the first area 620 and the second area 610 may vary depending on the shape of industrial equipment and the installation location of the photographing device, but the size of the first area 620 may be different from that of the second area ( 610) can be made larger than the size.

The safety enhancement device 410 notifies the user in a visual or auditory manner when the object 630 enters the first region 620 predefined based on the measurement data measured by the lidar. For example, in the case of the truck 100 of FIG. 1 , the safety enhancement device 410 may display a color or image indicating attention on a display device such as a HUD or output a predefined sound. Then, when the object 630 enters the second region 610, which is the shooting range of the depth camera, the safety reinforcement device 410 creates a virtual object based on the shooting data captured and measured by the depth camera and LIDAR, and creates it. It can be displayed in augmented reality or virtual reality and provided to the user. Accordingly, the user may be aware in advance that the virtual object is entering the industrial equipment before displaying the virtual object.

If the object 630 enters the first area 620, which is the measurement area, and does not enter the second area 610, which is the shooting area, and goes out of the measurement area again (ie, the safety zone), the safety reinforcement device 410 ) can release the attention step without creating a virtual object. That is, the safety reinforcement device 410 may notify the user of a safety accident risk situation while tracking the object from the time the object enters the danger zone of the first region 620 until it leaves the danger zone.

7 is a flowchart illustrating an example of a method for enhancing safety of industrial equipment according to an embodiment of the present invention.

Referring to FIG. 7 , the safety enhancement device 410 captures the surroundings of industrial equipment with a photographing device to create virtual objects for objects located in the surroundings (S700). An example of a method of generating a virtual object using a photographing device composed of a depth camera and lidar is shown below in FIG. 9 .

The safety enhancement device 410 displays the virtual object in augmented reality or virtual reality (S810). As an example, the safety enhancement device 410 may generate and display a virtual object for the object in real time when the object is located in a predefined risk area of industrial equipment. As another embodiment, the safety enhancement device 410 recognizes the type of virtual object located around the industrial equipment through an artificial intelligence model as shown in FIG. 5, and then determines whether the corresponding object type corresponds to a predefined safety accident element. Afterwards, an alarm may be provided to the user through visual or auditory information only when it corresponds to the safety accident factor.

8 is a diagram showing the configuration of an example of a safety reinforcement device according to an embodiment of the present invention.

Referring to FIG. 8 , the safety enhancement device 410 includes a virtual object generator 800 and a display unit 810 . As an example, the safety enhancement device 410 may be implemented as a computing device including a memory, a processor, and an input/output device. In this case, each configuration may be implemented as software, loaded into a memory, and then executed by a processor.

The virtual object creation unit 800 creates a virtual object for at least one or more objects existing in the blind spot of industrial equipment using at least one or more photographing devices installed in the blind spot. The photographing device may include a depth camera and lidar. In this case, the virtual object generator 800 corrects the depth value of the pixel of the area corresponding to the object in the image frame obtained by photographing with the depth camera with the distance value of the point of the measurement frame obtained by measuring with the LIDAR. A virtual object can be created based on the corrected pixel depth value. A method of generating a virtual object using a depth camera and LIDAR is shown below in FIG. 9 .

The display unit 810 displays virtual objects in augmented reality or virtual reality. As an example, the display unit 810 may generate an alarm by recognizing the type of the virtual object using the previously learned artificial intelligence model and determining whether the type of the virtual object is a safety accident factor. As another embodiment, the display unit 810 displays the existence of the moving object visually or by sound when there is a moving object in the first area, which is the measurement area of the lidar, through the measurement frame measured by the lidar, and the moving object is controlled. When entering the second area of the depth camera having an area narrower than the first area as a capturing area, a virtual object may be created and displayed or an alarm such as a warning may be output.

9 is a diagram showing an example of a schematic configuration of a system for creating objects in a virtual space according to an embodiment of the present invention.

Referring to FIG. 9 , a photographing device 900 includes a depth camera 902 and a LIDAR 904 . The depth camera 902 is a camera that captures a certain space in which the object 930 exists and provides a depth value of each pixel together. In this embodiment, the object 930 means an object to be created in a virtual space. The depth camera 902 itself is a well-known technology, and various types of conventional depth cameras 902 may be used in this embodiment. In this embodiment, the depth camera 902 may capture a still image or a video. However, hereinafter, for convenience of description, a case in which a video is captured by a depth camera will be mainly described. Photo data including a depth value of each pixel obtained by photographing with the depth camera 902 is called an image frame. That is, a video is composed of a certain number of video frames per second.

The LIDAR 904 emits a laser into a certain space, measures signals returned from each point in the space (ie, a reflection point), and outputs distance values for a plurality of points in the certain space. Data composed of distance values for a plurality of points measured by the LIDAR 904 in a certain space at a certain point in time is called a measurement frame. The resolution of a plurality of points measured by the lidar 904, that is, the measurement frame, may be different depending on the lidar 904. LiDAR 904 itself is already widely known technology, and various conventional lidars 904 may be used in this embodiment.

The photographing device 900 simultaneously drives the depth camera 902 and the LIDAR 904 to photograph and measure an object 930 in a certain space. In various embodiments, the expression 'photographing' of the photographing device 900 or 'measurement' of the photographing device may be interpreted as capturing the depth camera 902 and measuring the LIDAR 904 simultaneously. The number of video frames generated per second by the depth camera 902 and the number of measurement frames generated per second by the LIDAR 904 may be the same or different depending on embodiments. Also, the resolution of the video frame and the resolution of the measurement frame may be the same or different depending on the embodiment. However, since the depth camera 902 and the lidar 904 are simultaneously driven to generate an image frame and a measurement frame for a certain space, they can be mapped to the same time axis and synchronized.

The virtual object generating device 910 uses the image frame obtained by photographing with the depth camera 902 and the measurement frame obtained by measuring with the LIDAR 904 together to create an object in the virtual space for the object 930 in the real world (that is, , digital twin). 1 to 8 have been described on the assumption that the virtual object generating device 910 is implemented as a part of the safety enhancement device 410, but the virtual object generating device 910 is a device separate from the safety accident prevention device 410. can be implemented as

The virtual object generating device 910 may be connected to the photographing device 900 through a wired or wireless communication network (eg, WebRTC, etc.) and receive real-time video frames and measurement frames generated by the photographing device 900 . The virtual object generating device 910 may provide the result of object creation in the virtual space to the user terminal 920 so that the user can check it.

As another embodiment, the photographing device 900 and the virtual object generating device 910 may be implemented as one device. For example, the photographing device 910 and the virtual object device 910 may be implemented as part of various devices that display augmented reality or virtual reality, such as AR (augumented reality) glasses, HMD (Head Mounded Display), or wearable devices. . As another embodiment, the photographing device 900 is implemented as a part of AR glasses, HMD, or a wearable device, and the photographing device 900 is a virtual object generating device that connects real-time photographing and measuring image frames and measurement frames to a wired or wireless communication network. 910, the AR glasses, HMD, etc. implemented with the photographing device may receive the virtual object from the virtual object generating device 910 and display the virtual object in augmented reality or virtual reality. The user can immediately check the virtual object created in real time through augmented reality or virtual reality. A detailed method of generating an object in a virtual space will be reviewed again below in FIG. 10 .

10 is a diagram illustrating an example of a photographing method of a photographing device according to an embodiment of the present invention.

Referring to FIGS. 9 and 10 , at least one photographing device 900 may continuously photograph at least one object 1000 existing in a blind spot. At least one or

more objects

1000 and 1010 may exist at various points in time on the basis of the time axis in the image frame and the measurement frame obtained by capturing the photographing device 900 . This embodiment shows an example of an image frame for convenience of description.

The virtual object generating device 910 may classify the image frame and the measurement frame in units of objects. The virtual object generating apparatus 910 may assign the same identification information (or index) to the video frame and the measurement frame in which the same object exists in the video frame and the measurement frame. For example, first identification information (or first index, hereinafter referred to as identification information) is assigned to all of the plurality of image frames 1020, 1022, and 1024 in which the first object 1000 exists, and the second object ( Second identification information may be assigned to the plurality of image frames 1030 , 1032 , and 1034 in which 1010) is present. No identification information may be assigned to the image frames 1040 and 1042 in which no object exists, or third identification information may be assigned. In this embodiment, the image frames arranged along the time axis can be divided into three groups, that is, A (1050), B (1060), and C (1070). In another embodiment, if a plurality of objects exist in the video frame and the measurement frame, identification information corresponding to each object may be assigned to one video frame and the measurement frame. That is, a plurality of pieces of identification information may be assigned to one image frame and one measurement frame.

In another embodiment, since the photographing device 900 simultaneously drives the depth camera 902 and the lidar 904 to take pictures, the video frame generated by the depth camera 902 and the measurement frame measured by the lidar are synchronized in time. It can be. Therefore, the virtual object generating device 910 determines whether the same object exists only in the video frame, determines the time period of the video frame in which the same object exists, and assigns the same identification information to the video frame during that time period. The same identification information may be assigned by considering the measurement frame generated during the time period identified for the frame as a period in which the same object exists.

The virtual object generating device 910 may determine whether objects existing in an image frame are the same using various conventional image recognition algorithms. For example, the virtual object generating device 910 may use an artificial intelligence model as an example of an image recognition algorithm. Since the method itself for determining whether objects in an image are the same is a well-known technique, a detailed description thereof will be omitted.

11 is a flowchart illustrating an example of a method of generating an object in a virtual space according to an embodiment of the present invention.

Referring to FIG. 11 , the virtual object generating apparatus 910 divides a first background area and a first object area from an image frame obtained by photographing a certain space with a depth camera (S1100). The virtual object generating device 910 may distinguish a background and an object from a plurality of image frames in which the same object is photographed. For example, as shown in FIG. 10 , a background and an object may be distinguished for at least one image frame (eg, group A of 1020, 1022, and 1024) in which the same object exists. As another embodiment, when a plurality of objects exist in an image frame, the virtual object generating apparatus 910 may distinguish the background and the plurality of objects, respectively. An example of a method for distinguishing a background from an object in an image frame will be reviewed again in FIG. 12 .

The virtual object generating device 910 distinguishes a second background area and a second object area from a measurement frame obtained by measuring a certain space with lidar (S1110). The virtual object generator 910 may distinguish a background and an object for a plurality of measurement frames in which the same object is measured. For example, as shown in FIG. 11, a background and an object may be distinguished for a plurality of measurement frames to which the same identification information is assigned. As another embodiment, when a plurality of objects exist in the measurement frame, the virtual object generator may distinguish the background and the plurality of objects, respectively. An example of a method for distinguishing a background and an object in a measurement frame is shown in FIG. 12 .

As shown in FIG. 9 , the depth camera 902 and the LIDAR 904 are spaced apart from each other by a predetermined distance within the photographing device 900, and therefore, the photographing angles of the video frame and the measurement frame are different from each other. In addition, since the resolutions of the video frame and the measurement frame may be different from each other, the positions of each pixel of the video frame and each point of the measurement frame may not be mapped on a one-to-one (1:1, scale) basis. In order to map an image frame and a measurement frame having such a difference, this embodiment uses a grid space.

Specifically, the virtual object generator 910 arranges the pixels of the first object area divided from the image frame according to the depth value in the first grid space including the grid of the predefined size (S1120), and also The points of the second object area divided in the measurement frame are arranged according to the distance values in the second lattice space including the lattice of the same size (S1120). Since the image frame and the measurement frame are data obtained by photographing the same space, objects existing in the first object area and the second object area are the same object. The first lattice space and the second lattice space are spaces having grids of the same size in the virtual space. An example of a grid space is shown in FIG. 13 .

The virtual object generator 910 corrects the depth value of the pixel in the first grid space based on the distance value of the point in the second grid space (S1130). Since the pixels of the first object area and the points of the second object area exist in the grid space of the same size, if the position, direction, size, etc. of the first grid space and the second grid space match each other, each point of the second grid space and each pixel of the first lattice space may be mapped. A detailed method of correcting a depth value of a pixel in a grid unit of a grid space using a distance value of a point will be reviewed again in FIG. 15 .

The virtual object generating apparatus 910 creates an object (ie, a virtual object) in a virtual space having surface information based on the pixel whose depth value is corrected (S1140). The virtual object generating apparatus 910 corrects the pixel depth value of an object existing in a plurality of image frames to which the same identification information is assigned to the point distance value of an object existing in a plurality of measurement frames to which the same identification information is assigned, and then generates a plurality of A virtual object can be created using the corrected pixels of the image frame of . That is, a virtual object may be created by correcting a pixel depth value of an image frame of an object photographed at various angles and positions.

The virtual object generating device 910 may generate a 3D virtual object using various types of 3D modeling algorithms. For example, the virtual object generating apparatus 910 may generate a 3D object having surface information as an artificial intelligence model by using a pixel having a depth value. As another embodiment, when 3D modeling is performed using all pixels, since there is a disadvantage in that the amount of computation is large, the virtual object generator 910 is a point cloud representing corners and vertices among pixels constituting an object. A virtual object can be created by extracting and inputting the point cloud to a 3D modeling algorithm. An example of generating a virtual object using a point cloud will be reviewed again in FIG. 16 . A virtual object can be created based on the distance value of each point of the measurement frame, but generally the resolution of the measurement frame is lower than that of the video frame, so when creating a virtual object through the measurement frame, the corners of the object are crushed. In this embodiment, a virtual object is created using a distance value of a pixel of an image frame having a relatively high resolution.

12 is a diagram illustrating an example of a method of distinguishing a background and an object of a video frame and a measurement frame according to an embodiment of the present invention.

Referring to FIG. 12 , there is a first artificial intelligence model 1200 that distinguishes the background of an image frame from an object, and a second artificial intelligence model 1210 that distinguishes a background from an object of a measurement frame. Each of the

artificial intelligence models

1200 and 1210 is a trained model using pre-constructed learning data and may be implemented as a Convolutional Neural Network (CNN) or the like. Since the process of learning and generating an artificial intelligence model itself is already a well-known technology, a description thereof will be omitted.

The first artificial intelligence model 1200 is a model generated through machine learning to distinguish a background and an object in an image frame when an image frame is input. For example, if the first artificial intelligence model 1200 is an artificial intelligence model learned to recognize a chair, the first artificial intelligence model 1200 is a region in which a chair exists in an image frame (ie, a region of a chair in an image frame). pixels) can be distinguished.

The second artificial intelligence model 1210 is a model generated through machine learning to distinguish a background and an object in a measurement frame when a measurement frame is input. For example, if the first artificial intelligence model 1200 is an artificial intelligence model trained to recognize a chair, the second artificial intelligence model 1210 corresponds to a region where a chair exists in the measurement frame (ie, corresponds to a chair within the measurement frame). points) can be distinguished.

13 is a diagram showing an example of a lattice space according to an embodiment of the present invention.

Referring to FIG. 13 , the grid space 1300 is a space in which a region in the virtual space is divided into unit grids 1310 of a predetermined size. For example, the grid space 1200 may be a space composed of unit cells 1310 having width, length, and height of d1, d2, and d3, respectively. Depending on the embodiment, d1, d2, and d3 may all have the same size (eg, 1 mm) or different sizes.

14 is a diagram illustrating an example of displaying objects classified in a video frame in a grid space according to an embodiment of the present invention.

Referring to FIG. 14 , the virtual object generating apparatus 910 may display pixels of an object area divided in an image frame in the grid space 1400 using the depth values. In this embodiment, pixels 1410 are schematically illustrated for ease of understanding.

The virtual object generating apparatus 910 may determine the 3D coordinate value (or vector value of the pixel, etc.) of each pixel in the grid space by mapping the pixels of the object in the image frame to the grid space 1400 . That is, a 3D coordinate value of each pixel of the object may be generated using a predefined point in the lattice space as a reference point (0,0,0 or X,Y,Z).

In this way, the virtual object generating apparatus 910 may map points representing objects in the measurement frame to the grid space. If the point of the object in the measurement frame is displayed in the grid space, it can also be displayed in a shape similar to that of FIG. 14.

15 is a diagram illustrating an example of a method of correcting a depth value of a pixel of an object according to an embodiment of the present invention.

Referring to FIG. 15, a position corresponding to a first lattice 1500 in a first lattice space to which an object of an image frame is mapped and a first lattice 1500 in a second lattice space to which an object of a measurement frame is mapped denotes the second grating 1510, respectively. In this embodiment, it is assumed that the first lattice space in which the pixels of the object in the image frame are mapped and the second lattice space in which the points of the object in the measurement frame are mapped are first matched in position, direction, size, etc. In addition, this embodiment shows an example in which one pixel 1502 and one point 1512 are present in the first grid 1500 and the second grid 1510, respectively, for convenience of explanation, but one grid 1500 , 1510) may have a plurality of pixels or a plurality of points. Also, the number of pixels and points present in the first grid 1500 and the second grid 1510 may be the same or different.

The virtual object generator 910 corrects the depth value of the pixel 1502 of the first grid 1500 based on the distance value of the point 1512 of the second grid 1510. Since the distance value of the point measured by lidar is more accurate, the virtual object generator 910 corrects the depth value of the pixel based on the distance value of the point. For example, if the coordinate values of the pixels 1502 of the first grid 1500 in the grid space and the coordinate values of the points of the second grid 1510 are different from each other, the virtual object generator 910 operates the first grid The pixel 1502 of (1500) is corrected (1520) according to the coordinate values of the points of the second grid 1510.

Since the image frame and the measurement frame may have different resolutions, positions indicated by pixels of the first grid 1500 and points of the second grid 1510 may not be mapped one-to-one. Therefore, by using a plurality of points that exist in the second grid 1510 or a plurality of points that exist in the surrounding grids that exist in the upper, lower, left, right, etc. of the second grid 1510, the values existing between each point are interpolated, etc. Through this, it is possible to determine the distance value of the coordinate value of the point corresponding to the coordinate value of the pixel. In addition, coordinate values of pixels of the first grid 1500 may be corrected using distance values of points generated through interpolation.

16 is a diagram illustrating an example of a method of generating a 3D virtual object according to an embodiment of the present invention.

Referring to FIG. 16 , the virtual object generating apparatus 910 may generate a 3D virtual object 1620 including surface information by inputting a point cloud 1610 to a 3D modeling algorithm 1600 . A point cloud can be composed of points representing key points that can define an object, such as vertices and edges of an object. Various conventional methods of extracting a point cloud from an image frame captured by a depth camera may be applied to this embodiment. Since the method of extracting a point cloud itself is a well-known technique, an additional description thereof will be omitted.

The virtual object generating apparatus 910 maps the object extracted from the image frame to the lattice space, and corrects the distance value (or coordinate value) of each pixel mapped to the lattice space in the same manner as shown in FIG. 15 . Then, a point cloud to be used for generating a 3D virtual object is extracted from the corrected distance value (or coordinate value) of each pixel. An example of extracting a point cloud for the object of FIG. 14 is shown in FIG. 17 . The virtual object generating device 910 may use an artificial intelligence model such as machine learning as a 3D modeling algorithm 1600 . Various conventional algorithms for generating a 3D object based on a point cloud may be applied to this embodiment. Since the method of generating a 3D object using a point cloud itself is a well-known technology, a detailed description thereof will be omitted.

17 is a diagram illustrating an example of extracting a point cloud for generating a 3D virtual object according to an embodiment of the present invention. Referring to FIG. 17 , an example of extracting a point cloud 1710 for a chair is shown. Various conventional methods of extracting a point cloud may be applied to this embodiment. As another embodiment, a 3D virtual object may be generated by extracting a point cloud after correcting a depth value of a pixel in a plurality of image frames to which the same identification number is assigned, that is, a plurality of image frames in which the same object is photographed.

18 is a diagram illustrating an example of generating a 3D virtual object according to an embodiment of the present invention. Referring to FIG. 18 , the virtual object generator may create a virtual object 1800 including surface information using a point cloud.

Referring to FIG. 19 , the virtual object generating device 910 includes a first object extraction unit 1900, a second object extraction unit 1910, a first grid arrangement unit 1920, a second grid arrangement unit 1930, It includes a correction unit 1940 and an object creation unit 1950. As an example, the virtual object generator 810 may be implemented as a computing device including a memory, processor, input/output device, etc., or as a server, cloud system, etc. can be performed by

The first object extraction unit 1900 distinguishes a first background area and a first object area from an image frame obtained by photographing a certain space with a depth camera. The second object extraction unit 1910 distinguishes a second background area and a second object area from a measurement frame obtained by measuring the predetermined space with LIDAR. Classification of the background and the object can be performed using an artificial intelligence model, and an example thereof is shown in FIG. 12 .

The first grid arrangement unit 1920 arranges pixels of the first object area according to depth values in a first grid space including a grid having a predefined size. The second grid arranging unit 1930 arranges points of the second object area according to distance values in a second grid space including a grid having a predefined size. An example of the lattice space is shown in FIG. 13, and an example of mapping the pixels of an object extracted from an image frame to the lattice space is shown in FIG.

The correction unit 1940 corrects the depth value of the pixel in the first grid space based on the distance value of the point in the second grid space. An example of a method of correcting through grid space comparison is shown in FIG. 15 .

The object generator 1950 creates a virtual object having surface information based on pixels whose depth values are corrected. The object generator 1950 may create an object in a 3D virtual space using all pixels. However, in this case, since the amount of computation increases, the object generator 1950 may create a virtual object by creating a point cloud, examples of which are shown in FIGS. 16 to 18 .

Each embodiment of the present invention can also be implemented as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, SSD, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network to store and execute computer-readable codes in a distributed manner.

So far, the present invention has been looked at mainly with its preferred embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

Claims

Creating a virtual object for at least one or more objects existing in the blind spot using at least one photographing device installed in the blind spot of industrial equipment; and

The step of displaying the virtual object in augmented reality or virtual reality; safety reinforcement method for industrial equipment, characterized in that it comprises a.
According to claim 1,

recognizing the type of the virtual object using a pre-learned artificial intelligence model; and

The method of reinforcing safety of industrial equipment, characterized in that it further comprises; generating an alarm according to the type of the virtual object.
According to claim 1,

The photographing device includes a depth camera and lidar,

In the step of generating the virtual object, the depth value of a pixel of an area corresponding to the object in the image frame obtained by photographing with the depth camera is measured with the LIDAR and corrected with the distance value of the point of the measurement frame, and then corrected. A method for reinforcing safety of industrial equipment comprising the steps of generating a virtual object based on the depth value of a pixel.
The method of claim 1, wherein the displaying step,

If there is a moving object in the first region, which is the measurement region of the lidar, through the measurement frame measured by the lidar, visually or soundly indicating the existence of the moving object; and

and displaying the virtual object when the moving object enters a second area of the depth camera having a capturing area narrower than the first area.
a virtual object creation unit that creates a virtual object for at least one or more objects existing in the blind spot using at least one photographing device installed in the blind spot of industrial equipment; and

Safety reinforcement device comprising a; display unit for displaying the virtual object in augmented reality or virtual reality.
The method of claim 5, wherein the display unit,

A safety enhancement device characterized in that recognizing the type of the virtual object using the previously learned artificial intelligence model and generating an alarm according to the type of the virtual object.
According to claim 5,

The photographing device includes a depth camera and lidar,

The virtual object generator corrects the pixel depth value of the area corresponding to the object in the image frame obtained by photographing with the depth camera with the distance value of the point of the measurement frame obtained by measuring the LIDAR, and then corrects the corrected pixel depth. A safety enhancement device characterized in that for generating a virtual object based on the value.
The method of claim 5, wherein the display unit,

If there is a moving object in the first area, which is the measuring area of the lidar, through the measurement frame measured by the lidar, the existence of the moving object is displayed visually or by sound, and the moving object covers an area narrower than the first area. The safety reinforcement device, characterized in that for displaying the virtual object when entering a second area of the depth camera having a photographing area.
A computer-readable recording medium on which a computer program for performing the method according to claim 1 is recorded.