KR102492161B1 - Transfer Vehicle for Transfer System and Driving Method Thereof - Google Patents

Abstract

A transport vehicle for a transport system and a driving method of the transport vehicle are disclosed. A transport vehicle of a transport system according to the present invention includes a photographing unit for three-dimensionally photographing a front area when operating on a rail of the transport system; and a control unit that recognizes a designated object from a photographed image, measures a distance to a human object based on depth data obtained during photographing, and generates an emergency signal based on the measured value when the designated object as a recognition result is a human object. includes

Description

Transfer vehicle for transfer system and driving method of the transfer vehicle {Transfer Vehicle for Transfer System and Driving Method Thereof}

The present invention relates to a transport vehicle for a conveyance system and a driving method of the transport vehicle, and more particularly, uses a low-cost three-dimensional (3D) depth perception and color camera sensor to deep-learn a worker as one of the obstacles And a transfer vehicle for a transport system that performs object detection to discriminate, measures the distance between the differentiated operator and the transport object, and transmits a warning signal when the distance is less than a certain distance to stop the transport vehicle to prevent safety accidents. And it relates to a driving method of the transport vehicle.

The overhead hoist transport system is installed in large hospitals, semiconductors, LCDs, OLEDs, solar panels, and manufacturing plants with many objects to be transported. car, transport vehicle, ceiling transport vehicle) and rails for guiding OHT. The transfer truck, or OHT, runs along the rails installed on the ceiling. OHT is integratedly controlled and managed by OCS (OHT Control System), which issues a return order. When a ceiling transfer system is installed in an LCD or OLED production line, rails are arranged using the ceiling space in the clean room. transport from The running rail is supported from the ceiling of the clean room by supports or the like, and is composed of a running rail main body and a power supply rail. The OHT has a travel driving unit so that it can travel within the main body of the traveling rail, and communicates with the OCS by receiving signals from the power supply rail through the power receiving unit. In the ceiling transport system, objects are transported by a plurality of OHTs traveling on tracks. The track has a branching section that diverges so that OHT can move to another track while traveling in a straight line, and a joining section that allows OHT moving from the branching section to enter another track. The OHT is branched to various places and transports the object to the desired location by traveling on the connected track.

Conventionally, in the case of industrial transport vehicles such as AGV (Automated Guided Vehicles), OHT, and OHS (Over Head Shuttle System), laser scanner type sensors are mostly used to detect forward obstacles. Since this sensor is a type of irradiating laser on a plane, it can inspect only one plane among the spaces where the moving object passes. However, due to this, the obstacle detection blind area (or detection shadow area) is wider than the detection area, and accidents often occur in the field where people collide with work jigs or ladders.

Korea Patent Registration No. 10-0729986 (2007.06.13) Korea Patent Registration No. 10-0790817 (2007.12.26) Korean Registered Patent Publication No. 10-2010772 (2019.08.08) Korean Registered Patent Publication No. 10-2153050 (2020.09.01) Korean Registered Patent Publication No. 10-2062548 (2019.12.30)

Therefore, the technical problem to be achieved by the present invention is to distinguish, for example, a worker, which is one of the obstacles, by performing deep learning learning and object recognition using a low-cost three-dimensional (3D) depth perception and color camera sensor, and distinguish the worker and the transport vehicle It is to provide a transport vehicle for a transport system for preventing safety accidents by measuring the distance between the transport vehicles and stopping the transport vehicle by transmitting a warning signal when the vehicle gets closer to a certain distance or less, and a driving method of the transport vehicle.

According to one aspect of the present invention, the photographing unit for three-dimensionally photographing the front area when the rail of the transport system is operated; and recognizing a designated object from the photographed image, and when the designated object is a human object as a recognition result, by measuring a distance to the human object based on depth data obtained during the photographing, based on the measured value. A transfer vehicle of a transfer system comprising a control unit generating an emergency signal may be provided.

The control unit may extract a designated object from the captured image, and determine whether the extracted designated object is the human object through deep learning analysis of artificial intelligence (AI).

The control unit may calculate distances for each of a plurality of points preset in the human object, and may use an average of the calculated distance values as a measurement value of the distance.

The control unit may perform the average by excluding points having a difference value greater than or equal to a reference value compared to other points among the calculated plurality of points.

The controller may histogram the distance values for the plurality of points and then compare difference values between the points in a histogram-ized state.

The control unit may change a capturing area, a region of interest (ROI), or a capturing angle of the capturing unit based on a speed or a turning direction when the rail is operated.

The controller may automatically transmit a pre-stored recorded image related to the human object to an interlocking external device when the photographed human object disappears from the photographing area.

In addition, according to another aspect of the present invention, the photographing unit 3-dimensionally photographing the front area when the rail of the transport system is operated, and the controller recognizes a designated object from the photographed image, and as a result of the recognition, the designation When the object is a human object, measuring the distance to the human object based on the depth data obtained during the photographing and generating an emergency signal based on the measured value Driving the transport vehicle of the transport system A method may be provided.

In the generating step, a specified object is extracted from the captured image, and whether the extracted specified object is a human object may be determined through deep learning analysis of artificial intelligence (AI).

In the generating step, distances may be calculated for each of a plurality of points preset in the human object, and the average of the calculated distance values may be used as a measurement value of the distance.

In the generating step, the average may be performed by excluding points having a difference value greater than or equal to a reference value compared to other points among the calculated plurality of points.

In the generating step, distance values for the plurality of points may be histogrammed and then difference values of the points may be compared in a histogrammed state.

In the generating step, a capturing area, a region of interest (ROI), or a capturing angle of the photographing unit may be changed based on a speed or a turning direction when the rail is operated.

In the generating step, when the photographed human object disappears from the photographing area, a pre-stored recorded image related to the human object may be automatically transmitted to an interworking external device.

According to an embodiment of the present invention, a worker who cannot be detected by a conventional obstacle sensor can be recognized in advance, thereby preventing a safety accident caused by a conveying vehicle, that is, a conveying vehicle.

1 is a diagram showing a data processing system of a transport vehicle according to an embodiment of the present invention.
2 is a view showing various types of transport vehicles in which cameras are installed.
3 is a diagram for explaining a method of detecting a human body and measuring a distance.
4 is a diagram for explaining a method of matching image data and distance data.
5 is a diagram for explaining a method of efficiently calculating a distance between a human body and a conveying body within a body range rectangle.
FIG. 6 is a block diagram illustrating a detailed structure of the transport vehicle of FIG. 1 .
7 is a flowchart illustrating a driving process of the transfer vehicle of FIG. 1;
8 is a flowchart illustrating another driving process of the transfer vehicle of FIG. 1;

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only illustrated for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention It can be embodied in various forms and is not limited to the embodiments described herein.

Embodiments according to the concept of the present invention can apply various changes and can have various forms, so the embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are only for the purpose of distinguishing one component from another component, e.g., without departing from the scope of rights according to the concept of the present invention, a first component may be termed a second component, and similarly The second component may also be referred to as the first component.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle. Other expressions describing the relationship between elements, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", etc., should be interpreted similarly.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is an embodied feature, number, step, operation, component, part, or combination thereof, but one or more other features or numbers However, it should be understood that it does not preclude the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

In the drawings, the thickness of layers and regions is exaggerated for clarity. When a layer is referred to as being “on” another layer or substrate, or when a layer is referred to as being bonded or adhered to another layer or substrate, it may be formed directly on the other layer or substrate or as a third layer therebetween. may be intervened. Parts designated with like reference numerals throughout the specification mean like elements.

Terms such as top, bottom, top, bottom, front, back, or top, bottom are used to distinguish the relative positions of components. For example, in the case of naming the upper part in the drawing as the upper part and the lower part in the drawing as the lower part for convenience, in practice, the upper part may be named as the lower part and the lower part may be named as the upper part without departing from the scope of the present invention. . In addition, components in the drawings are not necessarily drawn to scale, and for example, the size of some components in the drawings may be exaggerated compared to other components to help understanding of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing a data processing system of a transport vehicle according to an embodiment of the present invention, and FIG. 2 is a diagram showing various types of transport vehicles in which cameras are installed.

As shown in FIG. 1 , a data processing system 90 of a transport vehicle according to an embodiment of the present invention includes a transport vehicle 100 and an external device 110 .

Rails constituting the conveying system according to an embodiment of the present invention may be installed on the ceiling of a place such as a semiconductor production line. Of course, in the embodiment of the present invention, the rail will not be particularly limited, and the rail will not be particularly limited to being installed on the ceiling. In other words, since it is possible for the transport vehicle 100 to autonomously drive in a place where there is no rail, for example, on the floor of a production line, the embodiment of the present invention will not be particularly limited to the rail. Alternatively, the rail in the embodiment of the present invention is not configured by installing a separate structure such as a railway rail, but means a road on which a line is drawn on the bottom surface, or a road that is installed below the surface of the road and is invisible. can include all As long as the transport vehicle 100 can run, it may be of any type.

However, for convenience of description, the embodiment of the present invention illustrates a rail installed on the ceiling of a production line, and the corresponding rail may be installed on the ceiling in various forms. Typically, it can be divided into a straight section and a merging section where a plurality of rails cross each other. Depending on the characteristics of the production line, it can be configured in various forms. In addition, a groove or home position in which transport vehicles 100 that do not carry out a transport operation are located may be formed on the rail. The transport vehicles 100 located in the groove do not interfere with the transport vehicle 100 during the transport operation on the rail, thereby preventing congestion in the production line. In general, when a vehicle enters, it can be called merging, and when a vehicle leaves, it can be called divergence. Of course, as a concept that integrates them, it can be named 'crossroads'. For example, various equipment for manufacturing semiconductors is installed in a clean room of a semiconductor production line, and the equipment includes a plurality of chambers for each process. In such a production line, rails can be installed and operated in a clean room.

The transportation vehicle 100 corresponds to a means of transport that runs or travels along rails. As shown in (a) to (c) of FIG. 2 , cameras may be installed and used in various types of vehicles such as AGV, OHT, and OHS. The transport vehicle 100 can perform a driving operation and a loading and unloading operation, and can be controlled by a merging controller installed in a merging section or a transport vehicle controller even in a merging section in a specific situation (eg, when the merging controller is abnormally operated). there is. For example, the transport vehicle 100 may transport a process object of a semiconductor process between equipment along a rail. Since the transport vehicle 100 is variously named such as a transport vehicle, a transport vehicle, a transport vehicle, and an OHT, the embodiments of the present invention will not be particularly limited to such names. However, the transport vehicle 100 may transfer the process object manufactured in A equipment to B equipment for the next process, and transfer the process object manufactured in B equipment to C equipment. perform the action Of course, it may be possible to supply substances or materials for manufacturing with each equipment.

Above all, the transport vehicle 100 of FIG. 1 according to an embodiment of the present invention may include a depth recognition camera, that is, a 3D camera 101 of FIG. The operation of the transport vehicle 100 may be controlled based on the analysis result of (or distance data or meta data). To this end, the controller 103 for analyzing the image and data of the 3D camera 101 may include a control unit such as a CPU, MPU or GPU. The controller 103 may recognize a human body shape in an image, generate a detection signal according to a distance to the human body, and perform an image recording operation. The controller 103 controls the operation of the transport vehicle 100 to transmit a communication confirmation RESPONSE signal (eg, controller ALIVE), transmission of a camera operation confirmation signal (eg, camera ERR), transmission of a conveyance control signal (eg, attention/deceleration) /stop, etc.) can be performed.

The controller 103 of the transfer vehicle 100 may transmit recorded images to an external device 110 such as a user PC. In the embodiment of the present invention, after detecting the transport vehicle 100, a recorded image may be transmitted through short-range wireless communication or the like when a specified time elapses. In FIG. 1, the 3D camera 101 or the controller 103 is shown as being separated from the transport vehicle 100 to explain the data processing process, but in reality, it is preferable to be provided inside the transport vehicle 100, Of course, in the case of the 3D camera 101, it is possible to use it separately, so the embodiment of the present invention will not be particularly limited to any one configuration. For example, the camera may be installed and operated in various forms, such as being installed and used in a designated place in a clean room, and the camera may perform wireless communication with the transport vehicle 100. However, for convenience of description, it will be described that they are integrated and used in the embodiments of the present invention.

More specifically, the transfer vehicle 100 according to an embodiment of the present invention uses a low-cost 3D depth perception and color camera sensor to save costs associated with manufacturing the transfer vehicle 100, for example, among obstacles that must be stopped immediately upon discovery. A single worker is distinguished by deep learning learning and object recognition, and the distance between the distinguished worker and the conveying object is measured, and when the distance is less than a certain distance, a warning signal is sent to stop the conveying object, that is, the conveying vehicle 100, for safety accidents. prevent To this end, in an embodiment of the present invention, a transport vehicle equipped with an RGB-D (Depth) camera and a GPU controller for deep learning-based object recognition, a worker object recognition and distance measurement program (eg, SW module, firmware, etc.), a schedule It may include a program that generates an emergency signal when a worker is found at a distance or less, a program that records and stores a situation when a worker is found, and the like. In particular, it also includes a method of reducing computational load and increasing accuracy so that a signal can be quickly transmitted to a moving object moving at high speed.

As will be discussed later, the transfer vehicle 100 according to an embodiment of the present invention recognizes a designated object by analyzing a captured image captured by the 3D camera 101, that is, a video frame of the captured image, that is, the video frame It is possible to extract an object from and determine whether the extracted object is a human object or an object object such as a forklift to be found by using an object detector applied with deep learning or the like. Therefore, when it is determined as a human object such as a worker or an object object such as a forklift, the depth or distance of the location where the corresponding object is extracted is calculated by matching or using the depth data generated when the 3D camera 101 is photographed. For example, in an embodiment of the present invention, when a specific object is recognized in a video frame of a captured image, bounding box processing may be performed on the object, and a plurality of points of a designated object within the bounding box are selected randomly or in a designated manner. You can calculate the distance value based on the point by designating it. That is, point sampling is performed to calculate distance values for each sampled point. In addition, the distance to the human object may be finally determined by calculating an average distance of each calculated distance value. As a result of the determination, when the value is below the reference value, an emergency signal may be generated so that an alarm sound may be generated.

Of course, the transport vehicle 100 according to the embodiment of the present invention may further perform various operations in the process of performing the above operations. Typically, distance values of sampled points are calculated and then histogrammed to calculate distances by excluding points having a large difference from other points, for example, a reference value or more. For example, when there are a total of 9 sampled points, the distances for each of the 9 points are calculated and converted into a histogram. In addition, when the difference between the distance values of the total nine points is too large, that is, when the difference between the distance values of other points is large, or when the difference is out of the reference range, the distance value of the corresponding point may be excluded. Afterwards, the distance average can be calculated by excluding the excluded points. For example, if 1 point is excluded from 9 points, the average can be calculated based on a total of 8 distance values. Through this, it is possible to increase the accuracy of distance calculation.

Furthermore, the transport vehicle 100 may adjust the shooting area of the 3D camera 101 based on the moving speed or the turning motion. In the embodiment of the present invention, the 3D camera 101 is used in a fixed form, but even when the 3D camera 101 is fixed, operations such as zoom in or zoom out may be possible. Therefore, when the speed is higher than the reference value, it is possible to prevent a sudden accident in advance by allowing an area farther from the front to be photographed. Of course, the zoom operation may be performed in proportion to the speed. Furthermore, as the 3D camera 101, it may be possible to use a rotatable camera. A zoom operation is performed in the straight section, but when there is a right or left turn operation, the 3D camera 101 may be rotated at a specified angle so that a photograph is taken. In addition, in order to reduce the burden of data calculation processing, the transport vehicle 100 moving on a fixed rail moves along the same route, rather than analyzing all the shooting areas captured by the lens of the 3D camera 101, so that accidents do not occur. It is possible to set a frequent region of interest (ROI) and analyze the captured image through this. Since the region of interest is smaller than the actual photographing region, it is possible to reduce the processing load of data by reducing the analysis region. This makes it possible to quickly recognize accidents.

Above all, since it is important to accurately recognize a human object such as a worker while the transport vehicle 100 is moving, artificial intelligence deep learning can be applied to accurately detect the object, and deep learning can be applied when recognizing the object. . Template-based images related to various types of workers can be set in advance, and learning can be performed based on them. there is. For example, even if it is difficult to determine whether or not a designated object is a human object in the (n-1)th video frame, whether or not the same object can be determined through tracking of the corresponding object in the (n-2)th or (n)th video frame. Through this, it is possible to increase the detection accuracy of an object by adding an object (or its shape or posture, etc.) that was difficult to discriminate. Since various methods may be possible, the embodiments of the present invention will not be particularly limited to any one form.

In addition, the transport vehicle 100 analyzes the object in the front area to determine whether it is a human object, and when it passes the location of the determined human object, for example, when the human object disappears from the shooting area, after a certain time or as soon as it passes The immediately recorded video can be transmitted to the external device 110 . For example, a captured image captured by the 3D camera 101 for a specified period of time before encountering a human object is recorded, the recorded captured image is stored, and the recorded captured image is transferred to an external device when passing the object. (110). Of course, depth data may be provided together with the captured image. Here, the depth data may be referred to as meta data, and the meta data may be regarded as including data such as a color attribute of a designated object in an image, as well as a simple depth value. In this regard, it is useful to apply a 3D camera 101, and it is preferable to use a low-cost 3D camera 101 to save cost. Of course, even if it is not the 3D camera 101, it is possible to generate depth data using two 2D cameras having different focal points. To this end, a separate process for generating 3D data such as depth data may be added, but a separate program may be unnecessary when a commercially available 3D camera 101 is used. Of course, in the embodiments of the present invention, it will not be particularly limited to any one form.

In summary, an image and distance data (or depth data) are provided from a depth-recognizing camera, for example, the 3D camera 101 to the controller 103 . The controller 103 performs an operation for controlling a conveying object, that is, the conveying vehicle 100, or a decelerator or accelerator of the conveying vehicle 100, by using the image and the distance data. To this end, the controller 103 can configure and execute three programs. Of course, the program herein may include a software (SW) module or firmware. One program finds the shape of a worker in image data transmitted from a depth-recognizing camera. Objects can be extracted through image analysis. In addition, another program calculates the relative distance between the worker formation found in the above step and the transport vehicle 100 using the distance data transmitted from the depth-recognition camera, and generates warning, warning, and error signals according to the distance. Furthermore, the controller 103 records a situation from a predetermined time before the worker's shape was discovered to a predetermined time after the worker's shape disappeared and stores it as image data. You can run the rest of the program for this. Accordingly, the transfer vehicle 100 interprets the signal generated by the controller 103 and performs normal running, deceleration, and stop of the transfer vehicle 100 . The type of controller system of the transfer vehicle 100 will not be particularly limited to any one type. Meanwhile, the external device 110 such as a user PC can check and store the video recorded by the controller 103 . Since the OS or PC hardware types of the PC for confirmation can be varied, it will not be particularly limited to any one form.

3 is a diagram for explaining a method of detecting a human body and measuring a distance.

Referring to FIG. 3 together with FIG. 1 for convenience of explanation, the transport vehicle 100 of FIG. 1 according to an embodiment of the present invention is an image (FIG. 3(a)) from a depth perception camera as shown in FIG. Left side) and distance (or depth) (right side of FIG. 3 (a)) data are received. In (a) of FIG. 3, the left side shows a captured image, and the right side shows images based on distance data. Of course, in reality, in the case of the right side, it may not be formed in the form of an image. However, if the color is dark when viewed from the image on the right, it means that it is close, but it will not be particularly limited thereto.

Subsequently, the transport vehicle 100 searches for a worker shape in the image using a deep learning algorithm. For example, it is possible to extract a designated object from a video frame of a photographed image and determine whether the corresponding object is a human object of a worker through an object detector or classifier. This object detection has already been described above. 3(b) shows a process of recognizing whether a designated object is a human object by detecting a designated object in a video frame of a photographed image. In this process, a bounding box can be set.

Also, the transport vehicle 100 may calculate an average distance by converting the searched location and area where the operator is located into a distance data space. Then, the results are synthesized and transmitted to the conveying body of the conveying vehicle 100 . The conveying body may mean a driving wheel or a reducer, but may mean the conveying vehicle 100 itself in the embodiment of the present invention. Since the transport vehicle 100 moves on rails, a deceleration device or an accelerator may be a transport vehicle. 3(c) and (d) show that the distance between the transport vehicle 100 and the operator in the image is calculated using depth data.

Therefore, the transport vehicle 100 calculates the average distance and outputs a warning sound to the outside when the average distance is within a designated range, thereby preventing a worker from colliding with the transport vehicle 100 . Alternatively, the conveying body of the conveying vehicle 100 may be controlled to decelerate.

4 is a diagram explaining a method of matching image data and distance data.

Referring to FIG. 4 together with FIG. 1 for convenience of explanation, among human body detection and distance measurement, there is only one depth-recognizing camera, but there are several cameras with different characteristics, such as a color image sensor and a sensor capable of processing depth like an image. (e.g. 2 camera modules, etc.) Therefore, it is necessary to align the color image and the distance data to accurately know the position of the human body found in the color image. Here, matching may include aligning, that is, aligning or matching two images with each other.

It is possible to simply match the entire image and distance data and use the distance data corresponding to the position of the human body, but the calculation load is high because the matching calculation for all pixels must be performed. In fact, in the case of conveying objects, it is different depending on the type, but since they move at a high speed of about 1 to 4 m/s, fast calculation and response speed are essential. Therefore, in the embodiment of the present invention, instead of matching the color image and the entire distance data, the four vertices of the rectangle corresponding to the range where the human body is located are first converted into the range rectangle of the distance data, and only the distance data within the range is used for matching. It reduces the calculation load by about 10 times and improves the response speed.

Referring to FIG. 4, (a) requires matching calculations of 640×480 pixels, a total of 307,200 pixels, when performing matching of the entire image. On the other hand, in (b), only the desired area is cropped and matched through the matching of 4 points, so that 300 × 100, a total of 30,000 pixels of matching calculation is required. Therefore, by using the method (b) of FIG. 4, the response speed can be increased by reducing the calculation load compared to (a) of FIG.

5 is a diagram for explaining a method of efficiently calculating a distance between a human body and a conveying body within a body range rectangle.

Referring to FIG. 5 together with FIG. 1 for convenience of description, the embodiment of the present invention does not calculate an average distance by using all distance data to improve calculation efficiency among calculating the average distance of the part where the human body is detected, but rather The distance between the moving object and the human body is calculated by sampling points at a rate and averaging them.

However, if the points are sampled at a constant rate, since the points sampled within the range rectangle where the human body is located may not actually be inside the human body but outside the body, the distance value of each point is histogrammed and points with a large difference from other points are excluded and the average distance is calculated.

5, (a) shows points sampled in the body range rectangle. A total of 9 are specified, but this number can be changed at any time. Figure 5 (b) shows the process of histogramming. Of course, this can be seen as an operation performed internally by the program. For example, it can be seen that the distance value of point 3 is excluded. Therefore, the average distance is calculated by summing the distance values of the remaining 8 points excluding the third point among the total 9 points and then dividing by the total number of points, i.e., 8.

Accordingly, when the average distance value is within a designated range, the transport vehicle 100 of FIG. 1 may perform a subsequent operation such as generating a warning sound. For example, when the designated range is set to 1 m, a warning sound may be generated while entering the 1 m distance. For example, it may be possible to perform a deceleration operation while generating a warning sound when entering a distance of 1 m. Since various operations are possible, the embodiment of the present invention will not be particularly limited to any one form.

FIG. 6 is a block diagram illustrating a detailed structure of the transport vehicle of FIG. 1 .

As shown in FIG. 6, the transport vehicle 100 of FIG. 1 according to an embodiment of the present invention includes a communication interface unit 600, a control unit 610, a safety management unit 620, and a photographing unit 630, or It includes all, and may further include a storage unit and the like.

Here, "including part or all" means that some components such as the communication interface unit 600 or the storage unit are omitted, or some components such as the safety management unit 620 are configured by other components such as the control unit 610. As meaning what can be integrated into the components and configured, it will be described as including all to help a sufficient understanding of the invention.

The communication interface unit 600 may communicate with an external device 110 such as a user PC of FIG. 1 to transmit a recorded image of an image captured through the photographing unit 630 . To this end, the communication interface unit 600 may include a short-distance communication module. The communication interface unit 600 may transmit recorded images to an external device through short-range wireless communication such as Bluetooth, ZigBee, and Wi-Fi, and may operate under the control of the controller 610 at this time.

Of course, the communication interface unit 600 further communicates with a controller for controlling the movement path of the plurality of transport vehicles 100 operating on rails, or further communicates with a merging controller that controls the transport vehicles 100 at an intersection. However, in the embodiment of the present invention, corresponding controllers are excluded from the description.

The communication interface unit 600 may be capable of wired or wireless communication. In other words, the 3D camera 101 of FIG. 1 may be connected through a wired port or the like. Since the communication interface unit 600 of FIG. 1 may be configured in the form of a chip on the main PCB, and the 3D camera 101 of FIG. 1 may be configured on the outer frame (or body) of the transport vehicle 100, In order to connect to each other in a circuit, it may be electrically connected to each other through a cable or the like.

The control unit 610 is in charge of overall control operations of the communication interface unit 600, the safety management unit 620, and the photographing unit 630 of FIG. For example, when the transportation vehicle 100 starts operating, the controller 610 may operate the photographing unit 630 to perform a photographing operation. In addition, the controller 610 may store the photographing data and depth data of the photographing unit 630 in an internal memory or in a separately configured memory.

In addition, the control unit 610 analyzes the captured image, and although this operation may be performed in the safety management unit 620, it is possible to extract objects from the video frame of the captured image and determine whether the extracted object is a human object such as a worker. there is. To this end, it is possible to determine whether or not a human object exists by setting a bounding box for a designated object in a video frame, extracting the corresponding part, and utilizing an object detector or classifier that performs a deep learning operation. In the case of using a deep learning-based object detector, more accurate object recognition of a human object may be possible. Of course, in the embodiment of the present invention, not only human objects but also object objects such as forklifts may be detected according to the intention of the system designer. If detection of a forklift is also required in a transport system installed in a production line, detection can be performed by designating an object for this as well.

Therefore, when the object detected in the video frame is determined to be, for example, a human object, the controller 610 calculates an average distance by averaging distance values for a plurality of points of the human object, or the average distance made by the safety management unit 620 Based on the value, a warning sound can be output or the speed can be reduced by controlling the speed reducer. Accordingly, the transport vehicle 100 may further include and control an audio output unit such as a speaker or a speed reducer.

In addition, the controller 610 may control various operations. For example, if it is set to control the zoom-in, zoom-out, and rotation operations of the 3D camera constituting the photographing unit 630, the zoom-in and zoom-out operations may be performed according to the speed, and during the rotation operation, the shooting angle of the 3D camera can be rotated to a specified angle. The camera of the photographing unit 630 can be controlled in various forms, and since this has been described above, further description will be omitted. For example, it is representative to perform analysis based on a region of interest among photographed regions.

The safety management unit 620 determines whether an accident such as collision with a worker occurs in the traveling direction of the transport vehicle 100 based on the photographed image and depth data captured by the photographing unit 630 when the transport vehicle 100 operates on the rail. watch over Accordingly, the safety management unit 620 extracts an object from the video frame of the photographed image and performs an operation of recognizing the extracted object. For object recognition, a deep learning-based object detector can be used, and through this, various types of human objects desired by the system can be determined. For example, although it is possible to detect an object based on a template, it is preferable to use an object detector that performs an artificial intelligence deep learning operation for more accurate detection of an object. In particular, in the embodiment of the present invention, when a low-cost 3D camera is used for cost reduction, the resolution may be lowered, and thus erroneous object detection may occur through this, and to compensate for this, deep learning-based object detection that may be desirable.

In addition, when the detection object is determined to be a human object, the safety management unit 620 may perform a distance calculation operation by matching a video frame of a corresponding captured image with depth data. For example, depth data may include attribute information, depth information, and location information (or coordinate information) on a video frame. Accordingly, it is possible to know where the designated object A is located in which video frame and what properties it has. In this respect, depth data may also be meta data. Accordingly, a distance may be calculated based on location information of a human object in a video frame through a matching operation.

For example, since the bounding box is set for the extracted object in the embodiment of the present invention, a plurality of predetermined points may be set for the object within the bounding box. Alternatively, a point may be set at a boundary region or a region corresponding to a feature point through pixel analysis of the corresponding object. Then, the distance between the corresponding points and the 3D camera is calculated. For example, it is possible to set the lower part of the video frame as the position of the camera and calculate the distance based on pixel coordinates on the video frame. When an actual distance value per unit pixel is assumed (or set), the distance in the real environment can be obtained by multiplying the number of pixels by the actual distance value.

In this way, the safety management unit 620 can calculate each distance value for a plurality of points, and when the calculation is completed, the distance values of the points that have a large difference from the distance values of other points are histogrammed. By excluding , the average distance can be calculated. Through this, precise distance measurement can be performed. Since this has been described above, further explanation will be omitted. Accordingly, the safety management unit 620 may request the control unit 610 to control the speed reducer to decelerate or generate a warning sound when the average distance reaches the reference value set by the system designer.

A method of calculating the distance in the conveyance system may be possible in various ways other than a method using pixel coordinates. For example, this is a case where fixed coordinates for a specific object included in a captured image are known in the front area where the rail is operated. If the distance is calculated by assuming the bottom center of the preceding video frame as the position of the 3D camera 101, the pixel distance to an object whose coordinates are known in the image is calculated, and then the actual distance value per pixel is calculated to obtain a real distance in the real environment. You may also know the actual distance. Since various methods may be possible, the embodiments of the present invention will not be particularly limited to any one form.

The photographing unit 630 may include a 3D camera capable of depth recognition. Normally, two cameras are used for depth recognition, and thus, 3D shooting may be possible by modularizing them. Of course, unlike 2D cameras, 3D cameras can generate more depth data. Accordingly, in the case of using two 2D cameras, a separate program for generating depth data must be further used, and therefore, a commercially available product can be used as the photographing unit 630 . However, in order to reduce the manufacturing cost or purchase cost of the transfer vehicle 100, a low-spec product may be used rather than a high-end (high-end) product. Of course, it will not be particularly limited to such things.

Meanwhile, as another embodiment of the present invention, the control unit 610 may include a CPU and a memory, and may be formed as a single chip. The CPU includes a control circuit, an arithmetic unit (ALU), a command interpreter, and a registry, and the memory may include RAM. The control circuit performs control operations, the operation unit performs operation of binary bit information, and the instruction interpretation unit performs an operation of converting high-level language into machine language and machine language into high-level language, including an interpreter or compiler. , the registry may be involved in software data storage. According to the above configuration, for example, at the beginning of the operation of the transport vehicle 100, the program stored in the safety management unit 620 is copied, loaded into a memory, that is, RAM, and then executed, thereby rapidly increasing the data operation processing speed. can

7 is a flowchart illustrating a driving process of the transfer vehicle of FIG. 1;

Referring to FIG. 7 together with FIG. 1 for convenience of description, when the transport vehicle 100 of FIG. 1 according to an embodiment of the present invention travels on a rail of a transport system, a front area is three-dimensionally photographed (S700). Here, it can be seen that 3D shooting includes generating depth data or meta data for a photographed object. For example, when two 2D cameras are installed, a separate program needs to be additionally installed to generate depth data. Therefore, even in this case, it can be viewed as a 3D photograph.

Subsequently, the transport vehicle 100 recognizes a designated object from the captured image and measures the distance to the human object based on depth data acquired (or acquired) when the designated object is a human object as a result of the recognition. An emergency signal is generated based on the measured value (S710). Here, the emergency signal may be a deceleration control signal for controlling the speed reducer, and may be a warning sound output control signal for outputting a warning sound. It can contain various signals.

In addition to the above, the transfer vehicle 100 according to the embodiment of the present invention can perform various operations, and since other details have been sufficiently described above, they will be replaced with those contents.

8 is a flowchart illustrating another driving process of the transfer vehicle of FIG. 1;

Referring to FIG. 8 together with FIG. 1 for convenience of description, when the human body sensor controller is operated, the transport vehicle 100 checks whether the depth camera is normally connected and periodically transmits an acknowledgment signal (S800, S810, S820). ). This signal is received by the transfer vehicle 100 and is used for the purpose of confirming whether the human body detection controller is operating normally.

In addition, the human body sensor controller of the transportation vehicle 100 waits to receive a frame from the depth recognition camera (S801). If frames are not received for a certain period of time or frames cannot be received due to camera disconnection, etc., reconnection is attempted until the camera is reconnected (S811, S812). At this time, it is possible to know whether or not the transport vehicle 100 has an abnormality through the camera error signal.

The human body detection controller of the transportation vehicle 100 receives a frame (eg, image, distance) from the depth camera and finds the human body using the image data through a deep learning algorithm (S802, S803). If the human body is not found, it waits for frame reception again, and if the human body is found, the relative distance between the human body and the conveying body is measured using the distance data (S804). In addition to the method described above, for example, even when the transport vehicle 100 includes a gyro sensor or an acceleration sensor, distance measurement may be possible using the sensing values of the corresponding sensors and the pixel coordinate values in the image.

In addition, the human body detection controller of the transport vehicle 100 stores the human body detection frames as images immediately after the human body is discovered, from a certain time before the human body is discovered until after the human body disappears from the (video) frame and a certain time passes (S813). .

Based on the relative distance measured by the human body sensor controller of the transport vehicle 100, the transport vehicle 100 performs a normal driving, deceleration or stop action (S822). In case of deceleration or stop, it can control the reducer, i.e. the brake system.

In addition to the above, the transfer vehicle 100 of FIG. 1 can perform various operations, and since other details have been sufficiently described above, they will be replaced with those contents.

Although preferred embodiments have been shown and described above, the present invention is not limited to the specific embodiments described above, and ordinary knowledge in the art to which the present invention belongs without departing from the gist of the present invention claimed in the claims Of course, various modifications are possible by those who have it, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

On the other hand, even if all the components constituting the embodiment of the present invention are described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all of the components may be selectively combined with one or more to operate. In addition, although all of the components may be implemented as a single independent piece of hardware, some or all of the components are selectively combined to perform some or all of the combined functions in one or a plurality of pieces of hardware. It may be implemented as a computer program having. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art. Such a computer program may implement an embodiment of the present invention by being stored in a computer-readable non-transitory computer readable media, read and executed by a computer.

Here, the non-transitory readable recording medium means a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is common in the art to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: transport vehicle 101: 3D camera
103: controller 110: external device
600: communication interface unit 610: control unit
620: safety management department 630: shooting department

Claims (14)

A photographing unit for three-dimensionally photographing the front area when the rail of the transport system is operated; and
A designated object is recognized from the photographed image, and when the designated object is a human object as a result of recognition, the distance to the human object is measured based on depth data obtained during the photographing, and the distance to the human object is measured. Including a control unit for generating a signal,
The control unit calculates distances for each of a plurality of points preset in the human object, and averages the calculated distance values and uses them as a measurement value of the distance;
The control unit performs the average by comparing distance values with other points among the calculated plurality of points, excluding distance values of points having a difference value greater than or equal to a reference value, and
The transport vehicle of the transport system of claim 1 , wherein the control unit automatically transmits a pre-stored recorded image related to the human object to an external device that interlocks with each other when the photographed human object disappears from the photographing area. According to claim 1,
The control unit extracts a designated object from the captured image, and determines whether the extracted designated object is the human object through deep learning analysis of artificial intelligence (AI). delete delete According to claim 1,
The transfer vehicle of the transport system according to claim 1 , wherein the control unit compares difference values of the points in a histogrammed state after converting distance values for the plurality of points into histograms. According to claim 1,
The control unit changes a capturing area, a region of interest (ROI), or a capturing angle of the capturing unit based on a speed or a turning direction when the rail is operated. delete 3-dimensionally photographing the front area when the photographing unit travels on the rail of the transport system; and
The control unit recognizes a designated object from the photographed image, and as a result of the recognition, when the designated object is a human object, the controller measures the distance to the human object based on the depth data obtained during the photographing, and jumps based on the measured value. Generating a signal; including,
The generating step is
calculating distances for each of a plurality of points preset in the human object, and averaging the calculated distance values and using them as a measurement value of the distance;
performing the average by comparing distance values with other points among the calculated plurality of points and excluding distance values of points having a difference value greater than or equal to a reference value; and
When the photographed human object disappears from the photographing area, automatically transmitting a pre-stored recorded image related to the human object to an interlocking external device.
A method of driving a transport vehicle of a transport system, comprising: According to claim 8,
The generating step is
A method of driving a transport vehicle of a transport system, characterized in that extracting a designated object from the captured image, and determining whether the extracted designated object is the human object through deep learning analysis of artificial intelligence (AI). delete delete According to claim 8,
The generating step is
A method of driving a transport vehicle of a transport system, characterized in that comparing the difference values of the points in the histogram after the distance values for the plurality of points are histogrammed. According to claim 8,
The generating step is
A method of driving a transport vehicle of a transport system, characterized in that changing a shooting area, a region of interest (ROI), or a shooting angle of the shooting unit based on a speed or a turning direction when the rail is operated. delete

Images