KR20230101505A - The around monitoring apparatus ofo the image base - Google Patents

Abstract

The present invention relates to a peripheral monitoring device for safe operation of a forklift truck, which can promote the driver's work convenience and safety by monitoring the rear side of the forklift truck and recognizing obstacles or people around the forklift truck when driving backwards. A device for monitoring the surroundings of a forklift comprising a vehicle body and a pair of forks provided to be able to move up and down in front of the vehicle body, which is installed on the rear side of the vehicle body and photographs the rear of the forklift. a photographing unit that outputs; a display unit installed adjacent to the driver's seat and displaying the photographed image on a screen; It is installed inside the vehicle body, detects an object included in the captured image by performing image recognition of the captured image, and confirms the type of the detected object, the distance to the object, and the driving state to confirm the non-warning step, first a control unit controlling to execute one of a warning step and a second warning step; and an alarm execution unit installed inside or outside the vehicle body and performing a warning notification through at least one of sound and lighting under the control of the control unit.

Description

Image-based peripheral monitoring device {THE AROUND MONITORING APPARATUS OFO THE IMAGE BASE}

The present invention relates to a peripheral monitoring device for safe operation of a forklift truck, and more particularly, by monitoring the rear side of the forklift truck to recognize obstacles or people around the forklift truck, the driver's work convenience and safety can be promoted. It relates to a peripheral monitoring device for the safe operation of a forklift truck.

In general, a forklift is a type of industrial vehicle that is widely used throughout industries such as civil engineering and construction. It is a vehicle that lifts and transports cargo to an arbitrary height in a loaded state using a pair of forks formed to protrude forward. , the driving stability is affected by the loading and unloading state of the cargo, for example, the weight of the cargo and the height of the fork loaded with the cargo.

In addition, when the forklift is reversing, in order to recognize an obstacle including a worker at the rear of the forklift, the driver must check the existence of the obstacle by looking in a rearview mirror or directly looking back. At this time, since the forklift has a high height, it is difficult for the forklift driver to check the rear obstacle when a nearby worker works while sitting or crouching, or when the height of the obstacle is low.

In consideration of these problems, Korean Patent Publication No. 10-2012-0084868 describes a method of generating a warning sound by using a distance sensor so that a person close to the rear of the forklift can recognize the operation of the forklift in advance.

However, the existing method has a problem in that the driver's attention is reduced because an excessive warning sound is generated to a person approaching the forklift not only when the forklift is in operation but also when it is stopped.

In addition, since a warning sound is generated at the time of proximity without distinguishing between an object and a person, there is a problem in that the meaning of the warning is faded because there are many cases where a warning sound is continuously generated in a forklift that needs to move an object close to the object.

On the other hand, forklifts often drive backwards due to the specificity of moving cargo using a forklift, and at this time, it is difficult for drivers to secure a rearward view.

Republic of Korea Patent Publication No. 10-2012-0084868 (2006.03.29.)

The present invention has been made to solve the above problems, and monitors the surroundings of the forklift in all environments, such as a wide open area as well as environments requiring high attention, such as intersections and inside factories where goods are densely loaded, to detect people and objects. Its purpose is to provide an image-based peripheral monitoring device capable of distinguishing and warning.

In addition, an object of the present invention is to provide an image-based peripheral monitoring device capable of preventing malfunctions caused by things loaded around, lights, sunlight, etc., and malfunctions that occur during stop or operation.

In order to achieve the above object, an apparatus for monitoring the surroundings of a forklift comprising a vehicle body equipped with a driver's seat and a pair of forks movably provided in front of the vehicle body according to an embodiment of the present invention a photographing unit installed on a rear surface of the vehicle body and outputting a photographed image by photographing the rear of the forklift; a display unit installed adjacent to the driver's seat and displaying the photographed image on a screen; It is installed inside the vehicle body, detects an object included in the captured image by performing image recognition of the captured image, and confirms the type of the detected object, the distance to the object, and the driving state to confirm the non-warning step, first a control unit controlling to execute one of a warning step and a second warning step; and an alarm execution unit installed inside or outside the vehicle body and performing a warning notification through at least one of sound and light under the control of the control unit, wherein the object includes an intersection, a work area, and a cargo loading area. It includes at least one of an identification marker, an object, and a person to be displayed, and the non-warning step is a step set to display an object detection image including an object on the display unit when the driving state is stopped, and the first warning step The step is set to execute the display unit and the alarm execution unit when the detected object is a person when the driving state is in operation, and the second warning step is set to execute the display unit and the alarm execution unit when the detected object is a person when the driving state is driving. It is characterized in that it is a step set to execute the display unit and the alarm execution unit.

In addition, the photographing unit is composed of at least one camera installed at the center of the rear surface of the forklift, and the camera is characterized in that it has a fish-eye lens having a 180° field of view and FHD resolution.

In addition, the photographing unit may further include at least one camera installed in the center of the front of the forklift, and the camera may include a fisheye lens having a 180° field of view and FHD resolution.

In addition, the display unit includes a monitor that blinks the screen for a predetermined time while displaying different colors of the screen when the first warning step and the second warning step are executed, and the alarm generating unit performs the first warning step and the second warning step. When the second warning step is executed, it is characterized in that it includes a speaker that outputs sound and a warning light that simultaneously outputs lighting and sound.

In addition, the control unit may include: an image collection unit for collecting and storing the photographed images taken by the photographing unit; an image recognizing unit that detects an object included in captured images collected through deep learning-based object recognition modeling and outputs an object detection image in which the object is detected; a distance measuring unit that calculates a distance to an object included in the collected captured images; a speed measurement unit that calculates a traveling speed of the forklift and determines one of a stationary state, a running state, and a rotational driving state based on the calculated traveling speed; and generating a control signal to perform one of a non-warning step, a first warning step, and a second warning step by combining the detected object information, the calculated distance information, and the determined driving state information, if the detected object is an identification marker. Regardless of the driving state, the object detection image is displayed on the display unit, but the control signal of the non-warning step is generated so that the screen color is changed and blinked, and if the detected object is an object, the non-warning step if the driving state is stopped When the driving state is driving or turning, if the distance to the object according to the calculated distance information is within a preset distance, it is determined that the entry into the danger zone is performed and the control signal is sent to execute the second warning step. It is characterized in that it comprises a; display unit and a processor that transmits to the alarm generating unit.

In addition, the speed measurement unit is provided with a 9-axis gyro sensor to calculate the speed and direction of movement, and if the driving speed of the forklift is 0, it is in a stopped state, and if the driving speed of the forklift is greater than or equal to a reference speed, it is in a running state, and the forklift While the driving speed is calculated, it is characterized in that if it is rotating, each is determined as a rotating operation state.

In addition, the processor transmits a control signal to the display unit to execute a non-warning step of displaying an object detection image on the display unit regardless of distance information when the detected object is a person and the current driving state is stopped, , If the detected object is a person, if the current driving state is driving or turning and the distance to the object according to the calculated distance information exceeds a preset distance, a control signal is transmitted to the display unit to execute the non-warning step, When the detected object is a person, the display unit sends a control signal to execute the first or second warning step when the current driving state is driving or turning and the distance to the object according to the calculated distance information is within a preset distance. And it is characterized in that each is transmitted to the alarm generating unit.

In addition, the processor increases the sound output of the alarm generating unit and shortens the flickering interval of lights as the distance between the rear of the forklift and the detected object gets closer and the calculated traveling speed of the forklift exceeds the reference speed. It is characterized by controlling to perform a warning notification.

According to the present invention, by providing a rear image, it is possible for the driver in the driver's seat to easily check the work situation and whether the operator is approaching the forklift with the naked eye during rear driving, thereby preventing safety accidents in advance. there is.

In addition, according to the present invention, a collision between a forklift and a person as well as a collision between an object located in a workplace and a forklift can be prevented by performing a warning by distinguishing a person, an object, and an identification marker, and also can prevent a person regardless of the distance. When this is recognized, an immediate warning is given to the driver, thereby preventing the occurrence of a safety accident such as a life-threatening accident in advance.

In addition, according to the present invention, by accurately distinguishing and recognizing people, objects, and identification markers using a deep learning technique, safety accidents due to departure from the work area are prevented by not leaving the work line displaying the work area. At the same time, There is an effect of warning the driver of a dangerous situation and the need for caution according to the surrounding environment.

1 is an exemplary diagram schematically illustrating a forklift to which an image-based peripheral monitoring device according to an embodiment of the present invention is applied.
FIG. 2 is a configuration diagram showing a detailed configuration of the image-based peripheral monitoring device shown in FIG. 1 .
FIG. 3 is an exemplary diagram for explaining object recognition modeling applied to the image recognition unit shown in FIG. 2 .
4 is an exemplary diagram illustrating an example of detecting a person recognized through the image recognition unit shown in FIG. 2 .
FIG. 5 is an exemplary diagram illustrating a monitor warning screen output according to a control signal of the processor shown in FIG. 2 .
6 is a flowchart illustrating a method for monitoring the surroundings of a forklift of an image-based peripheral monitoring device according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. At this time, the same components in each drawing are represented by the same reference numerals as possible. In addition, detailed descriptions of already known functions and/or configurations will be omitted. In the following description, parts necessary for understanding operations according to various embodiments will be mainly described, and descriptions of elements that may obscure the gist of the description will be omitted. In addition, some components in the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not entirely reflect the actual size, and therefore, the contents described herein are not limited by the relative size or spacing of the components drawn in each drawing.

In describing the embodiments of the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of a user or operator. Therefore, the definition should be made based on the contents throughout this specification. Terms used in the detailed description are only for describing the embodiments of the present invention, and should not be limiting. Unless expressly used otherwise, singular forms of expression include plural forms. In this description, expressions such as "comprising" or "comprising" are intended to indicate any characteristic, number, step, operation, element, portion or combination thereof, one or more other than those described. It should not be construed to exclude the existence or possibility of any other feature, number, step, operation, element, part or combination thereof.

In addition, terms such as first and second may be used to describe various components, but the components are not limited by the terms, and the terms are used for the purpose of distinguishing one component from another. used only as

Hereinafter, an embodiment of the present invention will be described in more detail with reference to the accompanying drawings.

1 is an exemplary diagram schematically illustrating a forklift to which an image-based peripheral monitoring device according to an embodiment of the present invention is applied.

As shown in FIG. 1 , the image-based peripheral monitoring device 100 according to the present invention is installed on the forklift 10, and includes a photographing unit 110 and a display unit 120 that photograph at least one of the front and rear. ), and a warning generating unit 130.

In addition, in the image-based peripheral monitoring device 100 according to the present embodiment, the control unit 140 for controlling the display unit 120 and the warning generating unit 130 is provided inside the vehicle body 11 of the forklift 10. .

Meanwhile, the forklift 10 according to the present embodiment has the structure of a known forklift in which the mast assembly 12 and the fork 13 are mounted on the front surface of the vehicle body 11, and thus a detailed description of the structure is omitted.

First, the photographing unit 110 is installed at the center of the rear surface of the forklift 10 so as to photograph at least the rear of the forklift 10 . In addition, the photographing unit 110 may be further installed at the center of the front of the forklift 10 to photograph the front. In addition, a plurality of photographing units 110 may be installed outside the forklift 10 so as to photograph the front, rear, left room, and right room, respectively.

That is, the photographing unit 110 may include at least one of a front camera for photographing the front, a rear camera for photographing the rear, a left chamber camera for photographing the left room, and a right camera for photographing the right room in order to secure a view desired by the driver. It may be appropriately selected to include the above.

The photographing unit 110 may be composed of a 180 ° fish-eye lens (FISH-EYE) camera to secure a wide angle of view in the photographing direction through a minimal camera, and blind spots can be eliminated through the wide angle of view.

In addition, each camera constituting the photographing unit 110 is electrically connected to the display unit 120 by wiring in a state of being coupled to the vehicle body of the forklift, or captures images through data communication through a wireless network to the display unit 120. will be forwarded to Here, the camera has been described as being installed in the center of one side of the forklift vehicle body, but is not limited thereto and may be installed on both sides of the one side of the vehicle body.

In particular, in this embodiment, the photographing unit 110 includes a rear camera as an essential camera, and additionally includes a front camera, a left room camera, and a right room camera, so that surrounding images can be obtained without a blind spot around the forklift.

Also, the display unit 120 receives and outputs the captured image captured by the photographing unit 110 . Since the display unit 120 is installed in the driver's seat of the forklift 10, the driver can drive while checking the image captured around the forklift 10 in real time from the driver's seat, thereby preventing safety accidents that occur while driving. there is.

The display unit 120 may be implemented as a monitor having a screen for outputting an image. Here, the screen configuration of the display unit 120 is configured to display the photographing directions (front, rear, left, right) of the displayed image together, so that the driver can easily see which side of the forklift 10 there are people can be verified.

In addition, the display unit 120 can divide the screen by the number of cameras, display images captured by each camera in each divided area, and display the object detection image including the object in the entire screen. It can be controlled through the control unit 140 so that

Also, when displaying an object detection image including an object, the display unit 120 may display the location of the object in the form of rectangular coordinates.

In addition, when an object is detected from an image captured around the forklift 10, the warning generating unit 130 issues a warning through sound, light, lighting, etc. to draw the driver's attention according to a control signal from the controller 140. allow this to happen

The warning generating unit 130 may include at least one speaker 131, warning light 132, and the like to output an alarm in a voice or visual manner. Here, the warning light 132 is installed in the driver's seat of the forklift 10 so that the driver of the forklift 10 can easily recognize the warning notification while driving in the driver's seat. To this end, the warning light 132 has a known warning light structure that simultaneously emits sound while emitting light.

The speaker 131 outputs an alarm sound to alert the driver to the existence of a worker around the forklift and to alert the operator to confirm the operation of the forklift.

In particular, in the present invention, when the forklift 10 moves backward, a notification message about the reverse operation of the forklift is output through the speaker 131 so that workers around the forklift can recognize it, thereby preventing safety accidents in advance.

Then, the control unit 140 receives the captured image around the forklift 10 captured by the photographing unit 110, analyzes the transferred captured image, and detects an object included in the captured image, and the detected object is a person, After confirming whether it is one of an object and an identification marker, the alert generating unit 130 is controlled to issue a warning notification according to a preset response process for each detected object.

The control unit 140 divides warning notifications into non-warning steps, first warning steps, and second warning steps to set a response process, and when an object is detected from a photographed image, if the driving state of the forklift is in a stop state, the non-warning step A warning notification is performed, and when an object is detected from the captured image, if the driving state of the forklift truck is running, a warning notification of the first warning step is performed, and when an object is detected from the captured image, when the driving state of the forklift truck is rotating, the second warning step can be controlled to perform warning notifications. Here, the non-warning step is performed by displaying a corresponding object detection image on the display unit, the first warning step is performed by warning notification using a speaker and a warning light, and the second warning step is a warning notification using a speaker and warning light. It can be performed by stopping the operation of the forklift at the same time.

For example, when an identification marker indicating an intersection, a work area, a cargo loading area, etc. is detected from a captured image, the controller 140 gives a non-warning warning so as to recognize that the driver needs attention since other workers or obstacles may exist. You can perform a warning notification of the step.

As described above, according to the present invention, by issuing a warning by distinguishing people, objects, and identification markers, it is possible to prevent collisions between forklifts and objects located in the workplace as well as collisions between forklifts and people, and also, regardless of distance, people When this is recognized, an immediate warning is given to the driver, so that safety accidents such as human accidents can be prevented in advance.

FIG. 2 is a configuration diagram showing a detailed configuration of the image-based peripheral monitoring device shown in FIG. 1 .

As shown in FIG. 2 , the control unit 140 of the image-based peripheral monitoring device 100 according to an embodiment of the present invention includes an image collection unit 141, an image recognition unit 143, and a distance measurement unit 144. , It may be configured to include a speed measuring unit 145 and a processor 146, and may further include a captured image DB 142 for storing captured images collected through the image collecting unit 141.

In detail, the image collection unit 141 collects images of the forklift's surroundings (hereinafter, referred to as "photographed images") from the photographing unit 110 . At this time, the photographing unit 110 transfers the captured image to the image collecting unit 141 in real time, the image collecting unit 141 stores the delivered captured image in the DB 142, and stores the data stored in the DB 142. When the storage time exceeds a predetermined capacity, it may be set in advance to perform a function of automatically deleting from the oldest order. Here, the image reception of the image collection unit 141 may be performed in real time while the forklift is operating.

Then, the image recognition unit 143 detects an object from the collected captured images, and if the detected object is a person, the object detection image including the person is immediately displayed on the display unit 120 under the control of the processor 146. can do.

This image recognition unit 143 may be made through object recognition modeling using deep learning-based artificial intelligence learning. Here, object recognition modeling using deep learning-based artificial intelligence learning first stores information about the proportions and various postures of the human body in advance, and detects a region similar to the stored human body information in a photographed image to perform human body detection learning. . In addition, object recognition modeling using deep learning-based artificial intelligence learning stores identification marker information for work lines that indicate intersections, work areas, and cargo loading areas in advance, and identifies areas similar to the saved identification marker information in captured images. By detecting, identification marker detection learning is performed. Similarly, object recognition modeling using deep learning-based artificial intelligence learning also performs learning for object detection.

That is, the image recognizing unit 143 detects the human body region using the object recognition pattern information learned from the captured image, stores the detection result, and uses it for learning again, thereby improving the accuracy of detecting the human body region.

In addition, the image recognition unit 143 checks the identification markers engraved on the floor of the workplace to recognize intersections and areas requiring attention (cargo loading area, work area, etc.), and transmits object recognition results to the processor 146, The processor 146 may output an identification marker recognition result to the display unit 120 to issue a warning notification to alert the driver.

As such, the image recognition unit 142 analyzes the transmitted captured image to perform object detection, and when an object is detected, the image recognition unit 142 transmits the object detection image to the distance measurement unit 144 and the processor 146, respectively.

Then, the distance measurement unit 144 calculates the distance from the object detection image transferred from the image recognition unit 143 to the object, and if the calculated distance is less than or equal to a preset threshold value, the distance is transferred to the processor 146 . In this case, the preset threshold may be displayed as a danger area (see D in FIG. 5 ) in the image output from the display unit.

That is, the distance measuring unit 144 transfers object distance information to the processor 146 to perform a first or second warning step when an object is detected within the set distance, and the driver can pay attention when an object is detected outside the set distance. It can be output to the display unit so that it can be recognized, and the screen flickers so that it can be recognized.

In addition, the speed measurement unit 145 calculates the driving speed, conering speed, etc. of the forklift, and determines driving conditions including stop, low-speed operation, rapid acceleration operation, and rotational operation using the calculated driving speed.

The speed measurement unit 145 may be implemented as a 9-axis gyro sensor and transmits the determined driving state information to the processor 146 .

In addition, the processor 146 outputs the captured image transmitted in real time from the image collection unit 141 to the display unit 120, or when the object detection image is transmitted from the image recognition unit 143, the object detection image is different from the existing screen. It can be output in color to draw the driver's attention.

In addition, when the photographing unit 110 is composed of a plurality of cameras, the processor 146 divides the screen of the display unit 120 as many as the number of cameras, divides the captured images for each camera, and outputs them in each divided area. can do.

In addition, the processor 146 is configured with a plurality of cameras in the photographing unit 110, and when the plurality of cameras take pictures of the front, left room, right room, and rear, respectively, the processor 146 synthesizes the captured images of the cameras in each area to create an around view. An image may be generated and output to the display unit 120 .

In addition, the processor 146 may control the configuration of a screen output to the display unit 120 in consideration of the driving state information of the speed measuring unit 145 . For example, if the driving state information of the speed measuring unit 145 is a stationary state, the processor 146 controls the images taken by each camera to be output on each split screen, and if the driving state information of the speed measuring unit 145 is a driving state, the driving direction The image taken by the camera corresponding to (for example, a front captured image when moving forward and a rear captured image when moving backward) may be controlled to be displayed on the screen of the display unit 120 .

In addition, the processor 146 may classify the warning notification into three steps based on the object recognition result and the driving state information. Here, step 3 may include a non-warning step, a first warning step, and a second warning step.

For example, the processor 146 determines the non-warning step regardless of the object recognition result, that is, if the driving state information is in a stationary state even if the object is recognized as a result of the object recognition, and outputs an object detection image to the display unit 120. and control not to perform any alarms.

For another example, the processor 146 controls to perform a first warning step when object recognition is confirmed as a result of object recognition and the driving state information is a driving state. At this time, the processor 146 checks whether the driving state is low-speed operation or higher than the standard speed, and then controls the alarm generating unit 130 to issue an alarm for 0.8 seconds in case of low-speed operation and for 0.2 second in case of operation in higher than standard speed. can

For another example, the processor 146 controls to perform a second warning step when object recognition is confirmed as a result of object recognition and the driving state information is a turning state. At this time, the processor 146 may control to immediately perform a warning notification through the alarm generating unit 130 when recognition of a person is confirmed as a result of object recognition.

In addition, the processor 146 increases the output size of the warning sound or shortens the blinking interval of lights as the distance between the rear of the forklift and the detected object increases, and the calculated driving speed of the forklift exceeds the reference speed to increase the driver's to draw the attention of

As described above, according to the present invention, a photographed image is analyzed through the image recognition unit 143 to detect an object including an identification marker, an object, and a person, and after confirming which object among the identification marker, object, and person is detected, When an identification marker or object is detected, a photographed image of the identification marker is output to the display unit 120, and when a person is detected, a warning notification may be immediately performed through the alarm generating unit 130.

Accordingly, the present invention can reduce the increase in driver's fatigue by frequently performing a warning notification to the driver whenever an object is detected, and can prevent a safety accident in advance by performing an immediate warning notification when a person is recognized.

In addition, the present invention can protect the forklift from surrounding dangerous objects or dangerous environments, prevent safety accidents that may occur when the forklift is moving backward, and allow the driver of the forklift to accurately recognize rear obstacles during work. This can improve work efficiency.

FIG. 3 is an exemplary diagram for explaining object recognition modeling applied to the image recognition unit shown in FIG. 2 .

As shown in (a) of FIG. 3, in order to build object recognition modeling in the image recognition unit 143, the present invention takes collected images as input and deep learning that uses the location of a person in the captured image as an output. Learning can be repeated. At this time, an output value can be designated by manually specifying (P) the position of the person in the captured image, and object recognition modeling can be built by performing deep learning learning using this as an input value again.

To this end, in the present invention, as shown in (b) to (e) of FIG. 3, information (P1, P2, P3, P4, P5) on the proportions and various postures of the human body is stored in advance, and in the captured image Deep learning is performed by detecting a region similar to the stored human body information.

FIG. 4 is an exemplary diagram illustrating an example of detecting a person recognized through object recognition modeling of the image recognition unit shown in FIG. 2 .

As shown in (a) to (f) of FIG. 4, in an embodiment of the present invention, the image recognition unit 143 is configured in various postures from captured images through object recognition modeling built by deep learning. It is possible to recognize the shape (P) of a person in .

For example, the image recognition unit has a shape of a person approaching within a preset distance (a), a shape of a person showing only a part of the person's body (b), a shape of a person approaching holding an object (c), a shape of a person lying or sitting (d, e ), and a human shape (f) in which only the upper body of a person is visible while being covered by an object, the human shape (P) in various postures can be recognized as a person.

FIG. 5 is an exemplary diagram illustrating a monitor warning screen output according to a control signal of the processor shown in FIG. 2 .

As shown in (a) of FIG. 5, the display unit 110 according to the embodiment of the present invention outputs a rear photographed image of the rear on the screen S1, and the output rear photographed image has a preset distance. (threshold) (D) is indicated by a red line.

In the present invention, the inside of the red line (D) is designated as a danger area, and a warning notification is performed for an object approaching within a preset distance, that is, the danger area.

As shown in (b) of FIG. 5, when a person (P) approaches within a preset distance (D) while passing by a forklift, the screen (S2) of the display unit ( ) is displayed as soon as the person's approach is detected. It is possible to focus the driver's attention by different colors or different colors and flickering at the same time. In this case, not only the output of the warning screen on the display unit but also the voice warning through the speaker may be simultaneously performed.

6 is a flowchart illustrating a method for monitoring the surroundings of a forklift of an image-based peripheral monitoring device according to an embodiment of the present invention.

As shown in FIG. 6 , a method for monitoring the surroundings of a forklift of an image-based peripheral monitoring device according to an embodiment of the present invention includes an image capturing step (S610) of generating a photographed image by photographing the surroundings of the forklift by a photographing unit; An image collection step of collecting and storing captured images (S620); An image recognition step of performing object detection by analyzing the collected captured images (S630); a distance measurement step of calculating a distance to the detected object (S640); A speed measuring step (S650) of determining the driving state by calculating the driving speed of the forklift; a warning stage determination step of determining one of a non-warning stage, a first warning stage, and a second warning stage based on the detected object information, the calculated distance information, and the determined driving state information (S660); and a warning notification step (S670) of performing a warning notification according to the determined warning stage.

In detail, the image capturing step (S610) is performed by a photographing unit installed at the rear of the forklift to photograph the rear to generate a rear photographed image. Here, the photographing unit is installed in the central part of the rear surface of the forklift so that the entire rear surface can be photographed. And, the photographing unit may be additionally installed in the forklift so that at least one or more of the front, left room, and right room may be photographed as well as the rear.

Then, the rear photographed image captured by the photographing unit is transferred to the controller, and the controller performs an image collection step of collecting the transmitted rear photographed image.

Accordingly, the image collection step (S620) is performed by the image collection unit of the controller receiving the captured rear image in real time and storing it in a database, and at the same time transmitting the rear image to the image recognition unit.

Accordingly, in the image recognition step (S630), the image recognition unit of the control unit analyzes the transmitted rear photographed image to perform object detection through object recognition modeling by deep learning learning, and when an object is detected, to calculate the distance of the detected object. The object detection image is transmitted to the distance measurement unit, and the distance measurement step is performed. On the other hand, if there is no object detection, it is made by transferring the rear photographed image to the processor. At this time, the processor controls the rear photographed image to be output on the screen of the display unit when the object is not detected.

Then, in the distance measuring step (S640), the distance to the object included in the object detection image transmitted from the image recognition unit is calculated, and when the calculated distance exceeds a preset distance, the distance is transmitted to the processor, and the processor This is achieved by outputting an object detection image in a different color on the screen of the display unit and not performing a separate warning notification. However, the driver can recognize that there is a person or object approaching the forklift through the changed screen color, and can take precautions in advance.

Subsequently, the speed measurement step (S650) is performed by calculating the driving speed of the forklift and determining whether the current driving state is stop, driving at or above the standard speed, and rotating.

Then, the warning level determination step (S660) and the warning notification step (S670) are performed by combining the detected object information, the calculated distance information, and the driving state information to select a warning step and performing the warning notification.

For example, if the detected object information is a person and the driving state information is a stationary state, a warning through the screen of the display unit regardless of the calculated distance information, that is, a warning output with a different screen color of the object detection image carry out Through this, the driver can prevent human accidents in advance by recognizing in advance that a person is approaching the forklift and proceeding with caution.

For another example, if the detected object information is a person and the driving state information is a driving state or a driving state, regardless of the calculated distance information, a warning through the screen of the display unit and a warning notification using a speaker and a warning light are issued. Perform the first warning step to perform. At this time, if the driving speed is low, a warning notification may be performed for 0.8 seconds, and if the driving speed is higher than a preset critical speed, a warning notification may be performed for 0.2 seconds. Through this first warning step, safety accidents can be prevented in advance by giving attention not only to the driver but also to people approaching the forklift truck at the same time.

For another example, if the detected object information is an identification marker, the identification marker information is an intersection, a cargo loading area, a work area (an area requiring attention), and the driving state information is a driving state, the driver is warned using a speaker and a warning light. notification can be performed.

As described above, according to the image-based monitoring method for the forklift surroundings according to the present invention, the photographing unit is installed on the rear of the forklift and the photographed image of the rear is output and provided to the monitor to secure the driver's work efficiency by securing a view according to rear driving. In addition, safety accidents that may occur during work can be prevented in advance by recognizing a person approaching from the rear and providing a warning notification.

The above description is merely an example of the technical idea of the present invention, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10. Forklift 11. Body
12. Mast assembly 13. Fork
100. Surrounding monitoring device 110. Shooting unit
120. Display unit 130. Alarm generating unit
140. Control

Claims (8)

A device for monitoring the surroundings of a forklift truck comprising a vehicle body equipped with a driver's seat and a pair of forks movable up and down in front of the vehicle body,
a photographing unit installed on the rear of the vehicle body and outputting a photographed image by photographing the rear of the forklift;
a display unit installed adjacent to the driver's seat and displaying the photographed image on a screen;
It is installed inside the vehicle body, detects an object included in the captured image by performing image recognition of the captured image, and confirms the type of the detected object, the distance to the object, and the driving state to confirm the non-warning step, first a control unit controlling to execute one of a warning step and a second warning step; and
An alarm execution unit installed inside or outside the vehicle body and performing a warning notification through at least one of sound and light according to the control of the control unit;
The object includes at least one of an intersection, an identification marker indicating a work area and a cargo loading area, an object and a person,
The non-warning step is a step set to display an object detection image including an object on the display unit when the driving state is stopped, and the first warning step is when the driving state is driving and the detected object is a person. A step set to execute the display unit and the alarm execution unit, and the second warning step is a step set to execute the display unit and the alarm execution unit when the detected object is a person when the driving state is in a turn. video-based peripheral monitoring device. The method of claim 1, wherein the photographing unit,
It consists of at least one camera installed at the center of the rear surface of the forklift, wherein the camera has a fish-eye lens having a 180 ° field of view and FHD resolution. The method of claim 2, wherein the photographing unit,
Image-based peripheral monitoring device, characterized in that it further comprises at least one camera installed in the center of the front of the forklift, wherein the camera is provided with a fish-eye lens having a 180 ° field of view and FHD resolution. According to claim 1,
The display unit includes a monitor that blinks the screen for a predetermined time while displaying different colors of the screen when the first warning step and the second warning step are executed,
The alarm generating unit comprises a speaker outputting sound and a warning light outputting light and sound simultaneously when the first warning step and the second warning step are executed. The method of claim 1, wherein the control unit,
an image collecting unit for collecting and storing the photographed images taken by the photographing unit;
an image recognizing unit that detects an object included in captured images collected through deep learning-based object recognition modeling and outputs an object detection image in which the object is detected;
a distance measuring unit that calculates a distance to an object included in the collected captured images;
a speed measurement unit that calculates a traveling speed of the forklift and determines one of a stationary state, a running state, and a rotational driving state based on the calculated traveling speed; and
A control signal is generated to perform one of the non-warning step, the first warning step, and the second warning step by combining the detected object information, the calculated distance information, and the determined driving state information, but if the detected object is an identification marker, driving Regardless of the state, the object detection image is displayed on the display unit, but the control signal of the non-warning step is generated so that the screen color is changed and blinked, and if the detected object is an object, the non-warning step is performed when the driving state is stopped When the driving state is driving or turning, if the distance to the object according to the calculated distance information is within a preset distance, it is determined that the entry into the danger zone is entered and a control signal is transmitted to execute the second warning step. An image-based peripheral monitoring device comprising a; The method of claim 5, wherein the speed measuring unit,
A 9-axis gyro sensor is provided to calculate the speed and direction of movement, but if the driving speed of the forklift is 0, it is in a stopped state, and if the driving speed of the forklift is higher than the reference speed, it is in a running state, and rotation while the driving speed of the forklift is calculated Image-based peripheral monitoring device, characterized in that for each determination in the middle of the rotation operation state. The method of claim 6, wherein the processor,
When the detected object is a person, if the current driving state is stopped, a control signal is transmitted to the display unit to execute a non-warning step of displaying an object detection image on the display unit regardless of distance information,
If the detected object is a person, if the current driving state is driving or turning and the distance to the object according to the calculated distance information exceeds a preset distance, a control signal is transmitted to the display unit to execute the non-warning step,
When the detected object is a person, the display unit sends a control signal to execute the first or second warning step when the current driving state is driving or turning and the distance to the object according to the calculated distance information is within a preset distance. and an image-based peripheral monitoring device, characterized in that each is transmitted to the alarm generating unit. The method of claim 7, wherein the processor,
As the distance between the rear of the forklift and the detected object gets closer, and as the calculated traveling speed of the forklift exceeds the reference speed, the sound output of the alarm generating unit is increased and the blinking interval of the light is shortened to perform warning notification. Image-based peripheral monitoring device, characterized in that for.

Images