KR102231244B1 - Bus information provision apparatus and bus information provision method - Google Patents

Abstract

According to the present invention, provided is a bus information guide device, which comprises: a transmission channel through which information including vehicle information is transmitted; an output unit outputting the vehicle information as visual information or sound information; a camera acquiring image information; an object detection unit detecting an object including a user from the image information; a face detection unit detecting the face of the object; a distance measuring unit obtaining an estimated distance between the object and the camera using information related to the size of the face detected through the face detection unit; an object tracking unit tracking the object; an object size measurement unit tracking the object and measuring a change in the detected object size; and a behavior determination unit using the object size and the estimated distance to determine the behavior of the user.

Description

Bus information provision apparatus and bus information provision method}

The present invention relates to a bus information guide device and a bus information guide method.

Bus Information Terminal (BIT) is installed at a bus stop and can perform the function of notifying the expected time of bus arrival. The bus information guide device of the present invention may include a bus guide.

The bus information terminal may be further extended from the basic function of notifying the expected time of bus arrival and further perform other functions. For example, in the cited literature, a camera may be further mounted on the bus information terminal. The camera captures an image of a road surrounding a bus stop and a surrounding image and provides it to a bus information guide. Using the surrounding images captured by the camera, it is possible to monitor whether a security incident occurs.

Since the bus information terminal is always exposed to the outside, it is frequently damaged. In fact, drunk people are often damaged by force.

Republic of Korea Patent Registration 10-1957637, illegal parking control system using bus information guide and its processing method

The present invention provides a bus information guide device capable of preventing damage to the bus information guide device.

The present invention further expands the usability of the bus information guide device, so that a user can use the bus information guide device more conveniently.

A bus information guide apparatus according to the present invention includes: a transmission channel through which information including vehicle information is transmitted; An output unit for outputting the vehicle information as visual information or sound information; A camera that acquires image information; An object detection unit that detects an object including a user from the image information; A face detection unit that detects the face of the object; A distance measuring unit that obtains an estimated distance between the object and the camera using related information of the face size detected through the face detection unit; An object tracking unit that tracks the object; An object size measuring unit that tracks the object and measures a change in the detected object size; And a behavior determination unit that determines the user's behavior by using the object size and the estimated distance.

When the size of the box indicating the object size suddenly changes and the estimated distance is close, it is determined that the user is deliberately damaging, and the user may be warned through the output unit.

The object detection unit may be a first learning model previously learned by artificial intelligence.

The face detection unit may be a second learning model previously learned by artificial intelligence.

The second learning model may be trained to distinguish between wearing/non-wearing a mask from an image.

를 이용하여 거리를 추정하고, 여기서, d는 객체와 카메라 간의 거리이고, y2, y1은 안면검출박스의 위치정보이고, a와 p는 환산변수일 수 있다.The distance measuring unit,

The distance is estimated by using, where d is the distance between the object and the camera, y2 and y1 are location information of the face detection box, and a and p may be conversion variables.

A counting unit for determining the number of users by counting the number of objects may be included.

A danger zone entry determination unit may be included to determine whether the object has entered the danger zone.

를 이용하여 상기 객체크기의 변화를 측정하고, T는 측정대상이 되는 전체 프레임이고, t는 각 프레임이고, x와 y는 박스의 좌표일 수 있다.The object size measuring unit,

The change in the size of the object is measured using, T is the entire frame to be measured, t is each frame, and x and y may be coordinates of the box.

The method for guiding bus information according to the present invention includes detecting an object in an image captured by a camera; Entering a low power mode when the object is not detected, and performing face detection and object tracking when the object is detected; Obtaining distance information between the object and the camera using the detected facial information; Tracking the object and finding out a change in size of the object; And if the size change of the object is large and the distance information is close, it may be determined that the user is deliberately damaging and warned.

When the mask is not worn in the face detection, warning of wearing the mask may be performed.

Using the object tracking, it is possible to determine whether an object enters a danger zone, and to warn a user.

According to the present invention, it is possible to prevent intentional damage to the bus information guide device.

According to the present invention, it is possible to prevent an accident before boarding a passenger.

According to the present invention, the overall safety of society can be further improved.

1 is a block diagram of a bus information guide device according to an embodiment.
2 is a flow chart showing a bus information guide method according to an embodiment.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. The spirit of the present invention is not limited to the following embodiments, and those skilled in the art who understand the spirit of the present invention easily propose other embodiments included within the scope of the same idea by adding, changing, deleting, and adding components. Although it may be possible, it will be said that this is also included within the scope of the inventive concept.

1 is a block diagram of a bus information guide device according to an embodiment.

Referring to FIG. 1, the bus information guide apparatus according to the present invention includes a transmission channel 2 through which information is transmitted inside a device. Vehicle information of vehicles including a bus may be input to the transmission channel. The vehicle information may be controlled by the controller 5 and output through the output unit 3. The output through the output unit 3 may include visual information using video and sound information using audio. Bus passengers can acquire bus-related information by using the visual information and the acoustic information.

A controller 5 for controlling the overall operation of the bus information guide device, and a memory 4 for temporarily or permanently storing necessary information may be further included. In addition, various devices and parts may be further provided to perform the fundamental operation of the bus information guide device. For example, a wired/wireless communication unit that receives vehicle information and transmits current information to the base station may be included, although not shown, of course.

A camera 6 may be provided in the bus information guide device of the embodiment. Images can be acquired using the camera. By analyzing the acquired image, protecting the device from intentional damage (warning to stop intentional damage), for the safety of users (warning to stop abnormal behavior, warning to stop entering the roadway), and for public safety (wearing a mask) Warning, abnormal behavior stop warning), and energy saving operation (low power entry) can be performed. The features of the present invention can be clearly revealed by each of the above functions, and the usability of the camera 6 mounted on the bus guide device can be greatly improved.

An object detection unit 1 may be included in the bus information guide device of the embodiment. The object detection unit may use a deep learning-based image object detection model such as YoLo, SSD, and R-CNN, or a modification thereof as a technology used for object detection. The object detection unit 1 may detect as'person' when the whole body of a person is seen in a certain ratio (eg, 50%) or more within the angle of view of the image.

As a result of detection, a pair of coordinates corresponding to the detection area per person in the field of view is returned, and multiple people can be detected. This can be expressed as Equation 1.

Here, P(0,W,0,H) is a set of pixels of the entire screen input to the camera 6, and the H function may refer to an object detection neural network that detects the presence area of a person within the angle of view.

In order to perform the function, the object detection unit 1 may accommodate a first learning model that has been previously learned by artificial intelligence using deep learning.

When an object is not detected by the object detection unit 1, the controller 5 may operate in a power saving mode. In the power saving mode, a function such as stopping video output may be performed.

When an object (person) is detected by the object detection unit 1, object tracking and face detection may be performed on the coordinates of the area of the object.

First, the operation of the face detection unit 11 for detecting the face will be described. The face detection unit 11 may detect a feature of a user's face. For example, it may be detected whether the mask is normally worn.

A second learning model previously learned by artificial intelligence using deep learning may be mounted on the face detection unit 11. In order to learn the second learning model, a deep learning-based object detection neural network that considers both wearing/non-wearing states of a mask may be trained using an image data set including a mask wearer and a non-wearer. In this case, it is possible to classify more cases by learning together even if the mask is not properly worn.

The second learning model may output probability information on whether or not the detected face mask is worn. The operation of the second learning model can be expressed by Equation 2 below.

는 각각 안면 검출 영역의 4개 좌표(검출된 안면영역) 및 마스크 착용여부를 알리는 식별자를 의미할 수 있다. Here, P(x1,x2,y1,y2) is a set of pixel values in an image belonging to the object detection area of a person, and the MF function may mean a face detection neural network that checks whether a mask is worn.

May denote the four coordinates of the face detection region (detected face region) and an identifier indicating whether or not the mask is worn.

When it is determined that the object is not wearing the mask, guidance on wearing the mask may be performed through the output unit 3 under the control of the controller 5.

Information on the detected facial area may be input to the distance measuring unit 13. The distance measuring unit 13 may perform'facial size-based distance estimation'. The distance estimation value (d) of the distance measuring unit 13 can be obtained as in Equation (3).

Where a and p are As a distance conversion variable, it can be adjusted according to the specifications and installation location of the camera installed inside or outside the bus information guide device. Here, a is a proportional factor to perform conversion adjustment, and p is an exponential factor to perform conversion adjustment. d is a distance, and y may be location information of the detected facial area (box). Preferably, the camera may be installed inside the bus information guide device.

Using the distance information estimated by the distance measuring unit 13, it is possible to determine whether a user has approached the bus information guide device. Furthermore, by using the distance information estimated by the distance measuring unit 13, it can be used to determine whether the'deliberate damage behavior'.

As described above, when an object (person) is detected by the object detection unit 1, object tracking may be performed on the coordinates of the area of the object.

The operation of the object tracking unit 21 that executes the object tracking will be described.

The object tracking unit 21 may assign a temporary identifier to the detected object (person). The object tracking unit 21 may track the movement of the object within the angle of view. By tracking the movement of objects within the angle of view, at least one of counting the number of users, determining whether to enter a vehicle, and performing abnormal behavior may be utilized to grasp.

The action of the object tracking unit 21 can be expressed by Equation 4 below.

Here, the T function is an object tracking algorithm, and a multiple object tracking (MOT) algorithm can be applied. In an embodiment, Simple Online Realtime Tracking (SORT) or Deep SORT may be used. ID may be an identifier of an object. Movement of the object may be tracked by the object tracking unit 21. The object tracking unit may provide each object in an x-coordinate range and a y-coordinate range.

The ID defined by the object tracking unit 21 may be counted by the counting unit 25. The user's counting information may be stored in the memory 4. By using the number of users as information, it can be used as information to grasp traffic flow.

With respect to the object being moved by the object tracking unit 21, the dangerous area entry determining unit 24 may determine whether a user has entered a dangerous area such as a roadway. Specifically, coordinate information (eg, area A) indicating a roadway area may be registered in advance within the area of the image captured by the camera 6. When an object to be tracked later enters the corresponding area (area A), the danger area entry determination unit 24 may warn the user of the dangerous fact through the output unit 3.

The information provided by the object tracking unit 21 may include box information of the x-coordinate range and y-coordinate range of each object. Using the box information, the object size measuring unit 22 may measure the size of the object. Specifically, the size of the object measured by the object size measuring unit 22 may be estimated through boxes of the x-coordinate range and y-coordinate range of each object.

The size of the object may vary. The change in the size of the object may be affected by a change in the distance between the object and the camera 6 and a change in the behavior of the object. For example, when an object approaches a camera, the size of the object can be measured. For example, when an object swings its arm or performs an action such as a kick, the size of the object can be largely measured. This is because the size of the box represented by the x-coordinate range and y-coordinate range increases due to the user's motion. If the box size increases due to the user's motion, it can be defined as an abnormal behavior.

As an example of defining the abnormal behavior, Equation 5 below may be applied.

According to Equation 5, the arithmetic mean of the amount of change in the box size between each t-th frame and the previous frame is calculated as much as T frames, and when the arithmetic mean exceeds a certain level, it can be determined that the user is acting radically.

The distance estimated by the distance measuring unit 13, the object size measured by the object size measuring unit 22, and the determination result according to Equation 5 may be input to the behavior determination unit 23. The behavior determination unit 23 may more accurately determine the state of the user.

For example, as a result of the abnormal behavior determination in Equation 5, it is determined that a specific person within the angle of view is causing the abnormal behavior, and as well as the distance estimation in the distance measuring unit 13, it is determined that the user approaches the camera. Can be. In this case, it can be determined that the user is doing the act of damaging the bus information guide device 1. In this case, a warning can be issued through the output unit 3.

As another example, as a result of the abnormal behavior determination in Equation 5, it is determined that a specific person within the angle of view is causing the abnormal behavior, but it may be determined that the user does not approach the camera in the distance estimation by the distance measuring unit 13. . In this case, it can be determined that the user is acting unrelated to the act of damage to the bus information guide device 1. In this case, observation and filming may be continued, the fact of violence may be notified to the outside, or the abnormal behavior may be stopped by judging the degree of continuation of the abnormal behavior.

The abnormal behavior that can be judged by Equation (5) cannot recognize a specific type of behavior as in deep learning-based pose estimation techniques. However, there is an advantage in that it is possible to identify abnormal behavior only by simple object tracking without applying a separate complex artificial intelligence model. This method is very suitable in the case of a bus information guide device that lacks computational resources.

A method of guiding bus information according to an embodiment will be described with reference to FIG. 2. The bus information guiding method of the embodiment may be operated using the bus information guiding apparatus described above. In the description of the bus information guiding method according to the embodiment, descriptions overlapping with those described in the bus information guiding apparatus will be omitted.

The following effects can be obtained through the bus information guidance method of the embodiment. First, to protect the device from intentional damage (warning to stop intentional damage), second, to ensure the safety of users (warning to stop abnormal behavior, warning to stop entering the vehicle), and third, to protect public safety (warning to wear a mask, Abnormal behavior stop warning), and fourth, energy saving operation (low power entry) is possible.

When an image is input (S1), an object may be detected first (S2). The object is a representative example of a person. If the object is not detected, the low power mode may be entered (S3). In the low power mode, only basic functions may be performed and other functions may be turned off. Through this, energy consumption can be reduced.

When the object is detected, detecting the face (S4) and tracking the object (S5) may be performed.

In the face detection step (S4), it is determined whether or not the mask is worn (S6), and when the mask is not worn, a mask wearing guide may be performed (S7) and returned. In the face detection step (S4), the distance between the object and the device may be estimated using the detected face information (S8). The face-based distance estimation method has been described in detail in the bus information guide device.

The distance estimated through the above process may be used later as a factor for determining the intentional damage intention of the object.

When the object tracking step S5 is performed, the number of users may be determined by counting objects (S10). The number of users can be used later as information such as traffic demand surveys.

When the object tracking step (S5) is performed, the object size change may be measured (S11). The object size may be the size of the box to which the object belongs. The box size may suddenly change significantly when a user swings his or her feet and legs (abnormal behavior). It can be said that a sudden change in the size of the box is likely to lead to abnormal behavior or intentional damage to the device.

The object size information and the estimated distance information estimated in the distance estimation (S8) between the object and the device are used as information in the user behavior determination step (S12). In the user behavior determination step (S12), when the object size information indicates an abnormal behavior and the user approaches the device in the estimated distance information, it may be determined that the user intentionally damages the device. In this case, a warning is made to stop the willful act of damaging the device (S13), and it can be returned (case B).

In the user behavior determination step (S12), if the object size information indicates an abnormal behavior and the user is not close to the device based on the estimated distance information, it may be determined that the user performs the abnormal behavior (S14). In this case, a warning can be made to stop the abnormal behavior. Here, the abnormal behavior may include a physical collision between people, a case where a person inflicts harm to another person, and the like. The abnormal behavior may be determined as abnormal behavior when the abnormal behavior continues by additionally determining whether the behavior continues for a certain period of time (C case).

The user behavior determination step (S12) may be returned immediately if there is no abnormal behavior in the object size information and the user is not close to the device in the estimated distance information (case A).

When the object tracking step S5 is performed, it may be determined that the object enters the roadway (S15). To determine whether to enter the roadway, a predetermined roadway area may be displayed on the image. Through this, even when a small storage capacity is used, sufficient operational performance can be drawn. Of course, if the storage capacity and computational capacity are allowed, it is also possible to find out dangerous areas such as roadways in real time through image analysis. In addition to the above roadway, it is also possible to judge the entry into the danger zone and warn the user.

When it is determined that the user has entered a dangerous area such as a roadway, it is possible to warn of entering the roadway (S16) and return.

The bus information guide device of the present invention can protect the device by preventing intentional damage.

In the present invention, power saving, number of users, danger warning, device damage warning, abnormal behavior warning, and mask wearing guide are possible by combining only face detection and object detection.

1: object detection unit
11: face detection unit
21: object tracking unit

Claims (10)

A transmission channel through which information including vehicle information is transmitted;
An output unit for outputting the vehicle information as visual information or sound information;
A camera that acquires image information;
An object detection unit that detects an object including a user from the image information:
A face detection unit that detects the face of the object;
A distance measuring unit that obtains an estimated distance between the object and the camera using related information of the face size detected through the face detection unit;
An object tracking unit that tracks the object;
An object size measuring unit that tracks the object and measures a change in the detected object size; And
Using the object size and the estimated distance, a behavior determination unit for determining the user's behavior is included,
When the size of the box indicating the object size suddenly changes and the estimated distance is close, it is determined that the user is deliberately damaging, and the bus information guide device warns the user through the output unit. The method of claim 1,
The object detection unit is a bus information guide device that is a first learning model previously learned by artificial intelligence. The method of claim 1,
The face detection unit is a second learning model that has been previously learned by artificial intelligence, and the second learning model is a bus information guide device that is learned to distinguish between wearing/non-wearing a mask from an image. The method of claim 1,
The distance measuring unit,

Estimates the distance using, where d is the distance between the object and the camera, y2, y1 are the location information of the face detection box, and a and p are conversion variables. The method of claim 1,
Bus information information device including a counting unit to determine the number of users by counting the number of the object. The method of claim 1,
A bus information guide device including a danger zone entry determination unit that determines whether the object has entered the danger zone. The method of claim 1,
The object size measuring unit,

The change in the size of the object is measured using, T is the entire frame to be measured, t is each frame, and x and y are the coordinates of the box. Detecting an object in the image captured by the camera;
Entering a low power mode when the object is not detected, and performing face detection and object tracking when the object is detected;
Obtaining distance information between the object and the camera using the detected facial information;
Tracking the object and finding out a change in size of the object; And
When the size of the object is large and the distance information is close, it is determined that the user is deliberately damaging, and a warning is performed. The method of claim 8,
In the case of not wearing a mask in the face detection, a method of providing a bus information warning is performed to warn of wearing a mask. The method of claim 8,
A bus information guidance method in which it is determined whether an object enters a dangerous area and warns a user using the object tracking.

Images