KR102189262B1 - Apparatus and method for collecting traffic information using edge computing - Google Patents

Abstract

According to one aspect of one embodiment of the present disclosure, provided is an edge computing device comprising: an input interface obtaining an input video photographed by a camera; at least one first processor; and a communication unit communicating with a traffic information collection server. The at least one first processor: inputs an input frame having a frame rate lower than a frame rate of an input video into a first machine learning model, and obtains identification information and speed information of at least one object detected in each frame from the first machine learning model; detects an event of detecting an object of interest whose speed exceeds a first reference value based on the identification information and speed information of each of the at least one object outputted from the first machine learning model; and transmits the event information and the input video to the traffic information collection server using the communication unit.

Description

Traffic information collection device and method using edge computing {Apparatus and method for collecting traffic information using edge computing}

Embodiments of the present disclosure relate to a traffic information collection system. Embodiments of the present disclosure relate to a computer program that performs an edge computing device of a traffic information collection system, a method for controlling an edge computing device, and a method for controlling an edge computing device. In addition, embodiments of the present disclosure relate to a computer program that performs a traffic information collection server, a traffic information collection server control method, and a traffic information collection server control method.

Current traffic condition measurement related technology uses a method of measuring section speed using a sensor installed on a road surface, a method of using a CCTV (closed-circuit television) image, and the like. However, when the section speed is measured using a sensor installed on the road surface, there is a disadvantage in that the installation and management cost of the sensor is high, and the section speed is provided instead of the speed of the vehicle itself. In the method using CCTV video, there is an inconvenience in that a person must directly view the video in the control system and determine whether a traffic law is violated. In addition, an unmanned enforcement system using drones is also being implemented. Drones were launched over the highway to crack down on traffic violations using drones. In the drone-based method, police officers directly judged whether or not a traffic law was violated by viewing the video captured by the drone, and then contacted the patrol car of a fellow police officer in the vicinity to crack down. The method using a drone has the inconvenience of having to read the video directly by the police officer.

Embodiments of the present disclosure are to provide an apparatus and method for easily collecting traffic information by detecting a vehicle and a speed using a machine learning model using a conventional CCTV image or a drone image.

In addition, embodiments of the present disclosure are to provide an apparatus and method for efficiently analyzing a very large amount of CCTV images and drone images using an edge computing method.

In addition, embodiments of the present disclosure are to provide an apparatus and method for collecting traffic information in real time and accurately measuring traffic information in real traffic flows of hundreds of scales.

According to an aspect of an embodiment of the present disclosure, there is provided an input interface for obtaining an input video captured by a camera; At least one first processor; And a communication unit communicating with a traffic information collection server, wherein the at least one first processor inputs an input frame having a frame rate lower than the frame rate of the input video into a first machine learning model, and the first machine learning Obtaining identification information and speed information of at least one object detected in each frame from the model, and based on the identification information and speed information of each of the at least one object output from the first machine learning model, the speed of the object is controlled. An edge computing device is provided that detects an event of detecting an object of interest exceeding one reference value and transmits the event information and the input video to the traffic information collection server using the communication unit.

In addition, according to an embodiment of the present disclosure, the traffic information collection server includes a second machine learning model that receives the input video and generates identification information and speed information of each of at least one object, and the first machine The learning model may be a machine learning model in which a bypass path is applied to at least one layer of the second machine learning model.

Also, according to an embodiment of the present disclosure, the first machine learning model may receive an input frame having a lower frame rate than the input frame input to the second machine learning model.

In addition, according to an embodiment of the present disclosure, the edge computing device further includes a video buffer for storing the input video, and the communication unit may transmit the input video stored in the video buffer to the traffic information collection server. have.

In addition, according to an embodiment of the present disclosure, the at least one first processor may obtain meta information including at least one or a combination of signal information, stop line information, or lane information, and detect the event. In this case, the event may be detected based on the identification information of each of the at least one object, the speed information, and the meta information.

In addition, according to an embodiment of the present disclosure, the at least one first processor performs a process of generating tracklet information defining an object location and an object region from the input frame, and the tracklet information Acquiring the identification information and the speed information of the at least one object input to the first machine learning model, output from the first machine learning model, and using the identification information and the speed information of the at least one object , A graph model including information on each of the at least one object and relation information with another object is generated, and using the relation information between the graph model and the at least one object, the cluster of the at least one object ( cluster) information can be created.

In addition, according to an embodiment of the present disclosure, the at least one first processor may perform event detection processing of another object in the same cluster by using the result of event detection processing for one object based on the cluster information. Can be done.

In addition, according to an embodiment of the present disclosure, the at least one first processor,

When the event is not detected from the input frame, event non-detection information indicating that the event is not detected is generated, and when the event is not detected, the event non-detection information is transmitted to the traffic information collection server at a predetermined period. Can be transferred to.

In addition, according to another aspect of an embodiment of the present disclosure, the step of obtaining an input video photographed by a camera; Inputting an input frame having a frame rate lower than the frame rate of the input video into a first machine learning model, and obtaining identification information and speed information of at least one object detected in each frame from the first machine learning model; Detecting an event in which the speed of the object exceeds a first reference value, based on identification information and speed information of each of the at least one object output from the first machine learning model; And transmitting the event information and the input video to the traffic information collection server.

In addition, according to another aspect of an embodiment of the present disclosure, a communication unit for communicating with at least one edge computing device; At least one second processor; And an output interface, wherein the at least one second processor includes information on an event in which the speed of an object exceeds a first reference value from the at least one edge computing device through the communication unit. , And an input video, extracting an interest frame section corresponding to the event information from the input video, and inputting the input frame of the frame of interest into a second machine learning model, and each frame from the second machine learning model Acquires identification information and speed information of at least one object detected in, and based on the identification information and speed information of each of the at least one object output from the second machine learning model, the speed of the object sets a second reference value. A traffic information collection server is provided that detects an event in which an object of interest is detected exceeding and outputs the event information and image information of the frame of interest through the output interface.

In addition, according to an embodiment of the present disclosure, the at least one edge computing device includes a first machine learning model that receives the input video and generates identification information and speed information of each of at least one object, and the The first machine learning model may be a machine learning model in which a bypass path is applied to at least one layer of the second machine learning model.

Also, according to an embodiment of the present disclosure, the first machine learning model may receive an input frame having a lower frame rate than the input frame input to the second machine learning model.

In addition, according to an embodiment of the present disclosure, the at least one second processor may acquire meta information including at least one or a combination of signal information, stop line information, or lane information, and detect the event. In this case, the event may be detected based on the identification information of each of the at least one object, the speed information, and the meta information.

In addition, according to an embodiment of the present disclosure, the at least one second processor performs a process of generating tracklet information defining an object location and an object area from the input frame, and the tracklet information Acquiring the identification information and the speed information of the at least one object input to the second machine learning model, output from the second machine learning model, and using the identification information and the speed information of the at least one object , A graph model including information on each of the at least one object and relation information with another object is generated, and using the relation information between the graph model and the at least one object, the cluster of the at least one object ( cluster) information can be created.

In addition, according to an embodiment of the present disclosure, the at least one second processor performs event detection processing of another object in the same cluster by using the result of event detection processing for one object, based on the cluster information. Can be done.

In addition, according to an embodiment of the present disclosure, the communication unit receives event non-detection information indicating that an event has not been detected from the at least one edge computing device, and the at least one second processor The detection information may be output through the output interface.

In addition, according to an embodiment of the present disclosure, when the event is not detected from the frame of interest, the at least one second processor generates event non-detection information indicating that the event has not occurred, The event non-detection information may be output through the output interface.

In addition, according to another aspect of an embodiment of the present disclosure, receiving, from at least one edge computing device, event information including information on an event in which the speed of an object exceeds a first reference value, and an input video; Extracting a frame of interest corresponding to the event information from the input video; Inputting the input frame of the frame of interest into a second machine learning model, and obtaining identification information and speed information of at least one object detected in each frame from the second machine learning model; Detecting an event of detecting an object of interest whose speed exceeds a second reference value, based on identification information and speed information of each of the at least one object output from the second machine learning model; And outputting the event information and image information of the frame of interest.

In addition, according to another aspect of an embodiment of the present disclosure, in a computer program stored in a recording medium, the computer program includes at least one instruction for performing an edge computing device control method when executed by a processor. The program is provided.

Further, according to another aspect of an embodiment of the present disclosure, in a computer program stored in a recording medium, the computer program includes at least one instruction for performing a traffic information collection server control method when executed by a processor, A computer program is provided.

According to embodiments of the present disclosure, there is an effect of providing an apparatus and method for detecting vehicle and speed using a machine learning model and easily collecting traffic information using an existing CCTV image or drone image. have.

In addition, according to embodiments of the present disclosure, there is an effect of providing an apparatus and method for efficiently analyzing a very large amount of CCTV images and drone images using an edge computing method.

In addition, according to the embodiments of the present disclosure, there is an effect of providing an apparatus and method for collecting traffic information in real time and accurately measuring traffic information in real traffic flows on a scale of hundreds of vehicles.

1 is a diagram showing the structure of a traffic information collection system according to an embodiment of the present disclosure.
2 is a diagram showing the structure of a traffic information collection system according to an embodiment of the present disclosure.
3 is a flowchart illustrating a control method of a traffic information collection system according to an embodiment of the present disclosure.
4A is a diagram illustrating a structure of a first processor according to an embodiment of the present disclosure.
4B is a diagram illustrating a structure of a second processor according to an embodiment of the present disclosure.
5 is a diagram illustrating an image processing process according to an embodiment of the present disclosure.
6 is a diagram illustrating a camera calibration process according to an embodiment of the present disclosure.
7 is a diagram illustrating a camera calibration process according to an embodiment of the present disclosure.
8A is a diagram for describing an object detection process according to an embodiment of the present disclosure.
8B is a diagram illustrating an object detection process 514 according to an embodiment of the present disclosure.
9 is a diagram for describing a process of generating tracklet information according to an embodiment of the present disclosure.
10 is a diagram illustrating a structure of a tracklet network according to an embodiment of the present disclosure.
11 is a diagram illustrating a graph model according to an embodiment of the present disclosure.
12 is a diagram illustrating a process of calculating a relationship between objects according to an embodiment of the present disclosure.
13 is a diagram illustrating a graph model generation process and a clustering process according to an embodiment of the present disclosure.
14 is a diagram illustrating a result image according to an embodiment of the present disclosure.

The present specification clarifies the scope of the claims of the present disclosure, and describes the principles of the embodiments of the present disclosure so that those of ordinary skill in the art to which the embodiments of the present disclosure belong may implement the embodiments of the present disclosure. And, the embodiments are disclosed. The disclosed embodiments may be implemented in various forms.

The same reference numerals refer to the same elements throughout the specification. This specification does not describe all elements of the embodiments, and general content in the technical field to which the embodiments of the present disclosure pertain or overlapping content between the embodiments will be omitted. The term'part, portion' used in the specification may be implemented in software or hardware, and according to embodiments, a plurality of'parts' may be implemented as one element or one It is also possible for'to contain multiple elements. Hereinafter, embodiments of the present disclosure and operating principles of the embodiments will be described with reference to the accompanying drawings.

1 is a diagram showing the structure of a traffic information collection system according to an embodiment of the present disclosure.

The traffic information collection system 100 according to an exemplary embodiment of the present disclosure is a system that detects a vehicle in violation of traffic laws by analyzing images collected from a plurality of cameras 150 disposed around a road. The traffic information collection system 100 includes a plurality of cameras 150, a plurality of edge computing devices 110, and a traffic information collection server 130. The camera 150 and the edge computing device 110 may correspond one-to-one, or a plurality of cameras 150 and one edge computing device 110 may correspond.

The camera 150 may correspond to a camera in the form of a CCTV camera or a drone camera. The camera 150 is arranged to have a field of view (FOV) facing a road in which traffic information is to be collected. The camera 150 may be fixed at a predetermined position or may be moved. According to an embodiment, the camera 150 may change the FOV by operations such as pan, tilt, and zoom. In addition, according to an embodiment, the camera 150 may be moved by a drone and may be moved to a predetermined position by a control signal from the edge computing device 110 or the traffic information collection server 130.

The edge computing device 110 receives the image of the camera 150, detects event information such as a regulation speed violation from the input video, and transmits the event information and the input video to the traffic information collection server 130. The edge computing device 110 processes the input video by using the first machine learning model and obtains object identification information and speed information.

The edge computing device 110 includes an input interface 112, a first processor 114, and a communication unit 118. According to an embodiment, the edge computing device 110 may further include a video buffer 116.

The input interface 112 receives an input video captured by the camera 150. The input video may have a first frame rate determined by the camera 150. The input interface 112 may correspond to an input device of a predetermined standard or a communication unit for receiving an input video from the camera 150. The video input through the input interface 112 is transmitted to the first processor 114 and the video buffer 116.

The first processor 114 controls the overall operation of the edge computing device 110. The first processor 114 may include one or more processors. The first processor 114 receives at least one instruction or command and performs a predetermined operation. The first processor 114 receives the input video, detects an object corresponding to the vehicle from at least some frames of the input video, and generates speed information of the detected vehicle. The first processor 114 executes the first machine learning model and generates identification information of the object and speed information of the object. The first processor 114 samples the frames of the input video at a second frame rate lower than the first frame rate of the input video and inputs them to the first machine learning model. Also, when the first processor 114 detects an event in which the speed of the object exceeds the first reference value, it generates event information.

Machine learning is an algorithm technology that classifies/learns the features of input data by itself, and element technology is a technology that simulates functions such as cognition and judgment of the human brain using machine learning algorithms such as deep learning. The machine learning model of the embodiments of the present disclosure may have, for example, a deep neural network structure. The machine learning model may be trained using training data based on one or more nodes and arithmetic rules between nodes. A structure of a node, a structure of a layer, and an operation rule between nodes may be variously determined according to embodiments. The machine learning model includes hardware resources such as one or more processors, memory, registers, summing processing units, parallel processing units, or multiplication processing units, and operates hardware resources based on a set of parameters applied to each hardware resource. To this end, the processor operating the machine learning model may perform a task of allocating hardware resources or resource management processing for each operation of the machine learning model. The machine learning model may have, for example, a structure such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). The first machine learning model and the second machine learning model according to an embodiment of the present disclosure may include a combination of a CNN and an RNN structure.

The event information is information indicating whether an object whose speed exceeds a first reference value or a second reference value is detected. The event information may include information on an object, information on a frame in which an event occurs, and detected speed information. The event information may include event detection information indicating that an event has been detected, and event non-detection information indicating that an event has not been detected. Event information may be generated for each frame. The information on the object may include, for example, information on coordinates, horizontal length, and vertical length of an area corresponding to the object within the frame. The information on the frame in which the event has occurred may include information such as a frame number or a time corresponding to the frame.

The video buffer 116 stores the input video 116. The video buffer 116 may include a predetermined storage medium capable of storing a moving picture. The video buffer 116 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD Memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), It may include at least one type of storage medium among magnetic memory, magnetic disk, and optical disk. The video buffer 116 may store an input video having a first frame rate.

The communication unit 118 may communicate with an external device by wire or wirelessly. The communication unit 118 transmits event information and input video to the traffic information collection server 130.

According to an embodiment of the present disclosure, the first processor 114 extracts only a frame of interest, which is a section in which an event is detected, from the input video and transmits it to the traffic information collection server 130. The edge computing device 110 detects the speed of an object using the first machine learning model and detects an event, so that it does not directly send all information to the traffic information collection server 130, but automatically determines whether to transmit the input video. It determines, extracts only the input video of the frame of interest that needs to be transmitted, and transmits it to the traffic information collection server 130. According to this embodiment, by minimizing the amount of data transmission of the edge computing device 110 and minimizing the data processing resources and storage resources used for storing the input video in the traffic information collection server 130, the system efficiency can be increased. I can. That is, the embodiment of the present disclosure reduces the load of the second processor 134 of the traffic information collection server 130, and determines and extracts the frame of interest in the edge computing device 110, and thus, the edge computing device 110 ) And the network cost between the traffic information collection server 130 may be reduced.

According to another embodiment of the present disclosure, the first processor 114 extracts an interest frame section, which is a section in which an event is detected, from the input video and transmits it to the traffic information collection server 130, and separates the input video of the entire frame section. To the control system of According to this embodiment, the traffic information collection server 130 minimizes data processing resources and storage resources used to store an input video, thereby increasing system efficiency. In addition, according to the present embodiment, the input video of the entire frame section is transmitted from the edge computing device 110 to a separate control system and stored in the control system, so that the input video can be preserved and managed while enhancing system efficiency. There is an effect.

The communication unit 118 may perform short-range communication, for example, Bluetooth, Bluetooth Low Energy (BLE), Near Field Communication, WLAN (Wi-Fi), Zigbee, and infrared (IrDA, infrared). Data Association) communication, WFD (Wi-Fi Direct), UWB (ultra wideband), Ant+ communication, etc. can be used. As another example, the communication unit 118 may use mobile communication, and may transmit and receive wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network.

The traffic information collection server 130 communicates with the plurality of edge computing devices 110 and receives input video and event information from each of the edge computing devices 110. The traffic information collection server 130 may correspond to a physical server disposed in a traffic control room or the like, or may correspond to a cloud server.

The traffic information collection server 130 includes a communication unit 132, a second processor 134, and an output interface 136.

The communication unit 132 may communicate with an external device by wire or wirelessly. The communication unit 132 receives event information and input video from a plurality of edge computing devices 110.

The communication unit 132 may perform short-range communication, for example, Bluetooth, Bluetooth Low Energy (BLE), Near Field Communication, WLAN (Wi-Fi), Zigbee, and infrared (IrDA, infrared). Data Association) communication, WFD (Wi-Fi Direct), UWB (ultra wideband), Ant+ communication, etc. can be used. As another example, the communication unit 132 may use mobile communication, and may transmit and receive wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network.

The second processor 134 controls the overall operation of the traffic information collection server 130. The second processor 134 receives at least one instruction or command and performs a predetermined operation. The second processor 134 may include one or more processors. The second processor 134 receives event information and an input video, detects an object corresponding to the vehicle from frames in the frame section in which the event occurs, and generates the detected vehicle speed information. The second processor 134 executes the second machine learning model and generates identification information of the object and speed information of the object. The second machine learning model receives and processes frames of the input video at a frame rate equal to or greater than the first frame rate and equal to or lower than the second frame rate. In addition, when the second processor 134 detects an event in which the speed of the object exceeds the second reference value, it generates event information. Since the second machine learning model of the second processor 134 is a model with higher accuracy than the first machine learning model of the first processor 114, even if an event is detected by the edge computing device 110, the traffic information collection server At (130), a result of determining whether an event is detected may be specified.

The first machine learning model is a model in which a bypass path for at least some layers is applied to the second machine learning model. Therefore, the first machine learning model has less throughput and faster processing speed than the second machine learning model. Also, since the first machine learning model receives an input video at a lower frame rate than the second machine learning model, the throughput is less than that of the second machine learning model. However, since the first machine learning model applies a bypass path to the second machine learning model, the accuracy of object identification information detection and speed information detection is inferior to that of the second machine learning model. According to embodiments of the present disclosure, the edge computing device 110 rapidly detects object identification information and speed with low accuracy for the entire frame of the input video, and the traffic information collection server 130 for the frame section in which the event occurs. ) Performs object identification information detection and speed detection processing with higher accuracy. With such a system structure, embodiments of the present disclosure have an effect of monitoring an input video in real time and remarkably reducing a system load.

The second processor 134 generates and outputs information on a frame of interest in which an event occurs and event information based on the event detection result.

The output interface 136 outputs data and various information. The output interface 136 may correspond to a display, a touch screen, a communication unit, or the like. The output interface 136 outputs frame of interest information and event information. According to an embodiment, the output interface 136 may display the object identification information and speed information together on the frame of interest, and output information on the object in which the event is detected together with the frame of interest. For example, a plurality of objects may be displayed on a frame of interest, and an indicator may be displayed on an object in which an event is detected.

2 is a diagram showing the structure of a traffic information collection system according to an embodiment of the present disclosure.

The traffic information collection system 100a according to an embodiment of the present disclosure includes an edge computing device 110a, a traffic information collection server 130a, and a control system 210. The control system 210 receives the input video collected by the edge computing device 110a, event information generated by the traffic information collection server 130a, and frame information of interest, and performs traffic control.

In FIG. 2, the first processor 114a of the edge computing device 110a, the video buffer 116, and the second processor 134a of the traffic information collection server 130 will be described, and other configurations will be omitted.

Blocks defined in the first processor 114a and the second processor 134a in the present disclosure are only examples of hardware blocks or software processing units for performing the embodiments of the present disclosure, and in various ways other than the processing units disclosed in the present disclosure. A processing unit that performs the embodiments of the present disclosure may be defined.

The first processor 114a of the edge computing device 110a may include a codec 222, a sampler 224, a first machine learning model 226, and a first speed analysis unit 228. Each block of the first processor 114a may correspond to a hardware block or a software block. For example, the first processor 114a may include a dedicated processor corresponding to the codec 222 or a dedicated processor corresponding to the first machine learning model 226.

The codec 222 decodes the input video input from the camera 150 to a predetermined standard. The codec 222 may support standards such as AVI, Moving Picture Experts Group (MPEG), MOV, and Window Media Video (WMV), for example. The codec 222 generates a plurality of input frames by decoding the input video.

The sampler 224 receives a plurality of input frames from the codec 222 and performs sampling processing on the plurality of input frames. For example, the sampler 224 may receive an input frame at 30 fps and sample at 3 fps. The sampling rate of the sampler 224 may vary according to embodiments.

The first machine learning model 226 generates and outputs identification information and speed information of an object in each input frame from a plurality of input frames input from the sampler 224. The plurality of input frames may be converted into input vectors required by the first machine learning model 226 through a predetermined pre-processing, and the input vectors may be input to the first machine learning model 226. For example, in pre-processing, at least one object corresponding to the vehicle may be detected from an input frame, and information on the at least one object may be input to the first machine learning model 226. Identification information and speed information of the object generated by the first machine learning model 226 are inserted into the input frame and output. For example, in the image data of the input frame, a box indicating an area of an object, information indicating identification information of an object (for example, a color of a box), and a speed of each object may be displayed together. The first processor 114a may insert object identification information and speed information into image data through post-processing of the output of the first machine learning model 226.

The first speed analysis unit 228 receives identification information and speed information of an object generated from the first machine learning model 226, determines whether the speed of each object exceeds a first reference value, and event information Create The first reference value may be determined based on a prescribed speed that becomes a reference value of the speed violation. For example, the first reference value may be determined as a value corresponding to 110% of the prescribed speed. The first speed analysis unit 228 detects an event in which the speed of each object in the input frame exceeds a first reference value. The first speed analysis unit 228 may generate event detection information indicating that an event has been detected or event non-detection information indicating that an event has not been detected according to a processing result. The event detection information may include identification information and frame information (or time information) of the object in which the event is detected.

When an event is detected, event information corresponding to the event detection information is transmitted to the traffic information collection server 130a. When the event is not detected, event information corresponding to the event undetected information is transmitted to the traffic information collection server 130a or the control system 210. Even if an event is not detected, event non-detection information may be transmitted to the traffic information collection server 130a or the control system 210 at a predetermined period.

When an event is detected, together with the event information, a plurality of frames of the frame of interest corresponding to the frame section in which the event has occurred are transmitted to the traffic information collection server 130a.

The video buffer 116 of the edge computing device 110a stores an input video including a plurality of input frames generated by decoding from the codec 222. The video buffer 116 transmits the input video to the traffic information collection server 130a or the control system 210.

The second processor 134a of the traffic information collection server 130a includes a second machine learning model 242 and a second speed analysis unit 244.

The traffic information collection server 130a receives a frame of interest in which an event occurs or information on a frame of interest (for example, a frame of interest) from a plurality of edge computing devices 110a, and second only the frame of interest in which the event is detected. By processing through the machine learning model 242, the throughput can be significantly reduced compared to the case where the entire input videos of the plurality of edge computing devices 110a are processed by the second machine learning model 242. In particular, since the second machine learning model 242 is a machine learning model with high precision compared to the first machine learning model 226, the throughput for the input frame is higher than that of the first machine learning model 226. The machine learning model 242 selectively processes only the frame of interest, concentrating processing resources on frames having a high probability of event detection, and significantly increasing the processing efficiency of the system.

The second machine learning model 242 generates and outputs identification information and speed information of an object in each input frame from a plurality of input frames. A plurality of input frames input to the second machine learning model 242 corresponds to a frame of interest. The second machine learning model 242 receives and processes an input frame at a higher frame rate than the first machine learning model 226. For example, the second machine learning model 242 may receive an input frame from the first machine learning model 224 at a frame rate of 10 times. The second machine learning model 242 may receive an input frame at the same frame rate as the input video generated by the camera 150 or may receive an input frame through a predetermined sampling process on the input video. The plurality of input frames may be converted into input vectors required by the second machine learning model 242 through a predetermined pre-processing, and the input vectors may be input to the second machine learning model 242. For example, in pre-processing, at least one object corresponding to a vehicle may be detected from an input frame, and information on at least one object may be input to the second machine learning model 242. Identification information and speed information of the object generated by the second machine learning model 242 are inserted into the input frame and output. For example, in the image data of the input frame, a box indicating an area of an object, information indicating identification information of an object (for example, a color of a box), and a speed of each object may be displayed together. The second processor 134a may insert object identification information and speed information into the image data through post-processing of the output of the second machine learning model 242.

The first machine learning model 226 and the second machine learning model 242 may be machine learned based on a plurality of object detection results, input frames, object identification information, and speed information. According to an embodiment, the first machine learning model 226 and the second machine learning model 242 may be trained using training data obtained from an intersection. In the case of using the learning data obtained from the intersection, the second machine learning model 242 having a target performance may be generated with only one day's worth of training data.

The second speed analysis unit 244 receives identification information and speed information of an object generated from the second machine learning model 242, determines whether the speed of each object exceeds a second reference value, and event information Create The second reference value may be determined based on a prescribed speed that becomes a reference value of the speed violation. The second reference value may be the same as the first reference value or may be greater than the first reference value. For example, the second reference value may be determined as a value corresponding to 110% of the prescribed speed. The second speed analysis unit 244 detects an event in which the speed of each object in the input frame exceeds a second reference value. The second speed analyzer 244 may generate event detection information indicating that an event has been detected or event non-detection information indicating that an event has not been detected according to a processing result. The event detection information may include identification information and frame information (or time information) of the object in which the event is detected.

According to an embodiment, at least one of the first speed analysis unit 228 or the second speed analysis unit 244, or a combination thereof, receives meta information, and provides an event based on the process result of the machine learning model and the meta information. Whether or not it is detected can be determined. The meta information may include signal information, lane information, stop line information, or crosswalk information. Meta information may be input from the outside or may be detected from an input frame by image processing. According to an embodiment, the first speed analysis unit 228 or the second speed analysis unit 244 may not perform event detection processing on a stop signal, and may perform event detection processing on a driving signal.

The traffic information collection server 130a transmits event information including event detection information or event non-detection information to the control system 210.

The control system 210 determines whether a traffic law is violated by a predetermined computing device or a person. The control room 212 includes at least one processor and a plurality of displays, and displays input video and event information through the plurality of displays. The control room 212 may display the input video as it is or by inserting event information into the frame of interest. The control room 212 displays input video and event information in various modes through a plurality of displays.

The control system 210 may include a storage unit 214 and may store input video, a frame of interest, and event information in the storage unit 214.

3 is a flowchart illustrating a control method of a traffic information collection system according to an embodiment of the present disclosure. In FIG. 3, the operation of the edge computing device 110 corresponds to a flowchart of the method for controlling the edge computing device, and the operation of the traffic information collection server 130 corresponds to the flowchart of the method for controlling the traffic information collection server. 3 shows a processing process for one frame, and the edge computing device 110 and the traffic information collection server 130 may repeatedly perform the steps of FIG. 3 for a plurality of frames.

According to an embodiment, the edge computing device 110 may perform steps S312, S314, S316, and S318 in parallel. The traffic information collection server 130 may perform steps S320, S322, S324, and S326 in parallel.

Each step of the method for controlling an edge computing device and a method for controlling a traffic information collection server of the present disclosure may be performed by various types of electronic devices including a processor and a communication unit, and using a machine learning model. The present disclosure will be described centering on an embodiment in which the edge computing devices 110 and 110a according to embodiments of the present disclosure perform a method of controlling an edge computing device. Accordingly, the embodiments described for the edge computing devices 110 and 110a can be applied to the embodiments of the edge computing device control method, and conversely, the embodiments described for the edge computing device control method are the edge computing devices 110 and 110a. ) Is applicable to the embodiments. The method for controlling an edge computing device according to the disclosed embodiments is performed by the edge computing devices 110 and 110a disclosed in the present specification, and the embodiment is not limited and may be performed by various types of electronic devices.

In addition, the present disclosure will be described centering on an embodiment in which the traffic information collection servers 130 and 130a according to the embodiments of the present disclosure perform a method of controlling the traffic information collection server. Therefore, the embodiments described for the traffic information collection servers 130 and 130a can be applied to the embodiments of the traffic information collection server control method, and conversely, the embodiments described for the traffic information collection server control method are the traffic information collection server. It is applicable to the embodiments for (130, 130a). The traffic information collection server control method according to the disclosed embodiments is performed by the traffic information collection servers 130 and 130a disclosed in the present specification, and the embodiment is not limited and may be performed by various types of electronic devices.

In addition, embodiments of the present disclosure described below with reference to FIGS. 4 to 15 may be applied to a method for controlling an edge computing device or a method for controlling a traffic information collection server. In addition, the operations of the edge computing device and the traffic information collection server described with reference to FIGS. 4 to 15 may be added as steps of the edge computing device control method or the traffic information collection server control method.

First, the edge computing device 110 obtains an input video from the camera 150 (S312). The edge computing device 110 may obtain a plurality of input frames by decoding an input video using a codec, and sample the plurality of input frames at a predetermined frame rate.

Next, the edge computing device 110 may input the sampled input frame into the first machine learning model to obtain object identification information and speed information of each object from the first machine learning model (S314). The first machine learning model receives an input vector including object detection information generated through pre-processing of an input frame, and outputs object identification information and speed information of each object.

Next, the edge computing device 110 detects an event in which the speed of the object exceeds the first reference value, based on the identification information of the object and the speed information of the object in each frame (S316). The edge computing device 110 generates event information including event detection information or non-event detection information based on the event detection result.

Next, the edge computing device 110 transmits the input video and event information to the traffic information collection server 130 (S318). When an event is detected, the edge computing device 110 may transmit information on a frame of interest, which is a frame section in which the event is detected, to the traffic information collection server 130.

The traffic information collection server 130 receives the input video and event information from the edge computing device 110 (S320).

The traffic information collection server 130 extracts a plurality of input frames corresponding to the frame of interest from the input video, and inputs the frame of interest into the second machine learning model. The second machine learning model receives the input frame and outputs identification information of the object and speed information of the object detected from the input frame (S322). The traffic information collection server 130 performs pre-processing on a plurality of frames input to the second machine learning model, generates an input vector including object information, and inputs the input vector as a second machine learning model. can do.

Next, the traffic information collection server 130 detects an event in which the speed of the object exceeds the second reference value, based on the identification information and speed information of the object output from the second machine learning model (S324). The traffic information collection server 130 generates event information including event detection information or non-event detection information based on the event detection result.

Next, the traffic information collection server 130 outputs the event information and the frame of interest through the output interface (S326). Event information and a frame of interest may be displayed on a display or transmitted to an external device through a communication unit.

4A is a diagram illustrating a structure of a first processor according to an embodiment of the present disclosure.

The first processor 114b according to an embodiment of the present disclosure includes a sampler 224, a preprocessor 410, a first machine learning model 226, a postprocessor 420, and a first speed analysis unit 228. It may include. Since the sampler 224, the first machine learning model 226, and the first velocity analysis unit 228 of FIG. 4A are the same as those described in FIG. 2, in FIG. 4A, the preprocessor 410 and the postprocessor 420 It will be explained mainly.

The preprocessor 410 receives the input frame output from the sampler 224, detects an object, and generates an input vector including object information. The preprocessor 410 may generate an object corresponding to the vehicle from the input frame. The preprocessor 410 may extract location information of an object, width and height information of an object area, and appearance information from the input frame, and generate an input vector including location information, width, height, and appearance information. When a plurality of objects are detected from the input frame, location information, width, height, and appearance information are generated for each object.

According to an embodiment, the preprocessor 410 may be implemented as a machine learning model. According to another embodiment, the preprocessor 410 may be implemented as a software module that performs a predetermined object detection algorithm.

The first machine learning model 226 receives an input vector and an input frame generated by the preprocessor 410 and generates identification information of an object and speed information of the object.

The post-processing unit 420 generates a plurality of connection lines in which image data of an object area corresponding to each object is accumulated, based on the identification information of the object and the speed information of the object output from the first machine learning model 226, An output data structure including relationship information indicating path similarity between a plurality of connection lines is generated. Each node corresponds to each object, and image data generated by cutting image data of the object region may be accumulated in each node. The output data structure may correspond to a model in which each node is defined and properties of a connection line between each node are changed according to relationship information between each node.

The post-processing unit 420 may additionally generate path information of each object based on the graph model. The path information of each object is information showing a trajectory that each object has moved for a predetermined time in the past. The post-processing unit 420 generates relevance information based on the similarity of the route information.

The first speed analysis unit 228 detects an event based on identification information of the object and speed information of the object generated from the first machine learning model 226. The first speed analysis unit 228 may detect an event in which the speed of the object exceeds the first reference value, based on the graph model generated by the post-processing unit 420. The first speed analyzer 228 may detect an event based on speed information of each object accumulated in each node of the graph model.

4B is a diagram illustrating a structure of a second processor according to an embodiment of the present disclosure.

The second processor 134b according to an embodiment of the present disclosure includes a sampler 430, a pre-processor 440, a second machine learning model 242, a post-processor 450, and a second speed analysis unit 244. It may include. Since the second machine learning model 242 and the second speed analysis unit 244 of FIG. 4B are the same as those described in FIG. 2, in FIG. 4B, the sampler 430, the preprocessor 440, and the postprocessor 450 It will be explained mainly.

The sampler 430 may sample the input video at a predetermined frame rate. When the second processor 134b processes the input video at the same frame rate as the input video, the sampler 430 may be omitted.

The preprocessor 440 receives the input frame output from the sampler 430, detects an object, and generates an input vector including object information. The preprocessor 440 may generate an object corresponding to the vehicle from the input frame. The preprocessor 440 may extract location information of an object, width and height information of an object area, and appearance information from the input frame, and generate an input vector including location information, width, height, and appearance information. When a plurality of objects are detected from the input frame, location information, width, height, and appearance information are generated for each object.

According to an embodiment, the preprocessor 440 may be implemented as a machine learning model. According to another embodiment, the preprocessor 440 may be implemented as a software module that performs a predetermined object detection algorithm.

The preprocessor 440 of the second processor 134b may have higher accuracy than the preprocessor 410 of the first processor 114b and may have a higher throughput. When the preprocessor 440 of the second processor 134b is implemented as a machine learning model including a plurality of layers, the preprocessor 410 of the first processor 114b is a preprocessor of the second processor 134b ( A machine learning model to which a bypass path for bypassing at least some layers of 440) is applied may be included.

The second machine learning model 242 receives the input vector generated by the preprocessor 440 and generates identification information of an object and speed information of the object.

The post-processing unit 450 generates a plurality of nodes in which image data of an object area corresponding to each object is accumulated, based on the object identification information and the object velocity information output from the second machine learning model 242, An output data structure including relationship information indicating path similarity between a plurality of nodes is generated. Each node corresponds to each object, and image data generated by cutting image data of the object region may be accumulated in each node. The output data structure may correspond to a graph model in which each node is defined and properties of a connection line between each node are changed according to relationship information between each node.

The post-processing unit 450 may additionally generate path information of each object based on the graph model. The path information of each object is information showing a trajectory that each object has moved for a predetermined time in the past. The post-processing unit 450 generates relevance information based on the similarity of the route information.

The second speed analysis unit 244 detects an event based on identification information of the object and speed information of the object generated from the second machine learning model 242. The second speed analysis unit 244 may detect an event in which the speed of the object exceeds the second reference value, based on the graph model generated by the post-processing unit 450. The second speed analyzer 244 may detect an event based on speed information of each object accumulated in each node of the graph model.

5 is a diagram illustrating an image processing process according to an embodiment of the present disclosure.

The first processor 114b and the second processor 134b process the input frame through pre-processing, machine learning model processing, and post-processing processes. In FIG. 5, a process of pre-processing 510, machine learning model processing 520, and post-processing 530 performed by the first processor 114b and the second processor 134b will be described.

According to an embodiment of the present disclosure, the preprocessing 510 includes a camera calibration 512, an object detection 514, and an embedding process 516. The machine learning model processing 520 includes processing by a first machine learning model or processing by a second machine learning model. The post-processing 530 includes a graph model generation process 532 and a clustering process 534.

6 is a diagram illustrating a camera calibration process according to an embodiment of the present disclosure.

The camera calibration process 512 is a process of converting a distance on an actual road with respect to an input frame into a rectangular shape. The camera calibration process 512 may generate reference lines 612a and 612c representing the same distance on the road from the input frame 610 and a reference line 612b representing a lane traveling direction on the road. According to an embodiment, the reference lines 612a, 612b, and 612c may be determined based on lane information detected in the input frame 610.

Next, the camera calibration process 512 may generate distance reference points 622 indicating a distance on a road based on a lane traveling direction. Based on the adjustment input frame 620 on which the camera calibration process 512 has been performed, there is an effect of accurately measuring a moving distance between frames of an object from the input frame.

7 is a diagram illustrating a camera calibration process according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the camera calibration process 512 may perform a camera calibration process based on an image type of an input video or a road marking line. To this end, the camera calibration process 512 first identifies an image type or a road marking line based on the input video 710 (712). The image type may include information on the type of camera equipment and a photographing composition. Road marking lines may include lanes, crosswalks, or stop lines. Next, the camera calibration processing 512 performs camera calibration based on the image type or road marking line (714).

The input video 710 may be acquired by being captured by various camera devices. In addition, when the input video is photographed, the photographing composition of the camera equipment may be variously determined. The camera calibration processing 512 performs a camera calibration processing 512 on the basis of image type information indicating information on at least one of a type of camera equipment or a photographing composition, or a combination of the types of camera equipment that has captured the input video.

The type of camera equipment may include, for example, a fixed CCTV camera, a rotating CCTV camera, or a drone camera. The photographing composition may include a diagonal composition, a front composition, or an aerial shot composition. The camera calibration process 512 may vary depending on whether the camera equipment is a fixed type, a rotating type, or a mobile type (eg, a drone). When the camera equipment is a fixed type, additional camera calibration processing 512 is not performed after the initial camera calibration processing 512, or camera calibration processing 512 is performed at a predetermined period (e.g., 1 hour). I can. When the camera device is of a rotation type, when the camera device is rotated (eg, pan, tilt, etc.), the camera calibration process 512 may be performed. In the case of the rotary camera device, in addition to the case where rotation has occurred, the camera calibration process 512 may be performed at a predetermined period after the camera calibration process 512. In the case of a mobile camera device such as a drone camera, the camera calibration process 512 may be performed in real time, and, for example, the camera calibration process 512 may be performed every input frame after the sampling process.

In addition, the camera calibration process 512 varies depending on the photographing composition. The camera calibration process 512 defines differently the shape of a rectangle generated by the reference lines 612a, 612b, 612c of the camera calibration process 512 depending on whether it is a diagonal composition, a front composition, or an aerial shot composition.

Further, the camera calibration process 512 may be determined based on the direction of the reference lines 612a, 612b, and 612c and the arrangement of the distance reference point 622 based on road marking lines such as lanes.

Referring back to FIG. 5, the preprocessing 510 includes an object detection process 514. The object detection process 514 detects an object corresponding to the vehicle from the input frame.

8A is a diagram for describing an object detection process according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the object detection process 514 detects at least one object 822 corresponding to a vehicle from the input frame 810. The object detection processing 514 defines a block 820 corresponding to the object area for the object detected in the input frame 810, and coordinates (x, y) of the reference point 824 of the block, and block 820 Define the width (w) of) and the height (h) of the block. The reference point 824 may be defined as, for example, a coordinate of the upper left corner of the block 820. Coordinates (x, y) represent coordinates based on the horizontal axis and the vertical axis within the input frame 810.

Also, the object detection process 514 may generate appearance information from the detected object. The appearance information may include, for example, information such as a vehicle type (eg, a car, a bus, a truck, a sport utility vehicle (SUV), etc.) of the object, and a color.

8B is a diagram illustrating an object detection process 514 according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the object detection processing 514 performs an inter-frame overlapping area processing 830 and an appearance similarity calculation processing 840 of the detected object from a plurality of sequentially inputted input frames.

The inter-frame overlapping area processing 830 detects an overlapping area between temporally adjacent frames, and uses the processing result of the previous frame for the overlapping area. The inter-frame overlapping area processing 830 detects an overlapping area 836 overlapping in the first frame 832 and the second frame 834 which is a frame following the first frame 832, and the overlapping area 836 For, object detection processing for the second frame 834 is performed based on the object detection processing result for the first frame 832. The overlapping area 836 may be defined based on the detected object or may be defined based on image similarity.

The appearance similarity calculation processing 840 detects objects having similar appearance information from a plurality of input frames. For example, the appearance similarity calculation processing 840 calculates an appearance similarity between an object detected in a first frame and an object detected in a second frame. According to an embodiment, the appearance similarity may be generated based on a feature map generated using a CNN from an input frame. According to another embodiment, the appearance similarity may be calculated based on appearance information of the object of the first frame and appearance information of the object of the second frame.

Tracklet information V is generated based on the object detection processing result generated based on the inter-frame overlapping area processing 830 and the appearance similarity calculation processing 840.

Referring back to FIG. 5, the preprocessing 510 performs a tracklet information generation process 516 based on the result of the object detection process 514 and the input frame on which the camera calibration process 512 has been performed. The tracklet information generation process 516 generates tracklet information 518 corresponding to an input vector input to the first machine learning model or the second machine learning model from the result of the object detection process 514 and the input frame.

9 is a diagram for describing a process of generating tracklet information according to an embodiment of the present disclosure.

The tracklet information generation process 516 receives an object detection result and an input frame, and performs a feature embedding process 910. The feature embedding process 910 generates tracklet information 518 from the object detection result and the input frame. The tracklet information 518 is information corresponding to an input vector input to the tracklet network 930 corresponding to the first machine learning model or the second machine learning model. The tracklet includes area information 924 and appearance information 926 for an area of an object detected in a format required by the treklet network 930. The area information 924 includes coordinates (x, y), width (w), and height (h) of a block corresponding to the object area. The appearance information 926 includes appearance information previously generated in the object detection process 514. The tracklet information 518 may include information 922 for each object in one frame. The tracklet information 518 may include information on a plurality of objects in a predetermined format.

Referring back to FIG. 5, the tracklet information 518 generated by the tracklet information generation process 516 is input to the machine learning model process 520 together with an input frame. The machine learning model processing 520 processes tracklet information and input frames by the first machine learning model or the second machine learning model described above.

10 is a diagram illustrating a structure of a tracklet network according to an embodiment of the present disclosure.

The tracklet network 1000 has a DNN structure including a plurality of layers. The tracklet network 100 may include a combination of a CNN structure and an RNN structure. The tracklet network 1000 includes an input layer 1010, a plurality of hidden layers 1020, and an output layer 1030.

The input layer 1010 includes at least one layer 1014. The input layer 1010 receives an input vector 1012 and an input frame, and generates at least one input feature map 1022.

At least one feature map 1022 is input to the hidden layer 1020 and processed. The hidden layer 1020 is generated by learning in advance by a predetermined machine learning algorithm. The hidden layer 1020 receives at least one input feature map 1022 and performs predetermined activation processing, pooling processing, linear processing, convolution processing, etc. to generate at least one output feature map 1026. According to an embodiment, the hidden layer 1020 may include a processing layer corresponding to each input feature map 1022. The feature map 1026 is converted into an output vector 1028 through additional processing.

The output vector 1028 is output from the tracklet network through the output layer 1030.

The second machine learning model included in the traffic information collection server 130 includes a tracklet network 1000. The first machine learning model includes a structure in which at least one bypass path 1040 is added to the structure of the tracklet network 1000. The start and end points of the bypass path 1040 and the number of bypass paths 1040 may vary according to exemplary embodiments. When the bypass path 1040 is arranged, the output value of the layer corresponding to the start point of the bypass path is transmitted to the layer corresponding to the end point of the bypass path, and the layers in the middle of the bypass path 1040 Processing is bypassed without being performed. The first machine learning model applies the bypass path 1040 to the tracklet network 1000 of the second machine learning model, thereby reducing throughput and remarkably improving processing speed.

Referring back to FIG. 5, object identification information, velocity information, and tracklet information 518 of each input frame generated by the machine learning model processing 520 are post-processed (530). The post-processing 530 includes a graph model generation process 532 and a clustering process 536.

11 is a diagram illustrating a graph model according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the graph model generation process 532 generates a graph model 1100 from object identification information, speed information, and tracklet information. The graph model 1100 includes at least one node 1110 and at least one connection line 1120 connecting the nodes 1110. Each node 1110 stores information about the same object in each frame. The graph model generation process 532 may identify and store information on the same object in each node based on identification information of the object generated by the machine learning model. For example, each node 1110 may include at least one of object region information, speed information, or image data 1112 in each frame, or a combination thereof, for the same object.

A connection line 1120 disposed between each node 1110 indicates a relationship between each object. Relevance is information indicating the similarity of paths between objects. The graph model 1100 may be represented by g(V,E), which is a function of tracklet information V and relevance information E.

12 is a diagram illustrating a process of calculating a relationship between objects according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the post-processing 530 includes a relevance calculation process 1200. The relevance calculation processing 1200 selects tracklet information of two different objects from the tracklet information and inputs it to the tracklet network 1220. The tracklet information of two different selected objects may include tracklet information extracted from a plurality of frames for each object.

The tracklet network 1220 calculates a degree of similarity of paths for two different objects, and generates correlation information E. When the relevance information E is high, the similarity of the paths of the two objects is high, and when the relevance information E is low, it may be determined that the similarity of the paths of the two objects is low. The relevance information E may be defined as a value of 0 or more and 1 or less. If tracklet information for the same object is input to the tracklet network, the relevance information (E) is 1, and if tracklet information of different objects is input to the tracklet network 1220, the correlation information (E) is Values less than 1 can be displayed as values.

13 is a diagram illustrating a graph model generation process and a clustering process according to an embodiment of the present disclosure.

As described above, the graph model generation process 532 performs a graph model generation process based on the tracklet information and the relevance information E. The graph model generation process 532 may change a property of a connection line between each node based on the relationship information E. For example, the graph model generation process 532 may change the thickness of the connection line or change the color of the connection line to display the relevance information E on the connection line. In addition, the graph model generation process 532 arranges related nodes adjacent to each other and connects each node according to the relevance information (E), and the non-relevant nodes are placed away so that connection lines are not placed between unrelated nodes. May not.

The clustering process 534 may cluster objects based on the relevance information E. For example, the clustering process 534 may define objects with high relevance as one cluster.

14 is a diagram illustrating a result image according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, identification information and speed information of an object may be inserted into an input frame. The resulting image 1400 may include a block 1410 indicating an object area, speed information 1420, and path information 1430. The object identification information may be represented by the color of the block 1410. The resulting image 1400 may represent a block 1410, speed information 1420, and path information 1430 for each of the detected at least one object. The route information 1430 may indicate route information tracked during a preset predetermined time period.

The resulting image 1400 may be generated for all frames of the input video, or may be generated only for a frame of interest in which an event is detected by the traffic information collection server. The second processor 134 may generate the result image 1400 and output it through the output interface 136.

Meanwhile, the disclosed embodiments may be implemented in the form of a computer-readable recording medium that stores instructions and data executable by a computer. The instruction may be stored in the form of a program code, and when executed by a processor, a predetermined program module may be generated to perform a predetermined operation. Further, when the command is executed by a processor, certain operations of the disclosed embodiments may be performed.

As described above, the disclosed embodiments have been described with reference to the accompanying drawings. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be practiced in a form different from the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are exemplary and should not be construed as limiting.

100 traffic information collection system
110, 110a edge computing devices
114 first processor
130 traffic information collection server
134 second processor
150 cameras
226 first machine learning model
228 first speed analysis unit
242 second machine learning model
244 2nd speed analysis unit

Claims (20)

An input interface for obtaining an input video captured by the camera;
At least one first processor; And
Including a communication unit that communicates with the traffic information collection server,
The at least one first processor,
By inputting an input frame with a frame rate lower than the frame rate of the input video into a first machine learning model, identification information and speed information of at least one object detected in each frame are obtained from the first machine learning model,
On the basis of identification information and speed information of each of the at least one object output from the first machine learning model, detect an event of detecting an object of interest in which the speed of the object exceeds a first reference value,
Using the communication unit, event information and the input video, which are information on the detected event, are transmitted to the traffic information collection server,
The traffic information collection server includes a second machine learning model that receives the input video and generates identification information and speed information of each of at least one object,
The first machine learning model is a machine learning model in which a bypass path is applied to at least one layer of the second machine learning model. delete The method of claim 1,
The edge computing device, wherein the first machine learning model receives an input frame having a lower frame rate than the input frame input to the second machine learning model. The method of claim 1,
The edge computing device further comprises a video buffer for storing the input video,
The communication unit transmits the input video stored in the video buffer to the traffic information collection server. The method of claim 1,
The at least one first processor obtains meta information including at least one of signal information, stop line information, or lane information, or a combination thereof,
When detecting the event, the edge computing device for detecting the event based on the identification information, the speed information, and the meta information of each of the at least one object. The method of claim 1,
The at least one first processor,
Performs a process of generating tracklet information defining an object location and an object area from the input frame,
Inputting the tracklet information into the first machine learning model,
Acquiring identification information and the speed information of the at least one object output from the first machine learning model,
Using the identification information of the at least one object and the speed information, a graph model including information on each of the at least one object and relation information with another object is generated,
An edge computing device that generates cluster information of the at least one object by using the relationship information between the graph model and the at least one object. The method of claim 6,
The at least one first processor, based on the cluster information, performs event detection processing of another object in the same cluster by using the result of event detection processing for one object. The method of claim 1,
The at least one first processor,
When the event is not detected from the input frame, event non-detection information indicating that the event has not been detected is generated,
When the event is not detected, the event non-detection information is transmitted to the traffic information collection server at a predetermined period. Acquiring an input video captured by a camera;
Inputting an input frame having a frame rate lower than the frame rate of the input video into a first machine learning model, and obtaining identification information and speed information of at least one object detected in each frame from the first machine learning model;
Detecting an event in which the speed of the object exceeds a first reference value, based on identification information and speed information of each of the at least one object output from the first machine learning model; And
Transmitting event information and the input video, which are information on the detected event, to a traffic information collection server,
The traffic information collection server includes a second machine learning model that receives the input video and generates identification information and speed information of each of at least one object,
The first machine learning model is a machine learning model in which a bypass path is applied to at least one layer of the second machine learning model. A communication unit communicating with at least one edge computing device;
At least one second processor; And
Includes an output interface,
The at least one second processor,
Receive event information including information on an event in which the speed of an object exceeds a first reference value, and an input video from the at least one edge computing device through the communication unit,
Extracting an interest frame section corresponding to the event information from the input video,
By inputting the input frame of the frame of interest into a second machine learning model, identification information and speed information of at least one object detected in each frame are obtained from the second machine learning model,
Based on identification information and speed information of each of the at least one object output from the second machine learning model, an event of detecting an object of interest in which the speed of the object exceeds a second reference value is detected,
Outputting event information, which is information on an event of detecting an object of interest in which the speed of the object exceeds a second reference value, and image information of the frame of interest, through the output interface,
The at least one edge computing device includes a first machine learning model that receives the input video and generates identification information and speed information of each of at least one object,
The traffic information collection server, wherein the first machine learning model is a machine learning model in which a bypass path is applied to at least one layer of the second machine learning model. delete The method of claim 10,
The traffic information collection server, wherein the first machine learning model receives an input frame having a lower frame rate than the input frame input to the second machine learning model. The method of claim 10,
The at least one second processor acquires meta information including at least one of signal information, stop line information, or lane information, or a combination thereof,
When detecting the event, the traffic information collection server for detecting the event based on the identification information, the speed information, and the meta information of each of the at least one object. The method of claim 10,
The at least one second processor,
Performs a process of generating tracklet information defining an object location and an object area from the input frame,
Inputting the tracklet information into the second machine learning model,
Acquire identification information and the speed information of the at least one object output from the second machine learning model,
Using the identification information of the at least one object and the speed information, a graph model including information on each of the at least one object and relation information with another object is generated,
The traffic information collection server for generating cluster information of the at least one object by using the relationship information between the graph model and the at least one object. The method of claim 14,
The at least one second processor, based on the cluster information, performs event detection processing of another object in the same cluster using a result of event detection processing for one object. The method of claim 10,
The communication unit receives event non-detection information indicating that no event has been detected from the at least one edge computing device,
The at least one second processor outputs the event non-detection information through the output interface. The method of claim 10,
The at least one second processor,
When the event is not detected from the frame of interest, event non-detection information indicating that the event has not occurred is generated,
Traffic information collection server for outputting the event undetected information through the output interface. Receiving, from at least one edge computing device, event information including information on an event in which the speed of the object exceeds a first reference value, and an input video;
Extracting a frame of interest corresponding to the event information from the input video;
Inputting the input frame of the frame of interest into a second machine learning model, and obtaining identification information and speed information of at least one object detected in each frame from the second machine learning model;
Detecting an event of detecting an object of interest whose speed exceeds a second reference value, based on identification information and speed information of each of the at least one object output from the second machine learning model; And
And outputting event information, which is information on an event of detecting an object of interest in which the speed of the object exceeds a second reference value, and image information of the frame of interest,
The at least one edge computing device includes a first machine learning model that receives the input video and generates identification information and speed information of each of at least one object,
The first machine learning model is a machine learning model in which a bypass path is applied to at least one layer of the second machine learning model. A computer program stored in a recording medium, wherein the computer program includes at least one instruction for performing the method of controlling the edge computing device of claim 9 when executed by a processor. A computer program stored in a recording medium, wherein the computer program comprises at least one instruction for performing the traffic information collection server control method of claim 18 when executed by a processor.

Images