KR102588729B1 - Hybrid ai image analysis system - Google Patents

Abstract

The present invention relates to a hybrid artificial intelligence image analysis system, and more specifically, to an image analysis system, which includes a general camera unit, an edge device camera unit, a media distribution unit, a background processing unit, and an image processing unit. It is possible to process artificial intelligence image analysis by combining a server type and an edge type.

Description

Hybrid artificial intelligence image analysis system {HYBRID AI IMAGE ANALYSIS SYSTEM}

The present invention relates to a hybrid video analysis system, and more specifically, to a hybrid artificial intelligence video analysis system combining edge type and server type.

Conventional video surveillance systems use CCTV cameras installed in each major facility area to store real-time video data on a server and display real-time video screens or record playback to provide the experience and continuous operation of multiple screens to the control room controller. It was a system managed through. Recently, with the significant development of artificial intelligence technology, attempts are being made to apply artificial intelligence or deep learning technology in the field of video surveillance.

Artificial intelligence video analysis systems can be divided into server type and edge type depending on the point at which events are detected by analyzing video. First, the edge type is a method of analyzing video from a camera and has the advantage of being able to analyze video without distortion of the signal that occurs during the process of compressing the video for transmission and has a fast response speed. Edge-type artificial intelligence video analysis generates reliable and meaningful information through stable linkage with related IoT equipment, but has the disadvantage of being economically unfeasible due to linkage with already installed equipment and the resulting difficulties. In particular, it is expensive because the camera must be replaced with an edge device, and because the embedded operation speed is limited, complex video analysis techniques cannot be applied, and future function upgrades are limited. If you want to introduce an artificial intelligence video analysis system, introducing the edge type requires replacing all existing cameras, which is expensive, and gaps in video surveillance may occur during replacement, so the timing of replacement must be considered.

On the other hand, the server type is a centralized structure that receives images from cameras and performs intelligent image analysis. Since there is no dependency on cameras, it is easy to apply to existing cameras. However, in order to support large-scale channels, expensive GPUs must be installed, making it expensive to build, and there are problems with scalability because the number of cameras that can be processed is limited. In addition, there are problems with recent server types such as high network bandwidth usage and the number of channels that can be processed per server. Therefore, when trying to introduce an artificial intelligence video analysis system, introducing a server type requires a large-capacity physical integrated control system that processes multi-channel CCTV simultaneously or separately, which may increase the burden on the server and limit performance. The cost of using a lot of the network must be considered.

The above-mentioned background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known technology disclosed to the general public before filing the application for the present invention.

Meanwhile, as prior art related to the present invention, Patent Registration No. 10-2058452 (title of the invention: IoT convergence intelligent video analysis platform system, registration date: December 17, 2019) has been disclosed.

The present invention was proposed to solve the above problems of previously proposed methods, and is a deep learning module in which an image processing unit estimates the first meta information by processing artificial intelligence image analysis from the first image captured by a general camera unit. Combining the server type and edge type in artificial intelligence video analysis by receiving and processing data from the edge device camera unit, which includes an artificial intelligence model and estimates the second meta information in real time from the captured second image. The purpose is to provide a hybrid artificial intelligence image analysis system that can process

In addition, the present invention combines a general camera unit that processes artificial intelligence image analysis in the image processing unit with an edge device camera unit that processes artificial intelligence image analysis as an edge type, so that existing hardware resources can be used without the need to replace all of the general camera unit. An artificial intelligence video analysis system can be introduced by installing a portion of the edge device camera unit while utilizing the edge device camera unit and the image processing unit to distribute and process the artificial intelligence calculations, enabling quick edge-type response and minimizing the burden on the server. Another purpose is to provide a hybrid artificial intelligence video analysis system that can maximize the effect by combining the advantages of server-type large-capacity processing.

A hybrid artificial intelligence image analysis system according to the characteristics of the present invention to achieve the above purpose,

As a video analysis system,

A general camera unit that captures a first image;

An edge device camera unit that captures a second image and uses an artificial intelligence model to estimate second meta information in real time from the second image;

A media distributor that receives and transmits the first image, second image, and second meta information from the general camera unit and the edge device camera unit;

a background processing unit that stores the second meta information received from the media distribution unit in a database unit; and

An image processing unit that decodes the first video and the second video received from the media distribution unit and processes them into an outputable format,

The image processing unit,

A deep learning module that estimates first meta information from the first image captured by the general camera unit using a deep learning model calculated using the GPU built into the processor and stores the first meta information in the database unit. including,

Its structural feature is that the image processing unit combines the general camera unit that processes artificial intelligence image analysis with the edge device camera unit that processes artificial intelligence image analysis in an edge type.

Preferably, the first meta information and the second meta information are:

It may include object images and analysis information extracted from the first image and the second image, respectively.

Preferably, the edge device camera unit,

Data can be transmitted to the media distribution unit according to a video transmission mode that transmits the second video and second meta information and a feature transmission mode that transmits only the second meta information and excludes the second video.

More preferably, the edge device camera unit,

The image transmission mode and feature transmission mode can be controlled according to the control signal received from the image processing unit.

Even more preferably, the image processing unit,

When an event is detected, a control signal to change to the video transmission mode can be transmitted to the edge device camera unit.

Preferably, the deep learning module,

The first meta information can be extracted from the first image based on openVINO.

Preferably,

Artificial intelligence image analysis processed in the image processing unit is more complex than artificial intelligence image analysis processed in the edge device camera unit and may require greater computing power.

According to the hybrid artificial intelligence image analysis system proposed in the present invention, the image processing unit includes a deep learning module that processes artificial intelligence image analysis from the first image captured by a general camera unit to estimate first meta information, and artificial intelligence By receiving and processing data from the edge device camera unit, which estimates the second meta information in real time from the second image captured with a built-in model, the server type and edge type can be combined and processed in artificial intelligence image analysis.

In addition, according to the hybrid artificial intelligence video analysis system proposed in the present invention, by combining a general camera unit that processes artificial intelligence video analysis in the image processing unit and an edge device camera unit that processes artificial intelligence video analysis as an edge type, a general camera An artificial intelligence video analysis system can be introduced by installing a portion of the edge device camera unit while utilizing existing hardware resources without the need to replace the entire unit. By distributing and processing the artificial intelligence calculations between the edge device camera unit and the image processing unit, The effect can be maximized by combining the advantages of the edge type's quick response and minimal server load with the advantages of the server type's high-capacity processing.

1 is a diagram showing the configuration of a hybrid artificial intelligence image analysis system according to an embodiment of the present invention.
Figure 2 is a diagram showing a case where a general camera unit and an edge device camera unit are used in a general video analysis system.
Figure 3 is a diagram showing a case of using a general camera unit and an edge device camera unit in a hybrid artificial intelligence video analysis system according to an embodiment of the present invention.
Figure 4 is a diagram showing the detailed configuration of a deep learning module in a hybrid artificial intelligence image analysis system according to an embodiment of the present invention.
Figure 5 is a diagram showing an example of a parking lot management system using a hybrid artificial intelligence image analysis system according to an embodiment of the present invention.
Figure 6 is a diagram illustrating an intruder detection system using a hybrid artificial intelligence image analysis system according to an embodiment of the present invention as an example.
Figure 7 is a diagram showing an example of a face search system using a hybrid artificial intelligence image analysis system according to an embodiment of the present invention.

Hereinafter, with reference to the attached drawings, preferred embodiments will be described in detail so that those skilled in the art can easily practice the present invention. However, when describing preferred embodiments of the present invention in detail, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the same symbols are used throughout the drawings for parts that perform similar functions and actions.

Additionally, throughout the specification, when a part is said to be 'connected' to another part, this is not only the case when it is 'directly connected', but also when it is 'indirectly connected' with another element in between. Includes. Additionally, ‘including’ a certain component does not mean excluding other components, but rather including other components, unless specifically stated to the contrary.

Figure 1 is a diagram showing the configuration of a hybrid artificial intelligence image analysis system according to an embodiment of the present invention. As shown in Figure 1, the hybrid artificial intelligence image analysis system according to an embodiment of the present invention includes a general camera unit 100, an edge device camera unit 200, a media distribution unit 300, and a background processing unit 400. ) and an image processing unit 500, and may further include a local image storage unit 600 and a database unit 700.

The present invention includes a deep learning module 510 in which the image processing unit 500 estimates first meta information by processing artificial intelligence image analysis from the first image captured by the general camera unit 100, and has a built-in artificial intelligence model. By receiving and processing data from the edge device camera unit 200, which estimates the second meta information in real time from the captured second image, the server type and edge type can be combined and processed in artificial intelligence image analysis. In other words, by applying edge-type cameras and server-type video analysis simultaneously, the budget burden when introducing an edge-type video analysis system and the server load when introducing a server-type video analysis system are reduced, and the edge-type fast By combining the advantages of minimizing response and server burden with the advantages of server-type large-capacity processing, deep learning functions can be focused.

With reference to FIG. 1, let us look at the artificial intelligence image analysis processing process of images captured by the general camera unit 100 and the edge device camera unit 200. First, looking at the artificial intelligence video analysis processing process of the first video captured by the general camera unit 100, ① the media distribution unit 300 receives RTSP streaming from the general camera unit 100, and ② the image processing unit 500 send to ③ The first decoder module 520 of the image processing unit 500 decodes the received first image and transmits it to the deep learning module (DL Module) 510. ④ The object image among the first meta information generated by the deep learning module 510 is stored in the local image storage unit 600, and ⑤ the object analysis information among the first meta information is stored in the database unit 700. ⑥ The decoded image is drawn on the screen through the render module 540.

Next, looking at the data processing process of the second image captured by the edge device camera unit 200 and the second meta information estimated in real time from the second image, ⓐ the media distribution unit 300 ) receives RTSP streaming of the second video and second meta information. ⓑ The media distribution unit 300 extracts the second meta information from the streamed data and transmits it to the background processing unit 400. ⓒ The background processing unit 400 downloads the object image included in the second meta information and stores it in the local image storage unit 600, and ⓓ stores the object analysis information included in the second meta information in the database unit 700. . ⓔ The media distribution unit 300 receives AI camera streaming and transmits it to the video processing unit 500. ⓕ The image decoded in the second decoder module 530 of the image processing unit 500 is drawn on the screen through the render module 540.

FIG. 2 is a diagram illustrating a case where a general camera unit 100 and an edge device camera unit 200 are used in a general video analysis system. As shown in Figure 2, when all 12 edge device camera units 200 and 4 general camera units 100 are connected to a general video analysis system, only 12 edge device camera units 200 are used as artificial intelligence cameras. You can use it.

FIG. 3 is a diagram illustrating a case where a general camera unit 100 and an edge device camera unit 200 are used in a hybrid artificial intelligence video analysis system according to an embodiment of the present invention. As shown in Figure 3, in the hybrid artificial intelligence video analysis system according to an embodiment of the present invention, when all 12 edge device camera units 200 and 4 general camera units 100 are connected, the general camera Since the images captured by the four subunits 100 can be processed through artificial intelligence image analysis through the deep learning module 510, all 16 units can be used as artificial intelligence cameras.

The hybrid artificial intelligence image analysis system according to an embodiment of the present invention can be implemented with 16 channels. For example, as shown in FIG. 3, 16 channels can be configured by combining 12 channels of the edge device camera unit 200 and 4 channels of the general camera unit 100. At this time, the deep learning module 510 of the image processing unit 500 can use a deep learning model calculated based on OpenVINO using the GPU built into the Intel processor. Based on the Intel i7 processor, openVINO's YOLOv4 -tiny fp32 When applying the algorithm, it has a performance of 76fps, so it can process 4 channels of video from the general camera unit 100.

Hereinafter, each configuration of the hybrid artificial intelligence image analysis system according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 3.

The general camera unit 100 can capture the first image. Here, the general camera unit 100 is a camera that cannot perform artificial intelligence image processing, in contrast to the edge device camera unit 200, which is a previously installed CCTV, IP camera, etc., capable of edge-type artificial intelligence image processing. It can mean. In particular, cameras that are already installed often do not have a GPU capable of performing artificial intelligence calculations or cannot be additionally installed, so the general camera unit 100 can be used as is without a GPU for calculation. The first image captured by the general camera unit 100 may vary depending on camera characteristics, such as black and white, color, infrared, and 3D images.

The edge device camera unit 200 can capture the second image and estimate the second meta information in real time from the second image by embedding an artificial intelligence model. As shown in FIG. 1, the edge device camera unit 200 includes an AI-camera, which is a camera with a built-in artificial intelligence model, an AI-BOX that analyzes images in an embedded box, and a license plate recognition camera (LPR-Camera, License). It can be implemented with a Plate Recognition Camera, etc.

Here, the second meta information may include object images and analysis information each extracted from the second image. For example, the edge device camera unit 200 can detect an object by analyzing the second image in real time, and produce a cropped image (object image) of the detected object portion and analysis information that classifies the detected object. These object images and analysis information can be configured as second meta information.

The media distribution unit 300 may receive and transmit the first image, the second image, and the second meta information from the general camera unit 100 and the edge device camera unit 200. More specifically, the media distribution unit 300 receives the first image from the general camera unit 100 using the RTSP protocol and transmits it to the first decoder module 520 of the image processing unit 500. ); And receiving the second image from the edge device camera unit 200 using the RTSP protocol and delivering it to the second decoder module 530 of the image processing unit 500, parsing the second meta information and delivering it to the background processing unit 400. It may include a second RTSP client 320 that includes a parser. That is, the hybrid artificial intelligence image analysis system according to an embodiment of the present invention includes not only the first image streaming of the general camera unit 100, but also an edge device camera unit (which includes a second image and second meta information). By including a media distribution unit 300 that can receive and process streaming information from 200, a hybrid system that effectively combines the general camera unit 100 and the edge device camera unit 200 can be implemented.

The background processing unit 400 may store the second meta information received from the media distribution unit 300 in the database unit 700. That is, the background processing unit 400 receives the second meta information (including the object image and analysis information), and stores the object image in the local image storage unit 600 and the analysis information in the database unit 700. You can.

The image processing unit 500 may decode the first and second images received from the media distribution unit 300 and process them into an outputable format. More specifically, the image processing unit 500 estimates the first meta information from the first image captured by the general camera unit 100 using a deep learning model calculated using the GPU built into the processor, and It includes a deep learning module 510 that stores information in the database unit 700, and may further include a first decoder module 520, a second decoder module 530, and a render module 540. Here, the first meta information may include an object image and analysis information extracted from the first image.

That is, the image processing unit 500 includes a deep learning module 510, so that the general camera unit 100 processes artificial intelligence image analysis as a server type through the deep learning module 510 and artificial intelligence image analysis as an edge type. The edge device camera unit 200 that processes can be combined.

Figure 4 is a diagram showing the detailed configuration of the deep learning module 510 in the hybrid artificial intelligence image analysis system according to an embodiment of the present invention. As shown in FIG. 4, in the hybrid artificial intelligence image analysis system according to an embodiment of the present invention, the deep learning module 510 of the image processing unit 500 includes a decoder module 511, a detection module 512, and It may be configured to include a sender module 513. Hereinafter, the data processing flow of the deep learning module 510 that performs object detection on a 4-channel basis will be described in detail with reference to FIG. 4.

First, ① the decoder thread of the decoder module 511 that receives the frame image of the first video from the general camera unit 100 may store the frame images in an image queue to perform object detection. ② The detection module 512 sequentially acquires frame images from the image queue, ③ performs object detection through the YOLOv4 deep learning model converted to OpenVINO for the acquired frame images, and then returns information about the detected objects to the image. Save to queue. ④ Obtain object detection information and frame images sequentially from the decoder module 511, ⑤ perform object tracking and event processing based on the acquired object detection information, then organize the results into packets and transmit them to TCP Save to queue. ⑥ The sender module 513 sequentially acquires packet information from the TCP queue, and ⑦ transmits the packet information acquired from the sender module 513 to the update DB to store/update the event DB contents.

Meanwhile, the deep learning module 510 can extract the first meta information from the first image based on OpenVINO. OpenVINO is an open source toolkit provided by Intel to build an inference engine optimized for deep learning inference. OpenVINO allows the use of an optimized and lightweight inference engine to perform calculations using the GPU built into Intel processors. Therefore, it is economical because it does not require an additional external GPU, and it can be used for general purposes because it is optimized for Intel processors, which have an absolutely high market share. In addition, through a deep learning model that has been lightweighted to operate only on the CPU without an external GPU, excellent results can be obtained in terms of inference speed and performance, and it is easy to convert to various models.

More specifically, by using OpenVINO, you can easily utilize deep learning models that have been widely used in existing deep learning, such as PyTorch and Tensorflow. In addition, models with excellent performance inevitably require high memory and resources, but optimization and lightweighting through inference engines help achieve excellent performance with small memory and resources. In particular, OpenVINO places the greatest emphasis on optimization and lightweighting, and through such lightweighting, the present invention can implement the inference function using the existing general camera unit 100.

Meanwhile, the edge device camera unit 200 includes a media distribution unit ( 300), data can be transmitted. At this time, the edge device camera unit 200 can control the image transmission mode and feature transmission mode according to the control signal received from the image processing unit 500.

In the feature transmission mode, the second meta information transmitted by the edge device camera unit 200 includes feature data extracted from the artificial intelligence model, image information of the specific area where the event occurred, the shape, size, speed, and shape of the detected object. It may consist of various information such as direction and color. More specifically, according to the image analysis role of the edge device camera unit 200 and the deep learning module 510 of the image processing unit 500, in the feature transmission mode, the edge device camera unit 200 uses the deep learning module 510. Data communication volume can be minimized by transmitting only the information necessary for video analysis. For example, when counting-based event detection is performed in the image processing unit 500, only the location, direction, and size information of the detected object is required, so unnecessary information such as the image and color of the object is not transmitted in the feature transmission mode. And through this, the amount of data transmitted can be reduced.

Additionally, when an event is detected, the image processing unit 500 may transmit a control signal to change the image transmission mode to the edge device camera unit 200. That is, the image processing unit 500 sets and controls the mode of the edge device camera unit 200, and detects events using characteristic information (second meta information) transmitted from the camera. When the image processing unit 500 detects an event, the transmission mode of the edge device camera unit 200 is changed to the video transmission mode, and the manager can verify the event on the second video transmitted in the video transmission mode. there is.

Artificial intelligence image analysis processed in the image processing unit 500 is more complex than artificial intelligence image analysis processed in the edge device camera unit 200, and may require greater computing power. In other words, by separating the feature extraction and event detection functions from the image into the camera and the server, the deep learning video analysis capacity of the image processing unit 500 can be increased and the computing capacity limitations of the edge device camera unit 200 can be overcome. In this way, simple deep learning video analysis is processed by the edge device camera unit 200, and highly complex video analysis can utilize only the advantages by applying a server-type video analysis system.

Figure 5 is a diagram showing an example of a parking lot management system using a hybrid artificial intelligence video analysis system according to an embodiment of the present invention. As shown in Figure 5, in the hybrid artificial intelligence image analysis system according to an embodiment of the present invention, the edge device camera unit 200 processes artificial intelligence image analysis to recognize vehicle license plates using an LPR model, and general The first image captured by the camera unit 100 can be implemented in a hybrid method by processing artificial intelligence image analysis that counts entering and exiting vehicles through the deep learning module 510 of the image processing unit 500.

Figure 6 is a diagram illustrating an intruder detection system using a hybrid artificial intelligence image analysis system according to an embodiment of the present invention as an example. As shown in FIG. 6, in the hybrid artificial intelligence image analysis system according to an embodiment of the present invention, the edge device camera unit 200 uses a face recognition model to recognize the face of the person entering and exiting the second image. Oriented artificial intelligence image analysis is processed, and the first image captured by the general camera unit 100 is used to process event-oriented artificial intelligence image analysis that detects intrusion through the deep learning module 510 of the image processing unit 500. You can. For example, if the face of an intruder that is not in the face DB is detected in face recognition using the edge device camera unit 200, the intruder can be detected and tracked through the general camera unit 100.

Figure 7 is a diagram showing an example of a face search system using a hybrid artificial intelligence image analysis system according to an embodiment of the present invention. As shown in Figure 7, in the hybrid artificial intelligence image analysis system according to an embodiment of the present invention, the edge device camera unit 200 detects the face area in the second image using a face recognition model, and the image processing unit ( Face recognition can be processed compared to the face DB of 500). In addition, with the first image captured by the general camera unit 100, the face area can be detected through the deep learning module 510 of the image processing unit 500 and compared with the face DB to process face recognition. In this way, face recognition can be processed quickly and simultaneously using all 16 cameras consisting of the general camera unit 100 and the edge device camera unit 200.

As described above, according to the hybrid artificial intelligence image analysis system proposed in the present invention, the image processing unit 500 processes artificial intelligence image analysis from the first image captured by the general camera unit 100 to generate first meta information. By receiving and processing data from the edge device camera unit 200, which includes a deep learning module 510 to estimate and estimates second meta information in real time from the second image captured with a built-in artificial intelligence model, artificial intelligence In video analysis, server type and edge type can be combined for processing. In addition, according to the present invention, by combining the general camera unit 100, which processes artificial intelligence image analysis in the image processing unit 500, with the edge device camera unit 200, which processes artificial intelligence image analysis as an edge type, a general camera An artificial intelligence video analysis system can be introduced by installing a portion of the edge device camera unit 200 while utilizing existing hardware resources without the need to replace the entire unit 100, and artificial intelligence calculations can be performed using the edge device camera unit ( By distributing processing between the image processing unit 200 and the image processing unit 500, the effect can be maximized by combining the advantages of quick response and minimal server burden of the edge type and the advantage of large-capacity processing of the server type.

The present invention described above can be modified or applied in various ways by those skilled in the art, and the scope of the technical idea according to the present invention should be determined by the claims below.

100: General camera unit
200: Edge device camera unit
300: Media distribution department
310: first RTSP client
320: Second RTSP client
400: Background processing unit
500: Image processing unit
510: Deep learning module
511: decoder module
512: detection module
513: Sender module
520: first decoder module
530: second decoder module
540: Render module
600: Local image storage unit
700: Database unit

Claims (7)

As a video analysis system,
A general camera unit 100 that captures a first image;
An edge device camera unit 200 that captures a second image and includes an artificial intelligence model to estimate second meta information in real time from the second image;
A media distributor 300 that receives and transmits the first image, the second image, and the second meta information from the general camera unit 100 and the edge device camera unit 200;
a background processing unit 400 that stores the second meta information received from the media distribution unit 300 in the database unit 700; and
It includes an image processing unit 500 that decodes the first and second images received from the media distribution unit 300 and processes them into an outputable form,
The image processing unit 500,
Using a deep learning model calculated using the GPU built into the processor, first meta information is estimated from the first image captured by the general camera unit 100, and the first meta information is stored in the database unit 700. Including a deep learning module 510 that stores,
The image processing unit 500 combines the general camera unit 100, which processes artificial intelligence image analysis, with the edge device camera unit 200, which processes artificial intelligence image analysis as an edge type,
The first meta information and second meta information are,
Each of the first image and the second image includes an object image and analysis information,
The edge device camera unit 200,
Transmitting data to the media distribution unit 300 according to a video transmission mode that transmits the second video and second meta information and a feature transmission mode that transmits only the second meta information and excludes the second video, ,
The media distribution unit 300,
A first RTSP client 310 that receives the first image from the general camera unit 100 using the RTSP protocol and transmits it to the first decoder module 520 of the image processing unit 500; and
The second image is received from the edge device camera unit 200 using the RTSP protocol and transmitted to the second decoder module 530 of the image processing unit 500, and the background processing unit 400 parses the second meta information. It includes a second RTSP client 320 configured to include a parser transmitted to,
The deep learning module 510,
Extracting the first meta information from the first image based on openVINO,
The artificial intelligence image analysis processed in the image processing unit 500 is more complex than the artificial intelligence image analysis processed in the edge device camera unit 200 and thus requires greater computing power, a hybrid artificial intelligence image analysis system. . delete delete delete delete delete delete

Images