KR20220082768A - Intelligent Video Big Data Analytics System in Cloud and Method thereof - Google Patents

Abstract

An intelligent video big data analysis system in a cloud and a method thereof are disclosed.
This system intelligently analyzes video big data in the cloud, and includes a real-time video stream acquisition unit that collects a real-time video stream collected from an external real-time video collection tool. In addition, the system provides a persistent and decentralized big data repository, a first big data repository for storing raw video streams of real-time video streams, managing large-scale structured data in a distributed environment, and processing data from real-time video streams. A second big data repository for storing intermediate results (IR) generated during real-time intelligent video analysis (RIVA) based on metadata and real-time video streams, and a real-time video stream acquisition unit, a first big data repository, and a second big data repository It contains the business logic that provides users' data access through web services. The real-time video stream acquisition unit includes an intermediate result manager that reads intermediate results generated during RIVA and stores them in a second big data storage for reuse avoiding recompilation.

Description

Intelligent Video Big Data Analytics System in Cloud and Method thereof

The present invention relates to an intelligent video big data analysis system in the cloud and a method therefor.

Videos are regularly recorded and uploaded to the cloud. Sources that are actively contributing to video generation include CCTV, smartphones, and drones, which have revolutionized big data in video management systems.

Processing large volumes of complex video data makes no good reason to use traditional data analysis approaches. Video big data analysis raises challenges for video management processing mining and manipulation. Therefore, a more comprehensive and sophisticated solution is needed to manage and analyze large-scale atypical image data. In addition, the volume of video data requires a flexible solution that can be stored and mined for possible decision-making.

However, with the rise of big data analysis and cloud computing technology, large-scale intelligent video analytics (IVA) has become a reality.

Big data technologies, such as the Hadoop or Spark ecosystem, are software platforms designed for distributed computing to process, analyze, and extract valuable insights from large data sets in a scalable and reliable way. Cloud computing is also a model that allows for rapid provisioning and release with minimal administrative effort or service provider interaction.

Recently, the deployment of distributed computing technology in the cloud for video big data analysis is attracting attention from academia and industry. However, these studies focus more on video analytics value extraction and overlook data curation issues such as managing value, volume, speed and diversity throughout the lifecycle of an IVA in the cloud. Likewise, understanding, configuring, and operating big data technologies for IVA in the cloud is tedious and time consuming, especially in educational settings. In addition, the IVA lifecycle in the cloud is approaching IVA centered, and existing solutions do not consider factors such as high-level and low-level function management while deploying IVA algorithms.

An object of the present invention is to provide an intelligent video big data analysis system and method in the cloud that can efficiently manage IR during the IVA life cycle.

The characteristic configuration of the present invention for achieving the object of the present invention as described above and for realizing the characteristic effects of the present invention to be described later is as follows.

According to one aspect of the present invention, there is provided an intelligent video big data analysis system, the system comprising:

A system for intelligently analyzing video big data in the cloud, providing a real-time video stream acquisition unit that collects a real-time video stream collected from an external real-time video collection tool, a permanent and distributed big data storage, and The first big data storage for storing raw video streams, managing large-scale structured data in a distributed environment, and real-time intelligent video analysis (Real-time Intelligent Video) based on metadata processed in the real-time video stream and the real-time video stream Analytics, RIVA) as an abstraction of a second big data storage for storing intermediate results (Intermediate Result, IR), and the real-time video stream acquisition unit, the first big data storage, and the second big data storage, the web Intermediate result manager that includes business logic for providing data access of users through service, wherein the real-time video stream obtaining unit reads the intermediate result generated during the RIVA and stores it in the second big data storage for reuse to avoid recompilation includes

Here, the real-time video stream acquisition unit is a broker client service module, a client application programming interface that collects the real-time video stream in a mini-batch format from a cluster of producers using a distributed messaging system and routes it to a user cluster, and provides a large-scale video stream. A video stream acquisition service module configured in the producer cluster for processing, a client application programming interface, wherein the video stream producer module receives records from the video stream acquisition service module and processes the received records to form a mini-batch; and Includes a lifelong video stream monitor that configures notifications for anomalies that occur during RIVA.

In addition, the broker client service module includes a topic manager that dynamically creates a new topic in a broker cluster when a new RIVA service is created, and automatically sets the number of partitions of topics according to the consumption of the video stream of the video stream data source and the stream of the consumer group. Includes a coordinating partition manager.

In addition, when the new RIVA service is created, the topic manager configures a RIVA_ID topic indicating the RIVA service, a RIVA_IR_ID topic indicating an intermediate result, and an RIVA_A_ID topic indicating an abnormal occurrence in the broker cluster.

In addition, the video stream acquisition service module includes: a video stream acquisition adapter that provides an interface for a device-independent video stream source, a video stream event producer that decodes the video stream, detects frames, and then performs metadata extraction; It includes a frame preprocessor that performs frame resizing on metadata, and a record constructor that defines a JavaScript Object Notation (JSON) object for communicating with a video stream source on the metadata.

In addition, the record in the video stream producer module has a basic message format, and includes fields of a topic name, a partition number, a key, and a value.

In addition, the intermediate result manager performs management in a distributed environment by transmitting and receiving intermediate results to and from the RIVA_IR_ID topic in the broker cluster.

In addition, when an abnormality occurs during the RIVA, the lifelong video stream monitor transmits it to the RIVA_A_ID in the broker cluster.

In addition, the first big data storage includes a user space corresponding to a user who has registered a service, and an access controller for managing the user space and data in the first big data storage, wherein the access controller includes: It includes a file system handler that manages file operations and controls according to logic, and an active data accessor that performs real-time and offline read-write operations.

In addition, the second big data storage provides a plurality of types of storage for storing data used in the intelligent video big data analysis system, and read-write access to the plurality of types of storage according to the business logic. including an access controller.

Additionally, the multiple types of repositories provide role-based access to users, user profiles and logs where user logs and role information are maintained through a user catalog, video stream sources that can subscribe to RIVA services, and batch IVA (Batch). IVA, BIVA) video data source metastore managing video data sets that can be subscribed to services, algorithm and service metastore managing IVA algorithms and services, intermediate results managing intermediate results delivered by the intermediate result manager It includes middleware, and a subscription meta store that manages subscription information of users who subscribe to video stream sources for RIVA services.

In addition, the access controller includes a connection configurator that maintains configuration parameters such as cluster configuration, connection information, drivers and security protocol parameters, a schema handler that maintains a schema structure of the second big data storage, and the second big data CRUD operators (Create, Read, Update, Delete operators) to perform create, read, update, and delete operations on persisted data in the repository, and read-write operations to load large amounts of data and keep it the same in distributed tables. It includes a manual accessor that provides

In addition, the business logic is a user manager that encapsulates all user-related tasks such as creating new user accounts, assigning access roles and managing sessions, enabling users to manage data in the distribution cloud of the intelligent video big data analytics system. A data source manager, an IVA algorithm and service manager that performs management, development and support of new IVA algorithms and services, an ontology data manager that enables developers to obtain intermediate results for decision-making, and an intelligent video big data manager that enables system administrators to Includes a cluster manager that allows you to monitor the status and functionality of the analytics system.

According to another aspect of the present invention, there is provided an intelligent video big data analysis method, the method comprising:

A method for a video big data analysis system to intelligently analyze video big data in a cloud, comprising: acquiring a video stream through a first cluster that provides an interface to an external video stream source according to a speed layer execution scenario; loading the video stream into a RIVA_ID topic of a second cluster, which is a broker cluster, and a third cluster that processes the video stream in real time while using a real-time intelligent video analytics (RIVA) service for the RIVA_ID video stream of the second cluster providing an RIVA service, wherein in the step of providing the RIVA service, an intermediate result generated during the RIVA service is transferred to the RIVA_IR_ID topic of the second cluster, and a fourth cluster that stores and manages the intermediate result is the Intermediate results are continuously read from the RIVA_IR_ID topic of the second cluster, and stored and managed.

Here, after loading into the RIVA_ID topic of the second cluster, the fifth cluster storing the video stream reads the video stream from the RIVA_ID topic of the second cluster, processing metadata in the video stream, and The method further includes storing the video stream in a first big data storage and storing the metadata in a second big data storage, respectively.

In addition, in the step of providing the RIVA service, when an abnormality occurs during the RIVA service, the generated abnormality is transferred to the RIVA_A_ID topic of the second cluster, and a sixth cluster notifying the occurrence of the abnormality occurs in the second cluster Anomalies are read from the RIVA_A_ID topic of the cluster, stored in the second big data storage, and delivered in real time to the owner of the video stream source through a web service.

According to another aspect of the present invention, there is provided an intelligent video big data analysis method, the method comprising:

A method for a video big data analysis system to intelligently analyze video big data in a cloud, comprising: acquiring external batch data through a first cluster that provides a batch video acquisition service according to a batch layer execution scenario; acquiring processing the batch data to extract batch video data and metadata; storing the batch video data in a first big data repository; storing the metadata in a second big data repository; and batch intelligent video analysis (Batch Intelligent Video Analytics, BIVA) comprising the step of providing a BIVA service for the batch video data stored in the first big data storage by a second cluster using the service, in the step of providing the BIVA service, the BIVA The intermediate result generated during the service is continuously stored in the second big data storage and read from the second big data storage again to be used when providing the BIVA service.

Here, before the step of providing the BIVA service, obtaining external batch model data through a third cluster providing a model storage service, processing the obtained batch model data to extract model data and file metadata; and storing the model data in the first big data storage and storing the file metadata in the second big data storage, wherein in the step of providing the IVA service, the second cluster The BIVA service is provided for batch video data and model data stored in the first big data storage.

In addition, in the step of providing the BIVA service, when an abnormal phenomenon occurs during the BIVA service, the generated abnormal phenomenon is transferred to the second big data storage and stored.

According to the present invention, IR can be efficiently managed during the IVA life cycle.

In addition, IVA algorithms can be easily pipelined to create domain-specific IVA services.

In addition, the implementation as a data source independent system is more scalable and compatible with IoT.

It is also facilitated through LVSM, which monitors video streams according to rules defined by domain experts in the IVA service.

In addition, distributed big data persistence is provided to support large amounts of raw video data model structured data IR, which enables the system to support data-driven knowledge generation techniques and predictive analytics visualization.

1 is a schematic block diagram of an intelligent video big data analysis system according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a detailed configuration of the RVSAS shown in FIG. 1 .
FIG. 3 is a diagram illustrating an operation example of the VSAS and VSP shown in FIG. 2 .
4 is a diagram illustrating a flow of video stream acquisition and synchronization according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a detailed configuration of the ISBDS shown in FIG. 1 .
FIG. 6 is a diagram illustrating a detailed configuration of the DPBDS shown in FIG. 1 .
FIG. 7 is a diagram illustrating a hierarchical structure of a user space of the DPBDS shown in FIG. 6 .
FIG. 8 is a diagram illustrating a detailed configuration of the business logic shown in FIG. 1 .
9 is a diagram illustrating user roles and use cases according to an embodiment of the present invention.
10 is a schematic flowchart of an intelligent video big data analysis method according to an embodiment of the present invention.
11 is a diagram illustrating a cluster example of an intelligent video big data analysis system according to an embodiment of the present invention.
12 is a diagram illustrating performance test results of VSAS and VSP of an intelligent video big data analysis system according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. have.

It should be understood that throughout the specification, the term “and/or” describes only an association relationship for describing a related object and indicates that three relationships may exist. For example, A and/or B may represent three cases in which only A exists, both A and B exist, and only B exists. Also, throughout the specification, the character "/" generally indicates an "or" relationship between related objects.

The devices described in the present invention are composed of hardware including at least one processor, a memory device, a communication device, and the like, and a program to be executed in combination with the hardware is stored in a designated place. The hardware has the configuration and capability to implement the method of the present invention. The program includes instructions for implementing the method of operation of the present invention described with reference to the drawings, and is combined with hardware such as a processor and a memory device to execute the present invention.

Hereinafter, a system for intelligent video big data analysis in the cloud according to an embodiment of the present invention will be described with reference to the drawings.

1 is a schematic block diagram of a system for intelligent video big data analysis according to an embodiment of the present invention.

1, the system 10 for intelligent video big data analysis according to an embodiment of the present invention includes a Real-Time Video Stream Acquisition and Synchronization (RVSAS) 100, Immediate Structured Big Data Store (ISBDS) (200), Distributed Persistent Big Data Storage (DPBDS) (300), Business Logic (400) and Web Services ( Web Service) (500).

The RVSAS 100 collects and processes a real-time video stream collected from an external, for example, an external real-time video collection tool.

RVSAS 100, for example, utilizes Apache Kafka, a Distributed Messaging System 101, to ingest video streams in a mini-batch format from a cluster of producers, and a Kafka Broker , and then routes the mini-batch to the user cluster.

The ISBDS 200 manages large-scale structured data in a distributed environment. It supports a distributed NoSQL big data store called Apache HBase according to data-intensive tasks and the requirements of different layers.

The DPBDS 300 is built on top of HDFS and provides a persistent and distributed big data storage.

Business logic 400 is a high-level abstraction for RVSAS 100 , ISBDS 200 , and DPBDS 300 , and is provided to customize data access according to the design philosophy of system 10 of the present invention. .

The web service 500 integrates the best functions into a simple integrated role-based web service in order to provide the functions of the framework of the system 10 according to an embodiment of the present invention through the web. This web service 500 is built based on the business logic 400 described above.

FIG. 2 is a diagram illustrating a detailed configuration of the RVSAS 100 shown in FIG. 1 .

2, the RVSAS 100 is a broker client service (Broker Client Services, BCS) 110, a video stream acquisition service (Video Stream Acquisition Service, VSAS) 120, a video stream producer (Video Stream Producer, VSP) ) 130 , an Intermediate Result-Manager (IR-Manager) 140 , and a Lifelong Video Stream Monitor (LVSM) 150 .

BCS 110 utilizes Apache Kafka to ingest video streams in mini-batch format from a producer's cluster, buffers it in a Kafka Broker, and routes the mini-batch to user clusters. The BCS 110 performs video stream collection according to the business logic 400 of the framework proposed to manage a Kafka broker for large-scale real-time video stream management.

The BCS 110 includes a topic manager (Topic Manager) 111, a partition manager (Partition Manager) 112, and the like.

The topic manager 111 is used to dynamically create a new Kafka topic in a Kafka broker cluster when a new RIVA (Real-time IVA) service is created. When a new RIVA service is created, three types of Kafka topics are automatically created in the Kafka broker cluster using the naming convention RIVA_ID, RIVA_IR_ID, and RIVA_A_ID. Here, ID is a unique identifier of the service. These items are used to detect anomalies detected by the actual video stream IR and IVA services.

The topic manager 111 allows the administrator role to list all topics, check their configuration, and override the configuration. Each topic consists of partitions where video stream records are distributed. In general, the degree of parallelism and the height throughput are proportional to the number of partitions in the topic.

The partition manager 112 automatically adjusts the number of partitions of a topic according to the consumption of a video stream of a video stream data source and a stream of a consumer group.

Next, the VSAS 120 is a client application programming interface (API), which can be configured in a cluster of producers to take a large-scale video stream.

Referring to FIG. 3, the VSAS 120 includes a video stream acquisition adapter 121, a video stream event producer 122, and a frame preprocessor 123. and a Record Composer 124 .

The video stream acquisition adapter 121 provides an interface for a device independent video stream source. When a specific video source is subscribed to the RIVA service, the Stream Acquisition service gets the configuration metadata from the Video Data Source (VDS) of the ISBDS 200 and configures the source device for video streaming.

The video stream event producer 122 decodes the video stream, detects frames, and then performs metadata extraction.

The frame preprocessor 123 performs frame size adjustment on the extracted metadata.

Record Configurator 124 defines a JavaScript Object Notation (JSON) object to communicate with the video stream source for metadata. The content of this object consists of five fields: the data source ID, the number of columns and rows of the frame, the timestamp where the data started from the data source, and the payload. This JSON object is passed to the VSP 130 as a record.

The VSP 130 is also a client API, which receives records from the VSAS 120, serializes the received records, constructs a mini-batch, and sends them to the Kafka broker.

When the video stream source is registered with the RIVA service, the producer handler 170 routes the mini-batch to the topic RIVA_1 of the broker cluster. Similarly, if you subscribe to multiple RIVA services, streams are routed to that topic.

The Kafka producer record in the VSP 130 has a basic message format and consists of four fields, namely, a topic name, a partition number, a key, and a value. Here, the topic name is dynamically provided by the subscription data source stored in the ISBDS 200 . The partition and key fields are set to the video stream source ID (camera ID provided in the video data source metastore of the ISBDS 200). The value field input represents a video stream record configured and provided by the record configurator 124 .

The VSP 130 is sent to the serializer 171 of the producer handler 170 to convert the producer record to a byte array for sending over the network. The VSP 130 collects the producer records and adds them to the batch of records for sending to the Kafka broker.

A fast compression algorithm is used to compress these mini-batches, and the VSP 130 sends the newly formed mini-batches to the broker server. BACK of the VSP 130 is activated to ensure message delivery.

The above-described sub-modules of the VSAS 120 and VSP 130 may be technically encapsulated in the videoStreamEventProducer high-level API as shown in the following algorithm.

4 , the obtained video stream now resides in Kafka broker 161 of various topics in the form of mini-batch, Video Stream Analytics Consumer Cluster to process this mini-batch of video stream ) (163), there are various computer clusters. Two types of client services are configured in each cluster: a RIVA service 1631 and a Video Stream Consumer Service (VSCS) 1633 . Each video stream analysis consumer cluster 163 has a different domain-specific RIVA service 1631, whereas the VSCS 1633 are all common. VSCS 1633 supports the RIVA service to read mini-batches of video streams in each topic of Kafka broker 161 for analysis.

The RIVA service 1631 should generate and persist the IR. Video analytics is always expensive and keep the IR as ISBDS 200 (IR middleware) to avoid recalculation.

The IR-manager 140 exchanges IR with the topic RIVA_IR_ID of the Kafka broker cluster 141, and performs appropriate management in a distributed environment. The IR-manager 140 also serves to read the IR on the topic and is maintained in the IR data store of the ISBDS 200 for reuse (eg mapping to video ontology, indexing for retrieval) avoiding recompilation. .

The domain-specific RIVA service processes video streams for abnormal or abnormal activity.

When an abnormality is detected, the LVSM 150 sends the same content to the Kafka broker topic, that is, (RIVA_A_ID). To generate an alert-based response, the LVSM 150 follows a standard observer-based implementation. Based on this approach, the LVSM 150 reads anomalies in the corresponding Kafka broker topic. That is, it is RIVA_A_ID, which informs the client in near real time and keeps it in the ISBDS at the same time. Create a template service to understand the VSAS 1433, IR-Manager) 140 and LVSM 150 (refer to the algorithm below).

These services are based on Apache Spark Streaming. Inputs to the RIVA service are serviceID and subscribed cameraID. The video stream mini-batch is obtained from the topic RIVA_ID using the VSAS high-level abstraction and assigned to the videoDataset (object of the spark dataset abstraction) as in step 4. RIVA operations (ad hoc or ad hoc analysis) are performed on videoData according to the developer's logic and requirements. The IR and abnormal RIVA outputs are delivered to the IRM 140 and LVSM 150 API, respectively.

FIG. 5 is a diagram illustrating a detailed configuration of the ISBDS 200 shown in FIG. 1 .

Referring to FIG. 5 , the ISBDS 200 hosts five types of data. The five types of data are User Profile and Logs (210), Video Data Source Meta-store (220), and Algorithm and Services Meta Store. 230 , Intermediate Result Middleware 240 , and Subscription Meta-store 250 are included.

First, user profiles and logs 210 provide role-based access to users. User logs and their role information are maintained through the user catalog. All sensitive user information is safely protected by supporting MD5 encryption scheme.

Second, the video data source meta store 220 manages two types of video sources: video stream sources and video data sets. The former may subscribe to the RIVA service, and the latter may subscribe to the BIVA (Batch IVA) service. The video data source meta store manages meta information of these sources along with access rights.

Third, the algorithm and service meta store 230 manages the IVA algorithm and service. In particular, the metadata of the IVA algorithm is managed through the algorithm metastore, and the data structure consists of the following 8 different data fields. i.e. AID, UID, algoType, algoName, input, output, requiredResources, sourceCodeAddress, description. The metadata of IVA services is managed through the service metastore, and includes attributes such as SID, UID, dsTypeID, serviceTypeID, serviceName, and description. The relationship between the service and the algorithm metastore is managed through a lookup table called SERVICE_ALGORITHM.

Fourth, the intermediate result middleware 240 is used to store the intermediate result.

Fifth, the subscription meta store 250 manages subscription information for a user to subscribe to a video stream source for the RIVA service.

The ISBDS 200 includes an ISBDS access controller (Access Controller) 260 . The ISBDS access controller 260 serves to securely provide read-write access to the underlying data according to the business logic 400 .

The ISBDS access controller 260 includes a Connection Configurator 261 , a Schema Handler 262 , a CRUD Operator (Create, Read, Update, Delete Operator) 263 , and a manual Passive Data Reader (DRW). , Writer) 264 .

The connection configurator 261 is used to maintain configuration parameters such as cluster configuration, connection information, drivers and security protocol parameters. The connection configurator 261 is used to establish and maintain a connection session with the Distributed Big Data Store 270 when there is a data access request.

The basic schema structure of the ISBDS 200 is maintained through the schema handler 262, and the same can be used for portability.

The CRUD operator 263 is designed and provided for CRUD operations on data persisted in the ISBDS 200 . The CRUD operator 263 is primarily designed to provide low latency real-time interactive CRUD operations via the distributed data store 270 . An instance of the CRUD operator 263 is used in the distributed RIVA engine and business logic 400 . Predefined generalized queries are hosted for CRUD operations. Upon receiving the request for the client, the CRUD operator 263 matches the request parameters with the available query set. If the parameters match, the selected query is executed in real time in response to the client. It is also designed to support offline analysis of large volumes of video while using a distributed in-memory processing engine such as Apache Spark.

A manual DRW 264 is provided to allow Apache Spark to load large amounts of data into RDDs (RDDs are Spark data structures) and keep them the same in distributed tables as needed. Execution of manual DRW 264 is a two-step process. first. The client application selects the schema parameters and then submits them to the manual DRW 264 . Passive DRW 264 dynamically creates a query upon receiving parameters. The query is then executed through the NoSQL distributed data store 270 and a response is generated and provided to the client program. Manual DRW 264 is provided to meet offline analysis requirements, and query execution time may take longer depending on the requested data size.

FIG. 6 is a diagram illustrating a detailed configuration of the DPBDS 300 shown in FIG. 1 .

6, data is systematically stored and mapped according to the business logic 400 of the system 10 of the present invention. The DPBDS 300 is designed to effectively and dynamically manage data workloads in the life cycle of the IVA algorithm. When newly registered in the system 10 of the present invention, a formal user space 310 is created on top of the HDFS (or Hadoop) 301 . Userspace 310 is managed through an appropriate hierarchical directory structure and the owner is granted special read and write access. All directories are synchronized and mapped according to the user ID in the user profile log. In HDFS, three types of distributed directories are created under each user space 310 as shown in FIG. 7 . That is, the raw video space (Raw Video Space) 320, the model space (Model Space) 330, and the project space (Project Space) 340 are them. The DPBDS 300 includes a DPB access controller 350 to manage and operate the user space 310 and corresponding data of the DPBDS 300, which includes an HDFS handler 351, active DRW. (Active DRW) 352 and passive DRW (Passive DRW) 353 . The HDFS handler 351 is designed through the low-level API of HDFS to manage file operations and permission control according to the business logic 400 . Active DRW 352 and passive DRW 353 perform real-time and offline read-write operations.

The aforementioned RVSAS 100 , ISBDS 200 , and DPBDS 300 are designed to support large-scale video data collection and domain-specific IVA in the cloud while utilizing advanced IVA algorithms.

FIG. 8 is a diagram illustrating a detailed configuration of the business logic 400 shown in FIG. 1 .

Referring to FIG. 8 , the business logic 400 is expandable according to future requirements, but in an embodiment of the present invention, a User Manager 410 , a Data Source Manager 420 , and an IVA It includes an algorithm and service manager (IVA Algorithm and Service Manager) 430 , an ontology data manager 440 , and a cluster manager 450 .

User manager 410 encapsulates all user related tasks such as creating new user accounts, assigning access roles, and managing sessions. The system 10 according to an embodiment of the present invention provides role-based access to users and provides three types of user accounts, namely, admin, developer, and consumer as shown in FIG. 9 . Referring to FIG. 9 for user roles and access rights, user data access logic is implemented in user manager 410 .

The data source manager 420 allows a user to manage data in the distribution cloud of the present system 10 . Users can manage three types of data sources: video stream sources, video data sets and models.

The IVA algorithm and service manager 430 manages, develops and supports new IVA algorithms and services based on Spark. The IVA algorithm provides developers with a aaS such as IVA-Algorithm-as-a-Service (IVAAaaS), and service management provides IVAaaS (IVA-as-a-Service) to consumers. Developers can create and publish new IVA algorithms. These IVA algorithms are provided as IVAAaaS for other developers to use. When an admin/developer creates and publishes an IVA service, users can subscribe to video sources for the provided IVA service.

The ontology data manager 440 enables developers to obtain intermediate results (IRs) for decision-making. The ontology data manager 440 provides a secure method of obtaining an IR, and maps it according to a video ontology of a knowledge curation layer (KCL).

Similarly, the cluster manager 450 allows the system administrator to monitor the status and functions of the system 10 .

Finally, the web service 500 integrates the best functions into a simple integrated role-based web service to provide the functions of the framework of the system 10 according to an embodiment of the present invention through the web. This web service 500 is built based on the business logic 400 described above.

Hereinafter, an intelligent video big data analysis method according to an embodiment of the present invention will be described. This method may be performed by the intelligent video big data analysis system 10 described above with reference to FIGS. 1 to 9 .

Using the system 10 according to an embodiment of the present invention, participating layers can register for distributed RIVA and BIVA services. The present system 10 follows the lambda architecture style and the execution scenarios go through two types of execution scenarios. Here, the two types of execution scenarios are a speed layer execution scenario and a batch layer execution scenario. Data in these two scenarios is managed through a common layer called Big Data Persistence (BDP) including the ISBDS 200 and the DPBDS 300, also known as the serving layer.

First, a speed layer execution scenario will be described with reference to FIG. 10 .

Cluster 'P' 901 hosts VSAS 120 and provides an interface to external video stream source 903 . Upon configuration, the video stream is loaded into the corresponding RIVA_ID 907 of cluster 'K' 905 .

Cluster 'K' 905 supports Kafka, a distributed messaging system 101 where the acquired video stream, IR and anomalies generated by LVSM 150 are buffered. Here, the cluster 'K' 905 includes RIVA_ID 907 , RIVA_IR_ID 909 , and RIVA_A_ID 911 . These topics are replicated to access cluster 'K' 905 to ensure high throughput. RF (Replication Factor) must be less than or equal to the number of workers in cluster 'K' 905 .

A mini-batch of video streams in topic RIVA_ID 907 of broker cluster 905 of distributed messaging system 101 should be maintained in DPBDS 300 and ISBDS 200 data stores. To this end, the cluster 'V' 913 has three types of modules: a Video Stream Consumer (VCS) module 915, a Video Preprocessor module 917 and a Persistence module ( 919) is developed.

Using the VCS module 915 , cluster 'V' 913 can read the video stream mini-batch in the RIVA_ID 907 topic of cluster 'K' 905 . The video preprocessor module 917 then processes, encodes and extracts metadata from the consumed video data. Finally, the persistence module 919 stores video data and corresponding metadata in the DPBDS 300 and the ISBDS 200, respectively.

Meanwhile, the cluster 'S' 921 serves to process a video stream in near real time while using the RIVA service. Various stream processing engines, such as Apache Spark Stream, are available for developing RIVA services. Cluster 'S' 921 has four types: Video Stream Consumer Service (VSCS) module 923, RIVA service module 925, IR Producer/Subscriber module 927 ) and an LSVM Producer module 929 .

VSCS module 923 is used to use RIVA_ID video stream 907 in cluster 'K' 905 .

The RIVA service module 925 is an actual RIVA service that analyzes a video stream (S915). These RIVA Services are loaded in accordance with the RIVA Services Subscription Agreement entered into by the User.

The IR producer/subscriber module 927 is used to send and receive IR according to the application logic with the topic IR 909 of cluster 'K' 905 .

A RIVA service instance deployed in cluster 'S' 921 should have some domain specific goals and should generate anomalies when analyzed. The system 10 supports a real-time anomaly delivery system. The RIVA service continues to send the anomaly via the LVSM producer 929 to each anomaly topic RIVA_A_ID 911 of cluster 'K' 905 .

Cluster group 'I' 931 continuously reads IR from topic RIVA_A_ID 909 of cluster 'K' 905 and transmits it to IR middleware 240 of ISBDS 200 for indexing.

Cluster 'N' 933 is an anomaly notification cluster. This cluster 'N' 933 reads the anomaly from the topic RIVA_A_ID 911 of cluster 'K' 905 and sends the same to the ISBDS 200 for persistence and communicates in real time to the video stream source owner of the alert. .

Next, a deployment layer execution scenario will be described with reference to FIG. 10 again.

The Batch Layer Service, or BIVA Service, is a aaS that can be used for batch video analytics. Batch video data sets are analyzed offline and execution time is proportional to the video data set size and the BIVA service you subscribe to.

A deployment layer execution scenario according to an embodiment of the present invention goes through three types of clusters: cluster 'R' 941 , cluster 'M' 943 , and cluster 'B' 945 .

Cluster 'R' (941) allows users to upload batch video data sets to system (10) and includes three types of libraries: Batch Video Acquisition Service (947) and Video Preprocessor (Video Preprocessor). ) 949 and Batch Video Persistence 951 can be configured.

Batch video acquisition service 947 is used to acquire a batch video data set.

When the batch video data set is uploaded to the node buffer by the batch video acquisition service 947, the batch data set is processed in the activated video preprocessor 949 to extract the metadata of the batch video.

Then, the batch video data and the extracted metadata are maintained in the DPBDS 300 and the ISBDS 200, respectively.

Likewise, cluster 'M' 943 works in the same way as cluster 'R' 941 , but cluster 'M' 943 is responsible for model management.

In batch video analysis, the support layer supports the BIVA service. Cluster 'B' 945 loads the BIVA service instance according to the user contract and processes the video offline. Once subscribed, this cluster 'B' 945 loads the batch video data set and model from the DPBDS 300 . Similarly, IR and anomalies are maintained in ISBDS 200 .

Next, an example of applying the intelligent video big data analysis system 10 according to an embodiment of the present invention will be described.

The application of the embodiment of the present invention was performed for two RIVA services, namely, face detection and motion recognition. Both of these applications are delivered as a service.

While deploying Hortonworks Data Platform (HDP) version 3.1.0, an indoor distributed cloud environment called cluster was built. As shown in FIG. 11, the cluster consists of 11 nodes. Each node has 5 parameters. The parameter on the top row indicates the operating system version in use. The second line shows the processor model, number of cores, RAM size in GB and hard disk size in GB, respectively. For high-density copper connections for networking purposes, a ProSafe GSM7328S fully managed switch was used that provides 24 and 48 ports with auto-sensing 1000Mbps interfaces.

Users can expose real-time video stream sources to RIVA. When registering, two services and three types of topics are automatically created by the topic manager 111 as shown in [Table 1] below. Set RF to 3 (since we have 3 broker servers) and set the number of partitions to 140 per topic (allowing maximum number of camera streams).

[Table 1]

In addition, different parameter values are set in the cluster configurator (not shown), the ISBDS 200 and the DPBDS 300, which are sub-modules of the BCS 110 as shown in Table 2 below.

[Table 2]

Since the VSAS 120 is device-independent, it registers different heterogeneous devices and offline video stream sources. Heterogeneous video stream sources include IP cameras, depth cameras, RTSPs, and iPhone6s Plus. By default, the frame rate for the first three data sources is 30, and the last data source is at 60 frames per second.

For offline video stream sources, the video files on the HDD (WDC WD10EZEX) were also made up of elements of the VSAS 120. The VSAS 120 sets the resolution of the acquired frame to 480 x 320 pixels. As a result, the size of each acquired frame was 614.495 KB.

The VSAS 120 converts the acquired frame into a format message at a rate of 6MS. VSAS 120 then forwards the message to VSP 130 . The VSP 130 compresses the message size to an average of 140.538 KB, and delivers it to the broker server at a speed of 12 MS. These two modules are configured to acquire and transmit streams to the broker server (Agent-6, Agent-7 and Agent-8, see Fig. 11). The performance test result is shown in FIG. 12 . From the results shown, it can be seen that on average, 34 and 54 frames per second can be obtained from heterogeneous video stream sources and offline video stream sources, respectively. The rates achieved are 36% and 116% higher than preferred. That is, in the case of RIVA analysis, it is 25 frames per second.

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also provided. is within the scope of the right.

Claims (19)

As a system for intelligently analyzing video big data in the cloud,
A real-time video stream acquisition unit that collects a real-time video stream collected from an external real-time video collection tool;
A first big data storage providing a persistent and distributed big data storage, storing a raw video stream of the real-time video stream;
Manage large-scale structured data in a distributed environment, and intermediate results generated during Real-time Intelligent Video Analytics (RIVA) based on metadata processed in the real-time video stream and the real-time video stream , a second big data storage that stores IR), and
As an abstraction of the real-time video stream acquisition unit, the first big data storage, and the second big data storage, a business logic that provides a user's data access through a web service
includes,
The real-time video stream obtaining unit includes an intermediate result manager that reads the intermediate result generated during the RIVA and stores it in the second big data storage for reuse to avoid recompilation,
Intelligent video big data analysis system. According to claim 1,
The real-time video stream obtaining unit,
a broker client service module for collecting and routing said real-time video streams from clusters of producers in mini-batch format to clusters of users using a distributed messaging system;
A client application programming interface comprising: a video stream acquisition service module configured in the producer cluster to process a large video stream;
A client application programming interface, comprising: a video stream producer module for receiving records from the video stream acquisition service module and processing the received records to form a mini batch; and
A lifelong video stream monitor that configures notifications for anomalies that occur during the RIVA
Including, intelligent video big data analysis system. 3. The method of claim 2,
The broker client service module,
A topic manager that dynamically creates a new topic in the broker cluster when a new RIVA service is created, and
A partition manager that automatically adjusts the number of partitions in a topic based on the consumption of a video stream from a video stream data source and a stream from a group of consumers.
Including, intelligent video big data analysis system. 4. The method of claim 3,
When the new RIVA service is created, the topic manager configures a RIVA_ID topic representing the RIVA service, a RIVA_IR_ID topic representing an intermediate result, and a RIVA_A_ID topic representing an abnormal occurrence in the broker cluster when the new RIVA service is created.
Intelligent video big data analysis system. 3. The method of claim 2,
The video stream acquisition service module,
a video stream acquisition adapter that provides an interface for a device-independent video stream source;
a video stream event producer that decodes the video stream, detects frames, and then performs metadata extraction;
a frame preprocessor that performs frame resizing on the extracted metadata; and
A record constructor that defines a JavaScript Object Notation (JSON) object for communicating with the video stream source for said metadata.
Including, intelligent video big data analysis system. 3. The method of claim 2,
The record in the video stream producer module has a basic message format, and includes fields of a topic name, a partition number, a key, and a value,
Intelligent video big data analysis system. 3. The method of claim 2,
The intermediate result manager performs management in a distributed environment by transmitting and receiving intermediate results to and from the RIVA_IR_ID topic in the broker cluster,
Intelligent video big data analysis system. 3. The method of claim 2,
The lifelong video stream monitor transmits to RIVA_A_ID in the broker cluster when an abnormality occurs during the RIVA,
Intelligent video big data analysis system. According to claim 1,
The first big data storage is
A user space corresponding to a user who has registered the service, and
an access controller for managing data in the user space and the first big data repository
includes,
the access controller,
A file system handler for managing file operations and related control according to the business logic, and
Active data accessor performing real-time and offline read-write operations
Including, intelligent video big data analysis system. According to claim 1,
The second big data storage,
A plurality of types of storage for storing data used in the intelligent video big data analysis system, and
an access controller providing read-write access to the plurality of types of storage according to the business logic
Including, intelligent video big data analysis system. 11. The method of claim 10,
The plurality of types of storage,
User profiles and logs that provide role-based access to users, and user logs and role information is maintained through a user catalog;
A video data source metastore that manages video stream sources that can subscribe to the RIVA service and video data sets that can subscribe to the Batch IVA (BIVA) service;
Algorithm and service metastore that manages IVA algorithms and services;
an intermediate result middleware for managing intermediate results delivered by the intermediate result manager; and
Subscription metastore managing subscription information of users who subscribe to video stream sources for RIVA service
Including, intelligent video big data analysis system. 11. The method of claim 10,
the access controller,
a connection configurator that maintains configuration parameters such as cluster configuration, connection information, drivers and security protocol parameters;
a schema handler that maintains the schema structure of the second big data storage;
A CRUD operator (Create, Read, Update, Delete Operator) that performs create, read, update, and delete operations on data persisted in the second big data storage, and
A passive accessor that provides read-write operations for loading large amounts of data and keeping them the same in distributed tables.
Including, intelligent video big data analysis system. According to claim 1,
The business logic is
a user manager that encapsulates all user-related tasks, such as creating new user accounts, assigning access roles, and managing sessions;
a data source manager that allows users to manage data in the distribution cloud of the intelligent video big data analysis system;
IVA Algorithms and Services Manager to perform management, development and support of new IVA Algorithms and Services;
An ontology data manager that enables developers to obtain intermediate results for decision-making, and
A cluster manager that enables system administrators to monitor the status and function of the intelligent video big data analysis system.
Including, intelligent video big data analysis system. A method for a video big data analysis system to intelligently analyze video big data in the cloud, comprising:
acquiring a video stream through a first cluster that provides an interface to an external video stream source according to a speed layer execution scenario;
loading the obtained video stream into the RIVA_ID topic of the second cluster, which is a broker cluster, and
A third cluster that processes a video stream in real time while using a real-time intelligent video analysis (RIVA) service provides a RIVA service for the RIVA_ID video stream of the second cluster
includes,
In the step of providing the RIVA service, the intermediate result generated during the RIVA service is transferred to the RIVA_IR_ID topic of the second cluster, and the fourth cluster storing and managing the intermediate result receives the intermediate result from the RIVA_IR_ID topic of the second cluster continuously reading, storing and managing,
Intelligent video big data analysis method. 15. The method of claim 14,
After loading into the RIVA_ID topic of the second cluster,
The fifth cluster storing the video stream reading the video stream from the RIVA_ID topic of the second cluster;
processing metadata in the video stream; and
Storing the video stream in a first big data storage and storing the metadata in a second big data storage, respectively
An intelligent video big data analysis method further comprising a. 16. The method of claim 15,
In the step of providing the RIVA service, when an abnormality occurs during the RIVA service, the generated abnormality is transferred to the RIVA_A_ID topic of the second cluster, and the sixth cluster notifying the occurrence of the abnormality Reading anomalies from the RIVA_A_ID topic, storing them in the second big data storage, and delivering them in real time to the owner of the video stream source through a web service,
Intelligent video big data analysis method. A method for a video big data analysis system to intelligently analyze video big data in the cloud, comprising:
According to a batch layer execution scenario, acquiring external batch data through a first cluster providing a batch video acquisition service;
processing the obtained batch data to extract batch video data and metadata;
storing the batch video data in a first big data repository and storing the metadata in a second big data repository; and
providing, by a second cluster using a Batch Intelligent Video Analytics (BIVA) service, a BIVA service for the batch video data stored in the first big data repository;
includes,
In the step of providing the BIVA service, the intermediate result generated during the BIVA service is continuously stored in the second big data storage and read from the second big data storage again to be used when the BIVA service is provided,
Intelligent video big data analysis method. 18. The method of claim 17,
Before providing the BIVA service,
acquiring externally deployed model data through a third cluster that provides a model storage service;
processing the obtained batch model data to extract model data and file metadata; and
storing the model data in the first big data storage and storing the file metadata in the second big data storage;
further comprising,
In the step of providing the IVA service, the second cluster provides the BIVA service for batch video data and model data stored in the first big data storage,
Intelligent video big data analysis method. 18. The method of claim 17,
In the step of providing the BIVA service, when an abnormality occurs during the BIVA service, the generated abnormality is transferred to the second big data storage and stored,
Intelligent video big data analysis method.

Images