KR102324804B1 - Monitoring system by generating context information of object in image data - Google Patents

Abstract

In order to provide a technique for reducing the construction cost of an object monitoring system by enabling identification and monitoring of an object, etc., without establishing a special communication network and installing a separate additional device, the object in the image according to an embodiment of the present invention A monitoring system through context information generation is implemented as a computing device including one or more processors and one or more memories for storing instructions executable in the processor, provided for each object, and object-related information to be monitored is stored. a database generator generating a first table that is a data table and a second table that is a data table in which context information of an object is stored; When the image data including the object is received, the region of interest, which is the region including the object, is set, and object state information, which is information that can determine the state of the object, is inferred from the image data in the set region of interest using the object state information inference model. an object state inference unit that stores the data in the first table; and a context information generator for generating context information of an object through time-series analysis of object state information, and storing the generated context information in a second table.

Description

Monitoring system by generating context information of objects in the image {MONITORING SYSTEM BY GENERATING CONTEXT INFORMATION OF OBJECT IN IMAGE DATA}

The present invention relates to a technology for identifying and monitoring objects such as operation equipment, signal display equipment, and human body that require motion and state detection on image data, and more specifically, a high-capacity data communication function and object identification and tracking function The present invention relates to a technology for enabling the use of photographing results of various image photographing devices that are not used for object monitoring, weight reduction of object monitoring information, and efficient management of a database.

Recently, as it is applied to smart factories and management systems, the number of facilities to be managed within a specific area is greatly increasing. As the number of objects such as equipment to be monitored in a factory or a general area increases, it takes a lot of manpower and time to monitor the state of the equipment.

Accordingly, in technologies such as smart factories and cities, by establishing a communication network on facilities and devices to be monitored, and using this to remotely receive and detect the status of facilities and devices, the manpower required for monitoring facilities and devices and Techniques are being applied to save time.

These technologies include, for example, Korean Patent Laid-Open Patent No. 10-2018-0137827, etc., which use a communication network between a client terminal such as a control terminal and a sensor such as a facility and device, and receive data using a sensor installed in the facility and use this to remotely monitor the status of facilities and devices.

However, such prior art and object monitoring of conventional equipment in general requires the installation of a completely new facility by setting up a communication network between the remote control server and each object, installation of a sensor for the object, etc. , and accordingly, there is a problem in that a lot of time and money are consumed to build a remote monitoring system for an object.

In particular, since it is very difficult to newly construct a communication network in an existing facility such as a building or a factory, there are many difficulties in practically applying this technology. Accordingly, it can be applied to existing facilities such as buildings or factories, and it is possible to remotely monitor objects such as each facility without the need to establish a separate communication network, but it is possible to accurately and conveniently monitor the operation status of each object. The need for existing technologies is pointed out.

Accordingly, an object of the present invention is to provide a technology for reducing the construction cost of an object monitoring system by enabling identification and monitoring of objects, etc., without establishing a special communication network and installing a separate additional device.

In particular, in analyzing image data and providing monitoring information through it, rather than transmitting and receiving a large amount of data such as image data, context information of an object that can be identified from an image is extracted, and based on this, very simple and lightweight data monitoring is performed. The purpose of the present invention is to provide a technology capable of building a very fast and efficient object monitoring system by enabling the network to be lightweight by making it usable as data.

In order to achieve the above object, a monitoring system through generation of context information of an object in an image according to an embodiment of the present invention is a computing device including one or more processors and one or more memories for storing instructions executable by the processor a database generation unit that is provided for each object and generates a first table that is a data table for storing information related to an object to be monitored and a second table that is a data table for storing context information of the object; When the image data including the object is received, the region of interest, which is the region including the object, is set, and object state information, which is information that can determine the state of the object, is inferred from the image data in the set region of interest using the object state information inference model. an object state inference unit for storing the data in the first table; and a context information generator for generating context information of an object through time-series analysis of object state information, and storing the generated context information in the second table.

The object state inference unit may include: a region of interest setting unit configured to set, as a region of interest, a bounding box in which an object exists in any one image data included in the image data; A request signal for generating the first table for each object within a set ROI is transmitted to the database generator, and object attribute information used to infer the object state information as information to be extracted from the object is transmitted to the first a table configuration unit that creates a table column; By applying the object state information inference model to the image data, the object on the region of interest in the image data is recognized based on the set region of interest, and after detecting the state of the object, a time stamp on the image data, video keyframe information, an object state information extraction unit for extracting coordinates of an object on the recognized region of interest; and object state information for storing a time stamp on the extracted image data, video keyframe information, coordinates of an object on a recognized region of interest, attribute information of the object, and information on the state of the detected object in the first table prepared for each object It is preferable to include a storage unit.

The context information generating unit analyzes the first table based on the time stamp, and extracts the periodic repetition state and change state of the object state information of the unit time before each time period and the object state information of the current state. It is preferable to generate context information on the situation and operation of the object in the state for the entire image data, and store a plurality of generated context information for each object in the second table provided for each object.

It is preferable to further include a monitoring information generating unit for processing the context information and object state information of the object stored in the first table and the second table to generate monitoring information of the object from which the image data is excluded.

The monitoring information generating unit selects identification information corresponding to the type of the analysis target object from among identification information provided for each type of object, creates a dashboard to which the selected identification information is applied, and then includes a first table for the analysis target object and a data preparation unit for loading the second table; And by applying some of the context information and object property information of the object from the loaded first table and the second table to the identification information so that some of the context information and object property information of the object are displayed on the dashboard, monitoring of the object It is preferable to include; a data generation unit for generating the monitoring information to express the information as context information excluding the image data of the object.

The data generating unit, as an editing input from the manager terminal, when receiving a processing input that is information for editing the content of the context information of the object into content recognizable by the control terminal, monitoring information processing the context information according to the processing input It is preferable to create

It is preferable to further include a model learning unit for updating the object state information inference model through learning of the object state information inference model.

The model learning unit may include: an image preprocessing unit that preprocesses image data using a preset image processing programming library when receiving image data including an identifiable object, and performs arbitrary attribute setting and labeling processing on the preprocessed image data; and a first process for detecting the state of an object using a preset pre-training model using a labeling centering on a keyword including attribute information included in the object state information, and a first process for detecting the state of the object detected by the first process. By repeating the second process of determining the accuracy of the result, a model for inferring object state information from the region of interest is learned through the update of the preset pre-trained model, and the learned model is used as a new object state information inference model. It is preferable to include; a model learning progress unit to update.

According to the present invention, rather than identifying and directly analyzing an object in the image data and providing the result, the image data is used only to identify the initial object and extract attribute and state information from the object, and the monitoring result is It is created based on context information of an object derived through time-series inference of the properties and state information of the extracted object, and is configured to provide the result of visualizing or dataizing the information at the same time.

According to this, even in a space where the communication network between the existing facility and the control server is not established, it is possible to effectively monitor the operation and status of each object using a video recording device such as a CCTV generally installed in the facility installation area. , there is an effect that the construction cost of the control system for monitoring objects is greatly reduced.

In particular, it is a technology that can build a very fast and efficient object monitoring system using a lightweight data network because it extracts context information of an object that can be identified from an image and uses very simple and lightweight data as monitoring data based on this. can provide

1 is a block diagram of a monitoring system through generation of context information of an object in an image according to an embodiment of the present invention;
2 is a detailed block diagram of an object state inference unit in the configuration of an embodiment of the present invention;
3 is a detailed block diagram of a monitoring information generating unit in the configuration of an embodiment of the present invention;
4 is a detailed configuration block diagram of a model learning unit in the configuration of an embodiment of the present invention.
5 is a diagram for explaining a flow in which object state information is inferred according to an implementation of an embodiment of the present invention;
6 is an implementation example of a dashboard output to a control terminal according to an embodiment of the present invention.
7 is a diagram for explaining a flow in which an object state information inference model is learned according to an embodiment of the present invention;
8 is an example of an internal configuration of a computing device according to an embodiment of the present invention.

Hereinafter, various embodiments and/or aspects are disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will also be recognized by one of ordinary skill in the art that such aspect(s) may be practiced without these specific details. The following description and accompanying drawings set forth in detail certain illustrative aspects of one or more aspects. These aspects are illustrative, however, and some of the various methods in principles of various aspects may be employed, and the descriptions set forth are intended to include all such aspects and their equivalents.

As used herein, “embodiment”, “example”, “aspect”, “exemplary”, etc. may not be construed as an advantage or an advantage in any aspect or design described herein over other aspects or designs. .

Also, the terms "comprises" and/or "comprising" mean that the feature and/or element is present, but excludes the presence or addition of one or more other features, elements and/or groups thereof. should be understood as not

Also, terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, are those commonly understood by those of ordinary skill in the art to which the present invention belongs. have the same meaning. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in an embodiment of the present invention, an ideal or excessively formal meaning is not interpreted as

1 is a block diagram of a monitoring system through generation of context information of an object in an image according to an embodiment of the present invention.

A monitoring system 10 (hereinafter referred to as a system) through generation of context information of an object in an image according to an embodiment of the present invention is a computing device including one or more processors and one or more memories for storing instructions executable by the processor 1 , as shown in FIG. 1 , an application or programs including a learning module included in performing a function of each configuration and a database 20 in which data to be processed and data to be processed are stored and a control terminal 30 for using monitoring data ) is configured separately from the system 10 and may be provided as an independent device. Alternatively, all devices may be configured as one server terminal according to the concept of the system 10 .

Described based on this system configuration, the system 10 generates a database generation unit 11, an object state inference unit 12, and a context information generation unit 13 according to the performance of each function as shown in FIG. It is characterized in that it further includes a monitoring information generating unit 14 according to an embodiment of the present invention.

The database generator 11 is provided for each object, and performs a function of creating a first table that is a data table in which information related to an object to be monitored is stored, and a second table that is a data table in which context information of an object is stored. That is, when a signal requesting analysis of image data is received in order to perform the function of the present invention, a data table in which the analyzed data is stored is generated by performing the function of the database generator 11 .

In each embodiment of the present invention, in each embodiment of the present invention, as described above, a sensor is attached to receive a state value by attaching a sensor for sensing to be monitored, such as various types of objects capable of inferring motion and state through an image. As a facility, it will mean a facility operated according to a specific function, a facility installed with a display module that can recognize a state value, etc. through an image. It will be understood as a concept that includes all analysis objects such as other equipment, human body, and animals that can infer the state of the state and can infer the operation according to the time-series analysis.

In the present invention, it is preferable that the first table and the second table are created for each object. That is, the database 20 is configured such that the first table and the second table are generated for each object, and the creation of the first table and the second table by the database generation unit 11 means that the database generation unit 11 itself contains data. The tables are created and stored, or the data table creation request input is generated in the database creation unit 11 and the data table described above is created in the database 20 that has received it, a storage space is allocated, and identification information of an object is allocated. This means that each data table is set for each object.

Although object-related information will be described later, it may be understood as a concept including attribute information of an object that may be derived as a result of analyzing image data of an object by the photographing device 40 or the object state inference unit 12 . It also means information that can be analyzed and extracted to derive context information.

Specifically, the attribute information of the object refers to information defining the object in order to enable specific identification and detection of the object, such as identification information of the object, the type of the object, the size, color, and shape of the object. On the other hand, as information that can be analyzed and extracted to derive context information, absolute or relative time stamps (time information) of image data, key frame information of a moving image that is image data, coordinate information of objects in the image, etc. may be included. have.

The first table is a data table in which information necessary for deriving the context information of the object is stored, and the second table is the context information of the object derived by time-series analysis of the information stored in the first table, that is, the operation of the object. It refers to a table in which information, such as the contents of a predicted or grasped operation, is stored.

The object state inference unit 12, upon receiving image data including an object, sets a region of interest that is a region including the object, and determines the state of the object using the object state information inference model from the image data within the set region of interest. It performs a function of inferring object state information, which is possible information, and storing it in the first table.

The object state inference unit 12 is a configuration for receiving image data in the present invention, and is characterized in that image data is not utilized in other configurations. That is, in the present invention, the image photographing apparatus 40 transmits image data in real time or periodically to the object state inference unit 12, or the image data photographed by the image photographing apparatus 40 is stored in a terminal such as a server. In this case, the object state inference unit 12 receives it and uses it to infer object state information.

As described above, the object state inference unit 12 receives or collects image data when receiving a signal requesting generation and provision of monitoring information for an object from the control terminal 30 or the like. A detailed configuration for performing a function of the object state inference unit 12 is illustrated in FIG. 2 .

Referring to FIG. 2 , the object state inference unit 12 includes a region of interest setting unit 121 , a table construction unit 122 , an object state information extraction unit 123 , and an object state information storage unit 124 . desirable.

The region of interest setting unit 121 performs a function of setting, as a region of interest, a bounding box in which an object exists in any one image data included in the image data. That is, an image analysis function for identifying and analyzing an object according to the region of interest is performed by identifying an object to be analyzed in the image data, setting a bounding box of a rectangular region including the object as the region of interest.

When setting the region of interest, the region of interest setting unit 121 stores a modeling algorithm for setting the region of interest, and when receiving image data, the algorithm is applied to set the region of interest. At this time, for example, a region of interest is set for a model such as R-CNN (region-based convolutional neural network).

The table configuration unit 122 transmits a request signal for generating a first table for each object in the set ROI to the database generator 11 or the database 20, but as information to be extracted from the object, object state information It performs a function of generating object attribute information as a column of the first table as in the above-described example used to infer .

That is, the table configuration unit 122 transmits/receives data to and from the database 20 and/or the database generation unit 11 to analyze the type and storage of data stored in the first table created as described above for object analysis. By defining the space, it performs a function of separately storing data in the first table.

The object state information extracting unit 123 applies the object state information inference model to image data, specifically, to an image region corresponding to the region of interest set by the region of interest setting unit 121 to obtain an object on the region of interest in the image data. Recognizes based on the set ROI, detects the state of the object by applying the object state information inference model, and extracts the time stamp on the image data, video keyframe information, and the coordinates of the object on the recognized ROI do.

The object state information inference model is a model of the state learned by the model learning unit 15 to be described later, and is a model for inferring state information of an object according to the color, size, shape, etc. of the object on the object attribute information or image data. am. For example, in the case of a facility that performs a function by a specific operation, information on the status of each operation is inferred, or in the case of a facility in which status information is displayed according to a specific display, whether power is applied, whether normal operation, each numerical value, etc. It can be inferred as this state information.

The object state information extraction unit 123 determines the state of objects for each frame on data such as a specific image or a video derived through a combination of images, in order to infer object state information and generate context information of the object based on the inferred object state information. Detect (inference) to derive information about the state of an object, and detect the state of an object for connection between states of objects, identification of objects, etc., as described above, for time-series analysis and specific coordinate analysis, image It performs the function of extracting the time stamp on the data, the video keyframe information, and the coordinates of the object on the recognized ROI.

The object state information storage unit 124 stores information on a time stamp on the extracted image data, video keyframe information, coordinates of an object on a recognized region of interest, attribute information of the object, and information on the state of the detected object for each object. 1 Executes the function of saving to a table. That is, the object state information storage unit 124 performs a function of storing the above-described data necessary for generating context information in the first table as a storage space of the corresponding data.

That is, according to each of the above-described configurations of the object state inference unit 12, object state information from image data, information about time and keyframe for time-series analysis, and attribute information of an object are stored in the first table, , therefore, high-capacity image data is not used for direct object state monitoring, but only the information stored in the first table is used in subsequent configurations, so information for object state monitoring is lightened, and the database 20 ), instead of storing image data, text information on the first table is stored, so that unnecessary expansion of the storage space is not required.

In this case, different AI algorithms may be applied to the function of the region of interest setting unit 121 and the object state extracting unit 123 according to the function of analyzing the object. When analyzing an object, an object detection function as a function of setting a region of interest including an object, and extracting features such as location information characteristics such as coordinate analysis of the above-described object, size, and texture, and the detected object As characteristic information, an object recognition function for extracting a posture such as color and motion characteristics and other instrument values is included. In this case, since different algorithms may be applied to the object detection function and the object recognition function, different algorithms may be applied to each.

The context information generator 13 generates context information of an object through time-series analysis of object state information, and performs a function of storing the generated context information in the above-described second table.

That is, as described above, the first table contains object state information on each frame according to the performance of the function of the object state information inference unit 12, a time stamp for identifying each frame, and information on keyframes of the image; In addition, property information of an object capable of identifying the object is stored.

The context information generating unit 13 uses the above-described data stored in the first table to analyze the object state information in a time-series manner, and generates and stores the context information of the inferred object in the second table.

Specifically, the context information generating unit 13 analyzes the first table based on the time stamp, and then periodically repeats and changes the object state information of the previous state and the object state information of the current state for each time period. By extracting , context information on the situation and operation of the current object is generated for the entire image data, and a plurality of generated context information for each object is stored in the second table provided for each object.

At this time, the context information generating unit 13 is processed with SQL and other programs, extracts the context information according to the above-described process according to a predetermined program and algorithm, and stores the extracted context information in the second table. In this case, as described above, all context information may be stored in the second table prepared for each object, or in the second table integrally configured as one table.

The plurality of context information for each object means that context information for each object to be analyzed is generated. Accordingly, the second table prepared for each object contains context information within the time stamp section of the received image data of each object. will be saved

The context and context information of the object is information that analyzes the state of the object or predicts how and how the state of the object has changed or how it will change as each object state information is repeated or changed periodically or for a certain time can mean In addition, in order to generate monitoring information based on the corresponding information, it may mean information to be provided to the control terminal 30 according to the situation of the object.

For example, if the status and context information of the object includes content indicating that the signal of the corresponding facility is in operation due to an error occurrence, guidance information such as notifying the need for inspection and repair of the facility is generated together. It can be used to generate monitoring information.

An example of performing the functions of the object state inference unit 12 and the context information generation unit 13 is shown in FIG. 5 . Referring to FIG. 5 , the image data 200 includes image data for each frame, and an object 201 and an ROI 210 accommodating the object 201 are set in each image data.

In this case, in the analysis result first table, time stamps t1 and t2 may be stored along with object state information of the inference result for the ROIs 210 and 211 . In addition, as the result data 220 analyzed therefrom, identification information (I) of the object, attribute information (Info) of the object, object state information, context information, etc. are generated and stored in the first and second tables as described above. can be saved.

Returning to the description of FIG. 1 again, the monitoring information generating unit 14 generates monitoring information of an object from which image data is excluded by processing the context information and object state information of the objects stored in the first table and the second table. perform the function

The exclusion of image data from the above description means that, as described above, among the components of the present invention, there is no configuration in which high-capacity image data is used or output to monitor the state of an object other than the object state inference unit 12. it means. That is, the monitoring information generating unit 14 processes the visual or audio content using only the text information stored in the first table and the second table, and provides it to the control terminal 30 .

A general control system uses high-capacity image data throughout the analysis system, and at the same time transmits it to the control terminal 30 as it is, and provides analysis results for the image. According to this, even if high-capacity image data is provided in a lightweight manner, the size of the data is inevitably larger than that of text data due to the characteristics of the image data. It is essential.

However, in the present invention, except for the object state inference unit 12 on the system 10, since almost only text data is transmitted and received between the respective components and is stored in the database 20 at the same time, for high-capacity data transmission and reception There is no need to build a network. That is, since the system 10 of the present invention can be implemented not only in a specific control system in which a dedicated network is built, but also in a management server of a general building, it is possible to construct a control system using object monitoring in a very general and efficient manner.

The monitoring information generating unit 14 may be implemented to include a detailed configuration as shown in FIG. 3 in order to perform a specific function of the corresponding configuration. Referring to FIG. 3 , the monitoring information generation unit 14 may be configured to include a data preparation unit 141 and a data generation unit 142 .

The data preparation unit 141 selects identification information corresponding to the type of the analysis target object from among the identification information provided for each type of object, generates a dashboard to which the selected identification information is applied, and then sets the first analysis target object It performs the function of loading the table and the second table.

The object attribute information may include information on the type of object, and corresponding data may be stored in the first table. For example, as the type of object, it can be divided into operation-type equipment, function display-type equipment, or various equipment types according to the operation function, and the corresponding information may be replaced with identification information for visualization of monitoring information. have. For example, different low-capacity icon images according to the type of each object are previously stored in the database 20 as identification information corresponding to the type of the object, and are selected according to the object attribute information.

The data preparation unit 141 creates a dashboard that is an interface for outputting visual information that can be checked in the control terminal 30, and primarily edits the dashboard so that identification information of objects such as icons are displayed in conjunction with each other. After that, the first table and the second table for the analysis target object are loaded for information application.

The data generator 142 applies some of the context information and the object property information of the object from the loaded first table and the second table to the identification information so that some of the context information and the object property information of the object are displayed on the dashboard. By doing so, it performs a function of generating monitoring information to express the monitoring information of the object as context information excluding the image data of the object.

That is, the data generating unit 142 extracts and applies some of the attribute information of the object required for monitoring information from the first table on the dashboard, that is, the interface generated by the data preparation unit 141, and the context of the object. By extracting and applying information from the second table, information necessary for monitoring is applied on the dashboard, thereby performing a function of generating monitoring information. The generated monitoring information may be output from the control terminal 30 as shown in FIG. 6 .

The screen 300 of FIG. 6 means an example in which a dashboard including monitoring information generated by the monitoring information generating unit 14 is implemented in the control terminal 30 or the like. Referring to the screen 300 of FIG. 6 , the constant state data 301 of an object included in an interface, etc., the context information 302 of the objects, the identification information 303 of each object for which the context information is processed, the operation or state State information 304 such as a variable value or a fixed value may be output.

At this time, when the data generating unit 142 extracts and applies the context information of the object, the content of the general context information so that the corresponding context information can be conveniently and intuitively recognized from the manager terminal (not shown) in the control terminal 30, etc. It is possible to additionally perform a function of processing context information by receiving an edit input for .

That is, the data generating unit 142 receives, as an editing input from the manager terminal, a processing input, which is information for editing the context information of an object into contents recognizable by the control terminal 30 , according to the processing input. Monitoring information by processing context information can be created.

Meanwhile, the object state information inference model used in the present invention may be provided to be advanced through a learning process such as machine learning. To this end, the model learning unit 15 may be additionally included on the system 10 of the present invention as shown in FIGS. 1 and 4 .

The model learning unit 15 is a configuration that updates the object state information inference model through learning of the object state information inference model, and a specific example thereof is shown in FIG. 4 .

Referring to FIG. 4 , the model learning unit 15 preferably includes an image preprocessing unit 151 and a model learning progressing unit 152 . First, the image pre-processing unit 151 pre-processes image data using a preset image processing programming library when receiving image data including an identifiable object, and performs arbitrary property setting and labeling processing on the pre-processed image data. perform the function

The image data used in the model learning unit 15 may be understood as a concept including image data included in some frames of the above-described collected image data or virtual image data prepared for learning. In addition, in order to increase the learning accuracy, the image is preprocessed using the image processing programming library as described above. For example, an open source computer vision library such as OpenCV may be used.

In the arbitrary property setting and labeling process, for learning through image data, data corresponding to the above-described object property information is arbitrarily allocated to the image data, etc., and the labeling process is performed for this, and the object state information inference model 410 It means processing for accuracy judgment and update of function execution.

The model learning progress unit 152 includes a first process for detecting a state of an object by using labeling centering on a keyword including attribute information included in the object state information, and a first process for detecting a state of an object detected by the first process. By repeating the second process of determining the accuracy of the result, a model for inferring object state information from the region of interest is learned, and the learned model is updated as a new object state information inference model.

In this case, when performing the object detection and object recognition algorithms while performing the first process, a preset pre-training model 101 may be adopted for each algorithm when necessary. In this case, the pre-training model may generate a pre-training model by using a general image processing algorithm or a deep learning algorithm, or adopt various general open pre-training models used for object detection and recognition.

In this case, the pre-training model 101 is an image processing algorithm used for image detection and recognition, and is a bulk-type model before the specific learning model used for object detection in the present invention, or is used for object detection in the present invention. As a specific learning model to be used, for example, it may mean a model that can be used for object detection in the present invention as well as not only being applied to a specific control position in the present invention because certain learning is in progress.

A function similar to that of the object state inference unit 12 is performed in the model learning progress unit 152 . That is, it performs a function of analyzing the state of an object in detail for learning. As described above, the first process is performed as a process of detecting the state of an object using a preset pre-training model, etc., at this time, the object state information As in the example described above, natural language processing is not necessary because it is enough to infer the degree of state display for an object, and learning is carried out using labeling centered on keywords.

The learning algorithm used at this time can utilize algorithms such as YOLO (You Only Look Once) algorithm, mobilenet SSD neural network algorithm, ResNet, HOG, BOW, etc. can be utilized.

Thereafter, the object state information inference model 410 is updated by continuously repeating the second process, such as comparing the result value for the state of the object detected (detected and recognized) by the first process with a pre-stored correct answer. . An example of this is shown in FIG. 7 . 7 is a diagram for explaining a flow of learning an object state information inference model according to an embodiment of the present invention. Referring to FIG. 7 , first, attribute information of an object in the sample image data 400 as a learning target and keyword information as status information and the like are analyzed. At this time, after comparing the correct answer for the object state information and the model 410 application result values (yy, zz), the model 410 is updated using the comparison.

According to another embodiment of the present invention, for example, when another artificial intelligence model similar to the above-described object state information inference model 410 may be applied to context information, the corresponding model is also the above-described object state information inference model 410 . Model learning can proceed through learning similar to the learning process of

8 illustrates an example of an internal configuration of a computing device according to an embodiment of the present invention, and in the following description, descriptions of unnecessary embodiments that overlap with those of FIGS. 1 to 7 will be omitted. do it with

8, the computing device 10000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input/output subsystem ( It may include at least an I/O subsystem) 11400 , a power circuit 11500 , and a communication circuit 11600 . In this case, the computing device 10000 may correspond to a user terminal connected to the tactile interface device (A) or the aforementioned computing device (B).

The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. have. The memory 11200 may include a software module, an instruction set, or other various data required for the operation of the computing device 10000 .

In this case, access to the memory 11200 from other components such as the processor 11100 or the peripheral interface 11300 may be controlled by the processor 11100 .

Peripheral interface 11300 may couple input and/or output peripherals of computing device 10000 to processor 11100 and memory 11200 . The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 10000 and process data.

The input/output subsystem 11400 may couple various input/output peripherals to the peripheral interface 11300 . For example, the input/output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor, keyboard, mouse, printer, or a touch screen or sensor as needed to the peripheral interface 11300 . According to another aspect, input/output peripherals may be coupled to peripheral interface 11300 without going through input/output subsystem 11400 .

The power circuit 11500 may supply power to all or some of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or a power source. It may include any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, if necessary, the communication circuit 11600 may include an RF circuit to transmit and receive an RF signal, also known as an electromagnetic signal, to enable communication with other computing devices.

The embodiment of FIG. 8 is only an example of the computing device 10000, and the computing device 11000 may omit some components shown in FIG. 8, or further include additional components not shown in FIG. 8, or 2 It may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 8 , and various communication methods (WiFi, 3G, LTE) are provided in the communication circuit 1160 . , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 10000 may be implemented in hardware, software, or a combination of both hardware and software including an integrated circuit specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the user terminal through a file provided by the file distribution system. As an example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of the user terminal.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be permanently or temporarily embody in The software may be distributed over networked computing devices and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and used by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result. Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the following claims.

Claims (8)

It relates to a monitoring system through creation of context information of an object in an image implemented as a computing device including one or more processors and one or more memories for storing instructions executable by the processor,
a database generator that is provided for each object and generates a first table that is a data table for storing information related to an object to be monitored and a second table that is a data table for storing context information of an object;
When the image data including the object is received, the region of interest, which is the region including the object, is set, and object state information, which is information that can determine the state of the object, is inferred from the image data in the set region of interest using the object state information inference model. an object state inference unit for storing the data in the first table;
a context information generator generating context information of an object through time-series analysis of object state information, and storing the generated context information in the second table; and
and a monitoring information generating unit that processes context information and object state information of objects stored in the first table and the second table to generate monitoring information of an object from which image data is excluded;
The object state inference unit,
a region of interest setting unit configured to set a boundary box in which an object exists in any one image data included in the image data as a region of interest;
A request signal for generating the first table for each object within a set ROI is transmitted to the database generator, and object attribute information used to infer the object state information as information to be extracted from the object is transmitted to the first a table configuration unit that creates a table column;
By applying the object state information inference model to the image data, the object on the region of interest in the image data is recognized based on the set region of interest, and after detecting the state of the object, a time stamp on the image data, video keyframe information, an object state information extraction unit for extracting coordinates of an object on the recognized region of interest; and
Object state information storing the time stamp on the extracted image data, video keyframe information, coordinates of the object on the recognized region of interest, object property information, and information on the state of the detected object in the first table prepared for each object including;
The context information generation unit,
After analyzing the first table based on the time stamp, the state of the object in the current state and the state of the object in the current state by extracting the periodic repetition state and change state of the object state information of the unit time before and the object state information of the current state for each time period generating context information for motion for the entire image data, and storing a plurality of generated context information in the second table;
The monitoring information generating unit,
Selecting identification information corresponding to the type of analysis target object from among identification information prepared for each type of object, creating a dashboard to which the selected identification information is applied, and loading the first and second tables for the analysis target object data preparation department; and
By applying some of the context information and object property information of the object from the loaded first table and the second table to the identification information to display some of the context information and object property information of the object on the dashboard, monitoring information of the object A monitoring system through generating context information of an object in an image comprising a;
delete delete delete delete According to claim 1,
The data generation unit,
As an editing input from the manager terminal, when receiving a processing input, which is information for editing the contents of the context information of an object into contents recognizable by the control terminal, generating monitoring information by processing the context information according to the processing input Monitoring system through creation of context information of objects in the video.
According to claim 1,
The monitoring system through creation of context information of an object in an image, further comprising a; a model learning unit that updates the object state information inference model through learning of the object state information inference model.
8. The method of claim 7,
The model learning unit,
an image pre-processing unit that pre-processes image data using a preset image processing programming library when receiving image data including an identifiable object, and sets arbitrary properties and labels the pre-processed image data; and
Results on the state of the object detected by the first process and the first process for detecting the state of the object using a preset pre-training model by utilizing the labeling centering on the keyword including the attribute information included in the object state information By repeating the second process of determining the accuracy of the value, a model for inferring object state information from the region of interest is learned through the update of the preset pre-trained model, and the learned model is updated as a new object state information inference model A monitoring system through generation of context information of an object in an image, characterized in that it comprises a; a model learning progress unit.

Images