KR20230077586A - System and method for storing video-objects contextualization data - Google Patents

Abstract

An object contextualization data storage system according to an aspect of the present invention stores object contextualization data obtained by contextualizing an object detected for one or more image files and object attribute information for each frame in the form of a text file in a big data storage, and additionally stores the image file It is also saved as a unit or set logical clip unit.

Description

Object contextualization data storage system and method { SYSTEM AND METHOD FOR STORING VIDEO-OBJECTS CONTEXTUALIZATION DATA }

The present invention relates to computer vision technology, and relates to a technology for contextualizing objects detected in a video and storing them in text form.

Recently, video data collected from CCTVs, black boxes, smartphones, etc., and unstructured data produced from social media such as Facebook are rapidly increasing, and the rate of increase is also rapidly accelerating. Accordingly, interest in the convergence approach between unstructured content big data of videos and social media and artificial intelligence technologies such as deep learning and various utilization methods based on this is greatly spreading. In particular, as R&D and commercialization of artificial intelligence disclosure engines such as deep learning that utilize video and social media big data of unstructured content as learning and training data, and service platforms that utilize them, have begun to draw attention, representative unstructured content big data Interest in research and development that applies artificial intelligence technologies such as deep learning to video search systems, which are application services, is increasing.

Conventional video retrieval systems based on physical contents and semantic objects of video frames do not show sufficient performance in constructing frame indexes or extracting core features, and some approaches extract even if they show relatively good performance. It has a problem that the dimension of the feature is too high or the implementation of the corresponding algorithm is very difficult. The most important step in video search approaches and systems is the feature extraction step to describe the video with the minimum descriptor, and as the most basic visual features, color and texture are considered. Recently, various texture expressions are being considered. Research and development on video object-based video search approaches and content-based video search approaches based on pattern recognition and computer vision according to the method are on the rise.

These video search approaches require a lot of computing resources to store and analyze the video itself, especially when CCTV images that continuously generate video data are used for analysis.

An object of the present invention is to provide a system that contextualizes objects appearing in a video and their properties in frame units and stores them in text form so that video search systems that search video based on content can search for video quickly and with little computing power. to be

An object contextualization data storage system according to an aspect of the present invention includes an input image processing unit, an object detection unit, a frame contextualization unit, an object contextualization unit, a clip designation unit, and a contextualization storage unit.

The input image processor generates input image data by assigning consecutive frame identifiers to each frame of one or more input image files.

The object detector detects a plurality of objects for each frame in the input image data using a deep learning model, classifies the type of object, and generates object attribute information including the type of object and location information of the object.

The frame contextualization unit generates frame contextualization data including detected frame information for each frame in which an object is detected, the type and number of objects, and object property information of each object.

The object contextualizer generates object contextualization data including a list of objects detected for the input image data and the frame contextualization data for each frame.

The clip specifying unit constructs a logical clip based on the set clip configuration information and adds it to the object contextualization data.

The contextualization storage unit stores the object contextualization data of the input image data as a text file in the big data storage.

A method for storing object contextualization data according to an embodiment of the present invention includes processing an input image, detecting an object, generating object property information, contextualizing a frame, contextualizing an object, specifying a clip, and storing contextualization data. Include steps.

In the input image processing step, one input image data is generated by assigning consecutive frame identifiers to each frame of one or more input image files.

In the object detection step, one or more objects are detected for each frame in the input image data using a deep learning model, and the object type is classified.

The object property information generating step generates object property information including object type and object location information for each detected object.

The frame contextualization step generates frame contextualization data including detected frame information for each frame in which an object is detected, the type and number of objects, and object property information of each object.

The object contextualization step generates object contextualization data including a list of objects detected in the input image data and the frame contextualization data for each frame.

The clip designation step configures a logical clip based on set clip configuration information and adds it to object contextualization data.

In the step of storing contextualization data, object contextualization data of the input image data is stored in a big data storage as a text file.

According to the present invention, video retrieval systems can quickly and with little computing power target a text file in which objects appearing in a video and their attributes are contextualized and stored.

1 is a block diagram of the object contextualization data storage system of the present invention.
2 illustrates the concept of contextualizing an object in the object contextualization data storage system of the present invention.
3 illustrates an example of frame contextualization structured for one frame by the object contextualization data storage system of the present invention.
4 illustrates an example in which the object contextualization data storage system of the present invention performs object contextualization on one or more image files.
5 shows an example of logical clip setting in the object contextualization data storage system of the present invention.
6 is a flowchart of a method for storing object contextualization data according to the present invention.

The foregoing and additional aspects are embodied through embodiments described with reference to the accompanying drawings. It is understood that the components of each embodiment are possible in various combinations within an embodiment unless otherwise stated or contradictory to each other. Each block in the block diagram may represent a physical component in one case, but in another case, it may be a logical representation of a function of a portion of a function of one physical component or a function across multiple physical components. Sometimes the substance of a block or part thereof may be a set of program instructions. All or part of these blocks may be implemented by hardware, software, or a combination thereof.

1 is a block diagram of the object contextualization data storage system of the present invention. An object contextualization data storage system 10 according to an aspect of the present invention includes an input image processing unit 100, an object detection unit 110, a frame contextualization unit 120, an object contextualization unit 130, and a clip designation unit. (140) and a contextualization store (150).

The object contextualization data storage system 10 according to one aspect of the present invention may consist of one computing device or a plurality of computing devices connected by a network. For example, the input image processing unit 100 of the object contextualization data storage system 10, the object detection unit 110, the frame contextualization unit 120, the object contextualization unit 130, the clip designation unit 140, The contextualization storage unit 150 may be included in one computing device, and as another example, the input image processing unit 100 of the object contextualization data storage system 10, the object detection unit 110, and the frame contextualization unit ( 120), the object contextualization unit 130, the clip designation unit 140, and the contextualization storage unit 150 may be composed of different computing devices. It is not limited thereto, and functional blocks may be combined to form two or more computing devices.

A computing device constituting the object contextualized data storage system 10 of the present invention is a device including a processor and a memory connected to the processor and including program instructions executable by the processor. A computing device constituting the object contextualization data storage system 10 of the present invention may be a computer device that further includes a storage device, a display, an input device, and the like in addition to a processor and a memory. The processor is a processor that executes program instructions that implement the functional blocks of the object contextualized data storage system 10, and the memory is connected to the processor and includes program instructions executable by the processor and data to be used for operation by the processor and processed by the processor. stored data, etc.

2 illustrates the concept of contextualizing an object in the object contextualization data storage system of the present invention. Referring to the concept of object contextualization with reference to FIG. 2 , the object contextualization data storage system 10 may receive and process one or more image files such as CCTV images.

In the example of FIG. 2 , four image files are input. The four video files may be related video files. For example, the four video files may be video files that are continuously photographed and stored by one CCTV camera or video files that are photographed and stored by four CCTV cameras located in the same area at similar times.

In the example of FIG. 2 , the object contextualization data storage system 10 arranges the four input image files in the input order and then allocates new frame identifiers (IDs) to all the image files (FRAME ₁ to FRAME _n ) to form one of the input image data. The object contextualization data storage system 10 detects objects frame by frame for all frames of input image data. Frame contextualization data is created by structuring the detected object information (object type and object attribute information) for each frame. In the example of FIG. 2, the frame contextualization data is structured as Object and the attribute as Item. The object contextualization data storage system 10 combines object contextualization data structured for each frame to generate object contextualization data for input image data.

In addition, the object contextualization data storage system 10 configures a logical clip according to clip configuration information set for a frame of input image data, adds the configuration result to object contextualization data, and then stores it in the form of a text file. In the example of FIG. 2, a total of three logical clips are configured, CLIP ₁ is designated as a part of the frame of FILE ₁ , CLIP ₂ is consecutively designated across FILE ₁ and FILE ₂ , and CLIP ₃ is designated as FILE ₃ and FILE ₄ are separately designated.

The input image processing unit 100, the object detection unit 110, the frame contextualization unit 120, the object contextualization unit 130, the clip designation unit 140, and the contextualization storage unit 150 have at least one of their functions. It is implemented as computer program instructions, some of which are stored in memory and executed on a processor.

The input image processing unit 100 generates input image data by assigning consecutive frame identifiers to each frame of one or more input image files. The input image processing unit 100 arranges one or more input image files in the input order and then sequentially assigns new frame identifiers (IDs) to all the image files to convert the one or more input image files into one input image data. create

The object detector 110 detects objects frame by frame from the input image data using a deep learning model learned to detect objects in the image. The object detection unit 110 detects a plurality of objects at once in every frame. The deep learning model used by the object detection unit 110 may be trained using a known data set such as the COCO data set.

The object detector 110 classifies the detected object, predicts the type of the object, and predicts the location information of the object in the image using a bounding box.

In addition, the object detection unit 110 generates object property information including the type of object classified for each frame and the location information of the corresponding object in the image. The object attribute information generated by the object detection unit 110 includes object location information, that is, object size information calculated from bounding box coordinates, object classification accuracy, object color (eg, car color if the object is a car). ) may include information such as For example, when the detected object is a human and the gender is recognized, the object detection unit 110 may create the object by adding the human gender as an object attribute. The object properties generated by the object detection unit 110 are not limited to those mentioned above and may be added in various ways according to properties recognizable from the image as needed.

The deep learning model used by the object detector 110 to detect objects is configured to be selectable. A user may select an object detection deep learning model through a user interface provided by the object contextualization data storage system 10 . The object contextualization data storage system 10 may further include a front end unit (not shown) providing a user interface. However, the corresponding deep learning model must be able to output a result in a prearranged format so that the object detection unit 110 can use the object detection result.

A deep learning model used by the object detector 110 to detect an object may be a deep learning model using a single-step algorithm. Deep learning models that use a single-step method include YOLO models, SSD models, and RetinaNet models, and it is particularly desirable to use YOLO models. The single-pass algorithm is known to outperform the two-pass algorithm in terms of speed.

A You Only Look Once (YOLO) model is a real-time object detection model that is most popularly used in the field of object detection and will be briefly described as a known technique. The YOLO model extracts the absolute size and position of each object in the form of coordinates from the camera image. The YOLO model receives 2D image data from camera images, uses a convolution layer to extract the absolute size and position of an object in the form of coordinates, and can determine the type of object. In general, YOLO consists of multiple convolutional layers and fully connected layers to extract features of an image. The fully connected layer determines the location and type of an object from the extracted results.

A single shot multibox detector (SSD) model is also briefly described as a known technique. The SSD model uses VGG-16 as a backbone, but extracts features using only some convolution layers, and the extracted features perform object detection by passing through several auxiliary detectors. YOLO has bounding box and classification information only in the final feature map, whereas SSD has information distributed in several hidden layers. The SSD has 6 layers corresponding to feature maps of different sizes. The large feature map can detect small objects and the small feature map can detect large objects. SSD is an algorithm that predicts a single object using bounding boxes of various sizes.

The RetinaNet model is also briefly described as a well-known technique. The RetinaNet model uses a focal loss function transformed from the cross entropy loss function. The RetinaNet model uses ResNet as a backbone and applies FPN (Feature Pyramid Networks). ResNet, a backbone network, plays a role in calculating feature maps for all input images. The RetinaNet model includes two subnetworks, the first subnetwork performs object classification on the results of ResNet, and the second subnetwork performs bounding box regression.

A deep learning model used by the object detector 110 to detect an object may be a deep learning model using a two-step algorithm. Deep learning models using the two-step method include the Faster R-CNN model and the R-FCN model, and it is particularly desirable to use the Faster R-CNN model. Two-pass algorithms are known to outperform single-pass algorithms in terms of accuracy.

The Faster R-CNN model and the R-FCN model are simply described as known techniques.

The Faster R-CNN model is a structure in which RPN (Region Proposal Network), which performs candidate region extraction, is added to solve the bottleneck caused by the selective search algorithm used for candidate region extraction. Faster R-CNN is a combination of RPN and Fast R-CNN. The Faster R-CNN model obtains a feature map by inputting an original image to a pre-trained CNN model, and passes the feature map to RPN to generate appropriate candidate regions (region proposals). After obtaining a feature map with a fixed size by performing RoI pooling on the feature map obtained through the candidate region calculation process and the CNN model, object classification and boundary are input to the Fast R-CNN model. Perform box prediction.

The R-FCN model has a structure in which operations are shared between regions of interest (RoIs) extracted through RPN and feature maps including location information are used. The R-FCN model is used by modifying the subnetwork at the post-RPN stage in the Faster R-CNN model into a Fully Convolutional Network (FCN).

According to an additional aspect of the present invention, the object property information generated by the object detector 110 may further include an object image in a corresponding frame. An object detected by the object detection unit 110 is displayed as a bounding box. The object detection unit 110 may extract an object image within the bounding box and include it in object property information. After the object image included in the object property information by the object detector 110 is stored in a separate storage location, the storage path may be included in the object property information, and the object image may be encoded in text form and included in the object property information. The object image included in object attribute information is included in object contextualization data to be described later.

The object images included in the object contextualization data are the object images included in the object contextualization data when the image retrieval system searches text-based images using the object contextualization data, but cannot play the images because the retention period of the searched images has elapsed. It can be used to restore a frame image from which the background part is removed by displaying it in consideration of location information, and collect the restored frames to reproduce an image from which the background part has been removed.

The frame contextualization unit 120 generates frame contextualization data including detected frame information for each frame in which an object is detected, the type and number of objects, and object property information of each object. The frame contextualization unit 120 creates frame contextualization data for each frame by structuring information including the type and number of detected and classified objects in the image frame data and object property information of each object.

3 illustrates an example of frame contextualization structured for one frame by the object contextualization data storage system of the present invention. The example shown in FIG. 3 assumes frame contextualization data for frame 10, and the frame contextualization data includes the frame identifier (ID), the type and number of objects detected in the frame (DetectedObjsDict), and the objects of individual objects. It is structured including attribute information (object_#). In the example of FIG. 3, the frame identifier for distinguishing frames in the input image data is 10, and the types of objects detected in the frame are person and car, and 13 people and 1 car appear. It is structured in the DetectedObjsDict item, and each object included in the frame contextualization data is structured from object_1 item to object_14 item, and the object structured in object_8 item is an object classified as a car, and classification accuracy (confidence) as object property information. and object location information (location) are included to create frame contextualization data. The example shown in FIG. 3 is one example, and object property information may be added in various ways according to the type of object.

According to an aspect of the invention, the frame contextualization unit 120 may generate frame contextualization data for frames sampled at set frame intervals. For example, if the set frame interval is 5, the frame contextualization unit 120 may generate frame contextualization data only for frames corresponding to the set frame interval, such as the 5th frame, the 10th frame, and the 15th frame.

According to another aspect of the present invention, the frame contextualization unit 120 may, if the type and number of objects detected for the current frame are the same as the types and number of objects detected for frames in which frame contextualization data was previously generated, the current frame. Frame contextualization data may not be generated for . That is, the frame contextualization unit 120 may generate frame contextualization data for the current frame when the type and number of objects detected for the current frame are different from the types and number of objects detected for the previous frame.

The object contextualization unit 130 generates object contextualization data including a list of objects detected with respect to the input image data and the frame contextualization data for each frame. The object contextualization unit 130 generates a list of all objects detected in at least one frame of the input image data, combines them with frame contextualization data of frames in which objects are detected, and structures the list.

4 shows an example in which the object contextualization data storage system of the present invention performs object contextualization on one or more image files. The example shown in FIG. 5 shows object contextualization as a result of performing object contextualization on input image data. This is an example of data structure. The object contextualization data includes list information (obj_to_detect) of objects detected in a corresponding image and frame contextualization data for each frame in which an object is detected.

The clip designator 140 constructs a logical clip for the input image data based on set clip configuration information and adds the corresponding information to the object contextualization data. The clip designator 140 constructs a logical clip in all or some frames of the input image data, and adds logical clip composition information, that is, a logical clip identifier, to object contextualization data of the frame composed of the logical clips. Frames belonging to one logical clip may also belong to another logical clip. That is, one frame may belong to multiple logical clips.

Clip composition information may be information including the number of frames constituting a logical clip. In this case, the clip designator 140 organizes consecutive frames corresponding to the number of frames included in the clip composition information into logical clips for all frames of the input image data, and adds them to the object contextualization data. For example, if the number of frames included in the clip configuration information is 20, a logical clip is created by dividing all frames of the input image data every 20 frames, and a logical clip identifier to which the corresponding frame belongs is assigned to the object contextualization data of all frames. Add.

Also, the clip configuration information may be information including a logical clip list divided into a start frame identifier and an end frame identifier. In this case, the clip designator 140 configures frames corresponding to the logical clip list included in the clip configuration information into logical clips and adds them to the object contextualization data. For example, if the logical clip list included in the clip configuration information is clip 1 (20, 40) and clip 2 (70, 100), clip 1 is from frame 20 to 40, and clip 2 is from frame 70. It consists of up to number 100 and adds a logical clip identifier to which the corresponding frame belongs to the object contextualization data of the frame included in the corresponding logical clips.

The logical clip list included in the clip composition information is arbitrarily set by the user, and it is preferable that related video frames are designated as logical clips. Logical clips can also be created using separate deep learning models or video analysis tools that analyze the video to identify relevant frames.

Also, the clip configuration information may be information including a logical clip list divided into a start time and an end time. An image file input to the object contextualization data storage system 10 of the present invention may include image capture time information. In this case, the clip assigning unit 140 organizes frames of video capture time corresponding to the logical clip list included in the clip composition information into logical clips and adds them to the object contextualization data. For example, if the logical clip list included in the clip composition information is clip 1 (start time 1, end time 1) and clip 2 (start time 2, end time 2), then clip 1 has the start time 1 and end time It is composed of frames corresponding to end time 1 and clip 2 is composed of frames whose captured times are between start time 2 and end time 2, and object contextualization data of frames included in corresponding logical clips corresponds to the frame. Adds the logical clip identifier to which it belongs.

5 shows an example of logical clip setting in the object contextualization data storage system of the present invention.

(a) shown in FIG. 5 is an example in which the clip composition information includes the number of frames, and logical clips are created from CLIP ₁ to CLIP _m for all frames by dividing them by the number of frames, and example (b) is This is an example in which CLIP ₁ to CLIP ₄ are created for the designated frame in the case where the clip configuration information includes 4 logical clip lists divided by the start frame identifier and the last frame identifier, and (c) example shows that the clip configuration information is the start time This is a case of including a logical clip list divided by and an end time, and an example in which frames in which the captured time of a frame in each video file falls between the start time and end time of the clip composition information are configured as logical clip CLIP ₁ (the logical clip list is example with only one logical clip).

The contextualization storage unit 150 stores object contextualization data of the input image data as a text file in a big data storage. The contextualization storage unit 150 stores a text file in which object contextualization data generated for an image is stored in a big data storage inside the device or transmitted to and stored in a cloud-based big data storage. The big data storage may be, for example, a storage using a Hadoop distributed file system or a storage using NoSQL (Not-Only SQL).

According to an aspect of the invention, the contextualization storage unit 150 may store a list of detected objects and object contextualization data for each frame as a text file in the form of a JSON file. A JSON file is a format that has the advantage of easily exchanging data between a client and a server using JavaScript object notation, and since it is a widely used format, a detailed description will be omitted.

According to an aspect of the invention, the contextualization storage unit 150 may store a list of detected objects and object contextualization data for each frame as a text file in the form of an XML file. XML is an abbreviation of eXtensible Markup Language. It is a language that describes the structure of data using tags, and free tag definition, that is, tag extension is possible. Since the XML file is also a widely used format for data exchange, a detailed description thereof will be omitted.

According to an additional aspect of the present invention, the contextualization storage unit 150 may additionally classify object contextualization data for each image file and store them in the form of a text file. That is, in the example of FIG. 2 , the contextualization storage unit 150 may additionally store object contextualization data for frames belonging to respective image files for FILE ₁ , FILE ₂ , FILE ₃ , and FILE ₄ .

According to an additional aspect of the present invention, the contextualization storage unit 150 may additionally classify object contextualization data for each logical clip and store them in the form of a text file. That is, in the example of FIG. 2 , the contextualization storage unit 150 may additionally store object contextualization data for frames belonging to respective logical clips for CLIP ₁ , CLIP ₂ , and CLIP ₃ .

A method of storing contextualization data in which an object is contextualized by the object contextualization data storage system 10 according to an embodiment of the present invention includes processing an input image, detecting an object, generating object attribute information, and contextualizing a frame. , object contextualization step, clip designation step, and contextualization data storage step.

Each step in which the object contextualization data storage system 10 of the present invention performs the object contextualization data storage method is implemented as computer program instructions, at least a part of which is stored in a memory and executed in a processor.

The input image processing step is a step in which the object contextualization data storage system 10 generates input image data by assigning consecutive frame identifiers to each frame of one or more input image files. The object contextualization data storage system 10 arranges one or more image files inputted in the input image processing step in the input order, and then sequentially allocates new frame identifiers (IDs) to all image files to one or more input image files. is generated as one input image data.

The object detection step is a step in which the object contextualization data storage system 10 detects an object frame by frame from an input image using a deep learning model learned to detect an object from the image. The object contextualization data storage system 10 detects a plurality of objects at once in every frame in the object detection step. The deep learning model used in the object detection step may be trained using a known data set such as the COCO data set.

The object contextualization data storage system 10 classifies the object detected in the object detection step, predicts the type of the object, and predicts the location information of the object in the image using a bounding box.

In the object detection step, a deep learning model used for object detection is configured to be selectable. A user may select an object detection deep learning model through a user interface provided by the object contextualization data storage system 10 . However, the corresponding deep learning model must be able to output a result in a prearranged format so that the object contextualization data storage system 10 can use the object detection result in the object detection step.

The deep learning model used to detect the object in the object detection step may be a deep learning model using a single step method algorithm. Deep learning models that use a single-step method include YOLO models, SSD models, and RetinaNet models, and it is particularly desirable to use YOLO models. The single-pass algorithm is known to outperform the two-pass algorithm in terms of speed.

The deep learning model used to detect the object in the object detection step may be a deep learning model using a two-step method algorithm. Deep learning models using the two-step method include the Faster R-CNN model and the R-FCN model, and it is particularly desirable to use the Faster R-CNN model. Two-pass algorithms are known to outperform single-pass algorithms in terms of accuracy.

The object property information generation step is a step in which the object contextualization data storage system 10 generates object property information including object type and object location information for each detected object. The object contextualization data storage system 10 generates object property information including the type of object classified for each frame and the location information of the corresponding object in the image in the object property information generation step. The object property information generated in the object property information generation step includes object location information, that is, object size information calculated from bounding box coordinates, object classification confidence, and object color (e.g., car color if the object is a car). ) may include information such as For example, when the detected object is a human and the gender is recognized, the object contextualization data storage system 10 may create the object by adding the human gender as an object attribute in the object property information generation step.

According to another embodiment of the present invention, the object property information generated in the object property information generating step may further include an object image in a corresponding frame. Objects detected in the object detection step are displayed as bounding boxes. The object contextualization data storage system 10 may extract the object image within the bounding box and include it in the object attribute information in the object attribute information generation step. In the object property information generation step, the object image included in the object property information is stored in a separate storage location, the storage path may be included in the object property information, and the object image may be encoded in text form and included in the object property information. An object image included in object attribute information may be included in object contextualization data.

The frame contextualization step is a step in which the object contextualization data storage system 10 generates frame contextualization data including detected frame information for each frame in which an object is detected, the type and number of objects, and object property information of each object. The object contextualization data storage system 10 creates frame contextualization data for each frame by structuring information including the type and number of objects detected and classified from image frame data in the frame contextualization step, and object property information of each object.

An example of frame contextualization performed in the frame contextualization step is shown in FIG. 3 and the example has been described above.

According to an embodiment of the present invention, the object contextualization data storage system 10 may generate frame contextualization data for frames sampled at set frame intervals in the frame contextualization step. For example, if the set frame interval is 5, the frame contextualization step may generate frame contextualization data only for frames corresponding to the set frame interval, such as the 5th frame, the 10th frame, and the 15th frame.

According to an embodiment of the present invention, the object contextualization data storage system 10 provides the type and number of objects detected for the current frame in the frame contextualization step, the type and number of objects detected for the frame in which frame contextualization data was previously generated. In the case of the same, frame contextualization data may not be generated for the current frame. That is, the frame contextualization step may generate frame contextualization data for the current frame when the type and number of objects detected for the current frame are different from the types and number of objects detected for the previous frame.

The object contextualization step is a step in which the object contextualization data storage system 10 generates object contextualization data including a list of objects detected with respect to the input image data and the frame contextualization data for each frame. In the object contextualization step, the object contextualization data storage system 10 generates a list of all objects detected in at least one frame of input image data, combines them with frame contextualization data of frames in which objects are detected, and structures the list.

An example of object contextualization performed in the object contextualization step is shown in FIG. 4 and the example has been described above.

The clip designation step is a step in which the object contextualization data storage system 10 configures a logical clip for the input image data based on set clip configuration information and adds the corresponding information to the object contextualization data. In the clip designation step, the object contextualization data storage system 10 constructs a logical clip in all or some frames of the input image data, and adds logical clip configuration information, that is, a logical clip identifier, to the object contextualization data of the frame composed of the logical clips. do. Frames belonging to one logical clip may also belong to another logical clip. That is, one frame may belong to multiple logical clips.

Clip composition information may be information including the number of frames constituting a logical clip. In this case, the object contextualization data storage system 10 organizes consecutive frames corresponding to the number of frames included in the clip configuration information into logical clips for all frames of the input image data in the clip designation step and adds them to the object contextualization data. do.

Also, the clip configuration information may be information including a logical clip list divided into a start frame identifier and an end frame identifier. In this case, the object contextualization data storage system 10 organizes frames corresponding to the logical clip list included in the clip configuration information into logical clips in the clip designation step and adds them to the object contextualization data.

The logical clip list included in the clip composition information is arbitrarily set by the user, and it is preferable that related video frames are designated as logical clips. Logical clips can also be created using separate deep learning models or video analysis tools that analyze the video to identify relevant frames.

Also, the clip configuration information may be information including a logical clip list divided into a start time and an end time. An image file input to the object contextualization data storage system 10 of the present invention may include image capture time information. In this case, in the clip designation step, the object contextualization data storage system 10 organizes frames corresponding to the logical clip list included in the clip configuration information into logical clips and adds them to the object contextualization data.

The contextualization storage step is a step in which the object contextualization data storage system 10 stores the object contextualization data of the input image data as a text file in the big data storage. The object contextualization data storage system 10 stores the text file in which the object contextualization data generated for the image is stored in the contextualization storage step is stored in a big data storage inside the device or transmitted to and stored in a cloud-based big data storage. The big data storage may be, for example, a storage using a Hadoop distributed file system or a storage using NoSQL (Not-Only SQL).

According to an embodiment of the present invention, the step of storing contextualization may store a list of detected objects and object contextualization data for each frame as a text file in the form of a JSON file. A JSON file is a format that has the advantage of easily exchanging data between a client and a server using JavaScript object notation, and since it is a widely used format, a detailed description will be omitted.

According to an embodiment of the present invention, the step of saving contextualization may store a list of detected objects and object contextualization data for each frame as a text file in the form of an XML file. XML is an abbreviation of eXtensible Markup Language. It is a language that describes the structure of data using tags, and free tag definition, that is, tag extension is possible. Since the XML file is also a widely used format for data exchange, a detailed description thereof will be omitted.

According to another embodiment of the present invention, the object contextualization data storage system 10 may additionally classify object contextualization data for each image file and store them in the form of a text file in the contextualization storage step. That is, in the example of FIG. 2 , the object contextualization data storage system 10 may additionally store object contextualization data for frames belonging to respective image files for FILE ₁ , FILE ₂ , FILE ₃ , and FILE ₄ .

According to another embodiment of the present invention, the object contextualization data storage system 10 may additionally classify object contextualization data for each logical clip and store them in the form of a text file in the contextualization storage step. That is, in the example of FIG. 2 , the object contextualization data storage system 10 may additionally store object contextualization data for frames belonging to respective logical clips for CLIP ₁ , CLIP ₂ , and CLIP ₃ .

6 is a flowchart of a method for storing object contextualization data according to the present invention. A procedure of a method for storing object contextualization data will be described with reference to FIG. 6 .

The object contextualization data storage system 10 arranges one or more input image files in the input order and then sequentially assigns new frame identifiers (IDs) consecutive to all the image files to combine one or more input image files into one single image file. It is created with input image data (S1000).

The object contextualization data storage system 10 classifies the type of the object by detecting the object for each frame in the input image using the deep learning model learned to detect the object (S1020). At this time, the object contextualization data storage system 10 classifies the detected object, predicts the type of the object, and predicts the location information of the object in the image using a bounding box.

The object contextualization data storage system 10 generates object property information including the type of the object and location information of the corresponding object in the image for each object detected for each frame (S1040).

The object contextualization data storage system 10 generates frame contextualization data for each frame by structuring information including the type and number of detected objects and object property information of each object (S1060).

The object contextualization data storage system 10 creates object contextualization data for the input image data by structuring information including a list of objects detected in the input image and frame-by-frame contextualization data (S1080).

The object contextualization data storage system 10 configures a logical clip for the input image data based on the set clip configuration information and adds the corresponding information to the object contextualization data (S1100). That is, the object contextualization data storage system 10 configures a logical clip in all or some frames of the input image data, and adds logical clip configuration information, that is, a logical clip identifier, to the object contextualization data of the frame composed of the logical clips.

The object contextualization data storage system 10 generates object contextualization data for an input image as a text file and stores it in a big data storage (S1120).

In the above, the present invention has been described through embodiments with reference to the accompanying drawings, but is not limited thereto, and should be interpreted to cover various modifications that can be obviously derived by those skilled in the art. The claims are intended to cover such variations.

10: Object Contextualized Data Storage System
100: input image processing unit
110: object detection unit
120: frame contextualization unit
130: object contextualization unit
140: clip designation unit
150: context storage unit

Claims (14)

an input image processor generating input image data by allocating consecutive frame identifiers to each frame of one or more image files;
an object detection unit that detects a plurality of objects for each frame in input image data using a deep learning model, classifies the type of the object, and generates object attribute information including the type of object and location information of the object;
a frame contextualization unit generating frame contextualization data including detected frame information for each frame in which an object is detected, the type and number of objects, and object property information of each object;
an object contextualization unit generating object contextualization data including a list of detected objects and the frame contextualization data for each frame;
a clip assignment unit configured to configure a logical clip based on the set clip configuration information and add it to object contextualization data; and
a contextualization storage unit that stores object contextualization data as a text file in a big data storage;
Object contextualization data storage system comprising a.
According to claim 1,
The clip composition information includes the number of frames constituting the logical clip;
An object contextualization data storage system in which the clip designator organizes consecutive frames corresponding to the number of frames included in the clip configuration information from all frames of the input image data into logical clips and adds them to the object contextualization data.
According to claim 1,
The clip configuration information includes a logical clip list divided into a start frame identifier and an end frame identifier;
The object contextualization data storage system that configures frames corresponding to the logical clip list included in the clip configuration information as logical clips and adds them to the object contextualization data.
According to claim 1,
The clip composition information includes a logical clip list divided by a start time and an end time;
The object contextualization data storage system that configures frames of video capture time corresponding to the logical clip list included in the clip configuration information into logical clips and adds them to the object contextualization data.
According to claim 1,
The object contextualization data storage system of claim 1 , wherein the object attribute information generated by the object detection unit further includes an object image in a corresponding frame.
According to claim 1,
The object contextualization data storage system, wherein the contextualization storage unit additionally classifies and stores object contextualization data for each image file.
According to claim 1,
The contextualization storage unit additionally classifies and stores object contextualization data for each logical clip. Object contextualization data storage system.
A method for generating and storing an object detected in an image by an object contextualization data storage system as contextualization data, the method comprising:
an input image processing step of generating input image data by allocating consecutive frame identifiers to each frame of one or more image files;
An object detection step of classifying object types by detecting one or more objects for each frame from input image data using a deep learning model;
object property information generation step of generating object property information including object type and object location information for each detected object;
a frame contextualization step of generating frame contextualization data including detected frame information for each frame in which objects are detected, types and numbers of objects, and object property information of each object;
an object contextualization step of generating object contextualization data including a list of detected objects and the frame contextualization data for each frame;
a clip designation step of constructing a logical clip based on set clip configuration information and adding it to object contextualization data; and
A contextualization data storage step of storing the object contextualization data as a text file in a big data repository;
Object contextualization data storage method comprising a.
According to claim 8,
The clip composition information includes the number of frames constituting the logical clip;
The clip designation step is a step of composing continuous frames corresponding to the number of frames included in the clip configuration information from all frames of the frame input image data into logical clips and adding them to the object contextualization data.
According to claim 8,
The clip configuration information includes a logical clip list divided into a start frame identifier and an end frame identifier;
The clip designation step is a step of composing frames corresponding to a logical clip list included in clip configuration information into logical clips and adding them to object contextualization data.
According to claim 8,
The clip composition information includes a logical clip list divided by a start time and an end time;
The object contextualization data storage system of claim 1 , wherein the clip designation step configures frames of video capture time corresponding to a logical clip list included in clip configuration information into logical clips and adds them to object contextualization data.
According to claim 8,
The object contextualization data storage method of claim 1 , wherein the object property information generated in the object property information generating step further includes an object image in a corresponding frame.
According to claim 1,
The method of storing the object contextualization data comprising the step of storing the object contextualization data separately for each image file.
According to claim 1,
The step of storing the contextualization data includes the step of dividing and storing the object contextualization data for each logical clip.

Images