KR20230165683A - An apparatus for processing video and image search of natural languages based on caption data and a method for operating it - Google Patents

Abstract

A device according to an embodiment of the present invention is a video image search device, comprising: an image information collection unit that collects image information; a caption generator that inputs visual data analysis information of the collected image information into a pre-built natural language processing model to generate caption sentence data describing the image information; an object classification search information constructor that compares visual data analysis information of the collected image information with the caption sentence data and configures object classification search information corresponding to the caption sentence data; And a search database unit that configures a search database for natural language search of images or videos using the caption sentence data and the object classification search information, wherein the object classification search information constructor extracts from the caption sentence data. and a weight setting unit for each information that compares object information and object identification information included in the visual data analysis information and assigns different search weights to the object classification search information for each object.

Description

An apparatus for processing video and image search of natural languages based on caption data and a method for operating it}

The present invention relates to a device and a method of operating the same. More specifically, the present invention relates to a video image search device that performs natural language search based on caption data and a method of operating the same.

Recently, with the development of electronic devices equipped with various functions such as smartphones and easy access to the Internet, it has become possible for individuals to simply and easily create and distribute video content such as images and videos as well as fragmentary text information. It has been done.

As a result, video content continues to be created and distributed in large quantities, and user demand for searching video content is also increasing due to the increase in the amount of related data.

Accordingly, users want to search for information about the image or video they want through not only text search using general text, but also the content itself, including video images.

However, most current general information retrieval systems provide documents or images containing text corresponding to the keyword when the user inputs a keyword based on most text. This allows the user to search for the exact video and image desired. There are limits to supporting .

To solve this problem, image search using automated search technology of artificial intelligence has been proposed recently, but since this is learning using feature information of images, only specific objects in the image can be searched in a partial and general form. As a result, users cannot naturally search based on natural language, and the accuracy is very low.

The present invention was developed to solve the above-mentioned problems. After constructing a search database using caption sentence data describing image or video information and performing a natural language-based search corresponding to the caption sentence data in the search database, The purpose is to provide a video image search device and method of operating the same that allows users to search for desired images and video information more accurately and quickly by providing the search results as images or video information.

An apparatus according to an embodiment of the present invention for solving the problems described above is a video image search device, comprising: an image information collection unit that collects image information; a caption generator that inputs visual data analysis information of the collected image information into a pre-built natural language processing model to generate caption sentence data describing the image information; an object classification search information constructor that compares visual data analysis information of the collected image information with the caption sentence data and configures object classification search information corresponding to the caption sentence data; And a search database unit that configures a search database for natural language search of images or videos using the caption sentence data and the object classification search information, wherein the object classification search information constructor extracts from the caption sentence data. and a weight setting unit for each information that compares object information and object identification information included in the visual data analysis information and assigns different search weights to the object classification search information for each object.

In addition, a method of operating a device according to an embodiment of the present invention to solve the above-described problem includes: collecting image information; Inputting visual data analysis information of the collected image information into a pre-built natural language processing model to generate caption sentence data describing the image information; Comparing visual data analysis information of the collected image information with the caption sentence data, and constructing object classification search information corresponding to the caption sentence data; and configuring a search database for natural language search of an image or video using the caption sentence data and the object classification search information. The step of configuring the object classification search information includes using the caption sentence data. Comparing extracted object information and object identification information included in the visual data analysis information, and assigning different search weights to the object classification search information for each object.

According to an embodiment of the present invention, visual data analysis information of the collected image information is input into a pre-built natural language processing model to generate caption sentence data describing the image information, and visual data of the collected image information is generated. By comparing analysis information and the caption sentence data, object classification search information corresponding to the caption sentence data is constructed, and the caption sentence data and the object classification search information are used to search for natural language search of an image or video. You can configure a database.

Accordingly, according to an embodiment of the present invention, a natural language-based search corresponding to caption sentence data in a search database is performed, and then the search results are provided as image or video information, thereby providing the user with more accurate and faster images and images desired by the user. A video image search device and method of operating the same that enable searching video information can be provided.

1 is a conceptual diagram schematically showing the entire system according to an embodiment of the present invention.
Figure 2 is a block diagram to explain in more detail a video image search device according to an embodiment of the present invention.
Figure 3 is a block diagram to explain in more detail the processing of visual data analysis information according to an embodiment of the present invention.
Figure 4 is a flowchart showing a video image search process according to an embodiment of the present invention.
Figure 5 is a diagram showing the output of search results according to an embodiment of the present invention.

The following merely illustrates the principles of the invention. Therefore, those skilled in the art will be able to invent various devices that embody the principles of the present invention and are included in the spirit and scope of the present invention, although not explicitly described or shown herein. In addition, it is understood that all conditional terms and embodiments listed herein are, in principle, expressly intended only for the purpose of ensuring that the concept of the invention is understood, and are not limited to the embodiments and conditions specifically listed as such. It has to be.

Additionally, it is to be understood that any detailed description reciting principles, aspects, and embodiments of the invention, as well as specific embodiments, is intended to encompass structural and functional equivalents thereof. In addition, these equivalents should be understood to include not only currently known equivalents but also equivalents developed in the future, that is, all elements invented to perform the same function regardless of structure.

Accordingly, for example, the block diagrams herein should be understood as representing a conceptual view of an example circuit embodying the principles of the invention. Similarly, all flow diagrams, state transition diagrams, pseudo-code, etc. are understood to represent various processes that can be substantially represented on a computer-readable medium and are performed by a computer or processor, whether or not the computer or processor is explicitly shown. It has to be.

Additionally, the clear use of terms such as processor, control, or similar concepts should not be construed as exclusively referring to hardware capable of executing software, and should not be construed as referring exclusively to hardware capable of executing software, including without limitation digital signal processor (DSP) hardware and ROM for storing software. It should be understood as implicitly including ROM, RAM, and non-volatile memory. Other hardware for public use may also be included.

The above-described purpose, features and advantages will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art will be able to easily implement the technical idea of the present invention. There will be. Additionally, in describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

1 is a conceptual diagram schematically showing the entire system according to an embodiment of the present invention.

Referring to FIG. 1, the entire system according to an embodiment of the present invention includes a video image search device 100, a user terminal 200, a relay server 300, and a web server 400.

Here, the video image search device 100 collects video information, inputs visual data analysis information of the collected video information into a pre-built natural language processing model, and generates caption sentence data describing the video information. Generates object classification search information corresponding to the caption sentence data by comparing visual data analysis information of the collected image information with the caption sentence data, and uses the caption sentence data and the object classification search information to generate object classification search information. , Constructs a search database for natural language search of images or videos.

In addition, the video image search device 100 can provide an AI-based search service using a search database, and for this purpose, an image processing model for image analysis and caption sentence data from the image analysis results are provided. A search service database can be built using a natural language processing model for creation.

In addition, the video image search device 100 provides object classification search information and sentence classification corresponding to the caption sentence data according to the object classification search information of the image analysis result and the importance and weight of each classification information of the caption sentence data. Accuracy is calculated, and the object classification search information and accuracy are used to select optimal caption sentence data that matches the search results among caption sentence data pre-configured to describe the information of the collected image or video, and the caption sentence In response to the data, image or video data stored in the database can be provided to the user as a search result.

The output data of the video image search device 100 may be transmitted to the user terminal 200, relay server 300, web server 400, etc. according to the performance of the service process of the video image search device 100.

In addition, the user terminal 200 may transmit a user input through an interface to the video image search device 100, and the video image search device 100 indexes appropriate caption sentence data corresponding to the user input, By providing response information including image or video information corresponding to the indexed caption sentence data to the user terminal 200, the user can check the image or video information provided as a search result.

To this end, the video image search device 100 can be connected to the user terminal 200, the relay server 300, and the web server 400 through a wired or wireless network, and each component communicates through the wired or wireless network. It may be equipped with one or more wired and wireless communication modules to perform. The wired/wireless communication network may use various widely known communication methods.

Meanwhile, the relay server 300 collects, stores and manages caption sentence data of image or video information from the video image search device 100 through the web server 400, and relays it to the user terminal 200. It could be a server.

In addition, the user terminal 200 receives and outputs search results based on caption sentence data provided from the video image search device 100, or inputs user query information into the video image search device 100. It may be equipped with communication means and input/output means.

Additionally, the user terminal 200 can input a natural language-based search word as query information into the video image search device 100. For example, a natural language-based search term may include “Find a picture of my child wearing light blue clothes,” and the video image search device 100 configures a search keyword based on natural language analysis corresponding to the search keyword, and the configured search keyword. Caption sentence data in a search database using can be indexed, image or video information corresponding to the indexed caption sentence data can be identified, and the identified image or video information can be transmitted to the user terminal 200.

Here, the image or video information may include file data of the image or video information uploaded to the web or the Internet, or may include address information of the image or video information uploaded to the web or the Internet. Additionally, according to an embodiment of the present invention, it may include access path information of images or video information stored in the user terminal 200 or other external devices.

Meanwhile, in order to accurately index the caption sentence data, the video image search device 100 according to an embodiment of the present invention compares the visual data analysis information of image information with the caption sentence data, and selects the caption sentence data corresponding to the caption sentence data. Object classification search information can be configured, and a search database for natural language search of images or videos can be configured using the caption sentence data and the object classification search information, which will be described in more detail later.

Figure 2 is a block diagram for explaining in more detail a video image search device according to an embodiment of the present invention, and Figure 3 is a block diagram for explaining in more detail the processing of visual data analysis information according to an embodiment of the present invention. am.

Referring to FIG. 2, the video image search device 100 according to an embodiment of the present invention includes an image information collection unit 110, a visual data analysis information processing unit 120, a caption generation unit 130, and object classification search information. It may include a configuration unit 140, a search word input unit 150, a search database unit 160, and a search result output unit 170.

First, the image information collection unit 110 collects image information according to a request from the user terminal 200. Here, the image information may be an image or video data stored in the user terminal 200, or may be an image or video data stored in another remotely connected external device, and the image or video data may be the image or video file itself, It may include image or video data obtained from file path information or file address information specified by the user.

In addition, the caption generator 130 inputs the object identification information extracted from the visual data analysis information processed by the visual data analysis information processor 120 into a pre-built natural language processing model for generating caption sentence data, Generate caption sentence data that explains video information.

Here, referring to FIG. 3, the visual data analysis information processing unit 120 includes a background modeling unit 121, a shape operation processing unit 122, an element connection processing unit 123, an object tracking unit 124, and a 3D viewpoint transformation. It may include a unit 125, an object classification processing unit 126, a visual data analysis result output unit 127, and a deep learning analysis processing unit 128.

First, the background modeling unit 121 can model the background area corresponding to the image information collected by the image information collection unit 110, and according to the background modeling, identify an object corresponding to the foreground and track it. You can calculate background area data that allows you to

The background modeling unit 121 can generate a background probability model based on the well-known Gaussian Mixutre Model from image data so that objects can be accurately detected in a complex environment. This allows the creation of background area data by reflecting many variables such as changes in lighting, objects added or removed from the background, backgrounds with movement such as swaying tree branches or fountains, and areas with high traffic.

More specifically, when consecutive image frames are input from the image information, the background modeling unit 121 can construct a background model per pixel according to statistical probability for time t through background subtraction processing. The background modeling unit 121 subtracts background information constructed according to a Gaussian mixture model from the image information of the current frame and creates the background according to the MOG (Mixtrue of Guassians) method, which cumulatively updates the background information to a per-pixel background model. Subtraction processing can be performed.

According to the background subtraction process of the background modeling unit 121, foreground pixel information remains in the background-subtracted image information, which can be referred to as a blob image, and the blob image is binary map data corresponding to the foreground pixel. It can also be said that

In addition, the shape operation processing unit 122 may perform shape operation processing to determine shape area information from binary map data of foreground pixel information that is subtracted and output according to background modeling in the background modeling unit 121, The shape area information may be grouping information that groups a plurality of pixels into one or more shape structural elements.

For example, the shape operation processing unit 122 may determine the shape area information by applying one or more morphological operation filters to the binary map data of the foreground pixel information.

Here, one or more morphological operation filters may include a binary erosion operation filter and a binary dilation operation filter, and each filter changes the size of the bright area in the binary map data to the size of the predetermined shape structural element. Computational processing that expands or reduces information in proportion to the information can be performed. When applying a dilation operation filter, dark areas smaller than the shape structure element can be removed, and conversely, when applying an erosion operation filter, bright areas smaller than the shape structure element can be removed, and at the same time, the size of large areas that are not removed can be reduced or increased. It can be.

In order to detect objects of a pre-specified size, the shape operation processing unit 122 uses a binary open filter and a closed filter to remove only small areas while maintaining the size of areas larger than the shape structural element. You can also perform .

Then, the element connection processing unit 123 acquires image information corresponding to the shape structural elements specified in the shape operation processing unit 122, connects and processes each shape structural elements, and classifies them into independent object connection areas, Connected component labeling processing can be performed to assign unique label values corresponding to the classified object connection area. This connected component labeling process can be used particularly effectively in binary map data.

Additionally, the element connection processing unit 123 may divide the classified object connection areas into unique label values, and determine and output feature values of the area, such as the size, position, direction, and perimeter of each object area. The element connection processing unit 123 may apply a recursive algorithm or a sequential algorithm to perform connection component labeling processing and classify object connection areas.

For example, a sequential algorithm may be an algorithm that determines the label of the current pixel by searching the pixel data of each foreground pixel for the labels of the top and left pixels of that pixel. Neighboring pixels including the top pixel and the left pixel may be pixels that have already been processed in the labeling process.

Here, the element connection processor 123 may assign a new label value to the current pixel if none of the neighboring pixels are foreground pixels. Additionally, when only one pixel among two neighboring pixels is a foreground pixel, the element connection processing unit 123 may assign the label value of that pixel to the current pixel.

Additionally, the element connection processing unit 123 may assign the same label value to the current pixel when both neighboring pixels are foreground pixels and have the same label value.

However, if two pixels are foreground pixels but have different label values, these two areas are connected to each other by the current pixel and must be merged with the same label value.

Accordingly, the element connection processing unit 123 may assign the smaller value of the label values of the two pixels as the label value of the current pixel, and register the two labels as equal labels in the equivalence table. When this first labeling process is completed, information about labels that must be merged into the same area can be stored in the equivalence table.

In addition, the element connection processing unit 123 can use this equivalence table to assign the same label to all pixels in each object connection area in the second labeling process.

Accordingly, the element connection processing unit 123 can output object connection area information according to the connection component labeling process along with blob image data including foreground pixel data, and the object connection area information is connected to each object. It may include characteristic information (position information, size information, direction information, perimeter information, etc.) corresponding to the area.

Additionally, the object tracking unit 124 may obtain tracking data for tracking object information in image information based on blob image data and object connection area information. Tracking data may include list information of objects found and tracked within the image.

More specifically, the object tracking unit 124 can cumulatively match the blob image at time t and the blob image at t-1, identify and process the object corresponding to the object connection area, and include it in the list. Tracking processing using the Kalman filter method may be an example. The Kalman filter is a filter that detects the state of a linear dynamical system by recursively accumulating input data based on measurements made over time.

Accordingly, the object tracking unit 124 may output object tracking data identified within the image information. The object tracking data may include size information and movement speed information of each identified object, along with object list information.

In addition, the 3D viewpoint conversion unit 125 converts the size information and movement speed information of each object in the above-described object tracking data into actual values according to the 3D viewpoint conditions for each camera device. That is, the tracking data of the object tracking unit 124 has size and speed information of the object, but is a relative value between pixels, so the 3D viewpoint conversion unit 125 converts the pixel coordinates of the image into coordinates in actual space. Conversion processing can be performed using camera characteristic information such as angle of view and height, which are previously specified through calibration.

Accordingly, the viewpoint conversion unit 125 can convert the size information and speed information of objects listed in the object tracking data into size information and speed information in actual space.

Meanwhile, the object classification processing unit 126 may perform processing to assign classification information to the object identified in the object tracking data based on preset classification criteria, and the visual data analysis result output unit 127 may perform the processing of assigning classification information to the object identified in the object tracking data. According to the assigned classification information, the visual data analysis information of the image information can be output as a visual data analysis result.

However, it may be more efficient for the process of setting the classification standard itself or accurately determining whether it corresponds to the classification standard to be processed in the deep learning analysis processing unit 128.

That is, in order to accurately classify and detect objects through cumulative learning of various external image information, the object classification processing unit 126 collects the tracking data and image information for object classification processing through the deep learning analysis processing unit 128. You can perform deep learning analysis and object classification processing using deep learning analysis result information.

And, the deep learning analysis processing unit 128 generates deep learning distributed processing data based on the tracking data and the image information according to the initial analysis obtained through the 3D viewpoint conversion unit 125, and Deep learning distributed processing data can be transmitted and processed to one or more deep learning distributed processing devices, and deep learning-based image information analysis results received from the deep learning distributed processing devices can be received and transmitted to the object classification processor 126. .

In addition, the deep learning analysis processing unit 128 constructs neural network data from image information pre-learned according to a deep learning method, detects object information corresponding to the distributed processing request data using the neural network data, and detects object information corresponding to the distributed processing request data. The classification information of the object information can be determined.

In addition, the deep learning analysis processing unit 128, for example, sets up a neural network according to a deep learning algorithm based on DNN (Deep Neural Network), and the neural network includes an input layer and one or more hidden layers. It can be composed of Layers and Output Layer. Here, the deep learning algorithm may be applied to a neural network other than DNN, for example, a neural network such as a Convolution Neural Network (CNN) or a Recurrent Neural Network (RNN).

Here, the deep learning analysis processing unit 128 can identify a target object in one or more specific images obtained from the image information through a deep learning-based neural network, and a specific object other than the target object is identified. In this case, the degree of similarity between the specific image and one or more of the images may be calculated.

And, when the similarity is greater than a preset standard value, error information about the error between the image processed output value and object information is generated based on the neural network for the specific image, and a preset back propagation algorithm is used. Parameters constituting the neural network can be adjusted based on the error information.

Here, the similarity comparison method between images is histogram matching (Histogram matching).

An example may be comparing similarity using template matching assigned to an object identified in the image, or comparing feature points extracted from the image through the neural network.

And, the deep learning analysis processing unit 128 sets a weight for the connection strength between the input layer, one or more hidden layers, and the output layer constituting the neural network through the backpropagation algorithm based on the error information, or By varying the bias of the units comprised in the input layer, hidden layer, and output layer, learning processing can be performed to minimize identification errors for the target object.

In addition, the deep learning analysis processing unit 128 can update the object information by repeatedly learning images continuously received from the video image search device 100 or other various imaging devices through the neural network, through which Objects corresponding to parameters for each object characteristic included in the object information can be accurately identified from the video and image.

And, the deep learning analysis processing unit 128, through the object detection unit, one or more objects identified from tracking data and image information obtained from deep learning distributed processing request data based on the learned neural network (or neural network) data. can be detected.

Referring again to FIG. 2, the caption generator 130 generates caption sentence data using object identification information extracted from the visual data analysis information output from the visual data analysis information processing unit 120 according to the above-described processing. can be created.

To this end, the caption generator 130 may include a caption sentence data constructor 133, a morpheme correction processor 135, a synonym DB constructor 137, and a face recognition DB constructor 139.

More specifically, the caption sentence data configuration unit 133 configures caption sentence data corresponding to the visual data analysis information output from the visual data analysis information processing unit 120.

Here, the caption sentence data may be sentence data that describes the image or video detected in response to the image information based on natural language, and is in the form of a string containing at least one of object information, motion information, and time information. It can be configured.

To generate such sentence data, the caption sentence data constructor 133 may pre-construct a visual data-based sentence generation learning model that generates sentences based on natural language from visual data analysis information. Visual data analysis information may include, for example, feature information and object classification information within an image, and as association learning is performed using learning data between caption data as output data and the visual data analysis information, the visual data is based on the visual data. A sentence generation learning model may be pre-built.

Additionally, the morpheme correction processing unit 135 may analyze the morphemes of the caption sentence data generated by the caption generation unit 130 and correct them to their original form if they correspond to verbs and adjectives.

For example, the morpheme correction processing unit 135 may analyze the morphemes of the caption sentence data into nouns, verbs, adjectives, and adverbs, and in the case of verbs and adjectives among each morpheme analyzed, their original forms, if applicable, It can be corrected with .

For example, if the caption data consists of “child raising hand,” the morpheme correction processing unit 135 may perform morpheme correction processing with “hand, raise, child.” This processing is to improve search performance and accuracy. The morpheme correction processing unit 135 identifies verbs and adjectives among the search words even when a natural language search word is entered according to the user's request, and provides information corresponding to the verbs and adjectives in the search word. Correction processing to the original shape can be further performed.

Meanwhile, the synonym DB configuration unit 137 identifies one or more words corresponding to the caption sentence data, indexes synonyms of the identified words, and converts the indexed synonyms into a database.

To this end, the thesaurus DB configuration unit 137 may access a pre-built thesaurus database or store and manage the thesaurus database in advance. Here, the index category of the synonym may be set according to a preset similarity range value, and the similarity calculation method may use, for example, a cosine similarity calculation process using Word2Vec.

In addition, the synonym DB configuration unit 137 may set synonyms of words extracted from the caption sentence data as search keywords in response to the caption sentence data, and include the search keyword as additional data of the caption sentence data. there is. Accordingly, search accuracy can be adjusted to suit each data characteristic by setting whether to index synonyms and setting the similarity range.

Meanwhile, the face recognition DB configuration unit 139 can identify one or more object face recognition words from the visual data analysis information, and can build, store, and manage a DB using the identified object face recognition words.

To this end, the face recognition DB configuration unit 139 may collect preset object face recognition words and visual data analysis information corresponding thereto from the user terminal 200, etc. For example, common visual data analysis information may be extracted from various images corresponding to a specific child of a specific user. In this case, the face recognition DB component 139 may use the child name input from the user terminal 200. The information can be set as an object face recognition word.

Accordingly, the face recognition DB configuration unit 139 sets, in response to the caption sentence data, an object face recognition word identified from visual data analysis information corresponding to the caption sentence data as a search keyword, and sets the search keyword as the search keyword. It can be included as additional data in the caption sentence data. Accordingly, search accuracy can be adjusted to suit each data characteristic and natural language conditions by setting the object face recognition.

Then, the object classification search information configuration unit 140 compares the object identification information identified in the visual data analysis information and the caption sentence data, and determines the object according to whether the object identification information is included in the caption sentence data. You can configure classification search information.

Here, the object classification search information is information added to improve search efficiency and accuracy in searching caption sentence data for indexing image or video information based on natural language, and includes the visual data analysis information and the caption sentence. It can be constructed conditionally based on comparison with data. And, as the object classification search information, the caption sentence data, and the corresponding image or video information are respectively mapped, search group data for search may be formed. Additionally, the object classification search information can be used to calculate classification accuracy of the caption sentence data.

To this end, the object classification search information configuration unit 140 can compare object identification information classified from caption sentence data and visual data analysis information, and perform corresponding exclusion processing or weight application processing for each keyword. .

More specifically, according to the first embodiment, the object classification search information configuration unit 140 excludes object identification information not included in the caption sentence data from the object classification search information, and excludes object identification information not included in the caption sentence data from the object classification search information. Object identification information can be added to the object classification search information.

For example, if the caption sentence data includes “child riding a bicycle” and the object identification information classified from the visual data analysis information identifies “road, bicycle, child,” according to the first embodiment. , When configuring the object classification search information, “road” may be excluded, and “bicycle, child” may be configured as the object classification search information.

Additionally, according to the second embodiment, the object classification search information constructor 140 determines whether the same object exists in object identification information classified from the caption sentence data and the visual data analysis information, It may include a weight setting unit 145 for each information that assigns different search weights to each classification search information.

According to the second embodiment, the information-specific weight setting unit 145 sets the information-specific weight if the first object information extracted from the caption sentence data does not exist in the object identification information extracted from the visual data analysis information. The setting unit 145 may set a search keyword corresponding to the first object information as the object classification search information, and may give the search keyword a weight relatively lower than that of other keywords.

Conversely, when the first object information is present in both the caption sentence data and the object identification information extracted from the visual data analysis information, the weight setting unit 145 for each information sets the search weight of the object identification information. It can be given relatively high.

This search weight corresponds to whether the same object exists in the object identification information classified from the caption sentence data and the visual data analysis information, and is preset according to the distribution of each search keyword within the data or similarity between keywords, etc. It may be determined differently depending on the table, etc.

For example, according to the second embodiment, if the caption sentence data includes “child riding a bicycle” and the object identification information classified from the visual data analysis information identifies “road, bicycle, child”, When constructing object classification search information, “road” does not exist in the caption sentence data, so a 50% weight is given based on the pre-configured table, and “bicycle, child” is given a relative weight based on the pre-configured table. A weightage of as high as 80% may be assigned.

In addition, for example, the caption generation unit 130 generates caption sentence data of “The baby is eating baby food at home,” and the visual data analysis information processing unit 120 generates caption sentence data such as “home,” “baby,” and “baby.” When object identification information of “mother” and “baby food” is output, “mother” does not exist in the caption sentence data, so the weight setting unit 135 for each information sets the object classification search information corresponding to “mother” However, the search weight of the “mom” keyword can be assigned a value that is relatively 50% lower than that of other keywords.

Conversely, since “house,” “baby,” and “baby food” in the object identification information are also commonly present in the caption sentence data, the weight setting unit 135 for each information may assign a search weight as high as 80%. there is.

Then, the search database unit 160 uses the caption sentence data and the object classification search information to calculate sentence classification accuracy and configures a search database for natural language search of images or videos. To this end, the search database unit 160 includes a sentence classification accuracy calculation unit 163 and a mapping table configuration unit 165.

More specifically, the sentence classification accuracy calculation unit 163 calculates the accuracy of the caption sentence data based on the overlap ratio of objects between the caption sentence data and the object classification search information or the weight for each object classification search information. You can.

For example, the sentence classification accuracy may be the sum of importance and weight for each object classification search information of the caption sentence data, or may be a number representing the average or matching rate, and the embodiment of the present invention is limited by the calculation method. It doesn't work.

Additionally, the search database unit 160 may adjust the search accuracy range of the video image search result based on the accuracy of the calculated caption sentence data.

In addition, the mapping table configuration unit 165 includes image information of an image or video and a caption sentence generated by the caption generator 130 in response to the image or video for more accurate sentence creation and accurate sentence indexing. The data, the object classification search information constituting the object classification search information configuration unit 140, and the sentence classification accuracy can be mapped, stored, and managed as one search group data.

Accordingly, when the search database unit 160 is configured, a natural language-based search word input from the user terminal 200 is input through the search word input unit 150, and the search database unit 160 responds to the input natural language-based search word. Thus, search group data is identified by performing an index based on a combination of the caption sentence data and the object classification search information mapped to the mapping table configuration unit 165, and image or video information included in the identified search group data. is extracted and output to the search result output unit 170.

Here, the search database unit 160 may index the caption sentence data and the object classification search information according to a preset accuracy category in response to a natural language-based voice or text search sentence input to the search word input unit 150. there is. In the case of this indexing method, the caption sentence data already composed is indexed based on text, so unlike the existing method of indexing the characteristics of the image itself as keywords, the learning process and indexing process are simplified and processed quickly, and a search suitable for natural language is performed. This has the advantage of outputting more accurate search results.

Accordingly, the search result output unit 170 can compare the search word input to the search word input unit 150 with the data included in the search database unit 160 and output a video or image to the user terminal 200. There will be.

Figure 4 is a flowchart showing a video image search process according to an embodiment of the present invention.

First, the video image providing device 100 collects video information for generating caption sentence data according to a request from the user terminal (S101).

Here, the video image providing device 100 may provide an interface to collect video information from the user terminal 200 in response to a request from the user terminal 200.

Then, the video image providing device 100 inputs the visual data analysis information of the collected video information into a pre-built natural language processing model to generate caption sentence data explaining the video information (S103).

Here, the caption generation unit 130 of the video image providing device 100 uses object identification information extracted from the visual data analysis information processed by the visual data analysis information processing unit 120, and pre-built the caption sentence data for generating caption sentence data. Caption sentence data explaining the image information can be generated by inputting it into a natural language processing model.

Then, the video image providing device 100 compares the visual data analysis information of the collected association information with the caption sentence data and configures object classification search information corresponding to the caption sentence data (S105).

Here, the object classification search information configuration unit 140 of the video image providing device 100 compares the object identification information identified in the visual data analysis information and the caption sentence data, and adds the object identification information to the caption sentence data. The object classification search information can be configured depending on whether is included.

Then, the video image providing device 100 uses the caption sentence data and the object classification search information to calculate sentence classification accuracy and configure a search database for natural language search of images or videos (S107).

Here, the search database unit 160 may calculate sentence classification accuracy using the caption sentence data and the object classification search information.

Additionally, the search database unit 160 may configure a search database for natural language search of images or videos, and may include a sentence classification accuracy calculation unit 163 and a mapping table configuration unit 165.

Then, the user terminal 200 receives a search sentence as a natural language-based voice or text and transmits it to the video image providing device 100 (S109).

Here, the search database unit 160 may further include a search word input unit 150 for inputting a natural language-based voice or text search sentence from the user terminal 200.

Then, the video image providing device 100 indexes caption sentence data corresponding to the input natural language-based search sentence using accuracy based on the object classification information (S111).

Here, using the sentence classification accuracy calculated by the sentence classification accuracy calculation unit 163, the search range and priority can be set to correspond to the search results desired by the user.

Thereafter, the video image providing device 100 outputs the indexed caption sentence data and video information mapped to the caption sentence data as a search result (S113).

Depending on the output image information, one or more images or videos may be output as a search result from the user terminal 200.

Figure 5 is a diagram showing the output of search results according to an embodiment of the present invention.

* Referring to FIG. 5, the user terminal 200 according to an embodiment of the present invention can receive an alarm service using caption data configuration from a separate device. In the case of an alarm service, caption data and corresponding image or video information may be configured, and the video image providing device 100 may build a search database 140 in advance from the caption data and the corresponding image or video information. You can.

And, as shown in Figure 5, when the user inputs “Find a picture of my two children looking at their cell phones” or “Find a picture of my grandfather taking medicine”, the input natural language search data is sent to the video image providing device. It can be entered into the search word input unit 150 of (100).

Then, the video image providing device 100 indexes the caption sentence data corresponding to the search term using accuracy based on the object classification search information, and video images mapped to the indexed caption sentence data. As information search results, each of Photo 1 and Photo 2 can be output to the user terminal 200.

The method according to the present invention described above can be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of computer-readable recording media include ROM, RAM, CD-ROM, and magnetic tape. , floppy disks, optical data storage devices, etc.

The computer-readable recording medium is distributed in a computer system connected to a network, so that computer-readable code can be stored and executed in a distributed manner. And, functional programs, codes, and code segments for implementing the method can be easily deduced by programmers in the technical field to which the present invention pertains.

In the above, preferred embodiments of the present invention have been shown and described, but the present invention is not limited to the specific embodiments described above, and may be used in the technical field to which the invention pertains without departing from the gist of the invention as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

100: video image search device 200: user terminal
300: relay server 400: web server

Claims (9)

In the video image search device,
A video information collection unit that collects video information;
a caption generator that inputs visual data analysis information of the collected image information into a pre-built natural language processing model to generate caption sentence data describing the image information;
an object classification search information constructor that compares visual data analysis information of the collected image information with the caption sentence data and configures object classification search information corresponding to the caption sentence data; and
It includes a search database unit that configures a search database for natural language search of images or videos using the caption sentence data and the object classification search information,
The object classification search information configuration unit,
It includes a weight setting unit for each information that compares object information extracted from the caption sentence data and object identification information included in the visual data analysis information and assigns different search weights to the object classification search information for each object. doing
Video image retrieval device. According to paragraph 1,
The caption generator,
a synonym DB component that extracts synonyms for each word included in the caption sentence data and adds them to the object classification search information; and
A face recognition DB component that extracts a face recognition word for a specific object from the visual data analysis information and adds it to the object classification search information; comprising a.
Video image retrieval device. According to paragraph 1,
The search database unit,
A sentence classification accuracy calculation unit that calculates accuracy for each caption sentence data based on an overlap ratio of objects between the caption sentence data and the object classification search information.
Video image retrieval device. According to paragraph 3,
The search database unit,
Adjusting the accuracy range of video image search results based on the accuracy of each caption sentence data calculated above.
Video image retrieval device. According to paragraph 1,
Further comprising a search result output unit that outputs a natural language search result of an image or video using the caption sentence data and the object classification search information.
Video image retrieval device. In a method of operating a video image search device,
collecting image information;
Inputting visual data analysis information of the collected image information into a pre-built natural language processing model to generate caption sentence data describing the image information;
Comparing visual data analysis information of the collected image information with the caption sentence data, and constructing object classification search information corresponding to the caption sentence data; and
Constructing a search database for natural language search of images or videos using the caption sentence data and the object classification search information,
The step of configuring the object classification search information is,
Comprising object information extracted from the caption sentence data and object identification information included in the visual data analysis information, and assigning different search weights to the object classification search information for each object.
How to operate a video image search device. According to clause 6,
The step of generating the caption sentence data is,
extracting synonyms for each word included in the caption sentence data and adding them to the object classification search information; and
Comprising: extracting a face recognition word for a specific object from the visual data analysis information and adding it to the object classification search information;
How to operate a video image search device. According to clause 6,
The step of configuring the search database is,
Comprising: calculating accuracy for each caption sentence data based on an overlap ratio of objects between the caption sentence data and the object classification search information;
How to operate a video image search device. According to clause 8,
Further comprising: adjusting the accuracy range of the video image search result based on the accuracy of each calculated caption sentence data.
How to operate a video image search device.