KR20230027945A - Apparatus and method for indexing of video - Google Patents

Abstract

Disclosed are an image indexing apparatus and a method for performing the same. The image indexing apparatus according to an embodiment comprises: an indexing unit which acquires a feature batch for an input image, extracts a first feature vector and metadata from the obtained feature batch, connects some of the elements of the first feature vector to generate a second feature vector, packages the second feature vector and the metadata to index the second feature vector; a query management unit which obtains a user's query and determines a rank for the obtained user's query based on the obtained user's query and the indexed second feature vector; and a load balancing unit which controls load distribution for the indexing and ranking at preset intervals. Accordingly, the present invention can measure similarity between a plurality of images.

Description

Video indexing device and method {APPARATUS AND METHOD FOR INDEXING OF VIDEO}

The disclosed embodiments relate to image indexing techniques for retrieving images.

In most images, pixel values adjacent to each other have a correlation, and accordingly, data constituting an image may contain more information than necessary.

Meanwhile, indexing plays a very important role in image search. An image may be searched based on text recognized from an image, audio features extracted from an audio stream, and visual features extracted from an image frame. Among them, various studies are being conducted on a Convolutional Neural Network (CNN) model to provide global features.

However, in studies on existing CNN models, image data is inherently large, the number of feature vectors extracted when processing big data is large, and the dimension of global feature vectors is high, resulting in excessive computation required for indexing and retrieval. There was a costly problem.

In particular, in the case of text-based retrieval, there is a limitation in that similarity between multiple images required for image retrieval cannot be calculated without conversion into an expression suitable for inverse-indexing to apply global features in reverse-indexing-based retrieval.

Republic of Korea Patent Registration No. 10-0359234 (published on October 18, 2002)

Disclosed embodiments are to provide an image indexing apparatus and method for searching an image for a target.

According to an embodiment, an image indexing apparatus obtains a feature batch for an input image, extracts a first feature vector and metadata from the obtained feature batch, and selects elements of the first feature vector. An indexing unit that connects parts to generate a second feature vector, packages the second feature vector and the meta data, and indexes the second feature vector; obtains a user's query; and a query management unit that determines a ranking of the user's query based on the indexed second feature vector, and a load balancing unit that controls load distribution for the indexing and ranking at every predetermined period.

The indexing unit may generate the second feature vector by quantifying elements of the first feature vector and then connecting some of the quantified elements of the first feature vector.

The indexing unit quantifies elements of the first feature vector based on a preset quantization factor, connects elements that satisfy a preset criterion among elements of the quantified first feature vector, and obtains the second feature. Vectors can be created.

The quantification factor may satisfy the following equation.

(mathematical expression)

: 상기 제1 벡터의 i(이때, i는 1≤j≤N을 만족하는 정수, N은 상기 제1 벡터의 차원)번째 차원의 원소를 정량화하기 위한 정량화 요소

: Quantification element for quantifying the element of the i-th dimension of the first vector (where i is an integer satisfying 1≤j≤N, and N is the dimension of the first vector)

: 자연로그

: natural logarithm

: 특징 벡터의 차원

: Dimensions of feature vectors

: 상기 i번째 차원의 원소

: element of the i-th dimension

The load balancing unit may control load distribution for at least one of the indexing and the ranking based on a weighted round robin method.

A similarity between the indexed second feature vector and the feature vector for the user's query is calculated using TF-IDF (Term Frequency-Inverse Document Frequency), and based on the calculated similarity, for the user's query An image search unit for searching for an image may be further included.

According to an embodiment, an image indexing method is a method performed in a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors, and includes disposing features of an input image. (feature batch) is obtained, a first feature vector and metadata are extracted from the obtained feature batch, a second feature vector is generated by connecting some of the elements of the first feature vector, and the second feature vector and indexing the second feature vector by packaging the metadata, obtaining a user's query, and ranking the obtained user's query based on the obtained user's query and the indexed second feature vector. and controlling the load distribution for the indexing and the ranking at every preset period.

In the indexing, after quantifying the elements of the first feature vector, a dimensionally reduced second feature vector may be generated by connecting some of the quantified elements of the first feature vector.

The indexing may include quantifying elements of the first feature vector based on a preset quantization factor, and connecting elements that satisfy a preset criterion among the quantified elements of the first feature vector to determine the indexing of the first feature vector. A second feature vector may be generated.

The quantification factor may satisfy the following equation.

(mathematical expression)

: 상기 제 1 특징 벡터의 i(이때, i는 1≤j≤N을 만족하는 정수, N은 상기 제1 벡터의 차원)번째 차원의 원소를 정량화하기 위한 정량화 요소

: Quantification element for quantifying the element of the i-th dimension of the first feature vector (where i is an integer satisfying 1≤j≤N, and N is the dimension of the first vector)

: 자연로그

: natural logarithm

: 특징 벡터의 차원

: Dimensions of feature vectors

: 상기 i번째 차원의 원소

: element of the i-th dimension

The controlling may control load distribution for at least one of the indexing and the ranking based on a weighted round robin method.

The image indexing method calculates a similarity between the indexed second feature vector and the feature vector for the user's query using TF-IDF (Term Frequency-Inverse Document Frequency), and based on the calculated similarity, The method may further include retrieving an image corresponding to the user's query.

According to the disclosed embodiments, image information is rearranged by integerizing the elements of the feature vector, and some elements of the feature vector are removed, thereby preserving important information contained in the image and removing less important information to be indexed and searched. can reduce the computational cost.

According to the disclosed embodiments, the similarity between a plurality of images may be measured by measuring the similarity of TF-IDF using feature vectors extracted from image information.

1 is a block diagram illustrating an image indexing apparatus according to an exemplary embodiment.
2 is an exemplary diagram of a pseudocode for explaining an indexing method according to an exemplary embodiment.
3 is an exemplary diagram of pseudocode for explaining a feature vector generation method according to an exemplary embodiment.
4 is an exemplary view of a graph illustrating a feature vector generation process according to an embodiment.
5 is a flowchart illustrating an image indexing method according to an exemplary embodiment.
6 is a block diagram illustrating a computing environment including a computing device according to an exemplary embodiment.

Hereinafter, specific embodiments will be described with reference to the drawings. The detailed descriptions that follow are provided to provide a comprehensive understanding of the methods, devices and/or systems described herein. However, this is only an example and disclosed embodiments are not limited thereto.

In describing the embodiments, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the disclosed embodiments, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the disclosed embodiments, which may vary according to intentions or customs of users or operators. Therefore, the definition should be made based on the contents throughout this specification. Terminology used in the detailed description is merely for describing the embodiments and should in no way be limiting. Unless expressly used otherwise, singular forms of expression include plural forms. In this description, expressions such as "comprising" or "comprising" are intended to indicate any characteristic, number, step, operation, element, portion or combination thereof, one or more other than those described. It should not be construed to exclude the existence or possibility of any other feature, number, step, operation, element, part or combination thereof.

1 is a block diagram illustrating an image indexing apparatus according to an exemplary embodiment.

The image indexing apparatus 100 according to an embodiment analyzes an image, preserves important information included in the image, and indexes the feature vector for image big data search using a feature vector from which unimportant information has been removed.

Specifically, according to an embodiment, the image indexing device 100 may be implemented on Apache Spark, for example, capable of analyzing an integrated engine for big data processing.

As shown, the image indexing apparatus 100 according to an embodiment includes an indexing unit 110, a query management unit 120, and a load balancing unit 130.

The indexing unit 110 obtains a feature batch for the input image, extracts a first feature vector and metadata from the obtained feature batch, connects some of the elements of the first feature vector, and then obtains a second feature batch. A feature vector is generated, and the second feature vector is indexed by packaging the second feature vector and metadata.

According to an embodiment, the indexing unit 110 may index a feature arrangement of an image.

Specifically, according to an embodiment, the indexing unit 110 may obtain image data through a service pool including various image analysis services running in the cloud.

At this time, the service pool may perform an image analysis task by obtaining image data from a source connected through the cloud, for example, IPCAM, a smartphone, a drone, and the like.

According to an embodiment, the indexing unit 110 may decode image data acquired from the service pool to obtain a feature arrangement expressed in the form of a property value pair through JSON (JavaScript Object Notation).

2 is an exemplary diagram of a pseudocode for explaining an indexing method according to an exemplary embodiment.

Specifically, FIG. 2 shows Algorithm 1 including a process of obtaining a feature arrangement and indexing performed by the indexing unit 130 .

Referring to FIG. 2, Algorithm 1 (200) includes input/output information and parameters 210 for performing Algorithm 1, an image data decoding function 220, an image data conversion function 230, and a mapping function 240 for parallel processing. ) and a partition function 250 for batch indexing.

First, the indexing unit 130 may decode image data through the image data decoding function 220 to extract feature vectors and meta data.

Thereafter, the indexing unit 130 may convert the image data into a basic unit provided by a big data processing service, for example, RDD (Resilient Distributed Dataset, a basic data structure used in Apache Spark) through the image data conversion function 230. .

Thereafter, the indexing unit 130 may package feature vectors and metadata in the converted RDDf through the mapping function 240 for parallel processing and map them to RDDp.

Thereafter, the indexing unit 130 may reduce network traffic by collectively indexing RDDs for each partition by mapping RDDp for each partition through the partition function 250 .

Meanwhile, according to an embodiment, the indexing unit 110 quantifies elements of the first feature vector based on a preset quantization factor, and satisfies a preset criterion among the quantified elements of the first feature vector. A second feature vector may be generated by concatenating the elements.

3 is an exemplary diagram of pseudocode for explaining a feature vector generation method according to an exemplary embodiment.

Specifically, FIG. 3 shows feature vector generation algorithm 2 performed by the indexing unit 130 .

Referring to FIG. 3 , Algorithm 2 (300) includes input/output information for performing Algorithm 2, an initialization function 310, a feature vector quantification function 320, and a feature vector generating function 330.

First, the indexing unit 130 may quantify elements of the first feature vector into integers based on a quantization factor preset through the feature vector quantization function 320 .

를 의미할 수 있다. According to an embodiment, the preset quantization factor (Q) is pfd included in the feature vector quantification function 320 of Algorithm 1.

can mean

By replacing the fixed constant value of the quantification factor with e ² and using the log of the order of the feature vector, it is possible to prevent image omission due to bin encoding caused by feature vector values close to '0'.

Thereafter, the indexing unit 130 removes some elements satisfying '0' among the elements of the first feature vector based on the elements of the first feature quantified through the feature vector generation function 330, and some elements are removed. A dimensionally reduced second feature vector may be generated by concatenating the first feature vectors.

In this case, the predetermined criterion may mean that an element of the feature vector is not '0'.

The feature vector generation function 330 includes a Soft(tfv) function for sorting the quantified first feature vector, a Prune(tsvp) function for removing elements having '0' elements of the quantified first feature vector, and converting a character string into Hex. It may include ToHex(v) that converts and ToSeries(hd, v), Concat(FV, ec) functions that connect the first feature vector from which some elements of which elements of the quantified first feature vector are '0' are excluded. there is.

4 is an exemplary view of a graph illustrating a feature vector generation process according to an embodiment.

In FIG. 4, for convenience of description, it is assumed that the first feature vector, f = {0.9, 0.4, 1.6, 0.8, 1.4, 0.3, 0.8, 0.5, 0.2, 1.5}.

4(a) is a graph showing the elements of the first feature vector.

4(b) is a graph showing the result of quantifying the elements of the first feature vector.

A quantification factor for quantifying an element of the first feature vector may be represented by Equation 1 below.

: 상기 제1 벡터의 i(이때, i는 1≤j≤N을 만족하는 정수, N은 상기 제1 벡터의 차원)번째 차원의 원소를 정량화하기 위한 정량화 요소

: Quantification element for quantifying the element of the i-th dimension of the first vector (where i is an integer satisfying 1≤j≤N, and N is the dimension of the first vector)

: 자연로그

: natural logarithm

: 특징 벡터의 차원

: Dimensions of feature vectors

: 상기 i번째 차원의 원소

: element of the i-th dimension

4(c) is a graph showing a result of sorting the elements of the quantified first feature vector.

4(d) is a graph showing the result of removing some elements having feature vector elements of '0' among the elements of the first feature vector based on the sorted feature vector values.

The query management unit 130 obtains a user's query and determines a rank for the obtained user's query based on the obtained user's query and the indexed second feature vector.

The load balancing unit 140 controls load distribution for indexing and ranking at each preset period.

According to an embodiment, the load balancer 140 may control load distribution for at least one of indexing and ranking based on a weighted round robin method.

In this case, the weighted round robin method means a method in which weights are set for each of the indexing unit and the query management unit, and priorities are given to places with high weights. Mainly, it can be applied when the traffic processing capability of the subject to be distributed is different.

Meanwhile, according to an embodiment, the image indexing apparatus 100 may further include an image search unit (not shown).

Specifically, according to an embodiment, the image search unit according to claim 1, calculates a similarity between the second feature vector and the feature vector for the user's query using TF-IDF (Term Frequency-Inverse Document Frequency) And, based on the calculated similarity, an image corresponding to the user's query may be searched.

At this time, the TF-IDF similarity may be calculated through Equation 2.

: TF-IDF 유사도

: TF-IDF similarity

: 인덱싱 된 제 2 특징 벡터

: indexed second feature vector

: 사용자의 쿼리에 대한 특징 벡터

: the feature vector for the user's query

5 is a flowchart illustrating an image indexing method according to an exemplary embodiment.

The method shown in FIG. 5 may be performed by, for example, the above-described image indexing apparatus 100 .

In step 510, the image indexing apparatus 100 acquires a feature batch for the input image, extracts a first feature vector and metadata from the acquired feature batch, and extracts some of the elements of the first feature vector. is connected to generate a second feature vector, and the second feature vector is indexed by packaging the second feature vector and the meta data.

In step 520, the video indexing apparatus 100 obtains the user's query and determines a rank for the obtained user's query based on the obtained user's query and the indexed second feature vector.

In step 530, the image indexing apparatus 100 controls load distribution for indexing and ranking determination at each preset period.

In the illustrated flowchart, the method is divided into a plurality of steps, but at least some of the steps are performed in reverse order, combined with other steps, performed together, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

6 is a block diagram illustrating a computing environment including a computing device according to an exemplary embodiment.

In the illustrated flowchart, the method is divided into a plurality of steps, but at least some of the steps are performed in reverse order, combined with other steps, performed together, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

6 is a block diagram illustrating and describing a computing environment 10 including a computing device according to an exemplary embodiment. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, computing device 12 may be image indexing device 100 .

Computing device 12 includes at least one processor 14 , a computer readable storage medium 16 and a communication bus 18 . Processor 14 may cause computing device 12 to operate according to the above-mentioned example embodiments. For example, processor 14 may execute one or more programs stored on computer readable storage medium 16 . The one or more programs may include one or more computer-executable instructions, which when executed by processor 14 are configured to cause computing device 12 to perform operations in accordance with an illustrative embodiment. It can be.

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. Program 20 stored on computer readable storage medium 16 includes a set of instructions executable by processor 14 . In one embodiment, computer readable storage medium 16 includes memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that can be accessed by computing device 12 and store desired information, or any suitable combination thereof.

Communications bus 18 interconnects various other components of computing device 12, including processor 14 and computer-readable storage medium 16.

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24 . An input/output interface 22 and a network communication interface 26 are connected to the communication bus 18 . Input/output device 24 may be coupled to other components of computing device 12 via input/output interface 22 . Exemplary input/output devices 24 include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices, and/or a photographing device. input devices, and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 24 may be included inside the computing device 12 as a component constituting the computing device 12, or may be connected to the computing device 12 as a separate device distinct from the computing device 12. may be

Meanwhile, embodiments of the present invention may include a program for performing the methods described in this specification on a computer, and a computer readable recording medium including the program. The computer readable recording medium may include program instructions, local data files, local data structures, etc. alone or in combination. The media may be specially designed and configured for the present invention, or may be commonly available in the field of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and specially configured to store and execute program instructions such as ROM, RAM, and flash memory. Hardware devices are included. Examples of the program may include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter.

Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications are possible to the above-described embodiments without departing from the scope of the present invention. . Therefore, the scope of the present invention should not be limited to the described embodiments and should not be defined, and should be defined by not only the claims to be described later, but also those equivalent to these claims.

10: Computing environment
12: computing device
14: Processor
16: computer readable storage medium
18: communication bus
20: program
22: I/O interface
24: I/O device
26: network communication interface
100: image indexing device
110: indexing unit
120: query management unit
130: load balancing unit

Claims (12)

A feature batch of an input image is acquired, a first feature vector and metadata are extracted from the acquired feature batch, and a second feature vector is generated by connecting some elements of the first feature vector. and an indexing unit for indexing the second feature vector by packaging the second feature vector and the meta data;
a query management unit that obtains a user's query and determines a rank for the obtained user's query based on the obtained user's query and the indexed second feature vector; and
And a load balancing unit for controlling load distribution for the indexing and the ranking for each predetermined period.
The method of claim 1,
The indexing unit,
After quantifying the elements of the first feature vector, some of the elements of the quantified first feature vector are connected to generate the second feature vector, an image indexing device
The method of claim 2,
The indexing unit,
quantifying the elements of the first feature vector based on a preset quantization factor, and generating the second feature vector by connecting elements satisfying a preset criterion among the elements of the quantified first feature vector; , image indexing device.
The method of claim 3,
The quantification factor satisfies the following equation, the image indexing device.
(mathematical expression)

: Quantification element for quantifying the element of the i-th dimension of the first feature vector (where i is an integer satisfying 1≤j≤N, and N is the dimension of the first vector)

: natural logarithm

: Dimensions of feature vectors

: element of the i-th dimension
The method of claim 1,
The load balancing unit controls load distribution for at least one of the indexing and the ranking based on a weighted round robin method.
The method of claim 1,
A similarity between the indexed second feature vector and the feature vector for the user's query is calculated using TF-IDF (Term Frequency-Inverse Document Frequency), and based on the calculated similarity, for the user's query An image indexing apparatus further comprising an image search unit for searching an image.
one or more processors; and
A method performed in a computing device having a memory storing one or more programs executed by the one or more processors,
A feature batch of an input image is acquired, a first feature vector and metadata are extracted from the acquired feature batch, and a second feature vector is generated by connecting some elements of the first feature vector. and indexing the second feature vector by packaging the second feature vector and the meta data;
obtaining a user's query and determining a rank for the obtained user's query based on the obtained user's query and the indexed second feature vector; and
And controlling the load distribution for the indexing and the ranking at every preset period.
The method of claim 7,
In the indexing step,
After quantifying the elements of the first feature vector, some of the elements of the quantified first feature vector are connected to generate a dimensionally reduced second feature vector.
The method of claim 8,
In the indexing step,
Elements of the first feature vector are quantified based on a preset quantization factor, and elements satisfying a preset criterion are connected among the quantified elements of the first feature vector to generate the second feature vector. , image indexing method.
The method of claim 9,
The quantification factor satisfies the following equation, image indexing method.
(mathematical expression)

: Quantification element for quantifying the element of the i-th dimension of the first feature vector (where i is an integer satisfying 1≤j≤N, and N is the dimension of the first vector)

: natural logarithm

: Dimensions of feature vectors

: element of the i-th dimension
The method of claim 7,
The controlling step controls load distribution for at least one of the indexing and the ranking based on a weighted round robin method.
The method of claim 7,
A similarity between the indexed second feature vector and the feature vector for the user's query is calculated using TF-IDF (Term Frequency-Inverse Document Frequency), and based on the calculated similarity, for the user's query An image indexing method, further comprising retrieving an image.

Images