KR20190069136A - Method and system for video retrieval based on image queries - Google Patents

Abstract

The present invention relates to a method for searching for an image based on an image query of a server which comprises: a step (a) of constructing an image feature information database using at least one image data stored in an image database; a step (b) of generating a feature information query using an image query when the image query is received; and a step (c) of searching for image feature information corresponding to the feature information query in the image feature information database. The step (a) includes: a preprocessing step of selecting a preset number of key frames among one or more key frames included in the image data; and a step of extracting a feature information query of input data using the key frames as the input data. The step (b) includes a step of extracting a feature information query of input data using the image query as the input data. The step of extracting the feature information query of the input data includes the steps of: extracting a square block from the input data; generating upper and lower threshold values using an average value of one or more pixel values constituting the block and a preset threshold value; generating upper and lower patterns using the difference between the pixel values of the block and the upper and lower threshold values; and converting the upper and lower patterns into a histogram to extract the feature information query of the input data.

Description

TECHNICAL FIELD [0001] The present invention relates to an image retrieval method and system based on image query,

The present invention relates to an image query-based image search method and system, and more particularly, to a method and system for searching similar images based on an image extracted from an image.

Augmented reality is a field of virtual reality. It is a computer graphics technique that combines virtual objects or information into actual environments and makes them look like objects existing in the existing environment. Recently, education, navigation, tourism, entertainment, electronic commerce, , And construction. Interest in e-learning, augmented reality-based museums and mobile applications, which are mobile augmented reality, is emerging. It is important for mobile augmented reality to recognize images in real time and search similar images. It can be searched based on a query of text, image or image recognized in the image database. However, feature extraction algorithms such as Scale Invariant Feature Transform (SIFT), Histogram of Oriented Gradient (HOG), Haar, Frens, Local Binary Pattern (LBP), and Modified Census Transform It is not suitable for real-time environment due to its complexity.

Conventional techniques for image retrieval include Korean Patent Registration No. 10-1713189, a HOG / HOF-based feature information extraction method and extractor for video object behavior classification. The above technique introduces the HOG algorithm, which includes a step of calculating the square root of the calculation process of the HOG algorithm and a step of calculating the normalization of the direction histogram, so that the calculation speed is not high.

Korean Registered Patent No. 10-1713189 (Feb. 28, 2017)

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to improve efficiency by reducing memory and calculation speed consumed in image processing using an upper frame extracted from an image.

Another object of the present invention is to shorten the time required for image processing by simplifying the operations used in the image processing and to secure a search accuracy higher than a certain level.

Another object of the present invention is to improve the efficiency by reducing the processing speed by retrieving similar images using a histogram generated by using a threshold value.

According to another aspect of the present invention, there is provided an image query-based image retrieval method for a server, the method comprising: a) constructing an image feature information database using at least one image data stored in an image database; A step of generating a feature information query using the feature information query, and a step c of retrieving image feature information corresponding to the feature information query from the image feature information database, wherein the step a) And a step of extracting a feature information query of input data by using the key frame as input data, wherein the step b) includes the steps of: Extracting a feature information query, Wherein the extracting of the feature information query of the input data comprises extracting a square block from the input data, generating upper and lower thresholds using an average value of the one or more pixel values constituting the block and a predetermined threshold value, Generating upper and lower patterns using the difference between the pixel values of the block and the upper and lower thresholds; converting the upper and lower patterns into histograms to extract feature information queries of the input data; The method comprising the steps of:

The preprocessing step may include converting one or more frames included in the image data to gray scale, and changing the size of the frame.

Further, the preprocessing step may include obtaining a histogram of each of the one or more frames, computing a histogram difference value of successive first frames and second frames, sorting the histogram difference values according to sizes, And setting a second frame corresponding to the selected difference value as a key frame.

The step of generating the upper and lower thresholds may further include calculating an average value of pixel values constituting the block, adding the threshold value to the average value to derive the upper threshold value, And deriving the lower threshold value by subtracting the lower threshold value.

The generating of the upper pattern may further include mapping 1 to each pixel if the pixel value of the block is greater than or equal to the upper threshold value, And maps 1 to each pixel if it is less than or equal to the lower threshold value, otherwise to 0.

The converting the upper and lower patterns into histograms may include generating the histogram by mapping the upper and lower patterns into n-bit values of binary numbers in a predetermined order and converting the n-bit values into decimal numbers. do.

In addition, the present invention provides an image query-based image search system, comprising: a generation unit that generates a feature information query using the image query upon receiving an image query; And a database unit including an image database for storing one or more image data and an image feature information database for storing image feature information.

The generating unit may include a preprocessor for selecting a preset number of key frames in one or more frames included in the image data, a database generating unit for constructing an image feature information database using one or more image data stored in the image database, And a feature information query extracting unit for extracting a feature information query of the input data by using the image query as the input data.

The feature information query extracting unit may further include a block extracting unit for extracting a square block from the input data, a threshold value generating unit for generating upper and lower threshold values using an average value of one or more pixel values constituting the block and a preset threshold value, A pattern generator for generating upper and lower patterns by using a difference between the pixel values of the block and the upper and lower thresholds, a histogram transforming unit for transforming the upper and lower patterns into a histogram, And a feature information query generating unit for extracting a feature information query.

According to the present invention as described above, the memory consumed for image processing and the calculation speed can be reduced by using the upper frame extracted from the image, thereby improving the efficiency.

In addition, the present invention simplifies the operations used in image processing, thereby shortening the time required for image processing and securing a certain level of search accuracy.

Also, the present invention can improve the efficiency by reducing the processing speed by searching the similar image with the histogram generated using the threshold value.

FIG. 1 is a diagram illustrating a configuration of an image retrieval system according to an embodiment of the present invention.
2 is a diagram illustrating a configuration of a feature information query extracting unit according to an embodiment of the present invention.
3 is a diagram for explaining an image search method according to an embodiment of the present invention.
4 is a diagram for explaining a database building method according to an embodiment of the present invention.
5 is a diagram for explaining a key frame extraction method according to an embodiment of the present invention.
6 is a diagram for explaining a feature information query generation method according to an embodiment of the present invention.
7 is a diagram illustrating an example of an image search method according to an embodiment of the present invention.

The above and other objects, features, and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings, which are not intended to limit the scope of the present invention. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

In the drawings, the same reference numerals are used to designate the same or similar components, and all combinations described in the specification and claims can be combined in any manner. It is to be understood that, unless the context requires otherwise, references to singular forms may include more than one, and references to singular forms may also include plural forms.

The terminology used herein is for the purpose of describing particular illustrative embodiments only and is not intended to be limiting. Singular representations as used herein may also be intended to include a plurality of meanings, unless the context clearly dictates otherwise. The terms "and / or" " and / or "include any and all combinations of the items listed therein. The terms "comprises," "comprising," "including," "having," "having," "having," and the like have the implicit significance, Steps, operations, elements, and / or components, and does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / Steps, processes, and operations of the methods described herein should not be construed as necessarily enforcing their performance in such specific order as discussed or illustrated unless specifically concluded the order of their performance . It should also be understood that additional or alternative steps may be used.

In addition, each of the components may be implemented as a hardware processor, the components may be integrated into one hardware processor, or a combination of the components may be implemented as a plurality of hardware processors.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating a configuration of an image retrieval system according to an embodiment of the present invention. Referring to FIG. 1, an image search system according to an embodiment of the present invention may include a generating unit 110, a searching unit 170, and a database unit 190.

The generating unit 110 may generate a feature information query using the received image query. More specifically, the generating unit 110 may include a preprocessing unit 120, a database generating unit 140, and a feature information query extracting unit 160.

The preprocessing unit 120 may select a preset number of key frames in one or more frames included in the image data. The preprocessing unit 120 may convert one or more frames included in the image data into grayscale and change the size of the frame. For example, the size of the modified frame may be 320 x 240, or may have other sizes.

The preprocessing unit 120 obtains a histogram of each of one or more frames, calculates histogram difference values of successive first and second frames, arranges the histogram difference values according to their sizes, And a second frame corresponding to the selected difference value can be set as a key frame. The key frame is a frame representing the image data, and has the effect of reducing memory and complexity required for image data processing. For example, assuming that the value of n is 4 and that the histogram values of the 10 frames constituting the image data are -10, 20, 90, 70, 30, 50, -30, -25, The histogram difference value of the frame may have a value of 30, 70, 20, 40, 20, 80, 5, 35, In this case, since the upper four difference values are 80, 70, 40, and 35, 3, 5, 7, and 9 frames can be set as key frames.

The database generation unit 140 may construct an image feature information database using one or more image data stored in the image database. The present invention can retrieve image feature information similar to the feature information query of the received image query based on the constructed image feature information database.

The feature information query extracting unit 160 may extract the feature information query of the input data using the key frame and the image query as input data. 2, the feature information query extraction unit 160 includes a block extraction unit 162, a threshold value generation unit 164, a pattern generation unit 166, and a feature information query generation unit 168 can do.

The block extracting unit 162 can extract a square block from the input data. The size of the square block may be 3 x 3, but may be more or less.

The threshold value generator 164 may generate the upper and lower threshold values using the average value of the one or more pixel values constituting the block and the predetermined threshold value. More specifically, the threshold value generating unit 164 calculates an average value of pixel values constituting the square block extracted by the block extracting unit 162, adds a preset threshold value to the calculated average value, and derives an upper threshold value , The lower threshold value can be derived by subtracting the predetermined threshold value from the average value. (0, 0) = 10, (0, 1) = 25, (0, 2) = 85, (1, 0) = 75 (2,1) = 35, (1,1) = 30, (1,2) = 20, (2,0) = 80, The average value of pixel values constituting the square block can be calculated as 40.6 which is a value of (10 + 25 + 85 + 75 + 30 + 20 + 80 + 5 + 35) / 9. Assuming that the predetermined threshold value is 10, the threshold value generator 164 may derive the upper threshold value as 40.6 + 10 = 50.6 and the lower threshold value as 40.6-10 = 30.6.

The pattern generation unit 166 may generate the upper and lower patterns using the difference between the pixel values of the block and the upper and lower threshold values. More specifically, the pattern generation unit 166 may generate an upper pattern by mapping 1 to each pixel if the pixel value of the block is greater than or equal to the upper threshold value, or to 0 otherwise. 7, when the upper threshold value is 50.6 and the pixel value at the (1,1) position of the square block is 30, since the pixel value is smaller than the upper threshold value, The value of the pixel may have a value of zero. The pattern generating unit 166 may generate an upper pattern by comparing all the pixel values of the square block with the upper threshold value.

In addition, the pattern generator 166 may generate a lower pattern by mapping 1 to 1 if the pixel value of the block is less than or equal to the lower threshold value, and 0 otherwise to each pixel. 7, when the lower threshold value is 30.6 and the pixel value at the (1,1) position of the square block is 30, since the pixel value is smaller than the lower threshold value, The value of the pixel may have a value of one. The pattern generation unit 166 may generate a lower pattern by comparing all the pixel values of the square block with the lower threshold value.

The feature information query generation unit 168 may convert the upper and lower patterns into a histogram, and extract a feature information query of the input data. More specifically, since the upper and lower patterns are composed of values of 0 and 1, the feature information query generation unit 168 associates the upper and lower patterns with n-bit values of binary numbers in a predetermined order, To a decimal number to generate a histogram. Referring to the example of Fig. 7, the order of the upper and lower patterns is (0,0), (0,1), (0,2) , (1,1) can be set to 5, (1,2) to 6, (2,0) to 7, (2,1) to 8, and (2,2) to 9. (1, 0) = 0, (0, 1) = 0, (0, 2) = 1, (2,2) = 0, (2,0) = 1, (2,1) = 0 and (2,2) = 0, the feature information query generator 168 converts the pixel value of the upper pattern into a 9- . The corresponding binary value may be 001100100.

When the feature information query generator 168 generates a 9-bit value for the pixel value of the upper pattern, the generated 9-bit value can be converted into a decimal value. 7, when the 9-bit value of the pixel value of the upper pattern is 001100100, the decimal value of 001100100 may be 76. [ At this time, the feature information query generator 168 may set the histogram value of the upper pattern to 76. [

The process of calculating the histogram value of the upper pattern by the feature information query generating unit 168 may be applied to the process of calculating the histogram value of the lower pattern.

The formulas of the operations performed by the feature information query extracting unit 160 are expressed by Equations 1 to 3. THOG in Equation 1 is a formula for calculating a histogram of a block, p represents a pixel, and HOG represents each pixel value of a block.

The search unit 170 can search the image feature information database for image data similar to the image feature information corresponding to the feature information query. Since all of the image data must be scanned to retrieve similar image data, a hashing algorithm can be used to speed up the computation. There are two types of hashing algorithms used to search similar image data: Single Bit Quantization (SBQ) and Double Bit Quantization (DBQ). Since SBQ has most of the threshold values in the high density region, many bits close to the threshold are hashed into completely different bits DBQ is used in the present invention. In addition, DBQ has a disadvantage that it has a faster execution speed and fewer processing times than SBQ. DBQ can generate code through double bits by setting two thresholds. The searching unit 170 may search similar image data using a code generated by applying an upper threshold value and a lower threshold value to DBQ.

The search unit 170 may specify a ranking of image data similar to the retrieved image query. At this time, the present invention can use the cosine similarity to set the ranking of similar image data. The cosine similarity is calculated by calculating the angle between each key frame of the candidate image data Dd and the upper and lower histogram values corresponding to the received image query Dq, and more specifically,

The database unit 190 may include an image database for storing one or more image data and an image feature information database for storing image feature information.

Hereinafter, an image search method according to an embodiment of the present invention will be described with reference to FIGS. 3 to 6. FIG. In the description of the image retrieval method, detailed embodiments overlapping with the above-described image retrieval system may be omitted.

3 is a diagram for explaining an image search method according to an embodiment of the present invention. Referring to FIG. 3, the server can construct a database (S100). The process of building the database is described in detail in the description of FIG. 4 below.

When the server establishes the database, it can receive the image query and generate the feature information query of the image query (S500). The process of generating the feature information query will be described in detail in the description of FIG. 6 below.

When the feature information query of the image query received by the server is generated, a feature information query similar to the feature information database stored in the feature information query of the image data can be retrieved using the generated feature information query (S900).

4 is a diagram for explaining a database building method according to an embodiment of the present invention. Referring to FIG. 4, the server may extract one or more frames from the image data (S110).

The server may convert the extracted frame into grayscale and change the size of the frame (S120).

When the size of the frame is changed, the server may extract the key frame in one or more frames (S130). The process of extracting key frames will be described in detail in the description of FIG. 5 below.

When the server extracts the key frame, the feature information query of the key frame can be extracted (S150).

The server may store the feature information query of the key frame in the feature information database (S160). Through the generated feature information database, the server can retrieve image data including feature information similar to the feature information query for the received image query.

5 is a diagram for explaining a key frame extraction method according to an embodiment of the present invention. Referring to FIG. 5, the server may obtain a histogram of a frame extracted from the image data (S151).

The server can calculate the difference value between the histograms of the first and second consecutive frames of one or more frames (S153).

If the server computes all of the difference values, it is possible to extract the difference values of the n highest one of the calculated difference values (S155).

When the difference value of the top n is extracted, the server can set the second frame corresponding to the extracted difference value as a key frame (S157).

6 is a diagram for explaining a feature information query generation method according to an embodiment of the present invention. Referring to FIG. 6, the server extracts a square block from the extracted frame (S510). The size of the square block may be 3 x 3, but it may have other sizes.

The server may generate the upper and lower threshold values by calculating an average value of pixel values constituting the extracted block (S520). More specifically, the server may generate an upper threshold value by adding a predetermined threshold value to the calculated average value, and generate a lower threshold value by subtracting the predetermined threshold value from the average value.

The server may generate the upper and lower patterns by comparing the generated upper and lower threshold values with the values of the pixels constituting the block (S530).

The server may extract the feature information query by associating the generated upper and lower patterns with 9-bit binary values (S540).

According to an embodiment of the present invention, it is possible to measure the time and accuracy required for searching similar image data. The image data used at this time may include, but is not limited to, UCSB data and UCF-50 data. The UCSB data may include 6889 images having 640 x 480 resolution, 6 different plane textures, and a large motion pattern, and the UCF-50 data may be composed of 6681 images having 320 x 240 resolution.

The time required to retrieve similar image data for UCSB data and UCF-50 data is shown in Table 1, and the accuracy is shown in Table 2.

algorithm Feature information extraction time Search Time HOG 60.71 1820.7 LBP 53.39 1569.8 SIFT 3420.2 2376.2 STPM 5167.8 2837.4 LAID 59.35 1765.2 THOG 56.63 1692.5

algorithm UCSB data accuracy UCF-50 Data Accuracy HOG 79.3% 85.6% LBP 81.8% 87.3% SIFT 87.5% 94.7% STPM 89.1% 95.3% LAID 83.1% 89.6% THOG 85.2% 93.8%

Table 1 and Table 2 show the results of retrieving similar image data by extracting randomly selected data from UCSB data and UCF-50 data and using HOG, LBP, SIFT, STPM, LAID algorithm as well as THOG algorithm of the present invention And the time and accuracy required when it is done.

Referring to Tables 1 and 2, the SIFT and STPM algorithms are highly accurate, but their efficiency is poor for computation speed. However, the THOG algorithm of the present invention is competitive with conventional algorithms because it has competitive accuracy and efficient computation speed.

The embodiments of the present invention disclosed in the present specification and drawings are merely illustrative of specific embodiments of the present invention and are not intended to limit the scope of the present invention in order to facilitate understanding of the present invention. It is to be understood by those skilled in the art that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein.

Claims (9)

1. An image query-based image retrieval method for a server,
A) constructing an image feature information database using one or more image data stored in an image database;
B) upon receipt of the image query, generating a feature information query using the image query;
And searching for the image feature information corresponding to the feature information query in the image feature information database,
In the step a,
A preprocessing step of selecting a preset number of key frames in one or more frames included in the image data;
Extracting a feature information query of input data using the key frame as input data,
Wherein the step (b) includes extracting a feature information query of input data using the image query as input data,
The step of extracting the feature information query of the input data
Extracting a square block from the input data;
Generating upper and lower thresholds using an average value and a predetermined threshold value of one or more pixel values constituting the block;
Generating upper and lower patterns using the difference between the pixel values of the block and the upper and lower thresholds;
And converting the upper and lower patterns into a histogram to extract a feature information query of the input data.
The method according to claim 1,
The pre-
Converting one or more frames included in the image data to gray scale;
And changing a size of the frame.
The method according to claim 1,
The pre-
Obtaining a histogram of each of the one or more frames;
Calculating histogram difference values of successive first and second frames;
Arranging the histogram difference values according to their sizes, and selecting predetermined difference values of n;
And setting a second frame corresponding to the selected difference value as a key frame.
The method according to claim 1,
Wherein generating the upper and lower thresholds comprises:
Calculating an average value of pixel values constituting the block;
Adding the threshold value to the average value to derive the upper threshold value, and subtracting the threshold value from the average value to derive the lower threshold value.
The method according to claim 1,
The step of generating the top pattern
Mapping 1 to each pixel if the pixel value of the block is greater than or equal to the upper threshold value,
The step of generating the lower pattern
Wherein if the pixel value of the block is less than or equal to the lower threshold value, 1 is mapped to 0 otherwise.
The method according to claim 1,
The step of converting the upper and lower patterns into a histogram
Wherein the histogram is generated by mapping the upper and lower patterns into n-bit values of binary numbers in a predetermined order, and converting the n-bit values into decimal numbers.
1. An image query based image retrieval system,
A generating unit, upon receipt of the image query, for generating a feature information query using the image query;
A searching unit for searching the image feature information database for image feature information corresponding to the feature information query;
An image database for storing at least one image data; and a database including an image feature information database for storing image feature information.
8. The method of claim 7,
The generating unit
A preprocessing unit for selecting a preset number of key frames in one or more frames included in the image data;
A database generation unit for constructing an image feature information database using at least one image data stored in an image database;
And a feature information query extracting unit for extracting a feature information query of input data using the key frame and the image query as input data.
9. The method of claim 8,
The feature information query extraction unit
A block extracting unit for extracting a square block from the input data;
A threshold value generation unit for generating upper and lower threshold values by using an average value and a predetermined threshold value of one or more pixel values constituting the block;
A pattern generating unit for generating upper and lower patterns using the difference between the pixel values of the block and the upper and lower thresholds;
And a feature information query generating unit for converting the upper and lower patterns into a histogram and extracting a feature information query of the input data.

Images