KR100430263B1 - Multimedia Query System Using Progressive Color Histogram - Google Patents

Abstract

PURPOSE: A color histogram capable of comparing and searching a partial portion gradually and a method for searching multimedia using the same are provided to perform a high quality search performance although a query image and an object image are compared and searched using only a partial portion of a color histogram. CONSTITUTION: It is assumed that a histogram corresponded to a query image is stopped and received up to the "P1" portion before the total is not reached during a transmission in an image search using a query image at a network environment. A client calculates a similarity rate using only bins located at the same position to the histogram of an objected image using a histogram up to the received "P1" portion. The bins of the created histogram may be constructed in the order of importance. A sample group constituted by various data is constructed. A color histogram of data corresponded to the constructed sample group is created. A variance is calculated according to each bin of the created color histograms, and the histogram is re-ordered in the order of the value of variance.

Description

Multimedia query method using color histogram with incremental partial comparison search {Multimedia Query System Using Progressive Color Histogram}

The present invention relates to a multimedia retrieval using a color histogram, which allows the color histogram to have relatively important information at the beginning when viewed in a time series of access sequences, thereby ensuring high retrieval performance even when only the front of the color histogram is used. It relates to a color histogram information structure and a multimedia retrieval method using the color histogram.

In particular, the present invention has a color histogram structure in which bins of the color histogram are reconstructed in order of significant bins so that the bins that are important in terms of the time series access order of the histogram are relatively leading. The present invention relates to a color histogram and a multimedia retrieval method characterized by enabling retrieval using only a gradual portion from the front of a stream.

In addition, the present invention provides a high histogram performance even if only a portion of the color histogram using the color histogram feature information including the color histogram containing the important bin order information of the color histogram, the information representing the color histogram A color histogram and a multimedia retrieval method.

Recently, as the technology of searching for multimedia based on the contents has emerged, researches on multimedia features that influence search performance have been actively conducted. Currently, the most popular search engines use global and local color information and texture information for image retrieval. Among them, color information is known as the most important factor in image retrieval. Therefore, more effective color features have been developed, and attempts have been made to develop more efficient color spaces.

The color histogram is the most widely used color information. The color histogram is information representing a color distribution of multimedia data such as an image, and the number of bins of the histogram is determined by how the color space is quantized. In general, when quantization is performed in more detail and expressed in many color bins, the search performance is increased, but the histogram size is increased, so it takes a lot of search time, and especially when searching through a network, the histogram size may be an important factor depending on the network capacity. .

In addition, when a histogram of data to be searched is transmitted due to a network problem and is interrupted at a specific part, the search performance is deteriorated when the search is performed using only a part of the transmitted histogram.

And not only the network environment but also the system performance of the client, the purpose of the search, etc., it is necessary to perform a rough search without the need for all the color histograms, but conventionally, the entire color histograms for the query image and the search target image are compared and searched. It did not meet this demand.

In addition, when the system performance is low, the comparative search of all the color histograms slows down the search speed, and when the general search alone is satisfactory, an unnecessary search time is required.

The present invention solves the problems described above, and its object is to enable high search performance even when comparing and searching a query image and a target image using only a part of a color histogram.

In order to achieve the above object, the present invention has a color histogram structure in which bins of the color histogram are reconstructed in the order of important bins. We propose a color histogram and a multimedia retrieval method using the reconstructed color histogram to enable multimedia retrieval that guarantees high retrieval performance by only incremental partial retrieval from the front of the histogram bit stream.

In addition, the present invention provides a color histogram and a multimedia retrieval method in which a color histogram includes important bin order information, and has a high retrieval performance even when only a part of the color histogram is used by using color histogram feature information including the bin order information. To present.

1 is a block diagram of a color histogram bit stream for explaining the present invention.

2 is a flowchart for explaining a first embodiment of a color histogram reconstruction method according to the present invention;

3 is a flowchart for explaining a second embodiment of a color histogram reconstruction method according to the present invention;

4 is a diagram showing an HMMD color space as an example of a color space (color space) to which the present invention is applied;

Fig. 5 is a diagram for explaining a 184 level quantization method viewed in the HMMD cross section.

Fig. 6 is a diagram showing an example of reordering information of a color histogram composed of 184 bins in the present invention.

7 shows an example of performance comparison when performing a partial comparison search using the present invention.

The present invention enables a high search performance even with comparative searches with only a portion of the histogram. This feature is very useful for two purposes:

In particular, when transmitting the histogram of the data as described above, if the histogram bit stream transmission is interrupted due to factors such as the network environment in the middle of the transmission, even if only the partial color histogram information already received can be searched for high search performance. For this purpose, high search performance must be ensured even if the histogram bins are searched using the transfer order, that is, only N (N <total histogram bins) bins from the beginning.

The second purpose is to determine whether to transfer part or all of the histogram according to the purpose of the search, the client's environment, and the network environment on the server side that has the data, so that the optimal search can be performed in each case.

For example, if the purpose of the search is to search for roughly the same kind of data, or if the network environment is not very good, the histogram of the data to be searched is transmitted instead of the whole, so that the network environment can be minimized while ensuring minimal search performance. You can perform a search. Conversely, if the network or computer environment is very good and you want a very accurate search, you can send the entire histogram.

In order to provide the above useful functions, it is necessary to optimize the histogram so that even if each bean constituting the histogram is searched with N bins from the front, the performance of the search is reduced. To do this, the histogram needs to be reconstructed into a structure that places the histogram in front of the bit stream, as the more important bins, that is, the bins that affect search more. As such, the histogram can be optimized for the purposes of the present invention through the reordering of the bins.

In the present invention, there are two methods for determining the important bins.

The first method is to statistically calculate how big each bin distinguishes the data from the sample group of data, and consider it as the more important bin when the bin is larger.

Determining the discriminating power calculates the variance of values for each bin in the sample population, so that the larger the variance, the more important the bin. If the data contained in the sample population are already grouped with similar data, the smaller the variance within the same group and the larger the variance between the data in different groups, the more important the bin.

Theoretically, the latter method can extract the importance of better accurate bins, but the former method can be used because it is not easy to group the sample in advance, and the experiment shows that this method also shows very good performance. Verified.

The second method considers the importance of the bin by considering the characteristics of the color space (color space) without a sample population. In general, when M is quantized in the color space, the importance of each quantized color according to chromaticity, purity, and brightness can be roughly known through prior knowledge.

In general, areas of low purity have a greater impact on search than areas of high purity, and areas of red have a higher effect than areas of blue. Therefore, if the color space is divided based on purity, the divided regions having low purity may be more important than the divided regions.

Therefore, the more bins belonging to the low purity region, the more important it is. When reordering the bins of the histogram, the bins corresponding to the regions belonging to the low purity region are placed more forward.

For example, suppose there are two areas S1 and S2 according to purity. Assuming that S1 is a lower purity area than S2 and that the ratio of importance is 2: 1, S2 is selected when two are selected when S1 is selected in order from the partition to be located forward to determine the bin order of the histogram. You can determine the empty order by selecting only one.

Based on the above-described concept, the features of the present invention will be described.

A first feature of the invention is; In the multimedia search using the color histogram, a progressive color histogram in which the histogram is reconstructed in the order of important bins so that the search can be performed using only the progressive part that comes first when the histogram is accessed (including transmission). It is a progressive multimedia retrieval method that searches using only a part of a histogram according to a search purpose and environment.

A second feature of the invention is; Reconstructing the histogram in the order of important bins described in the first feature is calculated based on the bin value variance for each bin calculated from the sample population, but the larger the variance is, the more important the bin is characterized.

A third feature of the invention is; Reconstructing the histogram in the order of the important bins described in the first feature above calculates the variance in the group for each bin and the variance between the different groups from a sample group in which the grouping of images similar to each other is made in advance. The larger the variance is, the more important the bin is considered.

A fourth feature of the invention is; Reconstructing the histogram in the order of the important bins described in the first feature comprises sampling bins representing the divided regions divided by color space quantization, and selecting the higher region probability for the divided regions. In other words, the partitions corresponding to the lower purity are reconfigured to appear more in front of the histogram.

A fifth feature of the present invention is; The color histogram described in the fourth feature uses an HMMD color space consisting of Hue, SUM (MAX (RGB) + MIN (RGB)), DIFF (MAX (RGB) -MIN (RGB)), and the HMMD color space is When having a DIFF value range of 0 to 255, the DIFF values designated for dividing based on the DIFF axis are 9, 29, 75, and 200, respectively, and each of the four divided regions divided by these values is S1 and S2. , S3, S4, S5, S1 is divided into 8 parts based on SUM, S2 is divided into 4 parts based on SUM, 8 parts based on Hue, S3 is divided into 4 parts based on SUM, 12 based on Hue S4 is divided into 4 equal parts based on SUM, 12 equal parts based on Hue, and S5 is divided into 2 equal parts based on SUM and 24 equal parts based on Hue. It is characterized by.

A sixth feature of the invention is; In order to change the sampling probability described in the fourth feature, the sampling probability in each of the areas S1, S2, S3, S4, and S5 described in the fifth feature is set to 24: 12: 6: 2: 1. do.

A seventh aspect of the invention is; In the multimedia retrieval system using the progressive color histogram described in the first feature, the progressive multimedia transmission or retrieval to the client using all or part of the histogram from the front according to the retrieval purpose and the hardware environment of the client to be retrieved Search method.

An eighth aspect of the invention is; In the multimedia retrieval system using the progressive color histogram described in the first aspect, when the histogram data of the query data is interrupted due to an unexpected result during transmission, the multimedia retrieval system performs retrieval using only the received histogram data.

The invention also features a ninth aspect; In the multimedia search using the color histogram, the search engine includes important empty order information and uses the empty order information so that the search can be performed using only the progressive part that comes first when the histogram is transmitted (bit stream). This is a multimedia retrieval method using a progressive color histogram that uses only a few bins of the color histogram to search for the bins in order of importance.

A tenth feature of the present invention is; What constitutes the important bin order information described in the ninth feature is calculated based on the bin value variance for each bin calculated from the sample population, but the larger the variance is, the more important the bin is considered.

An eleventh aspect of the present invention is; The variance within the group is small and the variance between the groups is calculated by calculating the variance in the group for each bin and the variance between different groups from the sample group in which the grouping of the images similar to the important bin ordering information described in the ninth feature is made in advance The larger is characterized as considered to be an important bin.

A twelfth feature of the invention is; The important bin order information described in the ninth feature is configured by sampling bins representing the divided regions divided by the color space quantization, but the lower regions are selected with a higher probability so that the lower regions are selected with higher probability. The partition area corresponding to purity is characterized in that it is reconfigured to appear more in front of the histogram.

A thirteenth feature of the invention is; The color histogram described in the twelfth feature uses an HMMD color space consisting of Hue, SUM (MAX (RGB) + MIN (RGB)), and DIFF (MAX (RGB) -MIN (RGB)). When the DIFF value ranges from 0 to 255, the DIFF values specified for dividing based on the DIFF axis are 9, 29, 75, and 200, respectively, and the four divided areas based on these values are respectively S1 and S2. , S3, S4, S5, S1 is divided into 8 parts based on SUM, S2 is divided into 4 parts based on SUM, 8 parts based on Hue, S3 is divided into 4 parts based on SUM, 12 based on Hue S4 is divided into 4 equal parts based on SUM, 12 equal parts based on Hue, and S5 is divided into 2 equal parts based on SUM and 24 equal parts based on Hue. It is characterized by.

A fourteenth feature of the invention is; In order to change the sampling probability described in the twelfth feature, the sampling probability in each of the areas S1, S2, S3, S4, and S5 described in the twelfth feature is set to 24: 12: 6: 2: 1. do.

A fifteenth feature of the invention is; In the multimedia retrieval system using the progressive color histogram described in the ninth aspect, the progressive multimedia transmission or retrieval of the histogram to the client using all or part of the histogram according to the retrieval purpose and the hardware environment of the client to be retrieved. Search method.

A sixteenth aspect of the invention is; In the multimedia retrieval system using the progressive color histogram described in the ninth aspect, the query histogram and the search target histogram are reconstructed according to the bin order information, and then the color histograms reconstructed according to the bin order information are compared with each other. It is characterized by searching.

A seventeenth aspect of the present invention is provided; In the multimedia retrieval system using the progressive color histogram described in the ninth aspect, the query histogram and the search target histogram are taken out one by one according to the bin order information, and the color histograms are compared and searched in the empty order.

Figure 1 shows an application of the present invention as an example.

In image retrieval using a query image in a network environment, suppose that the histogram corresponding to the query image was interrupted before the whole arrived during transmission and received only up to P1 as shown in the figure. The client calculates the similarity using only the bins in the same location for the histogram of the target image using the histogram up to the received P1. As such, even if only a part of the histogram is used, more efficient search is possible by guaranteeing a certain search performance.

FIG. 2 shows a sequence of reconstructing bins of the histogram generated in a general order in order of importance so that search performance is not significantly reduced even if only a part of the histogram is used.

First, a sample population composed of various data is composed, and a color histogram of data corresponding to the configured sample population is generated. Then, the variance is obtained for each bin of the generated color histograms, and the histogram is reordered in order of variance.

Figure 3 shows another example of reordering the bins of the histogram generated in the order of importance in order of importance.

First, a sample group composed of various data is composed, and information about similar data is known in advance and grouped using it. After generating the color histogram of the data corresponding to the configured sample group, the histograms of the variances within the group and the histograms of the data belonging to the different groups obtained the variance only for the histograms of the data belonging to the same group for each bin of the generated color histograms Find the variance between the groups where variance is obtained only for. At this time, the histogram is reordered in order of small variance in groups and large variance between groups.

4 illustrates an HMMD color space to explain a gradual color histogram using the HMMD color space as an example of realizing the above-described invention. The HMMD color space is double cone shaped and the center axis is represented by SUM (MAX (RGB) + MIN (RGB)), which corresponds to brightness. Purity increases from the center toward the edge of the cone, expressed as DIFF (MAX (RGB)-MIN (RGB)). The angle of the cone represents the color, which is represented by the usual Hue.

FIG. 5 shows an example in which the HMMD color space described with reference to FIG. 4 is quantized to 184 levels.

As can be seen from the figure, first, divided into five partial regions on the basis of DIFF and subdivided them based on the HUE and SUM, and all of them were represented by 184 divided regions. They generate a color histogram of 184 bins each.

FIG. 6 shows a bin at a probability ratio of 24: 12: 6: 2: 1 in each divided region when divided into five partial regions based on DIFF in the 184 level color histogram described in FIG. Represents generated reordering information when ordered.

Sampling a bin means selecting one of the 184 bins one after the other with a certain probability and reordering the color histogram in the selected order.

Therefore, the first bin selected by sampling should be the most important bin.

Therefore, the higher the probability, the higher the probability that it will be sampled first. For example, the two lowest purity regions are composed of 8 and 32 subdivided regions, respectively, in the example of FIG. 5. In the implementation of the present invention, since the probability ratio is 24:12, if one is sampled every two in the first region, In the next area, one is sampled every four, four in the first and eight in the next.

Figure 7 shows the performance change when only 16, 32, 64 and 128 bins are used when constructing a progressive color histogram by applying the present invention.

In this example, the bin values are not represented as a general decimal number when each N bins are used. In this example, the bin values are quantized to n bits for space efficiency.

For example, an empty value represented by 1 bit means that a decimal value between 0 and 1 is expressed as 0 or 1 based on a specific threshold.

Therefore, when represented by 1 bit and composed of 16 bins, the total number of bits of the histogram information used for the search becomes 16 bits.

In this experiment, the performance was obtained by expressing the bin values with 1 bit, 2 bits, and 4 bits for each N bins.

In the table of FIG. 7, the first column represents the number of bins selected, and the third column represents n bits of the bin values, indicating how many bits of histogram information are used (meaning that the bins are also quantized), and the middle column represents performance. Express. The same number of bins are represented by 4 bits, 2 bits, and 1 bit, respectively.

For example, if the number of bits 184 is represented by 4 bits (total number of bits 736), 2 bits (total number of bits 368), and 1 bit (total number of bits 184), respectively. Lower values indicate better cases.

As can be seen from the figure, the bin value represents high performance even when expressed in only 4 bits, and the smaller the number of bins used for searching in the full color histogram, the lower the performance but the smaller the decrease.

According to the present invention, in a multimedia search using a color histogram, even if only a part of the histogram is searched and compared, there is an effect of enabling a high search performance.

In particular, when transmitting the histogram of data, high search performance can be ensured even if only the received part is searched if the transmission is interrupted.

In addition, the present invention has the advantage that the optimal search can be performed in each case by determining whether to transfer part or all of the histogram according to the purpose of the search, the environment of the client, and the environment of the network. .

Claims (20)

Generating a color histogram for multimedia retrieval using a color histogram, and reconstructing the color histogram so that the more important bins are at the head of the access based on time-series access procedures. How to create a histogram. In a multimedia search using a color histogram, reconstructing the color histogram so that the more important bins are accessed at the time of accessing the histogram data for retrieval, and accessing the color histograms reconstructed in order of importance, and having higher priority bins. And performing a comparative search of the query color histogram and the search target color histogram according to the order. 3. The method of claim 2, wherein when the color histograms reconstructed in the order of importance are accessed, a comparative portion of the query color histogram and the target color histogram as information having a meaning sufficient to represent all of the corresponding color histograms is obtained. Multimedia search method, characterized in that made. 3. The multimedia retrieval method according to claim 2, wherein the reconstruction of the histogram in the order of important bins is calculated based on the bin value variance for each bin calculated from a sample population, but the larger the variance is regarded as an important bin. 3. The method according to claim 2, wherein the reconstructing the histogram in the order of the important bins calculates the variance in the group for each bin and the variance between the different groups from a sample group in which the grouping of similar images is made in advance. A multimedia retrieval method, characterized in that the larger the variance, the more important the bin. 3. The method of claim 2, wherein the reconstructing the histogram in the order of the important bins comprises: sampling bins representing the divided regions divided by color space quantization, and selecting a higher probability of the divided regions having lower purity. The retrieval method of claim 1, wherein a partition corresponding to a low purity is reconfigured to appear more in front of the histogram. 7. The color histogram described above uses a HMMD color space consisting of Hue, SUM (MAX (RGB) + MIN (RGB)), DIFF (MAX (RGB) -MIN (RGB)), and HMMD color. When the space has a DIFF value range of 0 to 255, the DIFF values specified for dividing by the DIFF axis are 9, 29, 75, and 200, respectively, and each of the four divided regions divided by these values is S1. , S2, S3, S4, S5, S1 is divided into 8 parts based on SUM, S2 is divided into 4 parts based on SUM, 8 parts based on Hue, S3 is divided into 4 parts based on SUM, and Hue is based on S4 is divided into 4 parts by SUM, 12 parts by Hue, and S5 is divided into 2 parts by SUM and 24 parts by Hue by using color quantization method. Multimedia search method characterized in that the configuration. 8. The multimedia retrieval method according to claim 7, wherein the sampling probability is set to 24: 12: 6: 2: 1 in each of the regions S1, S2, S3, S4, and S5 in order to change the above described sampling probability. . The multimedia retrieval system using the above-described progressive color histogram, wherein the histogram is transmitted or retrieved to the client using all or a part of the histogram according to the retrieval purpose and the hardware environment of the client to be retrieved. Progressive multimedia search method. The multimedia retrieval system using the above-described progressive color histogram, wherein when the histogram data of the query data is interrupted due to an unexpected result during transmission, a search is performed using only the received histogram data. Multimedia search method. Generating a color histogram for multimedia retrieval using a color histogram, and generating bin order information that determines the order for reconstructing the color histogram so that the more important bins are at the head of access based on time-series access procedures. Color histogram generation method comprising the steps of. In a multimedia search using a color histogram, generating bin order information for determining an access order such that bins that are important in accessing histogram data for searching are relatively first in access order, and using the bin order information Searching using only a portion of bins in the histogram, but selecting the bins in the order of important bins to perform a comparative search, characterized in that the multimedia search method using a progressive color histogram. The multimedia retrieval method according to claim 12, wherein the constituting important bin ordering information is calculated based on the variance of bin values for each bin calculated from a sample population, but the larger variance is regarded as an important bin. The variance in a group according to claim 12, wherein the constituents of the above-described bin information are calculated by calculating the variance in a group for each bin and the variance between different groups from a sample group in which grouping of similar images is made in advance. Multimedia retrieval method characterized in that the greater the distribution between the more important bins. The method according to claim 12, wherein the constituting the important bin order information described above is configured by sampling bins representing a divided region divided by color space quantization, and selecting a higher probability of the divided region corresponding to a lower purity. The retrieval method of claim 1, wherein a partition corresponding to a low purity is reconfigured to appear more in front of the histogram. 16. The color histogram described above uses a HMMD color space consisting of Hue, SUM (MAX (RGB) + MIN (RGB)), DIFF (MAX (RGB) -MIN (RGB)), and HMMD color. When the space has a DIFF value range of 0 to 255, the DIFF values designated for dividing by the DIFF axis are 9, 29, 75, and 200, respectively, and each of the four divided regions divided by these values is S1. , S2, S3, S4, S5, S1 is divided into 8 parts based on SUM, S2 is divided into 4 parts based on SUM, 8 parts based on Hue, S3 is divided into 4 parts based on SUM, and Hue is based on S4 is divided into 4 parts based on SUM, 12 parts divided by Hue, and S5 is divided into 2 parts based on SUM and 24 parts divided by Hue into 184 bins using color quantization method. Multimedia search method characterized in that the configuration. 17. The multimedia search according to claim 16, wherein the sampling probability is set to 24: 12: 6: 2: 1 in each of the regions S1, S2, S3, S4, and S5 in order to change the sampling probability described above. Way. The multimedia retrieval system according to claim 12, wherein in the multimedia retrieval system using the progressive color histogram, the histogram is transmitted or retrieved to the client using all or a part of the histogram according to the retrieval purpose and the hardware environment of the client to be retrieved. Progressive multimedia search method. 13. The multimedia retrieval system according to claim 12, wherein in the multimedia retrieval system using the progressive color histogram, the query histogram and the histogram to be searched are reconstructed according to respective bin order information, and then reconstructed so that the most important bin is first according to the bin order information. Multimedia search method characterized in that to compare the searched color histogram with each other. The multimedia retrieval method according to claim 12, wherein in the multimedia retrieval system using the above-described progressive color histogram, the query histogram and the search target histogram are taken out one by one according to the empty order information, and the color histogram is compared and searched in the empty order. .