RU2510935C2 - Method of indexing and searching digital images - Google Patents

Abstract

FIELD: physics, computer engineering.

SUBSTANCE: invention relates to systems for indexing and searching digital images contained in files of different graphics formats. The method involves finding a colour having the shortest Manhattan length from the found predominant colour in the selected colour coordinate system from a predetermined set of primary colours, classified according to brightness, saturation and hue; using the obtained colour as an identifier for organising the search procedure in an image database; determining colour association from the basic set of primary colours to form a visual colour similarity and visual colour contrast characteristic; generating an image index (metadata) according to the presented primary colour in RGB format and/or name and/or tag of the colour; searching images in the database. As a result of the search, a list of images is formed, having indices matching the presented index and/or indices for which primary colours are present in the list of associations of visual similarity or in the list of colour contrast for the primary colour in the image search index.

EFFECT: faster automatic and semi-automatic indexation of images and faster search of images with similar predominant colours in a database.

9 cl, 4 dwg

Description

The invention relates to systems for indexing and searching for digital images contained in files of various graphic formats located on a user's local computer, including the global Internet.

The proposed method can be widely used in search engines, as well as in hardware and software complexes of electronic commerce (online stores), where it is often required to search for goods from the store’s database for characteristics not only specified in text form, but also obtained as a result digital image analysis.

As part of the implementation of the system, two main tasks are distinguished: determining the prevailing color of the image while maintaining the index (metadata) of the image in the information database and searching for images containing similar predominant colors in the database. The predominant color of an image is defined as the color that most pixels in an image have. The predominant color of the image can be calculated by image analysis methods. Also, the predominant color of the image can be selected from a list of predefined colors by interacting with a computer operator with information database management software.

Known from the prior art, methods of content-oriented indexing to organize the search for files of various formats, including graphic, in which the coincidence of the content is evaluated by calculating the hash functions of the content [1]. The disadvantages of such methods include the lack of image analysis, which is why images with similar predominant colors can be classified as completely different images.

Known from the prior art, methods of content-oriented image retrieval mainly calculate the vector of characteristic values for each image and store it in the information database along with the image itself. In this case, the image is searched by calculating the vector of characteristic values for the presented image, followed by comparing the desired vector with all the vectors from the database. At the output of the search procedure, a series of images is obtained for which the error in the coincidence of the vectors with the desired vector is minimal (below a given threshold). Such methods include, for example, the method described in [2]. The method uses as a vector of characteristic features the value of histograms constructed from the values of the channels R, G, B, expressed by the RGB color model. Another method [3] uses as the vector of characteristic features of the image the values of histograms constructed from the values of channels H and S, expressed by the HSV color model.

Similar in technical essence is the method of searching for digital images from the image database [4], in which the histogram of the presented image is calculated, as well as the histograms of the images stored in the database, then the similarity measure of the images is calculated using the similarity assessment method of two probability distributions (Kullback- Leibler divergence) [5]. Then, the search results are ranked according to the degree of similarity with the input image.

The disadvantages of the prototype include the computational complexity of calculating histograms of images with the subsequent calculation of the metric of similarity. Another disadvantage of the prototype is the need for an input image to organize the search procedure for images from the database. The prototype analyzes the image content and outputs the result with the smallest histogram comparison error at the output of the search procedure. But this statement of the problem is not entirely suitable for e-commerce systems (online stores). The most typical task for e-commerce systems is to find the product image in the store’s product database for a given color and category. In this case, when using the prototype, a result can be achieved that does not quite meet the task of searching for goods by the selected color, because a histogram of the image is built over the entire area of the image, including the subject itself (product) along with the background, which may be heterogeneous. Distortions introduced into the histogram by background regions can introduce a significant error in the image search results by color.

The claimed invention allows to solve this drawback by defining a search query not in the form of an image of an object (product), but in the form of a color image painted in colors from a certain set of colors. Moreover, the inventive method of image search transforms the search query expressed in graphical form (color image) into a search query expressed in text form in the form of numbers and letters, which has a significant impact on the speed of the search. The number-letter string characterizing the color of the image is called a tag. At the same time, the selection of images is carried out according to the comparison of tags of small length (about 10-30 characters), which requires much lower computational costs compared to calculating the similarity of 8-bit histograms consisting of 256 values per channel. When using the most common RGB color space, you need to compare among themselves 768 = 256 · 3 values for each of hundreds or thousands of images stored in the image database.

Also, the prototype cannot be used to search for images of goods by given associations of color harmony, in which as a result of the image search procedure a series of images are obtained that have colors that visually harmonize with each other.

Thus, the primary task of building image search systems for electronic commerce is the task of searching for images according to a given color. Image attributes that describe color, image category, and other image parameters are called image index or metadata. Image index is stored in an information database. In this case, the index of the image should include, along with the tag, the name of the color, which should as accurately as possible characterize the color of the object in the image.

The problem is solved by developing a method for indexing (generating metadata) of images, followed by the formation of a database of image metadata and the implementation of the search procedure for image metadata in the database, while the indexing method includes the following operations:

- determine the basic set of primary colors in the format of the RGB color space;

- for each primary color in RGB format from the base set, the primary color in HSV format is presented, this association is saved, and a name and color tag are entered for each primary color;

- the original image is presented in RGB format;

- determine the predominant color of the image in RGB format,

- result in the prevailing color of the image in HSV format;

- determine the primary color of the image in HSV format from the basic set of primary colors of HSV according to the prevailing color of the HSV image;

- determine the primary color of the image in RGB format according to the association of the basic color set in HSV format to the basic color set in RGB format;

- form the index (metadata) of the image according to a specific primary color in RGB, HSV format, name and color tag;

- save the index (metadata) of the image in the information database.

The developed method for searching for images (image metadata) in the generated image database includes the following operations:

- determine the association of primary colors from the basic set to form a characteristic of the degree of visual similarity and harmony of colors;

- form the index (metadata) of the image search request according to the submitted primary color in RGB format, and / or the name and / or color tag;

- they search for images (image metadata) according to the index (metadata) of the image search request and, as a result of the search, form a list of images with indices that match the presented index and / or indices that match the given associations of visual similarity and color harmony.

Figure 1. The block diagram of the phased implementation of the method of indexing (generating metadata) images according to the invention.

Figure 2. A block diagram of the phased implementation of the image search method according to the invention.

Figure 3. An example of determining the prevailing and primary color of an image.

Figure 4. An example of searching images from the database in accordance with the specified primary color of the image.

Hereinafter, a preferred embodiment of the claimed invention is described in detail using graphic materials. However, the scope of protection of the claimed invention is not limited to the preferred embodiment of the invention, which can be implemented in various forms. The preferred embodiment disclosed in the description is only an example provided to disclose the essence and help specialists fully understand the claimed invention.

A flowchart of a method for indexing (creating and storing metadata in a database) of digital images is shown in FIG. 1. According to FIG. 1, in step 101 of the method, a basic set of primary colors is determined. As an illustration of the invention, the basic set of colors is presented in Table. 1. The basic set of colors is represented in the form of values of the RGB color space, i.e. three-dimensional coordinates specified by the RGB color model [6]. Also, as a more accurate determination of the prevailing color of the image, the basic set of colors can be represented in the form of HSV color space values, i.e. 3D coordinates specified by the HSV color model. A detailed description of the HSV color model is not given here due to fame from the current level of technology [7]. Moreover, the channels R, G, B have a range of values from 0 to 255, while the values of channel H vary from 0 to 360 °, and the values of channel S and V change from 0 to 1.

Tab. one Basic set of primary colors Color name RGB Hsv Tag Bright red (255,0,0) (0.1.1) red saturated bright Bright orange (255,128.0) (30,1,1) orange saturated bright Bright yellow (255,255,0) (60,1,1) yellow saturated bright Bright green (128,255,0) (90,1,1) light green saturated bright Bright green (0.255.0) (120,1,1) green saturated bright Bright blue green (0,255,128) (150,1,1) blue green saturated bright Bright cyan (0,255,255) (180,1,1) cyan saturated bright Bright blue (0,128,255) (210,1,1) blue saturated bright Bright blue (0,0,255) (240,1,1) blue saturated bright Bright purple (128,0,255) (270,1,1) purple saturated bright Hot pink (255,0,255) (300,1.1) pink saturated bright Bright raspberry (255,0,128) (330,1,1) raspberry saturated bright Dark red (102.41.41) (0,0.6,0.4) red halftone dark Dark orange (102.71.41) (30,0.6,0.4) orange halftone dark Dark yellow (102,102,41) (60.0.6,0.4) yellow halftone dark Dark green (71,102,41) (90, 0.6,0.4) light green semitone dark Dark green (41,102,41) (120,0.6,0.4) green halftone dark Dark blue green (41,102,71) (150,0.6,0.4) blue green halftone dark Dark cyan (41,102,102) (180,0.6,0.4) cyan halftone dark Dark blue (41.71.102) (210.0.6,0.4) blue halftone dark Navy blue (41,41,102) (240,0.6,0.4) dark blue halftone Dark violet (71,41,102) (270,0.6,0.4) purple halftone dark Dark pink (102,41,102) (300,0.6,0.4) pink halftone dark Dark raspberry (102.41.71) (330,0.6,0.4) raspberry halftone dark

Separately, we note that the choice of a color model for basic colors does not change the essence of the invention and depends only on ease of use to create a system for indexing digital images.

To determine the basic set of primary colors, presented in the form of the HSV color space, the values of the channels H, S and V can be divided into intervals for the convenience of choosing the basic colors when changing their number. In this case, the intervals for the description of the interval can be assigned names in the form of tags (Table 2). Hereinafter, the terms color channel and color space coordinates have the same meaning. In this case, the term “color channel” is used mainly when describing the reference system in the selected color space, while the term “color coordinate” is used in the context of the coordinate in the selected report system.

For a Hue channel, the number of slots is specified as N _HUE . If the Hue channel is evenly divided, each interval will have a length expressed as I _HUE / N _HUE , where I _HUE is the maximum value for the Hue channel (where I _HUE = 360 °).

For the Saturation channel, the number of slots is specified as N _SAT . If the Saturation channel is evenly divided, each interval will have a length expressed as I _SAT / N _SAT , where I _SAT is the maximum value for the Saturation channel (where I _SAT = 1 when using the normalized value).

For the Value channel, the number of slots is specified as N _VAL . If the Value channel is evenly split, each interval will have a length expressed as I _VAL / N _VAL , where I _VAL is the maximum value for the Value channel (where I _VAL = 1 when using the normalized value).

Moreover, for each primary color from the base set, the H coordinate of the primary color is selected as the value of H from the intervals into which the Hue channel is divided; the S coordinate of the primary color is selected as the S value from the intervals into which the Saturation channel is divided; the V coordinate of the primary color is selected as the value of V from the intervals into which the Value channel is divided. In Tab. Figure 2 shows an example of dividing Hue, Saturation, Value channels into intervals. Also in the table are the values of the primary colors specified within the selected ranges. To illustrate the method, for each primary color, select the Hue value as the middle of the interval of the mapped Hue channel interval. However, for the Saturation and Value channels, the primary color values do not correspond to the midpoints of the desired intervals.

To determine achromatic colors (colors with a low saturation value) of the basic set, a threshold T _S for the values of the Saturation channel and a threshold T _V for the values of the Value channel are entered. In this case, the primary color is considered achromatic if the value of the Saturation coordinate is less than T _S and the value of the Value coordinate is less than T _V.

To simplify indexing images, as well as forming groups of images, at step 101 of the method for indexing digital images, an association is made of the values of the primary colors with the color name, as well as with the tag. In this case, the color name is used to interact with the person in the graphical user interface. In turn, the tag is used to organize the search procedure. Tag categories can be selected as characterizing primary colors by brightness (intensity), and / or by color, and / or by color saturation. In Tab. Figure 1 shows examples of tags defined in accordance with the color tone, its brightness and saturation. Tags are composed by a combination of interval tags from Table. 2.

Tab. 2 Example of setting intervals for the Hue, Saturation, Value channel Channel Interval Tag Interval Interval primary color Hue red (-15.15) 0 Orange (15.45) thirty yellow (45.75) 60 light green (75,105) 90 green (105,135) 120 blue green (135,165) 150 cyanogen (165,195) 180 blue (195,225) 210 blue (225,255) 240 Violet (255,285) 270 pink (285,315) 300 crimson (315,345) 330 Saturation pastel (0,0.45) 0.3 semitone (0.45,0.9) 0.6 saturated (0.9.1) one Value dark (0,0.45) 0.4 light coloured (0.45,0.9) 0.7 bright (0.9.1) one

After determining the basic set of primary colors in step 102 of the method for indexing digital images, the desired digital image 100 is converted to RGB format. Ways to transfer images from well-known popular file storage formats such as JPEG, TIFF, PSD and others to a three-dimensional raster array of RGB values are not given here due to the fame of these methods from the current level of technology and technology.

After converting the input digital image to RGB in step 103, the prevailing color of the image 100 expressed in the RGB color space is determined. Finding the predominant color for automatic indexing of images in the information database is carried out using a software tool, by using which the computer operator enters the image into the database, then by clicking on the selected image area indicates the color of which pixel to consider the predominant color of the image. You can do without the participation of a computer operator and take the image center with coordinates (x _im / 2, y _im / 2) as the desired region of the image, where x _im is the image width in pixels, y _im is the image height in pixels. The center of the Cartesian coordinates with the axes X and Y is placed in the upper left corner of the image. In turn, the X axis is directed in the horizontal direction from left to right, and the Y axis is directed in the vertical direction from top to bottom.

Digitally presented images can often contain noisy areas due to signal processing in cameras and subsequent processing and / or compression. Noisy image areas are understood as areas in which individual pixels have values that are very different from the average value in a given image area. To reduce the influence of digital noise on the result of determining the predominant color, it is possible to use digital image filtering for the pixel region around the selected pixel. Filtering can be performed using the Gaussian function. A description of the operation of digital filtering using the Gaussian function is not given here due to the popularity of this method from the current level of technology. Any digital filtering method can be applied to eliminate the effect of image noise. However, the choice of filtering method does not affect the meaning of the invention. A pixel region is understood to mean a group of pixels located in the immediate vicinity of a given pixel with coordinates (x, y). The pixel area is usually set in the form of a rectangular image window with the coordinates of the upper left corner (x-x _w / 2, y-y _w / 2) and the lower right corner (x + x _w / 2, y + y _w / 2). In this case, x _w is the width of the rectangular window in pixels, y _w is the height of the rectangular window in pixels. The method of selecting a filtering window described above is purely illustrative. The choice of the shape of the filtering window does not change the essence of the invention. We want to note that the choice of filtering method falls on the shoulders of the developer of the image indexing system and does not change the essence of this invention.

After determining the predominant color of the image expressed in the RGB color space in step 104 of the method for indexing digital images, the primary color of the desired image is determined according to the basic set of primary colors. For this, the RGB value of the prevailing color is converted to the HSV color space. For the obtained HSV value, the Manhattan distance (1 norm) from each primary color from the base set is calculated according to formula 1

$${\Delta }_{hsv}={\Delta }_{hue}+{\Delta }_{sat}+{\Delta }_{val},\left(\mathrm{one}\right)$$

, $${\Delta }_{hue}\in \left\{0^{\circ },360^{\circ }\right\}$$

- абсолютное значение разницы координаты Hue для преобладающего цвета Р {Р_hue, P_sat, P_val} и основного цвета О {О_hue, O_sat, O_val};Where

, $${\Delta }_{hue}\in \left\{0^{\circ },360^{\circ }\right\}$$

- the absolute value of the difference in the coordinate Hue for the predominant color P {P _hue , P _sat , P _val } and the primary color O {O _hue , O _sat , O _val };

,

- абсолютное значение разницы координаты Saturation для преобладающего цвета Р и основного цвета О;

,

- the absolute value of the difference of the Saturation coordinate for the predominant color P and the primary color O;

,

- абсолютное значение разницы координаты Value для преобладающего цвета Р и основного цвета О.

,

- the absolute value of the difference of the coordinate Value for the predominant color P and the primary color O.

At the same time, the range of values of the Hue coordinate from 0 ° to 360 ° is normalized to a range from 0 to 1. The normalization function can look like in formula 2.

Thus, the following formula (formula 3) is used to calculate the distance between colors in the HSV color space

$${\Delta }_{hsvN}={\Delta }_{hueN}+{\Delta }_{sat}+{\Delta }_{val},\left(3\right)$$

The primary color of the presented image is determined from the basic set of HSV primary colors as the color that has the smallest Manhattan distance Δ _hsvN (formula 3) from the HSV value. Then, the primary color value in the RGB format is obtained from the set of associations of the primary colors RGB and HSV. An example of the primary color associations is shown in Table 1

After determining the primary color of the image expressed in the RGB color space in step 105 of the method for indexing digital images, an index (metadata) of the image 107 is formed. The desired index is used as an identifier when searching for images in the database. Along with information about the prevailing color of the image, the image index may contain information about the image file, for example, the system path to the image file. Also, the index may contain information on the photo shooting parameters (obtained, for example, from EXIF metadata of the image file) and any other information that characterizes the image, for example, to which category of goods does the subject recorded on the image belong.

After forming the index (metadata) of the image in step 106 of the method of indexing images, the index of the presented image is stored in the information database. Along with saving the index of images in the database, it is also possible to save the image itself in the database as a byte array.

Tab. 3 Base color associations from the base set Color name Hue Associations Contrast Associations Bright red Dark Red, Light Red Bright green Bright orange Bright Yellow, Dark Yellow, Light Orange Bright purple Bright yellow Bright Orange, Dark Yellow, Light Yellow Bright blue Bright blue Bright Blue, Light Blue, Navy Hot pink Bright blue Bright blue, light blue Bright yellow Bright purple Light Purple, Dark Purple Bright orange

An example implementation of the method of indexing images as part of the software database management system shown in Fig.3. According to Figure 3, the database operator can specify the categories of goods to which the image loaded into the database belongs. In addition to selecting categories, the software allows you to determine the prevailing and primary color of the image. The predominant color RGBA = {252, 22, 54, 255} was automatically calculated by filtering the central region of the image with a radius of several pixels. In this case, the main color of the image was defined as “red_light_saturated” according to the described method of indexing images.

A flowchart of a method for searching for images (image metadata) in the generated image database is shown in FIG. 2. According to FIG. 2, in step 201 of the method, associations of primary colors from a basic set are determined to form a characteristic of the degree of visual similarity and harmony of colors. An example of associations for some colors is presented in Table. 3.

After determining the associations of the primary colors from the base set, at step 202 of the image search method, an index (metadata) of the image search request is generated according to the presented primary color in RGB format and / or the name and / or color tag (200).

After forming the index in step 203 of the image search method, images (image metadata) are searched according to the index (metadata) of the image search request, and as a result of the search, a list of images (204) with indices matching the presented index and / or with indices that match with defined associations of visual similarity and color harmony.

An example implementation of a method for searching images as part of the firmware of a web interface (a web page placed on the Internet) of an electronic commerce system is shown in FIG. 4. According to Figure 4, the user of the web interface selects the desired color for the search by clicking on the color selection area (a rectangle on the top left with a rainbow scale). By clicking on the rainbow scale, the desired hue is set, in turn, by clicking on the rectangular area, the desired brightness and saturation are selected. After that, the web interface displays a list of product images according to the specified search criteria. Figure 4 shows a search result according to a given primary color without specifying a product category (e.g., blouse, dress, etc.).

The invention can be used as software and hardware for electronic commerce systems, in which the web user interface for choosing a product includes the search for goods not only by category, but also the search for images of goods indicating the desired color of the goods. It is also possible to search for images of goods according to given associations of color harmony, which can be very popular, for example, in electronic trading systems for clothes and shoes.

Literature

[1] WO 2005/033885, "Content oriented index and search method and system", 04/14/2005.

[2] "Image Retrieval using Color and Shape", Anil K. Jain and Aditya Vailaya, Pattern Recognition, 29 (8), 1996.

[3] "Supporting Similarity Queries in MARS", Michael Ortega, et al., ACM Multimedia 97.

[4] US 6163622 "Image retrieval system", 12.19.2000.

[5] http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler _ divergence.

[6] http://ru.wikipedia.org/wiki/RGB.

[7] http://m.wikipedia.org/wiki/HSV_ (color _ model).

Claims (9)

1. The method of indexing digital images, in which they analyze the digital representation of the images in order to create an index (metadata) of the image describing the primary color of the image, and use it as an identifier for organizing the image search procedure, characterized in that
- determine the basic set of primary colors in the format of the color space HSV;
- set the name and color tag for each primary color;
- the primary colors of the basic set are defined as achromatic if the value of the Saturation channel is less than a certain threshold T _S and if the value of the Value channel is less than a certain threshold T _V ;
- the original image is presented in RGB format;
- determine the prevailing color of the image in RGB format;
- make the conversion of the predominant color of the image from the RGB format to the HSV format;
- determine the primary color of the image in HSV format from the basic set of primary colors of HSV, as the color that has the smallest distance Manhattan (1 norm)

from the value presented by HSV, where

are the absolute differences of the Hue, Saturation, and Value coordinates, respectively;
- make the conversion of the primary color of the images from the HSV format to the RGB format;
- form the index (metadata) of the image according to a specific primary color in RGB, HSV format, name and color tag;
- save the index (metadata) of the image in the information database; 2. The method according to claim 1, characterized in that the list of tags is classified according to categories characterizing the primary colors according to brightness (intensity), and / or color, and / or color saturation. 3. The method according to claim 1, characterized in that the prevailing RGB color of the image is determined by selecting a pixel with corresponding coordinates. 4. The method according to claim 1, characterized in that the prevailing RGB color of the image is determined as the result of digital filtering of the selected pixel region with the corresponding coordinates. 5. The method according to claim 4, characterized in that the function of the Gaussian is used as the function of digital filtering. 6. The method according to claim 1, characterized in that for each primary color from the base set, the H coordinate of the primary color is selected as the value H from the intervals into which the Hue channel is divided, while the number of intervals for the Hue channel is determined as N _HUE ; the S coordinate of the main color is selected as the S value from the intervals into which the Saturation channel is divided, while the number of intervals for the Saturation channel is determined as N _SAT ; the V coordinate of the primary color is selected as the V value from the intervals into which the Value channel is divided, while the number of intervals for the Value channel is determined as N _VAL . 7. The method of searching for digital images using an image index as identifier (including information about the main color of the image in RGB format, tag and color name), characterized in that
- determine the association of colors from the basic set of primary colors to form a characteristic of visual similarity of colors;
- determine the association of colors from the basic set of primary colors to form characteristics of the visual contrast of colors;
- form the index (metadata) of the image according to the submitted primary color in RGB format, and / or the name and / or color tag;
- they search for images in the information database according to the generated image index and, as a result of the search, form a list of images with indices matching the presented index and / or with indices in which the primary colors are present in the list of visual similarity associations or in the color contrast list for the primary colors in the image search index. 8. The method according to claim 7, characterized in that the information database of digital images is stored on a server on the global Internet. 9. The method according to claim 7, characterized in that the database is accessed via a web interface (a website located on the global Internet), which is used by the user to form a search query into the image information database.

Images