KR102013372B1 - Apparatus and method of image saliency map - Google Patents

Abstract

An apparatus for generating a feature map of an image according to an embodiment of the present invention includes: a corner detector which receives an image acquired from an image sensor and detects feature points of a corner existing in the image; A blob detector for detecting a feature point of a blob present in the image; A corner-based feature map calculator configured to generate a corner-based feature map of the image based on the first image patch defined by the feature point of the corner and the area surrounding the feature point of the corner; A blob-based feature map calculator configured to generate a blob-based feature map of the image based on a second image patch defined as a feature point of the blob and an area surrounding the feature point of the blob; And a corner-blob integrated feature map calculator for normalizing and combining the corner-based feature map and the blob-based feature map, respectively, to generate a single integrated feature map.

Description

Apparatus and method for generating an image feature map {APPARATUS AND METHOD OF IMAGE SALIENCY MAP}

The present invention relates to an apparatus and method for generating an image feature map, and more particularly, to an apparatus and method for generating an image feature map for preserving feature information of corners and blobs existing in an image.

There is an increasing demand for transmitting image information acquired from a mobile device to a server, such as an Internet of things (IOT) environment or a smart phone, and performing image recognition or image retrieval using the received image information.

In this case, the end consumer of the image information received by the server is a computer (machine), not a human, and therefore, the image information required by the computer for computer image processing and recognition needs to be well preserved and transmitted to the server.

Considering that the end consumer of an image is a human as before, an image saliency map focusing on preserving the overall structure or image quality of the image is generated, and differential image transmission is performed according to the importance given to each pixel of the map. And processing could be applied, but considering that the end consumer of the image is a computer, the image feature map should be different from the existing image feature map considering that the end consumer of the image is a human.

With this point in mind, an example of creating an image feature map limited to the SIFT (Scale Invariant Feature Transform) and applying it to an application for transforming the size of an image has been presented (W. Tan, B. Yan, K.). Li, and Q. Tian, Image retargeting for preserving robust local feature: application to mobile visual search, IEEE Tr.On Multimedia, vol. 18, no. 1, pp. 128-137, Jan. 2016).

However, since SIFT considers only a feature corresponding to a blob among the feature information of the image, the feature map created based on the SIFT does not sufficiently reflect other features of the image such as a corner.

The present invention improves the efficiency of computer-based image recognition and image processing by generating an image feature map that can reflect both corner and blob feature information present in an image to solve the above problems. An object of the present invention is to provide an apparatus and method for generating an image feature map.

An apparatus for generating a feature map of an image according to an embodiment of the present invention includes: a corner detector which receives an image acquired from an image sensor and detects feature points of a corner existing in the image; A blob detector for detecting a feature point of a blob present in the image; A corner-based feature map calculator configured to generate a corner-based feature map of the image based on the first image patch defined by the feature point of the corner and the area surrounding the feature point of the corner; A blob-based feature map calculator configured to generate a blob-based feature map of the image based on a second image patch defined as a feature point of the blob and an area surrounding the feature point of the blob; And a corner-blob integrated feature map calculator for normalizing and combining the corner-based feature map and the blob-based feature map, respectively, to generate a single integrated feature map.

The corner based feature map calculator may calculate the importance of the pixels based on the number of the first image patches including the corresponding pixels of the image, and generate the corner based feature map based on the calculated importance of the pixels. Do.

The blob-based feature map calculator may calculate the importance of the pixels based on the number of the second image patches including the corresponding pixels of the image, and generate the blob-based feature map based on the importance of the calculated pixels. desirable.

The integrated feature map may further include a compression unit including importance information of each pixel included in the image, and determining a compression intensity of each pixel based on the importance information of each pixel.

The integrated feature map may include importance information of each pixel included in the image, and an image size reduction unit may reduce the data size of the image by deleting pixels having an importance less than a predetermined value among the pixels. It is preferable.

An image feature map generation method according to an embodiment of the present invention may include: a first step of receiving an image acquired from an image sensor and detecting feature points of a corner and feature points of a blob respectively present in the image; Generating a first image patch defined by an area surrounding a feature point of the corner and a second image patch defined by an area surrounding a feature point of the blob; A third step of generating a corner-based feature map of the image based on the feature point of the corner and the first image patch and generating a blob-based feature map of the image based on the feature point of the blob and the second image patch ; And a fourth step of normalizing and combining the corner-based feature map and the blob-based feature map, respectively, to generate one integrated feature map.

In the third step, the importance of the pixels is calculated based on at least one of the number of the first image patch and the number of the second image patches including the corresponding pixel of the image, and the corner-based feature map and the blob. At least one of the base feature maps is preferably generated based on the importance of the computed pixels.

The integrated feature map includes importance information of each pixel included in the image, and after the fourth step, the compression unit determines the compression intensity of each pixel based on the importance information of the respective pixels. It is preferable to further include.

The integrated feature map includes importance information of each pixel included in the image. After the fourth step, the image size reduction unit deletes pixels whose importance is less than a preset value among the pixels, thereby reducing the data size of the image. It is preferable to further include the step of reducing.

An image feature map generating apparatus according to another embodiment of the present invention receives an image obtained from an image sensor and detects a feature point of a blob present in the image by applying a MSS algorithm (Maximally Stable Extremal Region) algorithm. Detection unit; An image patch generator for generating an image patch defined by an elliptical region surrounding a feature point of the blob; An image patch extension unit extending an area of an image patch generated by the image patch generator; And a feature map calculator configured to generate a feature map of the image based on the feature points of the blob and the image patch expanded by the image patch extension.

The feature map calculator may calculate the importance of the pixels based on the number of the expanded image patches including the corresponding pixels of the image, and generate the feature map based on the calculated importance of the pixels.

The image patch extension unit preferably increases the length of the short axis and the long axis of the elliptical image patch generated by the image patch generator by a predetermined value.

The numerical value is preferably set to include the corner included in the image patch.

According to another aspect of the present invention, there is provided a method of generating an image feature map, which receives an image obtained from an image sensor and detects a feature point of a blob present in the image by applying a maximum stable extremal region (MSER) algorithm. step; Generating an image patch defined by an elliptical region surrounding a feature point of the blob; A third step of expanding an area of the generated image patch; And generating a feature map of the image based on the feature points of the blob and the extended image patch.

In the third step, the area of the image patch is expanded by increasing the length of the short axis and the long axis of the image patch by a predetermined value, wherein the value is set such that the image patch includes a corner included in the image. It is desirable to be.

In the fourth step, the importance of pixels is calculated based on the number of patches of the image including the corresponding pixel of the image, and the feature map is generated based on the calculated importance of the pixels.

The apparatus and method for generating a feature map of an image according to the present invention generates an image feature map that can include both a blob feature, a corner feature, and an image patch thereof that define features of the image data. The application of the specific feature point detection method is not limited, and various feature points can be detected, thereby improving generality and compatibility.

The effects of the present invention are not limited to those mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram illustrating an image processing system including a client and a server.
2 is a block diagram schematically illustrating an apparatus for generating an image feature map according to an embodiment of the present invention.
3 is an exemplary view of generating an image feature map according to FIG. 2.
4 is a flowchart illustrating a method of generating an image feature map according to an embodiment of the present invention in time series.
5 is a block diagram schematically illustrating an apparatus for generating an image feature map according to another exemplary embodiment of the present invention.
6 is an exemplary view of generating an image feature map shown in FIG. 5.
7 is a distribution diagram of elliptical image patches and corners (green points) of MSER blobs.
8 is an enlarged exemplary view of an ellipse of an MSER blob in the apparatus for generating a feature map according to another embodiment of the present invention.
9 is a flowchart illustrating a method of generating an image feature map according to another embodiment of the present invention in time series.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention.

In describing the present invention, when it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. In addition, numerals (eg, first, second, etc.) used in the description process of the present specification are merely identification symbols for distinguishing one component from another component.

In addition, throughout the specification, when one component is referred to as "connected" or "connected" with another component, the one component may be directly connected or directly connected to the other component, but in particular It is to be understood that, unless there is an opposite substrate, it may be connected or connected via another component in the middle.

In addition, throughout the specification, when a part is said to "include" a certain component, it means that it may further include other components, without excluding the other components unless otherwise stated. In addition, the terms "unit", "module", and the like described in the specification mean a unit that processes at least one function or operation, which means that it may be implemented by one or more pieces of hardware or software or a combination of hardware and software. .

Before describing embodiments of the present invention, an image processing system including a client and a server will be described first. 1 is a block diagram illustrating an image processing system including a client and a server.

As shown in FIG. 1, the image processing system includes a client 1, an image feature map generator 2, an image compression / transmission encoder 3, a transmission channel 4, and an image compression / transmission decoder 5. ) And the server 6.

The client 1 which acquires and provides the image data may be an image sensor terminal of various types of wired and wireless, and may be applied when the end consumer of the image data obtained from these terminals is a non-human computer, that is, a server 6. Can be.

In this case, transmitting or storing all information of the image acquired by the client 1 may be impossible by the transmission channel 4 of the limited band.

Therefore, it is important to divide the data to be stored and transmitted / stored according to the importance of the image data and the data that can be deleted even if it is deleted. ) Can generate a feature map of the image and use it for image compression and encoding (3).

In this case, a method of reducing overall image data (size) by deleting non-critical pixels according to the importance of each pixel determined by the image feature map may be applied.

In the present invention, by including both corners and blobs in the image feature information to be considered in order to generate the image feature map, the server 6 is not limited to a specific feature (corner or blob), but the corners or blobs, etc. Generalized to apply various feature detection methods.

Hereinafter, the image feature map generating apparatus according to an embodiment of the present invention will be described in detail with reference to FIGS. 2 and 3. FIG. 2 is a block diagram schematically illustrating an apparatus for generating an image specific map according to an embodiment of the present invention, and FIG. 3 is an example of generating an image feature map according to FIG. 2.

As shown in FIG. 2, the image feature map generating apparatus 120 according to an embodiment of the present invention includes a corner detector 121, a blob detector 122, a corner based feature map calculator 123, and a blur based It is configured to include a feature map calculator 124 and a corner-blob integrated feature map calculator 125.

The corner detector 121 and the blob detector 122 receive an image obtained from an image sensor and independently detect a feature point of a corner existing in the image and a feature point of the blob.

The corner-based feature map calculator 123 calculates a corner-based feature map of the image based on a first image patch defined as a corner feature point in the image detected by the corner detector 121 and an area surrounding the feature point of the corner. Perform the function to create.

In particular, the corner-based feature map calculator 123 calculates the importance of the pixels based on the number of first image patches including the corresponding pixels of the image, and generates the corner-based feature map based on the calculated importance of the pixels. .

Specifically, the importance, i.e., the feature intensity, is imparted to the first image patches in the area surrounding the corner feature points of the image, and if these first image patches overlap at a particular pixel in the image, the importance of that pixel is equal to the first image patch. The cumulative frequency is accumulated to calculate the final importance, that is, the feature strength, thereby generating a corner-based feature map.

Referring to FIG. 3, the first image 11 obtained from the image sensor 110 is a corner-based feature map calculator 123 after the corner feature points are derived by the corner detector 121. The corner-based feature map image 12 is generated by.

In the corner-based feature map image 12, as described above, the feature intensity is calculated according to the overlapping frequency of the corners and the first image patch surrounding the corners, and the higher the feature intensity is, the brighter the color is represented.

The blob-based feature map calculator 124 generates a blob-based feature map of the image based on a blob feature point in the image detected by the blob detector 122 and a second image patch defined as an area surrounding the blob feature points. Perform the function to create.

In particular, the blob-based feature map calculator 124 calculates the importance of the pixels based on the number of second image patches including the corresponding pixels of the image, and generates a corner-based feature map based on the calculated importance of the pixels. .

Specifically, the importance, i.e., the feature intensity, of the second image patches in the region surrounding the blob feature point of the image, and if these second image patches overlap at a particular pixel in the image, then the importance of that pixel is the second image patch. The cumulative frequency is accumulated to calculate the final importance, that is, the feature strength, thereby generating a corner-based feature map.

The second image patch is defined as an image patch of an ellipse proportional to the scale of the detected blob, and the importance, that is, the feature intensity, is calculated in proportion to the occurrence and duplication of the second image patch for each pixel.

Referring to FIG. 3, the first image 11 obtained from the image sensor 110 has a derivation point of a blob by the blob detector 122, and then a blob-based feature map calculator 124. ), The blob-based feature map image 13 is generated.

In the blob-based feature map image 13, as described above, the feature intensity is calculated according to the overlap frequency of the blob and the second image patch by the overlap frequency of the blob and the second image patch surrounding them, and the feature intensity is high. The more bright the color.

In addition, the corner-flop integrated feature map calculator 125 in the apparatus for generating an image feature map according to an embodiment of the present invention normalizes the corner-based feature map 123 and the blob-based feature map 125 described above, respectively. It combines and then creates a single integrated feature map.

That is, the corner-blob integrated feature map calculation unit 125 reflects specific intensities of pixels independently calculated through the corner-based specific map and the blob-based specific map, respectively, as shown in FIG. 3. The relatively bright part of the integrated feature map image 14 is the part with a high intensity of the feature. That is, it is selected as a pixel having high importance.

In the meantime, when the apparatus for generating an image feature map according to an embodiment of the present invention is further extended, the apparatus may further include a compression unit 131 and an image size reduction unit 132.

The compression unit 131 determines a compression intensity of each pixel based on the importance information of each pixel calculated by the corner-blob integrated feature map calculator 125 among the pixels in the image.

The image size reduction unit 132 reduces the size of the image data by discarding pixels whose importance is less than a predetermined value among pixels included in the image, and reconstructing the image using only pixels having a predetermined value or more.

As a result, the importance of the pixels included in the image is calculated by the corner-blob integrated feature map, and the compression unit 131 and the image size reduction unit 132 reduce the data size of the image based on the importance. Computer-based image recognition and image processing can be improved.

Hereinafter, a method for generating an image feature map according to an embodiment of the present invention will be described with reference to FIG. 4, but portions overlapping with the contents of the image feature map generating device according to the embodiment of the present invention will be described. The detailed description will be omitted. 4 is a flowchart illustrating a method of generating an image feature map according to an embodiment of the present invention in time series.

An image feature map generation method according to an embodiment of the present invention relates to an image feature map generation method using the image feature map generation device according to an embodiment of the present invention described above, and may be largely divided into four steps. have.

First, a first step (S110) of detecting a feature point of a corner and a blob of a corner existing in an image, a second step (S120) of generating a first image patch and a second image patch, and a corner-based feature map and a blob-based A third step S130 of generating a feature map is performed.

Specifically, the feature points of the corners and the blob feature points in the first step S110 are independently detected by the corner detector 121 and the blob detector 122, respectively.

The second step S120 is specifically divided into generating a first image patch defined as an area surrounding a feature point of a detected corner and generating a second image patch defined as an area surrounding a feature point of a detected blob. These two steps are also performed independently of each other.

The third step (S130) specifically generates a corner-based feature map of the image based on the feature points of the corners and the first image patch and generates a blob-based feature map of the image based on the feature points of the blob and the second image patch. The two steps are also performed independently.

In particular, the importance of the pixels is calculated based on the number of first image patches including corresponding pixels of the image, and the corner based feature map is generated based on the importance of the calculated pixels.

Similarly, importance of pixels is calculated based on the number of second image patches including corresponding pixels of the image, and a blob based feature map is generated based on the importance of the calculated pixels.

After the third step (S130) described above, a fourth step (S140) of generating a single integrated feature map by normalizing and combining the corner-based feature map and the blob-based feature map, respectively, is performed. Is the importance information of each pixel included in the image.

In the meantime, when the image feature map generation method according to an embodiment of the present invention is further extended before data transmission, the data size of the image is determined based on the importance information of the pixel after the fourth step S140. It may further include reducing steps.

Hereinafter, an image feature map generating apparatus according to another embodiment of the present invention will be described with reference to FIGS. 5 to 8. FIG. 5 is a block diagram schematically illustrating an apparatus for generating an image feature map according to another exemplary embodiment of the present invention, FIG. 6 is an example of generating a feature map according to FIG. 5, and FIG. 7 is an elliptic image patch and a corner of an MSER blob ( FIG. 8 is an extended example of an ellipse of an MSER blob in the apparatus for generating an image feature map according to another embodiment of the present invention.

As shown in FIG. 5, the apparatus for generating an image feature map according to another exemplary embodiment of the present invention may include an MSER blob detector 221, an image patch generator 222, an image patch expander 223, and a feature map calculator. 224).

The MSER blob detector 221 receives an image 21 obtained from the image sensor 210 and detects a feature point of the blob present in the image, and performs detection by applying an MSER (Maximally Stable Extremal Region) algorithm. In this configuration, it is possible to obtain the blob detection image 22 of FIG. 6.

The image patch generator 222 generates an image patch of an elliptical region including the center point of the detected blob and the corresponding blob area by applying an algorithm of the MSER.

The image patches of the elliptical area may overlap each other in the dense area of the blob, and the importance of each pixel among the pixels included in the image, that is, the intensity of the feature is calculated to be proportional to the number of overlapping image patches, so that the feature map will be described later. The calculator 224 may generate the MSER blob-based feature map 23, and the feature map corresponds to an image feature map from which corner features are excluded.

On the other hand, the image of overlapping the image patch (white) and the corner feature point (green) of the blob ellipse shape with respect to the original image of Figure 7 (a) is Figure 7 (b), as shown in Figure 7 (b) You can see that most corner points exist at the edge of the image patch.

When the area of the image patch is expanded as shown in FIG. 7C, the image patch may include not only a blob but also a corner area, and thus a feature map may be calculated in consideration of both the blob and the corner.

Accordingly, the apparatus for generating an image feature map according to another embodiment of the present invention extends an image patch that extends an area of an image patch generated by the image patch generator 222 to calculate an image feature map considering both a blob and a corner. The unit 223 is provided.

In particular, the image patch extension unit 223 may increase the length of the short axis and the long axis of the elliptical image patch generated by the image patch generator 222 by a predetermined value to expand the area of the image patch.

More specifically, referring to FIG. 8, when the length of the long axis of the image patch defined as an elliptical region surrounding the feature points of the blob is a and the length of the short axis is b, the image patch extension 223 is the length of the long axis. And by increasing the length of the short axis by Δa and Δb, respectively, the area of the image patch can be expanded.

However, Δa and Δb should be set to include corner feature points included in the image, and through this, the feature map calculated by the feature map calculator may consider all of the blobs and corners in the image.

In the meantime, when the apparatus for generating an image feature map according to another embodiment of the present invention is further extended, the apparatus may further include a compression unit 231 and an image size reduction unit 232.

The compression unit 231 determines a compression intensity of each pixel based on the importance information of each pixel calculated by the feature map calculator 224 among the pixels in the image.

The image size reducer 232 performs a function of reducing the size of the image data by discarding pixels whose importance is less than a predetermined value among pixels included in the image, and reconstructing the image using only pixels having a predetermined value or more.

As a result, the importance of the pixels included in the image is calculated by the feature map, and the compression unit 231 and the image size reduction unit 232 reduce the data size of the corresponding image based on the importance. It is possible to improve the efficiency of recognition and image processing.

Hereinafter, a method for generating an image feature map according to another embodiment of the present invention will be described with reference to FIG. 9, but portions overlapping with the contents of the image feature map generating device according to another embodiment of the present invention will be described. The detailed description will be omitted. 9 is a flowchart illustrating a method of generating an image feature map according to another embodiment of the present invention in time series.

An image feature map generation method according to another embodiment of the present invention relates to an image feature map generation method using the image feature map generation device according to another embodiment of the present invention, which can be largely divided into four steps. have.

First, a first step (S210) of receiving an image obtained from the image sensor 210 and detecting feature points of a blob present in the image by applying an MSER algorithm is performed.

Thereafter, a second step S220 of generating an image patch defined as an elliptical region surrounding the feature points of the blob detected in the first step S210 is performed.

Thereafter, a third step S230 of expanding the area of the image patch generated in the second step S220 is performed. Specifically, the area of the image patch is increased by increasing the length of the short and long axes of the image patch by a predetermined value. It is desirable to expand.

In addition, the above-described numerical value is preferably set so that the image patch may include corner feature points included in the image, so that the final feature map can consider not only the blob feature points but also the corner feature points.

Thereafter, a fourth step S240 of generating a feature map of the image based on the feature points of the blob and the expanded image patch is performed.

In particular, in the fourth step S240, the importance of pixels is calculated based on the number of image patches including corresponding pixels of the image, and a feature map is generated based on the importance of these pixels.

In the meantime, when the image feature map generation method according to an embodiment of the present invention is further extended before data transmission, the data size of the image is determined based on the importance information of the pixel after the fourth step S140. It may further include reducing steps.

The apparatus and method for generating an image feature map according to the present invention described above may be embodied as computer readable codes on a computer readable recording medium. Computer-readable recording media include all kinds of recording media having data stored thereon that can be decrypted by a computer system. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, and the like. The computer readable recording medium can also be distributed over computer systems connected over a computer network, stored and executed as readable code in a distributed fashion.

Although the above has been described with reference to embodiments of the present invention, those skilled in the art may variously modify the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. And can be changed easily.

110, 210: image sensor
121: corner detection unit
122: blob detector
123: corner-based specific map calculator
124: blob-based specific map calculation unit
125: corner-flop integration specific map calculation unit
221: MSER blob detection unit
222: image patch generator
223 video patch extension
224: feature map calculator

Claims (16)

An MSER blob detection unit that receives an image obtained from an image sensor and detects a feature point of a blob present in the image by applying a Maximumly Stable Extremal Region (MSER) algorithm; An image patch generator for generating an image patch defined by an elliptical region surrounding a feature point of the blob; An image patch extension unit extending an area of an image patch generated by the image patch generator; And a feature map calculator configured to generate a feature map of the image based on the feature points of the blob and the image patch extended by the image patch extension.
The feature map calculator may calculate an importance of pixels based on the number of extended image patches including corresponding pixels of the image, and generate the feature map based on the importance of the calculated pixels. .
The method of claim 1,
And the image patch extension unit increases the length of the short axis and the long axis of the elliptical image patch generated by the image patch generator by a predetermined value.
The method of claim 2,
And the numerical value is set so that the image patch includes a corner included in the image.
A first step of receiving an image obtained from an image sensor and detecting a feature point of a blob present in the image by applying an MSER algorithm (Maximum Stable Extremal Region); Generating an image patch defined by an elliptical region surrounding a feature point of the blob; A third step of expanding an area of the generated image patch; And generating a feature map of the image based on the feature points of the blob and the extended image patch.
In the fourth step, the importance of pixels is calculated based on the number of the image patches including the corresponding pixel of the image, and the feature map is generated based on the calculated importance of the pixels.
The method of claim 4, wherein
In the third step, the area of the image patch is expanded by increasing the length of the short axis and the long axis of the image patch by a predetermined value.
And the numerical value is set such that the image patch includes a corner included in the image.
delete delete delete delete delete delete delete delete delete delete delete

Images