JP2015191481A - Image processing apparatus and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus configured to extract an area of interest with high accuracy at a high speed, and an image processing program.SOLUTION: An image processing apparatus includes: a map creating section which creates a map indicating a distribution of a degree of attention of an input image; a mask creating section which creates a mask indicating an outer shape of an area of interest, on the basis of edge information of the input image; and an extraction section which extracts the area of interest, from the input image on the basis of the map created by the map creating section and the mask created by the mask creating section.

Description

  The present invention relates to an image processing apparatus and an image processing program.

  Conventionally, a technique for extracting a region where a main subject exists, that is, a region of interest from an image is known. For example, in Non-Patent Document 1, a saliency map is created from color information of an image and clustering based on the color information is performed on the image to divide the image into a plurality of regions. A technique for extracting a region of interest by extracting only a region having a high degree of attention) is described.

R. Achanta, S. Hemami, F. Estrada and S. Susstrunk, Frequency-tuned Salient Region Detection, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR 2009), pp. 1597-1604, 2009.

  The prior art has a problem that the calculation amount of clustering based on color information is large and requires processing time.

The image processing apparatus according to claim 1, wherein a map creation unit that creates a map that indicates a distribution of attention level of an input image, and a mask creation that creates a mask that indicates an outline of a region of interest based on edge information of the input image And an extraction unit that extracts the region of interest from the input image based on the map created by the map creation unit and the mask created by the mask creation unit. To do.
An image processing program according to claim 6, wherein a map creation step for creating a map showing a distribution of attention level of an input image, and a mask creation for creating a mask showing an outline of a region of interest based on edge information of the input image Causing the computer to execute a step, and an extraction step of extracting the region of interest from the input image based on the map created by the map creation step and the mask created by the mask creation step It is characterized by.

  According to the present invention, it is possible to extract a region of interest with high accuracy and at high speed.

1 is a schematic diagram of a configuration of an image processing system according to a first embodiment of the present invention. 2 is a diagram schematically showing a configuration of an image processing apparatus 3. FIG. 2 is a diagram illustrating a main configuration of an image processing apparatus 3. FIG. FIG. 3 is a diagram illustrating a main configuration of an attention area image creation unit 12. It is explanatory drawing of the map preparation part. It is explanatory drawing of the mask preparation part 22. FIG. It is explanatory drawing of the mask process part 23 and the extraction part 24. FIG. 5 is a flowchart of attention area image creation processing executed by a attention area image creation unit 12; It is explanatory drawing of the modification 1.

(First embodiment)
FIG. 1 is a schematic diagram of a configuration of an image processing system according to the first embodiment of the present invention. The image processing system 1 includes a camera 2, an image processing device 3, and a hard disk drive (HDD) 4. The camera 2 is a so-called digital still camera that captures a subject and stores captured image data (hereinafter simply referred to as an image) in a predetermined storage medium.

  The image processing device 3 is a device that manages images taken by the camera 2. The image processing device 3 acquires data from the camera 2 by performing data communication with the camera 2 by, for example, wired or wireless communication with the image taken by the camera 2. The image processing apparatus 3 stores and manages an image acquired from the camera 2 in the HDD 4 that is a large-capacity storage medium. The image processing device 3 displays a list of images stored in the HDD 4, for example, in a thumbnail format, searches based on the shooting date, shooting parameters, etc., displays (plays back) the image on a display device (not shown), Printing can be performed using the illustrated printer.

  In particular, the image processing apparatus 3 can automatically detect a set of similar images from a plurality of images taken by the camera 2 and manage the set of similar images as a unit of a block. For example, when a subject is photographed using a so-called continuous shooting function (continuous shooting function) or an exposure auto bracket function, the contents of each image are very similar. Even if such a special function is not used, if the user repeats photographing while fixing the angle of view or the like, the images will have very similar contents. The image processing device 3 identifies the main subject of the image by the process described later, and groups (classifies) images that match the main subject to some extent to facilitate operation by the user. The operation by the user is, for example, deletion, display, printing, or the like. In other words, the image processing apparatus 3 provides the user with a function for collectively deleting, displaying, printing, and the like of an image for each group.

  FIG. 2 is a diagram schematically illustrating the configuration of the image processing apparatus 3. The present embodiment provides an image processing apparatus 3 for storing and classifying images by causing the computer apparatus shown in FIG. 1 to execute an image processing program for storing and classifying images. When the program is loaded into the personal computer 100, the program is loaded into the data storage device of the personal computer 100 and then executed to be used as the image processing device 3 for storing and classifying images.

  The program may be loaded by setting a recording medium 104 such as a CD-ROM storing the program in the personal computer 100, or by loading the program into the personal computer 100 via a communication line 101 such as a network. Good. When passing through the communication line 101, the program is stored in the hard disk device 103 of the server computer 102 connected to the communication line 101. In this manner, the program is supplied as various types of computer program products such as provision via the recording medium 104 or the communication line 101.

  The image processing device 3 is not limited to being realized as a function of a personal computer, and may be realized as a function of a tablet computer, a photo viewer, a camera, or the like. Further, it may be realized as a function of a server connected to a network.

  FIG. 3 is a diagram exemplifying a main part configuration of the image processing apparatus 3 realized as a function of the personal computer 100 (FIG. 2). The image processing device 3 includes an image classification unit 11 and an attention area image creation unit 12. The image classification unit 11 inputs the image input from the camera 2 to the attention area image creation unit 12. The attention area image creation unit 12 creates an attention area image from the input image and outputs the attention area image to the image classification unit 11. The attention area image is an image obtained by extracting the attention area from the input image. The attention area refers to an area where the main subject exists in the entire image. In other words, the attention area image is an image obtained by removing an area that is not the main subject, such as a background reflected in the input image. The operation of the attention area image creation unit 12 will be described in detail later.

  The image classification unit 11 stores the image input from the camera 2 and the attention area image created by the attention area image creation unit 12 in association with each other in the HDD 4. At the time of storage, the image classification unit 11 searches the HDD 4 for an attention area image similar to the attention area image. When such an attention area image is found, the image that is actually stored and the image associated with the found attention area image are classified into the same group. That is, in the HDD 4, a plurality of images having attention area images similar to each other are classified into one group.

  FIG. 4 is a diagram illustrating a main configuration of the attention area image creation unit 12. The attention area image creation unit 12 includes a map creation unit 21, a mask creation unit 22, a mask processing unit 23, and an extraction unit 24. Hereinafter, the functions of these units will be described in order.

(Description of map creation unit 21)
The map creation unit 21 creates a saliency map of the input image (hereinafter simply referred to as a map) based on the color information of the input image. This map is a map in which, for each pixel of the input image, the attention level of the pixel, that is, the possibility that the pixel belongs to the attention area (likeness of attention area) is calculated and assigned to the position of the pixel. That is, this map is a map that represents the distribution of the attention level at each pixel position. The map creation unit 21 of the present embodiment first creates a filter image obtained by applying a predetermined smoothing filter (for example, a Gaussian filter) to the input image. Then, a map is created by performing the calculation shown in the following equation (1) using the input image and the filter image.

Here, S (x, y) is the value (attention level) of coordinates (x, y) in the map, I _μ is the average feature vector of the input image, and I _ωhc (x, y) is the coordinates (x, y) of the filter image. This is a pixel value vector of the pixel y). The above equation (1) means that the absolute value of the change in the pixel value of the coordinate (x, y) due to the application of the smoothing filter is the map value of the coordinate (x, y). For example, in an input image, an area such as the sky where color change is poor (low frequency) does not change much even when a smoothing filter is applied. Therefore, such a region has a low level of attention in the map. On the other hand, since it is considered that there is a fine texture (high frequency) in the main subject, such a region changes greatly from the input image when a smoothing filter is applied. Therefore, such a region has a high degree of attention in the map. That is, the map in the present embodiment has a characteristic that can be called a frequency map of the input image. In the following description, it is assumed that the attention level in the map is normalized to a numerical value in the range of 0 to 1. The greater the number, the greater the degree of attention. That is, the larger the numerical value, the higher the possibility that the position is within the region of interest.

  The map creation method described above is the same as the saliency map creation method described in Non-Patent Document 1. However, the map creation unit 21 may create a map (saliency map) differently. For example, the map may be created using luminance information instead of color information. It is also possible to calculate a so-called defocus amount for each part in the shooting screen by a so-called phase difference type focus detection method, and to map these defocus amounts on the shooting screen. In this case, the smaller the defocus amount, the greater the attention level. As described above, the map may be created by any method as long as it represents the distribution of the attention level in the image.

  FIG. 5A shows an example of the input image. The input image 40 includes a person 41 as a main subject. In addition, the sky 42, the mountain 43, and the ground 44 are shown as a portion (background portion) other than the main subject. An example of a map created from this input image 40 is shown in FIG. In FIG. 5B, the degree of attention is represented by the shade of color (hatching). The thinner the region, the higher the degree of attention, and the darker the region. The distant sky portion 42 has the lowest degree of attention in the map 50. The distant mountain portion 43 has a higher degree of attention than the sky portion 42, but the degree of attention is low when viewed from the entire map 50. The ground portion 44 has a higher degree of attention than the mountain portion 43, but the degree of attention is also low when viewed from the entire map 50. On the other hand, the portions 41a, 41b, 41c, and 41d of the person that is the main subject have higher attention than the portions other than the main subject. Specifically, the person's nose portion 41 a has the highest degree of attention in the entire map 50. The person's face part 41b and the person's torso part 41c have the second highest attention. On the other hand, the portion 41d around the person's neck has a slightly lower degree of attention, but the degree of attention is higher than the portion other than the main subject.

(Description of the mask creation unit 22)
The mask creation unit 22 estimates the outer shape of the attention area based on the luminance information of the input image 40. Then, a mask representing the estimated outer shape is created. The mask creation unit 22 detects edge information by converting the input image 40 to gray scale. And the local feature point of the input image 40 is detected from the edge information. As such local feature point detection methods, SIFT (Scale Invariant Feature Transform), SURF (Speed Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), and the like are known. The mask creation unit 22 detects local feature points of the input image 40 by such a known method. FIG. 6A shows an example in which a large number of local feature points 60 are superimposed on the input image 40. In FIG. 6B, the input image 40 is removed and only the local feature points 60 are shown. In FIG. 6A and FIG. 6B, only a part of a large number of local feature points 60 is denoted for convenience of drawing.

  The mask creation unit 22 identifies a region 71 that encompasses the outermost periphery of the local feature point 60 detected in this way. Then, a mask in which “1” is assigned to the area 71 and “0” is assigned to the other area 72 is created. An example of the mask created from FIG. 6B is shown in FIG. If the local feature point 60 exists only in the main subject (within the attention area), the mask 70 created here should completely match the outline of the main subject (attention area). That is, if this mask 70 is applied to the input image 40, the main subject (region of interest) should be able to be extracted from the input image 40. Actually, the local feature point 60 is detected from a position other than the main subject (region of interest) like the local feature point 60a in FIG. 6B. Therefore, the mask 70 created here is also incomplete including a portion other than the main subject (region of interest). That is, the mask 70 is an estimate of the outer shape of the main subject (region of interest).

(Description of mask processing unit 23)
The mask processing unit 23 creates a processed map 80 based on the map 50 created by the map creation unit 21 and the mask 70 created by the mask creation unit 22. FIG. 7 schematically shows a method for creating the processed map 80. As shown by (A) in FIG. 7, the map creation unit 21 creates a map 50 from the input image 40. As shown by (B) in FIG. 7, the mask creating unit 22 creates a mask 70 from the input image 40. The mask processing unit 23 creates a processed map 80 by multiplying the map 50 and the mask 70 as indicated by (C) in FIG. As described above, “1” is assigned to the portion of the mask 70 estimated as the main subject (region of interest), and “0” is assigned to the other portions. Therefore, the processed map 80 multiplied by the map 50 includes the attention level value in the base map 50 for the portion estimated as the main subject (attention area), and “0” for the other portions. "Will be assigned. That is, in the processed map 80, the degree of attention of the part other than the main subject (attention area) is zero.

(Description of the extraction unit 24)
Based on the processed map 80, the extraction unit 24 creates an attention area image in which the main subject (attention area) is extracted from the input image 40. As shown by (D) in FIG. 7, the extraction unit 24 multiplies the input image 40 and the processed map 80 to create the attention area image 90. As described above, since the degree of attention of the part other than the main subject (attention area) in the processed map 80 is 0, the pixel value of such a part in the attention area image 90 is 0. On the other hand, the pixel value of the main subject (region of interest) is attenuated (darkened) by the amount corresponding to the degree of attention of 0 to 1 in the pixel value of the input image 40. Specifically, the pixel position with a lower attention level becomes darker from the pixel value of the input image 40.

  For example, in FIG. 5B, the portions 41a to 41d of the person that is the main subject all have a high degree of attention. Therefore, as shown in FIG. 7, in the attention area image 90, the pixel values of those portions are not changed so much from the input image 40 (not darkened). On the other hand, in FIG. 5B, a low degree of attention is assigned to the ground portion 44. Therefore, as shown in FIG. 7, in the attention area image 90, the pixel value of that portion is greatly lowered from the input image 40 and is close to black.

  That is, in the attention area image 90, the main subject (attention area) portion is almost in the state of the input image 40 (the luminance may be slightly reduced), and the other portions are substantially black. The part that is not the attention area is not completely black, but is inconspicuous compared to the attention area. Such a region-of-interest image 90 is sufficient for searching for similar images by the image classification unit 11.

  Note that level correction may be performed on the attention area image 90 so that an area having a luminance value less than a certain value (an area where the attention level is less than a certain value) becomes completely black. Further, if a known edge detection process is performed on the attention area image 90, the outer shape of the attention area can be specified. The outer shape specified here is more precise than the outer shape estimated by the mask creating unit 22. Furthermore, it is possible to create a histogram of the attention level in the map 50 and exclude an area having an attention level less than the peak value from the attention area image 90 (set the luminance value of that area to 0).

  FIG. 8 is a flowchart of attention area image creation processing executed by the attention area image creation unit 12. First, in step S <b> 10, the attention area image creation unit 12 receives an input of the input image 40. In step S <b> 20, the map creation unit 21 creates a map 50 based on the input image 40. In step S <b> 30, the mask creation unit 22 detects local feature points 60 from the input image 40. In step S <b> 40, the mask creation unit 22 creates a mask 70 based on the detected local feature points 60. In step S50, the mask processing unit 23 creates a processed map 80 from the map 50 and the mask 70. In step S <b> 60, the extraction unit 24 creates the attention area image 90 from the input image 40 and the processed map 80.

According to the image processing system of the first embodiment described above, the following operational effects can be obtained.
(1) The map creation unit 21 creates a map 50 indicating the distribution of the attention level of the input image 40. Based on the edge information of the input image 40, the mask creation unit 22 creates a mask 70 that indicates the outer shape of the region of interest. The extraction unit 24 extracts a region of interest from the input image 40 based on the map 50 created by the map creation unit 21 and the mask 70 created by the mask creation unit 22. Since it did in this way, an attention area can be extracted with high precision and at high speed. In particular, as compared with the prior art described in Non-Patent Document 1, it is not necessary to perform a process that requires a huge amount of calculation such as clustering based on color information, so that a region of interest can be extracted at high speed.

(2) The mask processing unit 23 creates a processed map 80 (mask processed map) by performing multiplication processing of the map 50 created by the map creating unit 21 and the mask 70 created by the mask creating unit 22. To do. The extraction unit 24 extracts a region of interest from the input image 40 by performing multiplication processing of the processed map 80 created by the mask processing unit 23 and the input image 40. Since it did in this way, an attention area can be extracted with high precision and at high speed.

(3) The map creation unit 21 creates the map 50 by calculating the absolute value of the difference between the smoothed image obtained by smoothing the input image 40 and the input image 40. Since it did in this way, the distribution of the attention level in the input image 40 can be calculated accurately.

(4) The mask creation unit 22 detects a plurality of local feature points 60 from the edge information of the input image 40 and estimates a shape including the outermost periphery of the plurality of local feature points 60 as the outer shape of the attention area. Since it did in this way, the external shape of an attention area can be estimated easily with sufficient accuracy.

  The following modifications are also within the scope of the present invention, and one or a plurality of modifications can be combined with the above-described embodiment.

(Modification 1)
In the embodiment described above, the mask creating unit 22 estimates the outer shape of the region of interest by including the outermost periphery of the local feature point 60. However, the mask 70 may be created by other methods. For example, as shown in FIGS. 9A and 9B, a mask 70 may be formed by arranging a square of a predetermined size centered on the position of each local feature point 60. In this case, as shown in FIG. 9B, areas (areas 71, 72, etc.) to which “0” is assigned are generated inside the person portion, but the outermost periphery of many local feature points 60 It is possible to reduce the amount of calculation required to check A figure other than a square (for example, a rhombus) may be arranged.

  Note that the above-described regions 71 and 72 are generated because the local feature point 60 does not exist in the region. However, the fact that the local feature point 60 does not exist in that region means that no texture exists (no edge exists) in that region of the input image 40. Therefore, it can be estimated that the colors of the regions 71 and 72 are substantially the same as the surrounding colors. Therefore, in the finally created attention area image 90, if these areas 71 and 72 are filled with the surrounding color, the masked portion generated in the attention area, such as the areas 71 and 72, is interpolated. can do.

(Modification 2)
For example, consider a case where the outer shape of the region of interest represented by the mask 70 is very small. At this time, if there is a part having a very high degree of attention in the attention area, the other part in the attention area in the attention area image 90 becomes very dark. Thus, the magnitude of the influence of the attention level on the area image 90 changes according to the size of the outline of the attention area estimated by the mask creating unit 22. Therefore, the mask processing unit 23 may weight the map 50 according to the outer shape of the attention area when creating the processed map 80. Specifically, the weighted attention level S ′ (x, y) is calculated from the attention level S (x, y) by the following equations (2) and (3).
S ′ (x, y) = S (x, y) ^α (2)
α = (As / Ai) × C (3)
Here, As is the outer area of the region of interest represented by the mask 70, Ai is the entire area of the input image 40, and C is a constant. As in this example, the smaller the outer area of the attention area represented by the mask 70 is, the smaller the influence of the attention degree on the attention area image 90 is. The influence on clearness can be reduced.

(Modification 3)
In creating the map 50, it is not always necessary to assign one attention level to each pixel of the input image 40. For example, the input image 40 may be divided into a 4-pixel by 4-pixel area in a grid pattern, and one attention degree may be assigned to each area. Further, the input image 40 may be divided into several regions by other methods. The same applies to the mask 70.

(Modification 4)
In the above-described embodiment, an example in which the attention area image 90 is used for classification of similar images has been described. However, the attention area image 90 created according to the present invention can be used for other purposes. For example, by synthesizing the attention area images 90, an image in which a large number of main subjects are gathered can be created. Further, the attention area image 90 extracted from a certain image may be combined with the landscape image to create an image as if the main subject represented by the attention area image 90 exists in the landscape. it can.

  As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiments, and other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. .

DESCRIPTION OF SYMBOLS 1 ... Image processing system, 2 ... Camera, 3 ... Image processing apparatus, 4 ... Hard disk drive (HDD), 11 ... Image classification | category part, 12 ... Attention area image creation part, 21 ... Map creation part, 22 ... Mask creation part, 23 ... Mask processing unit, 24 ... Extraction unit

Claims (6)

A map creation unit that creates a map that shows the distribution of the attention level of the input image;
Based on the edge information of the input image, a mask creating unit that creates a mask indicating the outer shape of the region of interest;
An extraction unit that extracts the region of interest from the input image based on the map created by the map creation unit and the mask created by the mask creation unit;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The extraction unit creates a mask-processed map by multiplying the map created by the map creation unit and the mask created by the mask creation unit, and the created mask-processed map and An image processing apparatus, wherein the region of interest is extracted from the input image by performing a multiplication process with the input image. The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the map creation unit creates the map by calculating an absolute value of a difference between a smoothed image obtained by smoothing the input image and the input image. In the image processing device according to any one of claims 1 to 3,
The mask generating unit detects a plurality of local feature points from edge information of the input image, and estimates a shape including the outermost periphery of the plurality of local feature points as an outer shape of the attention area. Processing equipment. In the image processing device according to any one of claims 1 to 3,
The mask creation unit detects a plurality of local feature points from edge information of the input image, and estimates a shape in which a figure of a predetermined shape is arranged at each position of the plurality of local feature points as an outline of the attention area An image processing apparatus. A map creation step for creating a map showing the distribution of the attention level of the input image;
Based on the edge information of the input image, a mask creating step for creating a mask indicating the outer shape of the region of interest;
An extraction step for extracting the region of interest from the input image based on the map created by the map creation step and the mask created by the mask creation step;
An image processing program for causing a computer to execute.

Images