JP5158974B2 - Attention area extraction method, program, and image evaluation apparatus - Google Patents

Description

  The present invention relates to a region of interest extraction method, a program, and an image evaluation apparatus.

  In recent years, mainly as a method for preprocessing of image tagging in digital image understanding and image search, and as a method for determining the zoom center of automatic zoom in a slide show, a method for extracting subjects from images and main methods A technique for extracting a simple region has been studied (see, for example, Non-Patent Document 1 and Non-Patent Document 2). In addition to such a method, a method of extracting a specific image area based on a user instruction, for example, a user specifying a predetermined area on the screen has been proposed (for example, Patent Document 1). reference). These are all classified by the concept of “attention area extraction from image”.

JP 2005-78290 A

“Extraction of Interest Region from Image Based on Attention” Kimura et al. IEICE ICD 2005-221, 2005 “AUTO CROPPING FOR DIGITAL PHOTOGRAPHS” Zhang et al. IEEE ICME 2005, 2005

  However, any of the conventional methods has a problem that the processing speed is slow. For example, although the method described in Non-Patent Document 1 is relatively fast, it takes about 200 milliseconds in a general personal computer (PC) environment. In view of the current general user's photography behavior, each user has a large amount of image data. When such an image extraction process is performed in a built-in environment such as a digital still camera (DSC) for a certain number of images in the large amount of image data, it takes a total of 10 seconds or more. When considering a real application, it's still not practical. In the method described in Non-Patent Document 2, it takes several seconds to process one image even in a PC environment. This case is also not preferable when considering an actual application in an embedded environment such as DSC.

  Further, in the method described in Non-Patent Document 1, the extraction process is performed on the assumption that the region that the user is interested in exists in the vicinity of the center of the image. There are many cases where such assumptions do not hold. When extraction processing is performed on such an image, an area far from the user's sense is extracted as an attention area. On the other hand, in the invention described in Patent Document 1, in order to extract a region after specifying a region (or a point) to which a user pays attention on an image, a small device such as a DSC is compared with a PC. If only a simple user interface can be provided, it will not be an effective function. In addition, since region-based clustering is performed, the processing speed cannot be realized at a practical speed in an embedded environment such as DSC.

  The present invention has been made in view of such a problem, and is an attention area extraction method that is faster and more accurate, and that can extract a practical attention area even in an embedded environment such as DSC. It is an object to provide a program and an image evaluation apparatus.

  In order to solve the above-described problem, a region of interest extraction method according to the first aspect of the present invention extracts a region of interest in an extracted image, which is a digital image expressed by luminance and two chromaticities, for each pixel. In the extraction method, an edge component of an extracted image is extracted for each pixel from each of three planes corresponding to luminance and two chromaticities of the extracted image, and three planes are extracted for each pixel. A step of calculating an edge amount for each pixel by weighting and adding each edge component and an element corresponding to the number of pixels of the extracted image, and a weight value set in advance for each element Multiplying each of the elements of the region weighting map by the edge amount of the corresponding pixel of the extracted image to calculate a degree of attention for each pixel; and, in the extracted image, a pixel having a degree of attention greater than a predetermined threshold Contain everything And a step of extracting a region as a region of interest.

  Such a region-of-interest extraction method further includes a step of generating a resized image obtained by resizing the extracted image into at least one or more different dimensions, and the step of calculating the edge amount is performed for each of the extracted image and the resized image. The step of calculating the degree of attention is further configured to restore the edge amount calculated from the resized image to the same dimension as the number of pixels of the extracted image; Multiply each of the edge amount calculated from the resize image and the edge amount calculated from the resized image by each of the elements of the attention area weighting map, and each of the edge amounts multiplied by the attention area weighting map, It is preferable to include a step of multiplying a preset weight and adding each pixel to calculate a degree of attention for each pixel.

  Further, in this attention area extraction method, the step of extracting the extracted area in the step of extracting as the attention area from the extracted image, and resizing the extracted extracted image to at least one different size A step of calculating an edge amount for each image obtained by resizing the extracted image to be extracted and the extracted image to be extracted; and an edge amount calculated from an image obtained by resizing the extracted image to be extracted. A step of restoring to the dimension of the extracted image, an edge amount calculated from the extracted image to be extracted and an image obtained by resizing the extracted image to be extracted, and further restoring the restored edge amount. A step of calculating the degree of attention of the extracted image to be extracted In the step of binarizing the attention level of the extracted image as 1 when the attention level is equal to or higher than a predetermined threshold value and 0 when the attention level is smaller than the predetermined threshold value, Pixels for which 1 is set are grouped, the group including a predetermined number of pixels or more is determined as an attention area partial area, and an area including all of the attention area partial areas is extracted as a new attention area. Preferably comprising steps.

  An attention area extraction method according to the second aspect of the present invention is an attention area extraction method for extracting an attention area of an extracted image, which is a digital image expressed by luminance and two chromaticities, for each pixel, Fluctuates by obtaining the standard deviation and average value of the pixel value for each plane from each of the three planes corresponding to the brightness and two chromaticities of the image to be extracted and dividing the standard deviation by the average value. Calculating a coefficient, selecting a plane having the largest variation coefficient as a processing target plane, generating a resized image obtained by resizing the processing target plane of the extracted image into at least one or more different dimensions, In the processing target plane of the extracted image and the resized image, a step of extracting an edge component for each pixel and calculating an edge amount for each pixel; The image having the smallest dimension among the images is used as a reference image, and the edge amount calculated from the remaining extracted image and the resized image is restored to the same dimension as the number of pixels of the reference image, and the edge amount of the reference image and Each pixel of the restored edge amount has an element corresponding to the number of pixels of the reference image, and is multiplied by each element of the attention area weighting map in which a weight value is set in advance for each element. Multiplying each of the edge amounts multiplied by the area weighting map by a preset weight for each pixel and calculating the attention level for each pixel; and, in the extracted image, the attention level is a predetermined threshold value Extracting a region including all larger pixels as the region of interest.

  A program according to the present invention causes a computer to execute the above-described attention area extraction method.

  Further, the image evaluation apparatus according to the present invention stores, for each pixel, a storage unit that stores a digital image expressed by luminance and two chromaticities, and reads out the digital image from the storage unit as an extracted image. An attention area extraction unit that extracts an attention area of the extracted image by any one of the attention area extraction methods.

  When the region of interest extraction method, the program, and the image evaluation apparatus according to the present invention are configured as described above, faster region of interest can be extracted with higher accuracy, and in an embedded environment such as a digital still camera. In this case, a practical attention area can be extracted.

It is a block diagram showing the structure of an image evaluation apparatus. It is explanatory drawing which shows the data structure of a database. It is a flowchart of the image evaluation process performed with the above-mentioned image evaluation apparatus. It is a flowchart which shows the 1st attention area extraction method. It is explanatory drawing which shows the filter used for a Laplacian process, (a) shows an 8-neighbor Laplacian filter, (b) shows a 4-neighbor Laplacian filter. It is explanatory drawing for demonstrating an example of an attention area weighting map. It is explanatory drawing for demonstrating an example of the image displayed on the image display apparatus by the image evaluation process. It is a flowchart which shows the 2nd attention area extraction method. It is a flowchart which shows the 3rd attention area extraction method.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. First, the configuration of the image evaluation apparatus 100 in which the attention area extraction method according to the present embodiment is executed will be described with reference to FIG. As shown in FIG. 1, the image evaluation apparatus 100 includes a control unit 102 that controls the entire apparatus, a database 103 (storage unit) that stores image data including images and various types of information about the image, and an image. An image display unit 101 for displaying the image, an image data storage unit 104 for storing image data in the database 103 under the control of the control unit 102, and a region of interest extraction unit 105 for extracting a region of interest from the image under the control of the control unit 102. An attention area display section 106 that displays the attention area extracted by the control of the control section 102 on the image display section 101, an image storage switch (S / W) 107 that gives an instruction to store image data to the control section 102, An attention area extraction start switch (S / W) 108 which gives an instruction to start attention area extraction to the control unit 102, and an attention area to the image display section 101 Give shows instruction region of interest displayed a switch (S / W) 109 Metropolitan.

  As shown in FIG. 2, the database 103 stores an image information storage area 103 a in which the storage destination of image data, the coordinates of the attention area, and the like are stored, and an image that is stored as image data in association with the image itself and its attribute information. And a data storage area 103h. The image information storage area 103a has a data structure as shown in FIG. 2A, and as identification information (ID) for identifying image data, for example, a file name storage area 103b in which a file name is stored; An address storage area 103c in which the start address of the storage destination of the image data is stored, an image size (height) storage area 103d in which the size in the height direction (vertical direction) of the image is stored as the number of pixels, An image size (width) storage area 103e in which the size in the width direction (horizontal direction) is stored as the number of pixels; an upper left coordinate storage area 103f in which the upper left coordinates (top, left) of the extracted attention area are stored; And a lower right coordinate storage area 103g in which the lower right coordinates (bottom, right) of the attention area are stored. Note that, for these coordinates, the upper left corner of the image is the origin (0, 0), the height direction is the x axis, and the width direction is the y axis. Therefore, top and bottom indicate the relative position in the height direction from the origin in pixels, and left and right indicate the relative position in the width direction from the origin in pixels.

  On the other hand, the image data storage area 103h has a data structure as shown in FIG. 2B, and stores image storage area 103i in which the image itself is stored, and attribute information such as a header area and shooting time information. Attribute information storage area 103j, and information for one piece of image data is managed as one record. Here, the storage location (address) of the image data stored in the image data storage area 103h is stored in the address storage area 103c of the image information storage area 103a, and the image stored in the file name storage area 103b. The image data to be read is specified from the file name of the data, the start address of the storage area of this image data is extracted from the address storage area 103c, and the corresponding image data itself is acquired from the image data storage area 103h. be able to. The image storage area 103i and the attribute information storage area 103j may be configured as one file. As such a data structure, Exif (Exchangeable image file format) standardized by the Japan Electronics Industry Promotion Association (JEIDA) is known.

  Next, an image evaluation process using the attention area extraction method by the image evaluation apparatus 100 having the above configuration will be described with reference to the flowchart of FIG. This image evaluation processing is performed by the image data storage processing (steps S110 to S120) performed by the image data storage unit 104, the attention region extraction processing (step S210) performed by the attention region extraction unit 105, and the attention region display unit 106. The region-of-interest display processing (step S310) to be performed is executed by the user pressing one of the image storage switch 107, the region-of-interest extraction start switch 108, or the region-of-interest display switch 109. Is done.

  Specifically, when any one of the switches 107 to 109 is pressed, a processing start signal is transmitted from the switches 107 to 109 to the control unit 102. Receiving this, the control unit 102 determines which switch 107 to 109 is the start signal, and instructs each unit to execute processing according to the instruction of the determined start signal (steps S100, S200, and S300). . When the processes in the respective units are finished, the control unit 102 determines whether all the processes are finished (step S400), and when all the processes are finished, the image evaluation process is finished. Here, the determination of the end may be performed, for example, when an end operation such as turning off the power or pressing an end switch (not shown) is performed and the control unit 102 receives the end signal transmitted by the end operation. . While such an end operation is not performed, the control unit 102 waits for an instruction, and at the timing when the next instruction is made by pressing any switch, the control unit 102 performs image data storage processing, attention area extraction processing, or One of the attention area display processes is continued. Hereinafter, details of the processing of each unit will be described.

  First, when image data is newly stored in the image evaluation apparatus 100, a medium (for example, an SD card) storing image data is attached to an external medium reading apparatus (for example, an SD card reader) (not shown), and an image storage switch 107 is pressed. When the image storage switch 107 is pressed, the control unit 102 instructs the image data storage unit 104 to execute processing (step S100). Then, the image data storage unit 104 reads the image data from the medium via an interface (for example, an SD card interface), and assigns a file name (for example, file1.jpg) as an identification ID to the image data. The file name is stored in the image data storage area 103h of the database 103 together with the file name (step S110). Further, the image data storage unit 104 stores information relating to the stored image data in the image information storage area 103a of the database 103, and updates the database 103 (step S120). In this case, as information about the image data, the file name of the image data, the start address of the storage destination, and the size in the height direction and the width direction of the image are set in each storage area (103b to 103e). Further, the coordinates of the origin (0, 0) are stored in the upper left coordinate storage area 103f of the attention area, and the lower right coordinates (image height and image width) of the image are initially stored in the lower right coordinate storage area 103g. Set as a value.

  When the attention area extraction start switch 108 is pressed, the control unit 102 instructs the attention area extraction unit 105 to execute processing (step S200). Upon receiving this command, the attention area extraction unit 105 executes attention area extraction processing, which will be described later, for all image data stored in the image database 103, and the result (information on the extracted attention area). Are stored in the image information storage area 103a (upper left coordinate storage area 103f and lower right coordinate storage area 103g) of the corresponding image data (step S210).

  Finally, when the attention area display switch 109 is pressed, the control unit 102 instructs the attention area display section 106 to execute processing (step S300). Upon receiving this command, the attention area display unit 106 reads the image data stored in the image data storage area 103h of the database 103, and obtains the upper left and lower right coordinates of the attention area of each image from the image information storage area 103a. read out. Then, data corresponding to the region of interest specified by the upper left coordinates and the lower right coordinates is extracted from the read image data, synthesized as a predetermined display image, and then each image data is aligned with the image display unit 101. It is displayed (step S310). FIG. 7 shows a display example of image data in the image display unit 101. In the display example shown in FIG. 7, the entire image is displayed on the image display unit 101, and the attention area in each image is surrounded by a rectangle, but the display method on the image display unit 101 is limited to this. However, only the attention area may be extracted and displayed, or the entire image and the attention area may be displayed side by side.

  Then, as a detail of the attention area extraction processing executed by the attention area extraction unit 105, three methods will be described below. As described above, the attention area extraction unit 105 extracts the attention area for all the image data stored in the database 103. Hereinafter, the attention area extraction processing for one image data will be described. explain. Further, it is assumed that the image stored as image data is a digital image expressed by luminance (Y) and two chromaticities (Cb, Cr), that is, a digital image expressed by the YCbCr color system.

(First attention area extraction method)
As shown in FIG. 4, the attention area extraction unit 105 first performs attention level map creation processing (step S <b> 211). In this attention level map creation processing, first, resizing (reduction) processing of an image stored in the database 103 is performed. That is, for one extracted image (hereinafter referred to as “input image img1”) read from the image information storage area 103a of the database 103 and designated as a processing target, 1 in each of the width direction and the height direction. Generate images resized to / 2 and 1/4. Hereinafter, the image resized to ½ is called “input image img2”, and the image resized to ¼ is called “input image img3”.

  Then, a Laplacian process is performed on each YCbCr plane of the input images img1 to img3. Here, each plane of YCbCr of the input images img1 to img3 is defined as the following expressions (a1) to (a3). Note that (x, y) indicates pixel coordinates, and the definitions of the x-axis and the y-axis are as described above.

  Next, using the following formulas (b1) to (b9), the 8-neighbor Laplacian filter Δ (x, The Laplacian images (ΔY1-3, ΔCb1-3, ΔCr1-3) are obtained by performing a convolution operation on y).

  In the above formulas (b1) to (b9), the operator “*” means a convolution operation. When calculating the filter for the pixels at the top and bottom edges and the left and right edges of the image, the image is expanded by copying the rows and columns at the top and bottom edges, or the rows and columns at the left and right edges, and the computation is executed. It shall be.

  Then, for the Laplacian image of each plane obtained as described above, the following formulas (c1) to (c3) are used to calculate the edge component (contrast with adjacent pixels) of each plane for each plane. After multiplying by a predetermined coefficient, the values of the corresponding pixels of the three planes are added. Thereby, Laplacian images Δimg1, Δimg2, and Δimg3 corresponding to the input images img1 to img3 are obtained.

  In the above formulas (c1) to (c3), the operator “x” means that the pixel value of each image is multiplied by a common coefficient (wY, wCb, wCr). The coefficients (wY, wCb, wCr) are weighting coefficients (scalar values) for ΔY, ΔCb, ΔCr, respectively, and common values are used when calculating Δimg1, Δimg2, and Δimg3. Three items multiplied by the coefficient wM are obtained by subtracting the average value (represented by mean (*)) of each plane from the number of pixels of each plane (represented by abs (*)). Each coefficient is multiplied by a coefficient such as wM_Y, and the sum is multiplied by a coefficient wM. This is effective in removing the influence from the background image when calculating the attention level map obtained in the subsequent processing. It is added in anticipation.

  Further, the attention area weighting map wMap defined in advance and set in the image evaluation apparatus 100 for the Laplacian images Δimg1 to Δimg3 obtained by the above-described processing using the following expressions (d1) to (d3). To obtain attention map Sal1 to Sal3 for each Laplacian image. Thereafter, using the equation (d4) shown below, the attention level maps Sal1 to Sal3 are weighted by wd1 to wd3 and added for each corresponding pixel, whereby the input image composed of the attention level for each pixel. Is obtained.

  In the above formulas (d1) to (d3), resize (*, *) is a function for resizing (restoring to the original image size) the matrix (pixel) of the first argument by the magnification of the second argument. means. In other words, in the equations (d2) and (d3), the dimensions of the input image img1 are ½ and ¼, respectively. Therefore, these Laplacian images Δimg2 and Δimg3 are resized to the dimensions of the original image. ing. WMap is a region-of-interest weight map, and is a matrix having the same number of dimensions as Δimg1. For example, when the dimension of Δimg1 (that is, the dimension of the input image img1) is 80 × 120 (pixels), wMap similarly has a dimension of 80 × 120 (pixels). Note that wMap is a mixed Gaussian map represented by the following equations (d5) and (d6). In addition, the operator “x” for the matrix appearing in the defining expressions (d1) to (d3) of Sal1 to Sal3 means that each element of the matrix on both sides of the operator is multiplied. Therefore, inevitably, the dimensions of the two matrices must match. In the formula (d4), wd1 to wd3 are weighting factors (scalar values) for Sal1 to Sal3.

In the above formula (d5) and (d6), mu _n and gw _n represents the average vector of each Gaussian (as put index at n) to form the WMAP, and the weighting coefficients of the covariance matrix Σ respectively. Here, an example of this attention area weighting map wMap is shown in FIG. The attention area weighting map wMap shown in FIG. 6 shows a case where five Gaussian distributions are combined. The attention area weighting map wMap is generated, for example, by collecting and analyzing sample images.

  Once the attention level map is created as described above, a point of interest (POI) extraction process (step S212) is performed. Specifically, using the following formula (e1), points (pixels) having an attention level equal to or higher than a predetermined threshold th are extracted from the generated attention level map, and these are used as the attention point (POI). Define.

  When extraction of the point of interest (POI) is completed, a region of interest (ROI) determination process (step S213) is performed. First, an upper left coordinate (top, left) and a lower right coordinate of a region including all the POIs obtained by the attention point (POI) extraction process, that is, a minimum rectangular region including all the attention points (POI) in the image ( bottom, right) is determined, and this region is set as a final region of interest (ROI). Specifically, the procedures defined by the following formulas (f1) to (f5) are sequentially executed. In the following formula, x and y represent the x and y coordinates of the target point, and the subscript n represents the ID of the extracted target point.

  The values of the upper left coordinates (top, left) and lower right coordinates (bottom, right) of the region of interest (ROI) obtained by the above formulas (f2) to (f5) are the upper left coordinate storage area 103f of the image information storage area 103a. And stored in the lower right coordinate storage area 103g, and the database 103 is updated (step S214).

  As described above, two resized images obtained by resizing (reducing) one extracted image are generated as three images having different dimensions, and edge extraction processing is performed on the three planes constituting each image. Combine the images after weighting them, weight each of the three images with different dimensions with the attention area weighting map wMap, and generate the attention map Sal by weighting and combining these three images. By doing so, the accuracy of extracting the attention area can be improved.

(Second attention area extraction method)
Next, the second attention area extraction method will be described. In this method, the attention area is roughly extracted by the above-described first attention area extraction method (hereinafter referred to as “rough extraction process”), and then the attention area extraction process is performed again based on the extracted attention area. (Hereinafter referred to as “fine extraction”) to improve the accuracy of the region-of-interest extraction process.

  Details of the second attention area extraction method will be described with reference to the flowchart of FIG. As shown in FIG. 8, first, rough extraction processing (steps S <b> 221 to S <b> 223) is performed by the attention area extraction unit 105. In this process, attention level map creation processing is first performed (step S221). The details of this processing are the same as those in the first attention area extraction method. Image data is read from the database 103, and the extracted image (input image img1) designated as the processing target is processed in the width direction and the height direction. An input image img2 and an input image img3 resized to 1/2 and 1/4 in the vertical direction are created. Then, Laplacian processing is performed on the YCbCr planes of these input images img1 to img3 using the above-described equations (a1) to (a3).

  Further, similarly to the first attention area extraction method, the above-described equations (b1) to (b9) are used to determine the YCbCr planes of the input images img1 to img3 defined above with reference to FIG. The Laplacian images (ΔY1 to 3; ΔCb1 to 3; ΔCr1 to 3) are obtained by performing a convolution operation on an 8-neighbor Laplacian filter Δ (x, y) as shown in FIG.

  Then, the Laplacian image of each plane obtained in this way is multiplied by a predetermined coefficient for each pixel for each plane using the above formulas (c1) to (c3). Thus, by adding each value, Laplacian images Δimg1, Δimg2, and Δimg3 corresponding to the input images img1 to img3 are obtained. Thereafter, the first attention level map Sal of the input image is obtained using the above-described equations (d1) to (d6).

When the first attention level map Sal is created as described above, the attention point (POI) extraction process (step S222) is then performed. In this process, similarly to the first attention area extraction method, points (pixels) having an attention degree equal to or higher than the threshold th are extracted using the above-described formula (e1), and these are extracted as the first attention points ( POI). Then, in the next attention area (ROI) determination process (step S223), the first expression (f1) to (f5) described above is used based on the extracted first attention point (POI). The attention area is required. The first region of interest obtained by this rough extraction process is hereinafter referred to as “ROI _rough ”.

Next, the fine extraction process (steps S224 to S226) will be described. Fine extraction processing is performed based on the first region of interest ROI _rough extracted by the rough extraction processing. In this process, first, the attention level map is regenerated (step S224). In this case, a region corresponding to the first region of interest ROI _rough is cut out from the input image img1, and this region is set as an input image img1 _fine for fine extraction processing. At this time, if the width and height of the region defined by ROI _rough are not a multiple of 4, a copy of the lower end row and the right end column is assigned to img1 _fine . For the lower right corner, information on the lower right corner of the copy source can be used. Based on the input image img1 _fine , the input image img2 _fine and the input image img3 _fine resized (restored) to 1/2 and 1/4 in the width direction and the height direction, respectively, are generated in the same manner as the rough extraction process. . Based on these, the second attention level map Sal of the input image is obtained using the above-described equations (d1) to (d4). This series of processing is basically the same processing as the rough extraction processing, but the parameters in each equation have different values from the rough extraction processing. Further, the attention area weighting maps wMap (x, y) of the expressions (d1) to (d3) for obtaining the attention degree maps Sal1 to Sal3 are all set to 1 regardless of the values of x and y (that is, this In the fine extraction process, weighting by the attention area weighting map wMap is not performed).

When the regeneration of the second attention level map is completed, the binarization process of the second attention level map is performed (step S225). That is, for the second attention level map Sal (hereinafter referred to as “Sal _fine ”) obtained as a result of performing the attention level map regeneration process, the following is shown based on a predetermined threshold th _fine. By performing binarization processing using the expressions (g1) to (g2) (when the value of each pixel in the attention level map is equal to or greater than the threshold th _fine, “1” is set, and when smaller than that, “0” is set. And a binarized attention map (hereinafter referred to as “Sal _bin ”).

In the binarization processing of the attention degree map, next, the 4-neighbor Laplacian filter Δ (x, y as shown in FIG. 5B is applied to the Sal _bin generated by the above formula (g1) or (g2). by) to the convolution operation, the extraction of binary saliency map Sal _bin edge component (substantially equal to the contour component) is performed. Then, a labeling process is performed on the pixels having a superior value whose result of the convolution operation is not 0 by the following procedures (1) to (3).

(1) A label value is defined as λ, and λ = 0 is an initial value, and forward raster scanning is performed from the upper left on Sal _bin .
(2) When a pixel having no label value and a pixel value other than 0 is found in the scanning process, the pixel is labeled. At this time, the label value λ to be applied is changed as follows according to the label value assigned to the already scanned eight neighboring pixels.
(I) When the pixel values of all the eight neighboring pixels that have been scanned are 0, the label value λ is incremented by 1 (λ = λ + 1), and the label value λ is given to the pixel.
(Ii) When there is only one type of label value, the label value λ is given to the pixel without updating λ.
(Iii) When there are two types of label values (λ, λ ′, λ <λ ′), a label value λ is assigned to the pixel, and further, a pixel to which a label value of λ ′ is assigned to the scanned pixel Are all changed to the label value λ.
(3) The process of (2) is performed on all pixels on Sal _bin .

Next, in the binarization process of the attention level map, the labeling result is evaluated. First, after the labeling process for all pixels not zero on Sal _bin as described above, for each label value, it counts the number n _lambda of pixels that are granted. This n _λ is evaluated by a predetermined threshold th _λ , and a label in which n _λ is larger than th _λ is extracted. Thereafter, the pixel (set) to which the label is _assigned is defined as the attention area partial area POI _fine .

When the binarization processing of the attention level map ends, next, the attention region (ROI) determination processing (step S226) is performed. In this process, the upper left coordinates (top, left) and the lower right coordinates (bottom, right) of the minimum rectangular area including all the attention area partial area POI _fine obtained as described above are obtained, and this area is the second attention area. This is the final region of interest (ROI) in the region extraction method. Specifically, the procedures defined by the above formulas (f1) to (f5) are sequentially executed. The values of the upper left coordinates (top, left) and lower right coordinates (bottom, right) thus determined are stored in the upper left coordinate storage area 103f and the lower right coordinate storage area 103g of the image information storage area 103a, and are stored in the database. 103 is updated (step S227). As described above, according to the second attention area extraction method, the attention area extraction process is performed in two stages of the rough extraction process and the fine extraction process, so that the attention area can be extracted with higher accuracy.

(Third attention area extraction method)
Next, a third attention area extraction method will be described. In the first attention area extraction method, all three planes of YCbCr are evaluated when creating the attention level map. However, in the third attention area extraction method, only one of the YCbCr planes is processed. As a target, attention areas are extracted after creating an attention level map. Thus, in the third attention area extraction method, the extraction process of the attention area is accelerated.

The attention area extraction processing in the attention area extraction unit 105 will be described below with reference to the flowchart of FIG. As shown in FIG. 9, first, the attention area extraction unit 105 calculates a variation coefficient and selects a processing target plane (step S <b> 231). In this process, first, image data is read from the database 103, and the variation coefficient of each plane is calculated for the extracted image (input image img1) designated as the processing target. Here, when the variation coefficient of the Y plane is defined as V _Y , the variation coefficient of the Cb plane is defined as V _Cb , and the variation coefficient of the Cr plane is defined as V _Cr , these variation coefficients are expressed by the following equations (h1) to (h3). ).

  In the above formulas (h1) to (h3), sd (*) means an operation for obtaining the standard deviation of the edge component of the pixel in the plane, and mean (*) obtains the average value of the edge component of the pixel in the plane. Means an operation.

  When the variation coefficient is obtained in this way, the processing target plane SP is selected using the following equation (i1).

In the above formula (i1), max (*) is a function that returns the maximum element as an argument, and arg determines whether the return value of max (*) is a plane of V _Y , V _Cb , or V _Cr. It is an operator. That is, the plane with the largest variation coefficient is selected as the processing target plane. For example, when the variation coefficient V _Y of the Y plane is the largest value among V _Y , V _Cb , and V _Cr , the processing target plane is the Y plane. Hereinafter, this processing target plane is referred to as “SP”. For example, the processing target plane selected for the input image img1 is referred to as “SP1”.

  When the selection of the processing target plane SP is completed, an attention level map creation process is performed on the processing target plane SP (step S232). First, the input image is resized. The input image img2 and the input image img3, which are resized to 1/2 and 1/4 in the width direction and the height direction, respectively, with respect to the input image img1 designated as the processing target. Generated. Note that the processing target planes of the input image img2 and the input image img3 are referred to as “SP2” and “SP3”, respectively.

  Next, an 8-neighbor Laplacian filter Δ as shown in FIG. 5A is applied to the processing target planes SP1 to SP3 of the input images img1 to img3 using the following expressions (b1 ′) to (b3 ′). A Laplacian image (ΔSP1 to ΔSP3) is obtained by performing a convolution operation on (x, y).

  In the above formulas (b1 ′) to (b3 ′), the operator “*” means a convolution operation. When calculating the filter for the pixels at the top and bottom edges and the left and right edges of the image, the image is expanded by copying the rows and columns at the top and bottom edges, or the rows and columns at the left and right edges, and the computation is executed. It shall be.

  Then, by using the following formulas (d1 ′) to (d3 ′), the Laplacian images Δimg1 to Δimg3 obtained as described above are multiplied by a preset attention area weighting map wMap, Attention level maps Sal1 to Sal3 for each Laplacian image are obtained. Thereafter, using the equation (d4) shown below, Sal1 to Sal3 are weighted by wd1 to wd3 and added for each pixel, thereby obtaining an attention level map Sal for the entire input image.

  In the above formulas (d1 ′) to (d3 ′), “resize (*, *)” means a function for resizing the matrix (pixel) of the first argument by the magnification of the second argument. That is, in the first attention area extraction method, the input image img1 which is the extracted image is resized to the dimension, but in the third attention area extraction method, the input image img3 having the smallest dimension is used as the reference image. The remaining input images img1 and img2 are resized to the dimensions of this reference image. Therefore, the attention area weighting map wMap is a matrix having the same number of dimensions as the reference image (Δimg3). For example, when the dimension of Δimg3 (that is, the dimension of the input image img3) is 20 × 30 (pixels), wMap similarly has a dimension of 20 × 30 (pixels). The meaning of the operator of each expression is as described in the description of the expressions (d1) to (d4). Note that wMap is a mixed Gaussian map represented by the above equations (d5) and (d6) (see FIG. 6).

Once the attention level map is created as described above, a point of interest (POI) extraction process (step S233) is performed. Using the following formula (e1 ′), points having an attention level equal to or higher than a predetermined threshold th _SP are extracted from the generated attention level map Sal, and defined as attention points (POI).

Note that the threshold th _SP has a different value depending on whether the selected processing target plane is Y, Cb, or Cr. This threshold th _SP may be a fixed value determined in advance for each plane, or may be obtained dynamically using equations (j1) and (j2) shown below. In the above formula (j1), V _SP is a coefficient of variation corresponding to S, and param _SP is expressed by the following formula (j2) and means a predetermined parameter having a fixed value according to SP. To do.

  When the extraction process of the attention point (POI) is completed, in the next attention area (ROI) determination process (step S234), based on the extracted attention point (POI), the above formulas (f1) to (f1) to ( Using f5), the upper left coordinates (top, left) and upper right coordinates (bottom, right) of the region of interest ROI are obtained. As described above, since the dimensions of the reference image (the image having the smallest dimension and in the above case, the input image img3) are resized, the dimensions of the reference image and the extracted image (input image img1) are the same. If they are different, it is necessary to convert (restore) the coordinates of the attention area ROI into the dimensions of the extracted image.

  The values of the upper left coordinates (top, left) and lower right coordinates (bottom, right) thus determined are stored in the upper left coordinate storage area 103f and the lower right coordinate storage area 103g of the image information storage area 103a, and are stored in the database. 103 is updated (step S235). According to the third attention area extraction method, it is possible to extract the attention area at a higher speed by using only one YCbCr plane as a processing target.

  The image evaluation apparatus 100 includes a central processing unit (CPU), a memory, and the like, and the above-described image evaluation process and the attention area extraction method executed in the image evaluation process are programs executed by the CPU. Can be implemented. Here, this program is stored in a storage medium or storage device such as a flexible disk, a CD-ROM, a magneto-optical disk, a semiconductor memory, or a hard disk. In some cases, it may be distributed as a digital signal via a network or the like. At this time, intermediate processing results are temporarily stored in a storage device such as a main memory. Alternatively, the above processing can be configured as a logic circuit in an ASIC, DSP, or the like without providing a special CPU or memory. The above-described image evaluation process and the attention area extraction method executed in this image evaluation process are performed when determining an automatic zoom center in a slide show, or when performing automatic trimming with image editing software, The present invention can be applied to the case where an automatic exposure (AE) determination area is determined, but is not limited thereto.

DESCRIPTION OF SYMBOLS 100 Image evaluation apparatus 103 Database 105 Region of interest extraction part

Claims (6)

An attention area extraction method for extracting an attention area of an extracted image, which is a digital image expressed by luminance and two chromaticities, for each pixel,
For each pixel, an edge component of the extracted image is extracted from each of the three planes corresponding to the luminance and two chromaticities of the extracted image, and each of the three planes is extracted for each pixel. Calculating the edge amount for each pixel by weighting and adding the edge components of:
Each of the elements of the attention area weighting map having elements corresponding to the number of pixels of the extracted image, and having a weight value set in advance for each of the elements, the edge amount of the pixel corresponding to the extracted image Multiplying to calculate the degree of attention for each pixel;
A region of interest extraction method comprising: extracting, from the image to be extracted, a region including all pixels having a degree of attention greater than a predetermined threshold as the region of interest. Generating a resized image obtained by resizing the extracted image into at least one or more different dimensions;
The step of calculating the edge amount is configured to calculate the edge amount for each of the extracted image and the resized image,
The step of calculating the attention level further includes:
Restoring the edge amount calculated from the resized image to the same dimension as the number of pixels of the extracted image;
Each of the edge amount calculated from the extracted image and the edge amount calculated and restored from the resized image is multiplied by each of the elements of the attention area weighting map, and further, the attention area weighting map The attention area according to claim 1, further comprising: multiplying each of the edge amounts multiplied by a predetermined weight for each pixel and calculating the attention degree for each pixel. Extraction method. further,
Cutting out the region extracted in the step of extracting as the region of interest from the extracted image;
Resizing the extracted image to be extracted to at least one different size;
Calculating the edge amount for each image obtained by resizing the extracted image to be extracted and the extracted image to be extracted; and
Restoring an edge amount calculated from an image obtained by resizing the extracted image to be extracted to a dimension of the extracted image to be extracted;
The edge amount calculated from the extracted image to be extracted and the edge amount calculated from the image obtained by resizing the extracted image to be extracted and further restored by multiplying a preset weight by the pixel amount. And calculating the degree of attention of the extracted image to be extracted,
Binarizing the degree of attention of the extracted image to be extracted with 1 when the degree of attention is greater than or equal to a predetermined threshold and 0 when smaller than the predetermined threshold;
In the binarized attention level, adjacent pixels set to 1 are grouped, the group including a predetermined number of pixels or more is determined as the attention area partial area, and the attention area partial area The region of interest extraction method according to claim 2, further comprising a step of extracting a region including all as a new region of interest. An attention area extraction method for extracting an attention area of an extracted image, which is a digital image expressed by luminance and two chromaticities, for each pixel,
From each of the three planes corresponding to the luminance and two chromaticities of the extracted image, a standard deviation and an average value of the pixel values are obtained for each plane, and the standard deviation is divided by the average value. Calculating a coefficient of variation, and selecting the plane having the largest coefficient of variation as a processing target plane;
Generating a resized image in which the processing target plane of the extracted image is resized to at least one or more different dimensions;
Extracting an edge component for each pixel and calculating an edge amount for each pixel in the processing target plane of the extracted image and the resized image;
Of the extracted image and the resized image, an image having the smallest dimension is set as a reference image, and the edge amount calculated from the remaining extracted image and the resized image is set to the same dimension as the number of pixels of the reference image. Steps to restore,
An attention area weighting map in which each pixel of the edge amount of the reference image and the restored edge amount has an element corresponding to the number of pixels of the reference image, and a weight value is set in advance for each element. Each of the elements is multiplied, and each of the edge amounts multiplied by the attention area weighting map is multiplied by a preset weight and added for each pixel, and the degree of attention for each pixel is obtained. A calculating step;
A region of interest extraction method comprising: extracting, from the image to be extracted, a region including all pixels having a degree of attention greater than a predetermined threshold as the region of interest.   The program for making a computer perform the attention area extraction method as described in any one of Claims 1-4. A storage unit for storing a digital image expressed by luminance and two chromaticities for each pixel;
An attention area extracting section that reads out the digital image from the storage section as an extracted image and extracts the attention area of the extracted image by the attention area extraction method according to claim 1. Image evaluation device.