JP5006839B2 - Trimming rule learning apparatus and method, and program - Google Patents

Description

  The present invention relates to a trimming rule learning device and method for learning a trimming rule when automatically trimming an image, and a program for causing a computer to execute a trimming rule learning method.

  When shooting, the camera is set so that a desirable composition is obtained. However, since it takes skill to appropriately determine the composition at the time of shooting, an image obtained by shooting always has a desired composition. Often not a thing. For example, there may be a case where an irrelevant subject is included in the image, or a subject that is desired to be included in the image is positioned at the end of the image. For this reason, trimming is performed so that a partial region of an image acquired by photographing has a desired composition.

  Trimming can be performed by manually cutting out a desired region while viewing the image, but if the number of images is large, the operation becomes very troublesome.

  For this reason, various methods for automatically trimming an image have been proposed (see Patent Documents 1 and 2). The method described in Patent Document 1 calculates the attention level of an attention area of an image and an object index using a predefined model, and performs a trimming technique based on the attention level, the object index value, and a predefined trimming rule. It is determined and the image is trimmed.

The method described in Patent Document 2 learns the user's individual sensibility and uses the learning result to perform trimming according to the user's sensibility from the image.
Japanese Patent Application Laid-Open No. 2004-228995 JP 2006-134153 A

  However, Patent Document 1 does not disclose how to learn trimming rules. Further, since the technique described in Patent Document 2 performs trimming by learning the user's sensibility, the composition of the trimming region may not necessarily be the composition desired by the user.

  The present invention has been made in view of the above circumstances, and an object thereof is to enable trimming to achieve a composition desired by a user.

The trimming rule learning device according to the present invention includes display means for displaying images classified in advance for each object included therein,
Input means for accepting designation of a trimming region by the user for the displayed image;
Learning means for learning a trimming rule for an image for each object on a user basis based on the composition of the trimming region for a plurality of images is provided.

  In the trimming rule learning device according to the present invention, the learning means extracts an object included in the trimming area, and based on the composition of the trimming area, the position of the object in the original image before trimming, trimming The position of the object in a region, the area ratio of the object with respect to the original image, and the area ratio of the object with respect to the trimming region may be acquired as composition information, and the composition information may be learned as the trimming rule.

  In this case, the learning unit may be a unit that extracts a region of interest in the trimming region as the object.

The trimming rule learning method according to the present invention displays images classified in advance for each included object,
Accepting a user to specify a trimming area for the displayed image;
Based on the composition of the trimming area for a plurality of images, a trimming rule for the image is learned for each object for each user.

  In addition, you may provide as a program for making a computer perform the trimming rule learning method by this invention.

  According to the present invention, an image classified in advance for each object included is displayed, and a user designates a trimming area for the displayed image. Then, based on the composition of the trimming area for a plurality of images, a trimming rule for the image is learned for each object for each user. Therefore, by automatically trimming an image using the learning result of the trimming rule, it is possible to trim an object included in the image so as to have a composition desired by the user.

  Further, by extracting the attention area in the trimmed area as an object, the object can be easily extracted.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a trimming rule learning device according to an embodiment of the present invention. As shown in FIG. 1, the trimming rule learning device 1 according to the present embodiment includes an image recording unit 2 that records a plurality of images to be learned, a display unit 3 such as a liquid crystal monitor that displays images, and various instruction inputs. An input unit 4 including a keyboard and a mouse for performing, a learning unit 5 for learning trimming rules, and a control unit 6 for controlling each unit are connected to each other by a bus 7.

  A plurality of images to be learned are recorded in the image recording unit 2, and each image is classified in advance for each object included. For example, it is classified for each object including sky, flower, sea, mountain, group photo, front face portrait, oblique face portrait, signboard, sign, building, landmark, and the like. The classification may be performed by manual operation while viewing the image, or may be performed automatically by performing object recognition. An image including a plurality of objects is classified across a plurality of groups.

  The learning unit 5 includes an object extraction unit 8 and a learning result database DB1 that registers trimming rules that are learning results.

  As will be described later, the object extraction unit 8 extracts a region of interest within a trimming region designated by the user as an object. Here, the object extraction unit 8 extracts a portion that is noticed when the image is visually confirmed as an attention region, that is, an object. For example, when a part of the color on the image is different from the surrounding colors, a part of the image is very bright compared to the surroundings, or a straight line that appears on a flat screen is noticed when viewing the image. It becomes an area. For this reason, the object extraction unit 8 determines the degree to which the feature of each part in the image is different from the feature of the part located around the part based on the color and brightness of the image and the direction of the linear component appearing in the image. In search of these, areas where these different degrees are large are extracted as attention areas, that is, objects.

  In this way, the region of interest that is visually noticeable has elements that make up the image, such as color, brightness, and linear components that appear in the image, different from the surroundings. Therefore, using the color of the image (Color), the intensity of the image (Intensity), and the direction of the linear component appearing in the image (Orientation), the characteristics of each part in the image are the characteristics of the part located around that part. And a portion having a large difference degree can be extracted as a region of interest that is visually noticed. Specifically, using the method proposed by Itti and Koch et al., It is possible to automatically extract a visually noticeable area of interest (for example, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 20 , NO. 11, NOVEMBER 1998 “A Model of Saliency-Based Visual Action for Rapid Scene Analysis”, Laurent Itti, Christof Koch and Emst Neibur).

  A flow of processing for extracting a region of interest using this method will be described with reference to FIG.

  First, linear filtering processing is performed on the image to generate an image representing brightness and an image of color components divided into a plurality of color components (Step 1).

  Specifically, a brightness image I is generated from the image, and a Gaussian pyramid of the brightness image I is further generated. An image of each layer of this Gaussian pyramid is assumed to be I (σ) (σ represents a pixel scale, and σ∈ [0..8]).

  Next, the image is divided into four color component images R (red), G (green), B (blue), and Y (yellow). Further, four types of Gaussian pyramid are generated from these images R, G, B, and Y, and the images of each layer are defined as R (σ), G (σ), B (σ), and Y (σ).

  Therefore, the difference between the feature of each part on the image and the feature of the surrounding part is calculated from these images I (σ), R (σ), G (σ), B (σ), and Y (σ). Thus, a feature amount map is generated (Step 2).

A place where the brightness of a certain part on the screen is different from the brightness of the surrounding area is a place where a dark part exists in a place where the surroundings are bright, or a place where a bright part exists in a place where the surroundings are dark. Therefore, how much the lightness of the central portion is different from the surrounding lightness is obtained using the image I (c) represented by fine pixels and the image I (s) represented by rough pixels. Since the value of one pixel of the rough image I (s) represents a value obtained by collecting a plurality of pixels of the fine image I (c), the value of the pixel (center portion) of the image I (c) And the value of the pixel of the image I (s) at the position corresponding to this pixel (surrounding brightness) (referred to as center-surround) It can be determined whether the degree is different. For example, the scale of the image I (c) represented by fine pixels is cε {2, 3, 4}, and the scale of the image I (s) represented by rough pixels is s = c + δ (δε {3, 4}), a brightness feature map M _I (c, s) is obtained. The brightness feature map M _I (c, s) is expressed as in the following equation (1).

Similarly, for each color component, a feature map is generated from R (σ), G (σ), B (σ), and Y (σ). A place where the color of a certain part on the screen is detected as different from the surrounding color can be found from a combination of colors (opposite colors) positioned in opposite directions in the hue circle. For example, the feature map M _RG (c, s) is acquired from the combination of red / green and green / red, and the feature map M _BY (c, s) is acquired from the combination of blue / yellow and yellow / blue. This color feature map is represented by the following equations (2) and (3).

Further, with respect to the direction of the linear component appearing on the image, the part where the difference between the direction of the linear component appearing in each part and the surrounding linear component is detected detects the direction of the linear component from the brightness image I. It can be found using a Gabor filter. A Gabor filter is used for images in each layer of I (σ) to detect a linear component in each direction of θ∈ {0 °, 45 °, 90 °, 135 °}, and a feature map M _O (c, s, θ). The feature map in this direction is expressed as the following equation (4).

  If c∈ {2, 3, 4} and s = c + δ (δ∈ {3,4)}, there are 6 brightness feature maps, 12 color feature maps, and direction feature maps. Twenty-four maps are obtained, and these maps are viewed comprehensively to extract a region of interest that is visually noted.

These 42 pieces of feature maps _{M I, M RG, M BY} , M O is the difference of information that difference and extraction of the dynamic range, which does not appear the parts and so large that those surrounding the difference appears greater There is. Therefore, if the determination is made using the values of the 42 feature maps M _I , M _RG , M _BY , and M _O as they are, the feature map information with a small difference is reflected by the feature map having a large difference. May not be. Therefore, these 42 amino feature maps _{M I, M RG, M BY} , combined after normalizing _{M O,} it is preferable to extract the region of interest.

Specifically, for example, a brightness element-specific attention map M ^C _I obtained by standardizing and combining six brightness characteristic maps M _I (c, s) is acquired, and a color feature map M _RG (c, s) and 12 of M _BY (c, s) are standardized and combined to obtain a color element-specific attention map M ^C _C, and 24 feature map M _O (c, s, θ) related to direction are obtained to obtain the orientation of the elemental saliency map ^M _{C O} which is combined normalized (Step3). Further, the attention level map M ^S representing the distribution of the attention level of each part of the image is obtained by linearly combining the element-specific attention level maps M ^C _I , M ^C _C and M ^C _O (Step 4). A region where the degree of attention exceeds a predetermined threshold Th1 is extracted as a region of interest, that is, an object (Step 5).

In addition, when extracting the region of interest, the color, brightness, and slope of the linear component that appears in the image are changed so that the influence of the degree of the color component, brightness, and the slope of the linear component that appears in the image differ from the surroundings. The attention area to be extracted can be changed by changing the degree of each and the weighting degree weighted for each degree. For example, the attention area extracted can be changed by changing the weight when linearly combining the element-specific attention maps M ^C _I , M ^C _C , and M ^C _O. Alternatively, when the element-specific attention maps M ^C _I , M ^C _C , and M ^C _O are acquired, the feature map M _I (c, s) for each brightness, the color feature map M _RG (c, s), M _BY (c, s), to alter the effect of each map feature with respect to the direction map _{M O (c, s, θ} ), each feature map _{M I (c, s),} M RG (c, s), The weights for M _BY (c, s) and M _O (c, s, θ) may be changed.

  Note that the method of extracting the attention area, that is, the object is not limited to the above-described method, and any known method can be used.

  The control unit 6 includes a CPU, a RAM serving as a work area, and a ROM storing a program for operating the trimming rule learning device 1.

  In addition, the process performed at the time of learning is demonstrated in the operation | movement of this embodiment below.

  FIG. 3 is a flowchart showing processing performed at the time of learning performed in the present embodiment. Note that learning is performed on a user-by-user basis, and it is assumed that the user ID of the user who performs the learning is input to the trimming rule learning device 1 from the input unit 4 in advance. When the instruction to start learning is given from the input unit 4 by the user, the control unit 6 starts processing, and displays an image to be learned on the display unit 3 (step ST1). Note that the order in which the learning target images are displayed may be classified object units or random. Next, the control unit 6 starts monitoring whether or not the trimming area has been designated by the user (step ST2). If step ST2 is affirmed, the object extraction unit 8 of the learning unit 5 extracts the attention area of the trimming area designated by the user (step ST3).

  FIG. 4 is a diagram for explaining trimming, and FIG. 5 is a diagram for explaining object extraction. As shown in FIG. 4, an image before trimming (hereinafter referred to as an original image) S0 includes a flower, and a trimming area designated by the user is denoted as T0. In this case, the subject included in the trimming area T0 is a part of a mountain that is a flower and a background, but the flower is an attention area. For this reason, a flower region is extracted as an object. Note that, compared with the region A0 surrounded by the contour shape including the petals (indicated by a broken line in FIG. 5), the region of interest is a region A1 corresponding to only the central portion of the flower. The object extraction unit 8 extracts a region of interest A1 corresponding to the central portion of the flower as an object O1.

  Next, the learning unit 5 calculates the position of the object in the original image S0, the position of the object in the trimming region T0, the area ratio of the object with respect to the original image S0, and the area ratio of the object in the trimming area T0 based on the composition of the trimming area T0. Then, it is acquired as composition information representing the composition of the trimming area T0 (step ST4).

  Here, the position of the object O1 in the original image S0 is distances L1 to L4 from the upper, lower, left, and four sides of the original image S0 of the object O1, as shown in FIG. The position of the object O1 in the trimming area T0 is the distances LT1 to LT4 from the four upper and lower left and right sides of the trimming area T0 of the object O1 as shown in FIG. The area ratio H1 of the object O1 with respect to the original image S0 is the area of the object O1 with respect to the area of the original image S0, and is expressed as a percentage, for example, 10%. The area ratio H2 of the object in the trimming region T0 is the area of the object O1 with respect to the area of the trimming region T0, and is expressed as a percentage, for example, 20%.

  And the learning part 5 learns a trimming rule by updating the trimming rule which is a learning result with the acquired composition information (step ST5). Note that the trimming rule update is the average of the composition information included in the trimming rule for a specific object registered in the learning result database DB1 up to the present time and the newly acquired composition information for the user who is currently learning. Is calculated. For example, as shown in FIG. 8, when the trimming rule for a certain object registered in the learning result database DB1 obtained by only one composition information is Rold and the newly obtained composition information is K0, the trimming rule Rold The updated trimming rule Rnew is obtained by calculating the average value of the value of each composition information and the value of the newly acquired composition information K0. In addition to calculating the average, the trimming rule R may be updated by storing all the composition information acquired for the learning target image.

  Next, the control unit 6 determines whether or not learning has been performed for all images recorded in the image recording unit 2 (step ST6). If step ST6 is negative, the learning target is changed to the next image. (Step ST7), the process returns to Step ST1. If step ST6 is affirmed, the learning of the trimming rule for the user is terminated.

  FIG. 9 is a diagram showing a trimming rule learning result database. As shown in FIG. 9, user IDs (001, 002...) Of a plurality of users are registered in the learning result database DB1, and each user ID has a sky, a flower, the sea, a mountain, a group photo, and a front face. Objects such as portraits and oblique face portraits are registered. In each object, a trimming rule obtained by learning is registered.

  FIG. 10 is a schematic block diagram illustrating a configuration of a trimming apparatus that trims an image using the trimming rules learned by the trimming rule learning apparatus 1 according to the present embodiment. As shown in FIG. 10, the trimming apparatus 20 includes a recording control unit 22 that controls reading of an image from a medium 21 on which an image to be trimmed is recorded and recording of the image on the medium 21, and the above-described trimming rule learning. A learning result database DB1 in which trimming rules learned by the apparatus 1 are registered, a trimming unit 23 that performs trimming, a display unit 24 such as a liquid crystal monitor that performs various displays, an input unit 25 that performs various inputs, and each unit And a control unit 26 that controls the above-described components, and each unit is connected by a bus 27.

  The control unit 26 includes a CPU, a RAM serving as a work area, and a ROM that stores a program and the like for operating the trimming rule learning device 1.

  Hereinafter, processing performed in the trimming apparatus 20 will be described. FIG. 11 is a flowchart showing a trimming process performed by the trimming apparatus 20. When the user gives an instruction to start trimming from the input unit 25, the control unit 26 starts processing, and receives the user ID and the designation of the image to be trimmed from the input unit 25 (step ST11). Then, the processing target image S1 designated by the recording control unit 22 is read from the medium 21 (step ST12), and the trimming unit 23 recognizes an object included in the processing target image S1 (step ST13). Note that the user may input the type of the object included in the image to be trimmed from the input unit 25 without recognizing the object.

  Then, the trimming unit 23 refers to the learning result database DB1 based on the user ID and the recognition result of the object, and acquires a trimming rule corresponding to the user who is working and the object included in the image to be trimmed (step) ST14). When the learning result database DB1 has acquired all the composition information for the learning target image, the composition information is selected at random, and the selected composition information is acquired as a trimming rule. Then, the trimming unit 23 trims the image S1 to be processed based on the acquired trimming rule (step ST15), displays the trimming result on the display unit 24 (step ST16), and ends the process.

  As described above, in the present embodiment, an image classified in advance for each included object is displayed, the designation of the trimming area by the user for the displayed image is received, and based on the composition of the trimming area for a plurality of images, A trimming rule for an image is learned for each object on a user basis. For this reason, by automatically trimming an image using the learning result database DB1 which is a learning result, an object included in the image can be trimmed so as to have a composition desired by the user.

  The apparatus 1 according to the embodiment of the present invention has been described above. However, a program that causes a computer to function as a unit corresponding to the learning unit 5 and performs processing as illustrated in FIG. 3 is also an embodiment of the present invention. It is one of. A computer-readable recording medium in which such a program is recorded is also one embodiment of the present invention.

1 is a schematic block diagram showing the configuration of a trimming rule learning device according to an embodiment of the present invention. A diagram for explaining a method of extracting a region of interest, that is, an object The flowchart which shows the process performed at the time of the learning performed in this embodiment Diagram for explaining trimming Diagram for explaining object extraction The figure for demonstrating the position of the object in an original image The figure for demonstrating the position of the object in a trimming area | region Diagram for explaining trimming rule learning The figure which shows learning result database DB1 Schematic block diagram showing a configuration of a trimming apparatus that trims an image using a trimming rule learned by the trimming rule learning apparatus according to the present embodiment Flow chart showing trimming process

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Trimming rule learning apparatus 2 Image recording part 3 Display part 4 Input part 5 Learning part 6 Control part 7 Bus 8 Object extraction part DB1 Learning result database

Claims (4)

Display means for displaying pre-classified images for each included object;
Input means for accepting designation of a trimming region by the user for the displayed image;
Based on the composition of the trimming area for a plurality of images, when learning the trimming rule for the image for each object in units of users, the object included in the trimming area is extracted, and based on the composition of the trimming area, The position of the object in the original image before trimming, the position of the object in the trimming area, the area ratio of the object to the original image, and the area ratio of the object to the trimming area are obtained as composition information, and the composition information A trimming rule learning device comprising: learning means for learning as a trimming rule. Said learning means, trimming rule learning device according to claim 1, characterized in that the means for extracting a region of interest of the trimming area as the object. Display pre-classified images for each included object,
Accepting a user to specify a trimming area for the displayed image;
Based on the composition of the trimming area for a plurality of images, when learning the trimming rules for the image for each object in units of users, the objects included in the trimming area are extracted,
Based on the composition of the trimming area, the position of the object in the original image before trimming, the position of the object in the trimming area, the area ratio of the object to the original image, and the area ratio of the object to the trimming area Acquired as composition information,
A trimming rule learning method comprising learning the composition information as the trimming rule . A procedure for displaying pre-classified images for each included object;
A procedure for accepting designation of a trimming region by the user for the displayed image;
A procedure for extracting an object included in the trimming area when learning a trimming rule for the image for each object on a per-user basis based on the composition of the trimming area for a plurality of images ;
Based on the composition of the trimming area, the position of the object in the original image before trimming, the position of the object in the trimming area, the area ratio of the object to the original image, and the area ratio of the object to the trimming area The procedure to get as composition information,
A program for causing a computer to execute a trimming rule learning method comprising learning the composition information as the trimming rule .

Images