JP5962491B2 - Angle of view adjustment apparatus, method, and program - Google Patents

Description

  The present invention relates to an angle-of-view adjusting apparatus, method, and program for adjusting an angle of view for a main subject in an image.

  Sometimes you want to know the name of a flower you saw on Noyama or a roadside. Therefore, a flower image as an object is extracted from a digital flower image obtained by photographing or the like using a clustering method, and information obtained from the extracted flower image is used as a feature amount. Proposed a technique to determine the type of wild grass by calculating one or more feature quantities and analyzing the obtained feature quantities and the feature quantities of various plants registered in the database in advance using statistical methods. (For example, the technique described in Patent Document 1).

  In addition, a conventional technique is known in which an image including a main subject is divided into a main subject and a background using the Graph Cuts method (for example, techniques described in Non-Patent Document 1 and Patent Document 2). When region segmentation is performed, there may be a portion where the boundary is unclear due to the relationship between the main subject and the background, and it is necessary to perform segmentation optimally. Therefore, in this prior art, region division is regarded as an energy minimization problem, and a minimization method is proposed. In this prior art, the energy function is minimized by creating a graph that matches the region division and obtaining the minimum cut of the graph. This minimum cut realizes efficient area division calculation by using a maximum flow algorithm.

  In the method of dividing the main subject and the background using the Graph Cuts method, the region label indicating the main subject or background to be given to each pixel in the image is updated, and the region is based on the region label and the pixel value of each pixel. A technique for performing division is known. In this case, for example, an energy function including the following cost term is defined. First, for example, a cost term that includes a cost value that decreases as the value of a histogram for each color pixel value calculated from an image showing the main subject increases is included. Further, for example, a cost term that decreases as the value of the histogram for each color pixel value calculated from the image indicating the background increases is included. Then, the main subject and the background are divided into regions in the image by the energy function minimization process (for example, the method described in Non-Patent Document 1).

Further, it may be difficult to divide the main subject and the background only by the Graph Cuts method. For this reason, for example, in mounting on a so-called smartphone or the like, for example, an input device such as a touch panel is used for an approximate region where an object (for example, a flower) to be recognized exists for an image captured by the imaging device. The function to specify the rectangular frame is implemented.
Alternatively, the user adjusts the angle of view of the camera so that the recognized object is large within a predetermined rectangular frame.

Japanese Patent Laid-Open No. 2002-203242 JP 2011-35636 A

Y. Boykov and G. Funka-Lea: “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in ND Images”, Proceedings of “Interference Convence CV”. I, p. 105-112, July 2001.

However, as described above, the operation of designating the rectangular frame by the user and the operation of adjusting the angle of view of the camera so that the recognized object is large within the predetermined rectangular frame are both cumbersome and easy to operate. It had the problem of losing.
On the other hand, if the rectangular frame and the angle of view are specified appropriately, there is a problem that the identification accuracy of a flower image or the like is lowered.

  An object of the present invention is to make it possible to adjust the angle of view with respect to a main subject with a simple operation.

An example of the aspect is an angle-of-view adjusting device that adjusts an angle of view with respect to the main subject so that the main subject has an appropriate angle of view in the image, the position specifying unit for specifying the center position of the target angle of view, A first region that divides the main subject and the background other than the main subject into regions based on each pixel value in the image and divides the surrounding pixels at the center position into the regions of the main subject. A second area division process for repeatedly executing a division process and a process for increasing the degree of area division for each pixel around the main subject divided into the main subject area until a predetermined end condition is reached; And a circumscribing rectangular frame surrounding the main subject divided by the second region dividing process as a reference so that the circumscribing rectangular frame fits within the image. Up, the the angle adjusting means for adjusting the angle of view with respect to the main object, provided with a shortest line segment of the four line segments extended toward the upper and lower left and right edges of the image from the center position The zoom area is a rectangle that is widened from the center position in the up, down, left, and right directions by the length of the center area, and zoom-in is performed so that the center ratio of the zoom area is maintained while maintaining the aspect ratio of the zoom area. An initial zoom means for setting an initial zoom area after the zoom-up, the adjustment of the angle of view is performed on the initial zoom area, and the area of the main subject divided by the second area dividing process The angle-of-view adjusting apparatus is characterized in that the predetermined end condition is that any one of the above reaches the end of the initial zoom area.

  According to the present invention, it is possible to improve the accuracy of area division.

It is a functional block diagram of an angle-of-view adjustment device according to an embodiment of the present invention. It is operation | movement explanatory drawing of the angle-of-view adjustment apparatus of FIG. It is a block diagram which shows the hardware structural example of the angle-of-view adjustment apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the whole operation | movement of a view angle adjustment process by this embodiment. It is explanatory drawing of an initial zoom process. It is explanatory drawing of an end determination. It is explanatory drawing of the last zoom process. It is explanatory drawing of a weighted directed graph. It is explanatory drawing of histogram (theta). It is a characteristic view of h _uv (X _u , X _v ). It is the figure which showed typically the relationship between the graph which has t-link and n-link, the area | region label vector X, and the graph cut. It is a flowchart which shows the process of the data update. It is a flowchart which shows the process of the data update.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a functional block diagram of an angle of view adjusting apparatus according to an embodiment of the present invention.

  The present embodiment is an angle-of-view adjusting device that adjusts an angle of view with respect to a main subject so that the main subject has an appropriate angle of view in an image.

  The position designation unit 101 designates the center position of the target angle of view. For example, the position specifying unit 101 is a unit that allows the user to touch-specify the center position of the target angle of view with a finger or the like on a display with a touch input function of a smartphone or a digital camera that displays an image.

  The first area dividing unit 102 executes a process of dividing the main subject and the background other than the main subject into areas based on each pixel value in the image. Then, the first area dividing unit 102 divides the surrounding pixels at the center position into areas of the main subject. Specifically, for example, the first region dividing unit 102 updates the main subject or background region label indicating the background to be given to each pixel in the image, and determines the main subject based on the region label and the pixel value of each pixel. A first cost term that decreases as the value of the first histogram for each pixel value calculated from the image to be displayed is larger, and a value that increases as the value of the second histogram for each pixel value calculated from the image that indicates the background increases. Includes a second cost term that decreases, and for the surrounding pixels related to the center position, the main subject and the background in the image are reduced by an energy function that minimizes the value of the first cost term to, for example, the Graph Cuts method. Divide the area.

  The second area dividing means 103 executes a process of dividing the main subject and the background other than the main subject into areas based on each pixel value in the image. Then, the second area dividing unit 103 performs a process for increasing the degree of area division of each pixel around the main subject divided by the first area dividing unit 102 into the main subject area under a predetermined end condition. Repeat until it reaches. Specifically, the second region dividing unit 103 updates the first label based on the region label and the pixel value of each pixel while updating the region label indicating the main subject or background to be given to each pixel in the image. A first cost term that decreases as the value of the histogram increases and a second cost term that decreases as the value of the second histogram increases, and the pixel value of the pixel is a bin value. The first cost term is smaller when the value of the histogram is larger than the predetermined histogram threshold and the saturation of the pixel value is larger than the predetermined saturation threshold. For example, energy by the Graph Cuts method By the function minimization process, the process of further dividing the main subject and background in the image is repeated.

  The angle-of-view adjusting unit 104 adjusts the angle of view with respect to the main subject with reference to a circumscribed rectangular frame surrounding the main subject divided by the second region dividing unit 104. For example, the angle-of-view adjustment unit 104 adjusts the angle of view for zooming up so that the circumscribed rectangular frame can be accommodated in the screen including a certain margin area in the vertical and horizontal directions.

In the above configuration, the following initial zoom means may be further provided. This initial zoom means expands in the vertical and horizontal directions from the central position by the length of the shortest of the four line segments extending from the center position of the target angle of view toward the top, bottom, left and right edges of the image. A rectangle of a certain size is set as a zoom area. Then, the initial zoom means executes an initial angle adjustment of zooming up so as to fit in the center of the screen while maintaining the aspect ratio of the zoom area, and sets the initial zoom area after zooming up.
Then, the adjustment of the angle of view according to the configuration of FIG. 1 is performed on the above-described initial zoom region.
At this time, the predetermined end condition described above is that one of the boundaries of the divided main subject area has reached the end of the initial zoom area.

  FIG. 2 is an operation explanatory view of the angle of view adjusting apparatus of FIG.

  First, as shown in FIG. 2A, the user touches the center position 201 of the target angle of view, that is, the center of the main subject on the image on which the flower is displayed, for example.

  As a result, first, the first area dividing unit 102 extracts only the central part 202 of the flower shown in FIG. 2B as the main subject area.

  Next, the second area dividing unit 103 sets the pixel value of each pixel in the surrounding area 203 as shown in FIG. 2C to the main object area 202 in FIG. The main subject and background region dividing process is executed so as to capture the subject region. Then, the second area dividing means 103 performs this process until the predetermined end condition is reached while gradually expanding the area of the main subject as shown by 204 in FIG. 2D (for example, the initial zoom area). Until it reaches the end of).

As a result of the above processing, the user can automatically adjust the angle of view so that the main subject fits the entire screen just by one-touching the center position of the target angle of view. A rectangular frame is automatically set.
This frees the user from the hassle of setting a rectangular frame and adjusting the angle of view, and improves operability.
In addition, in executing an application that recognizes and searches for a main subject, it is possible to improve the identification accuracy of the main subject.

  FIG. 3 is a block diagram illustrating a hardware configuration example of the field angle adjustment device 301 according to the embodiment of the present invention.

  The angle-of-view adjustment device 301 is realized on a computer system that is a portable information terminal such as a so-called smartphone or a digital camera, for example.

  The image area dividing device 301 includes a CPU (Central Processing Unit) 302, a ROM (Read Only Memory) 303, and a RAM (Random Access Memory) 304. The image area dividing device 301 includes an external storage device 305 such as a solid storage device, a communication interface 306, an input device 307 such as a touch panel display device, and a display device 308. Further, the image area dividing device 301 includes a portable recording medium driving device 309 capable of setting a portable recording medium 310 such as a micro SD memory card or a USB (Universal Serial Bus) memory card. The imaging device 312 is a digital camera mechanism that can capture still images and video images, and includes a lens, an autofocus drive control device, an exposure control device, an imaging sensor, and the like. The devices 302 to 309 and 312 described above are connected to each other by a bus 311.

  The ROM 303 stores a program for controlling general operations of the smartphone and the entire digital camera, as well as a control program for image region division processing shown by flowcharts of FIGS. 4, 12, and 13 described later. The CPU 302 reads this control program from the ROM 303 and executes the RAM 304 as a work memory. Thereby, the function of the angle-of-view adjusting device having the configuration of FIG. 1 is realized. As a result, for example, the angle at which the flower is captured on the image obtained by the user imaging the flower or the like with the imaging device 312. However, the angle of view is automatically adjusted so that the flowers are displayed full screen. For the image whose screen is automatically adjusted in this way, an image area dividing process for dividing a main subject such as a flower from other backgrounds is executed. The image data of the main subject area such as a flower thus obtained is used as an image search server computer connected to the Internet from the communication interface 306 via the Internet (not shown) in particular so that the user can search for, for example, the type of flower. Sent to. On this computer, a flower database is searched based on the sent flower image data of the main subject area. As a result, the pictorial book information of the flower hit by the search is received by the communication interface 306 via the Internet together with the image data of the flower and displayed on the display device 308.

  FIG. 4 is a flowchart showing the overall operation of the angle of view adjustment process according to the present embodiment. The processing of this flowchart is realized as processing in which the CPU 302 in FIG. 3 executes the control program stored in the ROM 303 while using the RAM 304 as a work memory together with the processing in the flowchart showing the detailed processing in FIGS. The

  First, the CPU 302 displays an image that has been captured and stored in, for example, the RAM 304, on the display device 308 that is a display with a touch input function in FIG. 3, for example, to the center of the zoom target area (the center position of the target field angle). Encourage touch. When the user touch-inputs the center position from the input device 307 that is a display with a touch input function, the CPU 302 stores the coordinate information of the pixel at the center position in the RAM 304, for example (step S401 in FIG. 4).

Next, the CPU 302 executes an initial zoom process (step S402 in FIG. 4). FIG. 5 is an explanatory diagram of the initial zoom process. First, the center position 501 of the target angle of view is designated by the user. Next, the CPU 302 moves in the vertical and horizontal directions from the central position 501 by the length of the shortest line segment 502 (bottom) among the four line segments 502 extending from the central position 501 toward the top, bottom, left and right edges of the image. A rectangle having a size expanded while maintaining the aspect ratio is defined as a zoom area. Then, the CPU 302 performs an initial angle adjustment for zooming up so as to fit in the center of the screen of the zoom area, and sets the initial zoom area 503 after zooming up.
The following angle of view adjustment processing is executed for this initial zoom area.
The process of step S402 executed by the CPU 302 realizes the function of the initial zoom means described above in connection with the description in FIG.

Next, the CPU 302 executes data item update 1 for the initial zoom area calculated in step S402 (step S403 in FIG. 4).
The data item update 1 process executed by the CPU 302 realizes a part of the function of the first area dividing means 102 of FIG.
That is, the CPU 302 updates the main subject or background region label indicating the background to be applied to each pixel in the image of the initial zoom region, and calculates the pixel calculated from the image indicating the main subject based on the region label and the pixel value of each pixel. The first cost term (see equation 6 below and step S1203 in FIG. 12) that decreases as the value of the first histogram for each value increases, and the second for each pixel value calculated from the image indicating the background. The second cost term (see equation (7) and step S1204 in FIG. 12), which decreases as the histogram value increases, is calculated.
At this time, the CPU 302 performs control so that the value of the first cost term regarding the pixel touched by the user and the surrounding pixels is zero (see step S1206 in FIG. 12 described later).
Here, the surroundings refer to pixels within a radius of 3 pixels from the touched pixel position, for example.
Note that the surrounding area includes the touched pixel position itself.
Details of the processing in step S403 will be described later with reference to the flowchart of FIG.

Next, the CPU 302 executes the Graph Cuts algorithm while calculating an energy function (see Equation 3 described later) including the first and second cost terms calculated in the data term update 1 process in step S403. (Step S404 in FIG. 4).
Details of this processing will be described later.
In this case, the process of step S404 executed by the CPU 302 realizes a part of the function of the first area dividing unit 102 in FIG.

After step S404, the CPU 302 makes an end determination (step S405 in FIG. 4). FIG. 6 is an explanatory diagram of the end determination. As shown in FIG. 6, the end condition in this end determination is that when one of the boundaries of the divided main subject region 601 reaches the end 602 of the initial zoom region, or the main subject region is a predetermined region. One of the cases where the change does not change within the error range.
This end determination is a countermeasure against a case where the background pixel around the main subject area is similar to the foreground color and is determined to be the main subject area in the update 2 of the data term 406 described below.

As a result of the end determination process in step S405, the CPU 302 executes the data item update 2 process on the initial zoom area calculated in step S402 (FIG. 4). Step S406).
The data term update 2 process executed by the CPU 302 realizes a part of the function of the second area dividing unit 103 in FIG.
That is, the CPU 302 updates the main subject or background region label indicating the background to be applied to each pixel in the image of the initial zoom region, and calculates the pixel calculated from the image indicating the main subject based on the region label and the pixel value of each pixel. The first cost term (see Equation 6 below and step S1203 in FIG. 13) that decreases as the value of the first histogram for each value increases, and the second for each pixel value calculated from the image indicating the background. The second cost term (see equation (7) described later and step S1204 in FIG. 13), which decreases as the histogram value increases, is calculated.
At this time, the CPU 302 determines that the value of the first histogram having the pixel value as a bin value is larger than a predetermined histogram threshold and the saturation of the pixel value is larger than a predetermined saturation threshold. (See step S1301 in FIG. 13 described later), the first cost term is set to a smaller value (see steps S1302 to S1304 in FIG. 13 described later).
Details of the processing in step S406 will be described later with reference to the flowchart of FIG.

Next, the CPU 302 executes the Graph Cuts algorithm while calculating an energy function (see Equation 3 described later) including the first and second cost terms calculated in the data term update 2 process in step S406. (Step S404 in FIG. 4).
Details of this processing will be described later.
In this case, the process of step S404 executed by the CPU 302 realizes a part of the function of the second area dividing unit 103 in FIG.

  Thereafter, the end determination process is executed again (step S405 in FIG. 4), and the above-described data item update 2 process (step S406 in FIG. 4) is repeatedly executed to expand the main subject area.

When the end is determined in step S405, final zoom processing is executed (step S407 in FIG. 4). FIG. 7 is an explanatory diagram of the final zoom process. As shown in FIG. 7, zoom-in angle adjustment is performed so that the circumscribed rectangle of the main subject area 701 fits within the screen including a certain margin area in the vertical and horizontal directions.
The processing in step S407 executed by the CPU 302 realizes the function of the angle of view adjustment unit 104 in FIG.

As described above, as described above with reference to FIG. 2, in step S <b> 401, the user performs one-touch operation on the center 201 of the zoom target area as illustrated in FIG. In step S403, only the central part 202 of the flower shown in FIG. 2B is extracted as the main subject area. Further, by repeating step S406, the pixel value of each pixel in the surrounding region 203 as shown in FIG. 2C is taken as a new main subject region with respect to the main subject region 202 in FIG. 2B. An area division process of the main subject and the background, which is going to be performed, is executed. This process is repeatedly executed until the condition for the end determination in step S405 is reached while gradually increasing the area of the main subject, as indicated by 204 in FIG.
Then, the user simply touches the center position of the target angle of view first, and after the final zoom process in step S407, the angle of view is automatically adjusted so that the main subject such as a flower fits on the screen. . The user is freed from the hassle of setting the rectangular frame and adjusting the angle of view, and can improve the operability.

In the following, the GraphCuts algorithm that is the basis of the processing of steps S403, S404, and S406 of FIG. 4 will be described.
Now
And an element X _v is a region label vector indicating an area label for pixels v (1 ≦ v ≦ V) in the image V. This area label vector is, for example, a binary vector having element X _v = 0 if the pixel v is in the main subject area and element X _v = 1 if it is in the background area. That is,
It is.

The area dividing process executed in the present embodiment is a process for obtaining an area label vector X of Formula 1 that minimizes an energy function E (X) defined by the following expression in the image V.
As a result of executing the energy minimization process, the main subject region is obtained as a set of pixels v having the region label value X _v = 0 on the region label vector X. Note that the set of pixels v with the region label value X _v = 1 on the region label vector X is the background region.

In order to minimize the energy of equation (3), the following equation and a weighted directed graph (hereinafter abbreviated as “graph”) shown in FIG. 8 are defined.
Here, V is a node and E is an edge. When this graph is applied to image area division, each pixel of the image corresponds to each node V. Also, as nodes other than pixels, the following formula and shown in FIG.
A special terminal called is added. Consider the source s in association with the main subject region and the sink t in the background region. The edge E represents the relationship between the nodes V. An edge E representing the relationship with surrounding pixels is n-link, and an edge E representing the relationship between each pixel and the source s (corresponding to the main subject region) or sink t (corresponding to the background region) is t-link. Call.

Now, each n-link connecting the source s and a node corresponding to each pixel is regarded as a relationship indicating how much each pixel seems to be a main subject region. Then, a cost value indicating the likelihood of the main subject area is associated with the first term of Equation 3 and
It is defined as Here, θ (c, 0) is function data indicating a histogram (number of appearances) for each color pixel value c calculated from the area of the main subject of the image. For example, as shown in FIG. Has been obtained. It is assumed that the sum total of θ (c, 0) over all color pixel values c is normalized to be 1. I (v) is a color (RGB) pixel value at each pixel v of the input image. Actually, this is a value obtained by converting a color (RGB) pixel value into a luminance value, but for the sake of simplicity of explanation, it will be described as “color (RGB) pixel value” or “color pixel value”. In Equation 6, the cost value decreases as the value of θ (I (v), 0) increases. This is because, as the number of appearances of color pixel values in the main subject region obtained in advance increases, the cost value obtained by Equation 6 becomes smaller, and the pixel v seems to be a pixel in the main subject region. This means that the value of the energy function E (X) in Equation 3 is pushed down.

Next, each t-link connecting the sink t and the node corresponding to each pixel is regarded as a relationship indicating how much each pixel is a background region. Then, the cost value indicating the likelihood of the background region is associated with the first term of Equation 3 and
It is defined as Here, θ (c, 1) is function data indicating a histogram (appearance frequency) for each color pixel value c calculated from the background region of the image, and is obtained in advance as shown in FIG. 9B, for example. It has been. It is assumed that the sum total of θ (c, 1) over all color pixel values c is normalized to be 1. I (v) is a color (RGB) pixel value at each pixel v of the input image, as in Equation 6. In Equation 6, the cost value decreases as the value of θ (I (v), 1) increases. This means that the more frequently appearing color pixel values of the background area obtained in advance, the smaller the cost value obtained by Equation 7 is, and the pixel v seems to be a pixel in the background area. As a result, the value of the energy function E (X) in Formula 3 is pushed down.

Next, the cost value of t-link indicating the relationship between the node corresponding to each pixel and its surrounding pixels is associated with the second term of Equation 3;
It is defined as Here, dist (u, v) indicates the Euclidean distance between the pixel v and the surrounding pixel u, and κ is a predetermined coefficient. Further, I (u) and I (v) are color (RGB) pixel values in the pixels u and v of the input image. When the region label values X _u and X _{v of the} pixel v and the surrounding pixels u are selected to be the same (X _u = X _v ), the cost value of Formula 8 is set to 0, and the energy E ( It does not affect the calculation of X). On the other hand, the cost value of Equation 8 when the region label values X _u and X _{v of} the pixel v and the surrounding pixels u are selected to be different (X _u ≠ X _v ) is, for example, the characteristic shown in FIG. It has a function characteristic having That is, when the region label values X _u and X _v of the pixel v and the surrounding pixel u are different, and the difference I (u) −I (v) between the pixel v and the surrounding pixel u is small. In this case, the cost value obtained by Equation 8 is large. In this case, the value of the energy function E (X) in Equation 3 is pushed up. In other words, when the difference in color pixel value between neighboring pixels is small, the area label values of those pixels are not selected to be different from each other. In other words, in this case, the region label values are controlled to be the same between neighboring pixels and the main subject region or the background region is not changed as much as possible. On the other hand, when the region label values X _u and X _v of the pixel v and the surrounding pixel u are different and the difference I (u) −I (v) between the color values of the pixel v and the surrounding pixel u is large. Therefore, the cost value obtained by Equation 8 is small. In this case, the result is that the value of the energy function E (X) in Equation 3 is pushed down. In other words, if the color pixel value difference between neighboring pixels is large, it means that the boundary between the main subject region and the background region is likely, and the region label values in the direction where the pixel v and its surrounding pixels u are different. Be controlled.

  Using the above definition, the cost value of t-link (likeness of main subject area) connecting the source s and each pixel v is calculated for each pixel v of the input image by Equation (6). In addition, the cost value (likeness of background area) of t-link connecting the sink t and each pixel v is calculated by Expression 7. Further, for each pixel v of the input image, eight n-link cost values (likeness of boundary) connecting the pixel v and its surroundings, for example, each of eight pixels in eight directions, are calculated by Equation (8).

Theoretically, for each combination of region labels 0 or 1 of the region label vector X in Equation 1, the above Equation 6, Equation 7, and Equation 8 according to each region label value. The energy function E (X) of Formula 3 is calculated while the calculation result of is selected. Then, by selecting the region label vector X that minimizes the value of the energy function E (X) among all the combinations, as a set of pixels v with the region label value X _v = 0 on the region label vector X The main subject area can be obtained.

  However, since the number of combinations of 0 or 1 of all region label values of the region label vector X is actually the number of pixels multiplied by 2, calculation of the energy function E (X) minimization process in a realistic time is calculated. Can not do it.

Therefore, the Graph Cuts method executed in step S404 in FIG. 4 makes it possible to calculate the energy function E (X) minimization processing in a realistic time by executing the following algorithm. .
FIG. 11 is a schematic diagram showing the relationship between a graph having t-link defined by the above-described Equation 6 and Equation 7 and n-link defined by Equation 8, and the region label vector X and the graph cut. It is the figure shown in. In FIG. 11, the pixel v is shown one-dimensionally for easy understanding.

  In the calculation of the first term of the energy function E (X) in Expression 3, in the pixel in the main subject area where the area label value in the area label vector X should be 0, among Expression 6 and Expression 7, The cost value of equation (6), which is a smaller value depending on the pixel in the main subject area, is smaller. Therefore, in a certain pixel, the t-link on the source s side is selected and the t-link on the sink t side is cut (case 1102 in FIG. 11), and E (X) in Equation 3 using Equation 6 If the calculation result becomes small when the first term is calculated, 0 is selected as the region label value of the pixel. Then, the graph cut state is adopted. If the calculation result does not become small, the graph cut state is not adopted and another link search and graph cut are attempted.

  Conversely, for the pixels in the background region where the region label value in the region label vector X should be 1, among the equations (6) and (7), the equation The cost value is smaller. Therefore, in a certain pixel, t-link on the sink t side is selected, and t-link on the source s side is cut (case 1103 in FIG. 11), and E (X) in Equation 3 using Equation 7 When the first term is calculated, if the calculation result becomes small, 1 is selected as the region label value of the pixel. Then, the graph cut state is adopted. If the calculation result does not become small, the graph cut state is not adopted and another link search and graph cut are attempted.

  On the other hand, by the above-described graph cut processing relating to the calculation of the first term of the energy function E (X) of Formula 3, the area label value in the area label vector X is 0 or 1, and the main subject area should be continuous or the background area The cost value of Formula 8 is 0 between the internal pixels. Therefore, the calculation result of Equation 8 does not affect the calculation of the cost value of the second term of the energy function E (X). Further, the n-link between the pixels is maintained without being cut so that Equation 8 outputs a cost value of 0.

  However, when the region label value changes between 0 and 1 between neighboring pixels by the graph cut processing relating to the calculation of the first term of the energy function E (X), the color pixel value between those pixels is changed. If the difference is small, the cost value of Equation 8 is large. As a result, the value of the energy function E (X) in Equation 3 is pushed up. Such a case corresponds to a case where the determination of the region label value by the value of the first term happens to be reversed in the same region. Therefore, in such a case, the value of the energy function E (X) becomes large, and as a result, such inversion of the region label value is not selected. In this case, the n-link between the pixels is maintained without being cut so that the calculation result of Expression 8 maintains the above result.

  On the other hand, when the region label value changes between 0 and 1 between neighboring pixels by the graph cut processing relating to the calculation of the first term of the energy function E (X), If the difference between the color pixel values is large, the cost value of Equation 8 is small. As a result, the value of the energy function E (X) in Equation 3 is pushed down. Such a case means that these pixel portions are likely to be the boundary between the main subject area and the background area. Therefore, in such a case, the region label value is varied between these pixels, and the control is performed in the direction in which the boundary between the main subject region and the background region is formed. In this case, in order to stabilize the boundary formation state, the n-link between these pixels is cut, and the cost value of the second term of Equation 3 is set to 0 (FIG. 11). 1104 case).

  The above determination control processing is repeated starting from the node of the source s while sequentially tracing the nodes of each pixel, so that a graph cut as indicated by 1101 in FIG. 11 is executed, and in step S404 of FIG. The process of minimizing the energy function E (X) is calculated in a realistic time. As a specific method of this processing, for example, the method described in Non-Patent Document 1 can be adopted.

  If t-link on the source s side remains for each pixel, 0 is given as the area label value of that pixel, that is, a label indicating the pixel of the main subject area is given. On the contrary, if t-link on the sink t side remains, 1 is given as the area label value of the pixel, that is, a label indicating the pixel in the background area is given. Finally, the main subject area is obtained as a set of pixels having the area label value of 0.

  FIG. 12 is a flowchart showing details of the data item update 1 process in step S403 of FIG. 4 based on the above-described operation principle.

  First, the color pixel values I (V) are read one by one from the image of the initial zoom area obtained in step S402 (step S1201 in FIG. 12).

  Next, it is determined whether or not the pixel read in step S1201 is a surrounding pixel related to the center of the zoom target area touch-designated by the user in step S401 in FIG. 4 (step S1202 in FIG. 12).

  If the determination in step S1202 is NO, the cost value indicating the main subject area likelihood, the cost value indicating the background area likelihood, and the boundary likelihood are calculated based on the above-described Expression 6, Expression 7, and Expression 8. The cost values shown are respectively calculated (steps S1203, S1204, and S1205 in FIG. 12). The initial value of θ (c, 0) is calculated from the areas of a plurality of (approximately several hundred) main subjects prepared for learning. Similarly, the initial value of θ (c, 1) is calculated from a plurality (several hundreds) of background regions prepared for learning.

On the other hand, if the determination in step S1202 is YES, it can be said that the surrounding pixels including the touch-designated pixel are surely the main subject region. Therefore, in order to ensure that the surrounding area including the touch-designated pixel is determined as the main subject area, the cost value g _v (X _v ) indicating the likelihood of the main subject area is expressed by the following equation: It is set to zero (step S1206 in FIG. 12).

Further, in order to ensure that the surrounding area including the touch-designated pixel is determined as the main subject area, the cost value g _v (X _v ) indicating the likelihood of the background area is a value K as shown in the following equation: It is said.
Here, as shown in the following equation, K is set to a value larger than the sum of smoothing terms of arbitrary pixels (step S1207 in FIG. 12).

Further, since all the surrounding areas including the touch-designated pixel are main subject areas, the value of h _uv (X _u , X _v ) is set to 0 (step S1208 in FIG. 12).

  After the above processing, it is determined whether or not there remains a pixel to be processed in the image (step S1209 in FIG. 12).

  If there is a pixel to be processed and the determination in step S1209 is YES, the process returns to step S1201 and the above process is repeated.

When there is no more pixel to be processed and the determination in step S1209 is NO, the data item update 1 process in step S403 in FIG. 4 ends.
Thereafter, in step S404 of FIG. 4, the Graph Cuts algorithm is executed while calculating the energy function E (X) of Formula 3 using the cost values obtained for all the pixels in the image of the initial zoom region, The main subject and the background are divided into regions.

  FIG. 13 is a flowchart showing details of the data item update 2 process in step S406 of FIG. 4 based on the operation principle of the Graph Cuts algorithm described above. In this flowchart, the same processing steps as those in FIG. 12 are denoted by the same step numbers.

  First, the color pixel values I (V) are read one by one from the image of the initial zoom area obtained in step S402 (step S1201 in FIG. 12).

  Next, whether or not the color pixel value I (V) read in step S1201 seems to be a pixel of the main subject is determined as follows. Now, let Sat (I (V)) be the saturation with respect to the color pixel value I (V), and θ (I (V), 0) be the histogram showing the main subject likeness of the color pixel value I (V). It is determined whether the degree Sat (I (V)) is greater than or equal to a predetermined saturation threshold Sth and the histogram θ (I (V), 0) is greater than or equal to the predetermined histogram threshold θth (step in FIG. 13). S1301).

  If the determination in step S1301 is NO, as in the case of FIG. 12, the cost value indicating the likelihood of the main subject region and the likelihood of the background region are calculated based on the above-described Equation 6, Equation 7, and Equation 8. The cost value to be indicated and the cost value to indicate the likelihood of the boundary are respectively calculated (steps S1203, S1204, and S1205 in FIG. 12).

On the other hand, if the determination in step S1301 is YES, the color pixel value I (V) read in step S1201 is a pixel around the main subject area obtained by the previous processing, and is also a main pixel value. It can be said that there is a high possibility of being a subject area. For this reason, the cost value g _v (X _v ) indicating the likelihood of the main subject area is corrected as follows to ensure that the surrounding area including the touch-designated pixel is surely determined as the main subject area. .
First, it is determined whether or not the cost value g _v (0) indicating the likelihood of the main subject area is greater than the cost value g _v (1) indicating the likelihood of the background area (step S1302 in FIG. 13).
If g _v (0)> g _v (1) and the determination in step S1302 is YES, the cost value g _v (0) indicating the likelihood of the main subject area is the cost value g _v (1) indicating the likelihood of the background area. The value is corrected to half (step S1303 in FIG. 13). That is, from Equation 7,
g _v (X _v ) = g _v (0) = g _v (1) / 2 = −log θ (I (V), 1) / 2
It is said.
On the other hand, if g _v (0)> g _v (1) is not satisfied and the determination in step S1302 is NO, the cost value g _v (0) indicating the likelihood of the main subject area is corrected to half the original value. (Step S1304 in FIG. 13). That is, from Equation 6,
g _v (X _v ) = g _v (0) = g _v (0) / 2 = −log θ (I (V), 0) / 2
It is said.

Further, the cost value g _v (X _v ) indicating the likelihood of the background area is calculated in the above-described step S1204.

The value of h _uv (X _u , X _v ) is calculated in step S1205 described above.

  After the above processing, it is determined whether or not there remains a pixel having a color similar to the vicinity of the currently processed pixel in the image (step S1305 in FIG. 13). The similar color is, for example, a color having the same RGB value when the 256 gradation RGB values are reduced to 10 gradations for each color.

  If there is a pixel to be processed and the determination in step S1305 is YES, the process returns to step S1201 and the above process is repeated.

If there is no more pixel to be processed and the determination in step S1305 is NO, the data item update 2 process in step S406 of FIG. 4 ends.
Thereafter, in step S404 of FIG. 4, the Graph Cuts algorithm is executed while calculating the energy function E (X) of Formula 3 using the cost values obtained for all the pixels in the image of the initial zoom region, The main subject and the background are divided into regions.

  As described above, in this embodiment, it is possible to improve the accuracy of area division.

  In the present embodiment, a flower has been described as an example of a main subject. However, the present invention is not limited to a flower, and various objects can be employed.

  In this embodiment, a system for searching for a main subject has been described. However, the present invention can be applied to various systems that require angle-of-view adjustment.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
An angle-of-view adjusting device that adjusts an angle of view with respect to the main subject so that the main subject has an appropriate angle of view in the image,
Position specifying means for specifying the center position of the target angle of view;
A first region that performs a region division process on the main subject and a background other than the main subject based on each pixel value in the image, and divides the surrounding pixels at the center position into the region of the main subject. Dividing means;
Based on each pixel value in the image, the main subject and a background other than the main subject are divided into regions, and each pixel around the main subject divided by the first region dividing unit is processed. Second area dividing means for repeatedly executing a process for increasing the degree of area division into the area of the main subject until a predetermined end condition is reached;
An angle-of-view adjusting unit that adjusts an angle of view with respect to the main subject with reference to a circumscribed rectangular frame surrounding the main subject divided by the second region dividing unit;
An angle-of-view adjustment apparatus comprising:
(Appendix 2)
The angle-of-view adjustment apparatus according to appendix 1, wherein the position specifying unit is a unit that allows a user to specify a center position.
(Appendix 3)
The position specifying means causes the user to specify the center position of the target angle of view on the display with a touch input function for displaying the image.
The angle-of-view adjustment apparatus according to appendix 2, characterized in that:
(Appendix 4)
A rectangle having a size expanded from the center position in the up, down, left, and right directions by the length of the shortest of the four line segments extending from the center position toward the top, bottom, left, and right edges of the image. A zoom area, and zooming up so as to fit in the center of the screen while maintaining the aspect ratio of the zoom area, further comprising an initial zoom means as an initial zoom area after the zoom-up,
Adjusting the angle of view with respect to the initial zoom area;
The predetermined end condition is that any boundary of the area of the main subject divided by the second area dividing means reaches the end of the initial zoom area.
The angle-of-view adjustment apparatus according to any one of appendices 1 to 3, characterized in that:
(Appendix 5)
The first region dividing means updates the main subject or the region label indicating the background to be applied to each pixel in the image, and determines the main subject based on the region label and the pixel value of each pixel. A first cost term that decreases as the value of the first histogram for each pixel value calculated from the image to be displayed is smaller, and the value of the second histogram for each pixel value calculated from the image to indicate the background is A second cost term that decreases as the value increases, and for the surrounding pixels related to the center position, the main subject within the image is identified by the energy function minimization process in which the value of the first cost term is zero. Dividing the background into regions,
The angle-of-view adjustment device according to any one of appendices 1 to 4, wherein
(Appendix 6)
The second region dividing means updates the region label indicating the main subject or the background to be given to each pixel in the image, and based on the region label and the pixel value of each pixel, A first cost term that decreases as the value of the histogram increases, and a second cost term that decreases as the value of the second histogram increases, and the pixel value of the pixel is a bin value An energy function of which the first cost term is smaller when the value of the first histogram is larger than a predetermined histogram threshold and the saturation of the pixel value is larger than the predetermined saturation threshold. By the minimization process, the main subject and the background are divided into regions in the image.
The angle-of-view adjustment apparatus according to appendix 5, characterized in that:
(Appendix 7)
The first region dividing unit and the second region dividing unit execute the energy function minimization processing by a Graph Cuts method.
The angle-of-view adjustment apparatus according to appendix 6, wherein
(Appendix 8)
An angle of view adjustment method for adjusting an angle of view with respect to a main subject so that the main subject has an appropriate angle of view in an image,
A position specifying step for allowing the user to specify the center position of the target angle of view;
A first region that performs a region division process on the main subject and a background other than the main subject based on each pixel value in the image, and divides the surrounding pixels at the center position into the region of the main subject. A splitting step;
Based on each pixel value in the image, the main subject and a background other than the main subject are divided into regions, and each pixel around the main subject divided in the first region dividing step is processed. A second area dividing step of repeatedly executing a process for increasing the degree of area division into the main subject area until a predetermined end condition is reached;
An angle-of-view adjusting step of adjusting an angle of view with respect to the main subject with reference to a circumscribed rectangular frame surrounding the main subject divided by the second region dividing step;
The angle-of-view adjustment method comprising:
(Appendix 9)
A computer that adjusts the angle of view of the main subject so that the main subject has an appropriate angle of view in the image;
A position specifying step for allowing the user to specify the center position of the target angle of view;
A first region that performs a region division process on the main subject and a background other than the main subject based on each pixel value in the image, and divides the surrounding pixels at the center position into the region of the main subject. A splitting step;
Based on each pixel value in the image, the main subject and a background other than the main subject are divided into regions, and each pixel around the main subject divided in the first region dividing step is processed. A second area dividing step of repeatedly executing a process for increasing the degree of area division into the main subject area until a predetermined end condition is reached;
An angle-of-view adjusting step of adjusting an angle of view with respect to the main subject with reference to a circumscribed rectangular frame surrounding the main subject divided by the second region dividing step;
A program for running

DESCRIPTION OF SYMBOLS 101 Position designation means 102 1st area division means 103 2nd area division means 104 View angle adjustment means 301 View angle adjustment apparatus 302 CPU
303 ROM
304 RAM
305 External storage device 306 Communication interface 307 Input device 308 Display device 309 Portable recording medium driving device 310 Portable recording medium 311 Bus 312 Imaging device

Claims (9)

An angle-of-view adjusting device that adjusts an angle of view with respect to the main subject so that the main subject has an appropriate angle of view in the image,
Position specifying means for specifying the center position of the target angle of view;
A first pixel that divides the main subject and the background other than the main subject into regions based on the pixel values in the image and divides the surrounding pixels at the center position into the main subject region. A region dividing process, and a second region dividing process that repeatedly executes a process for increasing the degree of area division for each pixel around the main subject divided into the main subject area until a predetermined end condition is reached. Region dividing means for executing
An image for adjusting the angle of view with respect to the main subject by zooming in so that the circumscribed rectangular frame fits in the image with reference to the circumscribed rectangular frame surrounding the main subject divided by the second region dividing process. Angle adjustment means;
With
A rectangle having a size expanded from the center position in the up, down, left, and right directions by the length of the shortest of the four line segments extending from the center position toward the top, bottom, left, and right edges of the image. A zoom area, and zooming up so as to fit in the center of the screen while maintaining the aspect ratio of the zoom area, further comprising an initial zoom means as an initial zoom area after the zoom-up,
Adjusting the angle of view with respect to the initial zoom area;
The predetermined end condition is that any one of the main subject areas divided by the second area dividing process has reached the end of the initial zoom area.
An angle-of-view adjusting device characterized by that. An angle-of-view adjusting device that adjusts an angle of view with respect to the main subject so that the main subject has an appropriate angle of view in the image,
Position specifying means for specifying the center position of the target angle of view;
A first pixel that divides the main subject and the background other than the main subject into regions based on the pixel values in the image and divides the surrounding pixels at the center position into the main subject region. A region dividing process, and a second region dividing process that repeatedly executes a process for increasing the degree of area division for each pixel around the main subject divided into the main subject area until a predetermined end condition is reached. Region dividing means for executing
An image for adjusting the angle of view with respect to the main subject by zooming in so that the circumscribed rectangular frame fits in the image with reference to the circumscribed rectangular frame surrounding the main subject divided by the second region dividing process. Angle adjustment means;
With
As the first region dividing process, while updating the main subject or the region label indicating the background to be given to each pixel in the image, the main subject is changed based on the region label and the pixel value of each pixel. A first cost term that decreases as the value of the first histogram for each pixel value calculated from the image to be displayed is smaller, and the value of the second histogram for each pixel value calculated from the image to indicate the background is A second cost term that decreases as the value increases, and for the surrounding pixels related to the center position, the main subject within the image is identified by the energy function minimization process in which the value of the first cost term is zero. Dividing the background into regions,
An angle-of-view adjusting device characterized by that. As the second region dividing process, while updating the region label indicating the main subject or the background to be given to each pixel in the image, the first region is divided based on the region label and the pixel value of each pixel. A first cost term that decreases as the value of the histogram increases, and a second cost term that decreases as the value of the second histogram increases, and the pixel value of the pixel is a bin value An energy function of which the first cost term is smaller when the value of the first histogram is larger than a predetermined histogram threshold and the saturation of the pixel value is larger than the predetermined saturation threshold. By the minimization process, the main subject and the background are divided into regions in the image.
The angle-of-view adjusting apparatus according to claim 2. The region dividing unit performs the energy function minimization process by a Graph Cuts method.
The angle-of-view adjustment apparatus according to claim 3.   The angle-of-view adjustment apparatus according to claim 1, wherein the position designation unit is a unit that allows a user to designate a center position. The position specifying means causes the user to specify the center position of the target angle of view on the display with a touch input function for displaying the image.
The angle-of-view adjustment apparatus according to claim 5. An angle of view adjustment method for adjusting an angle of view with respect to a main subject so that the main subject has an appropriate angle of view in an image,
A position specifying step for specifying the center position of the target angle of view;
A first pixel that divides the main subject and the background other than the main subject into regions based on the pixel values in the image and divides the surrounding pixels at the center position into the main subject region. A region dividing process, and a second region dividing process that repeatedly executes a process for increasing the degree of area division for each pixel around the main subject divided into the main subject area until a predetermined end condition is reached. A region dividing step for performing
An image for adjusting the angle of view with respect to the main subject by zooming in so that the circumscribed rectangular frame fits in the image with reference to the circumscribed rectangular frame surrounding the main subject divided by the second region dividing process. Corner adjustment step;
With
A rectangle having a size expanded from the center position in the up, down, left, and right directions by the length of the shortest of the four line segments extending from the center position toward the top, bottom, left, and right edges of the image. A zoom area, and performing zoom-up so as to fit in the center of the screen while maintaining the aspect ratio of the zoom area, further comprising an initial zoom step as an initial zoom area after the zoom-up,
Adjusting the angle of view with respect to the initial zoom area;
The predetermined end condition is that any one of the main subject areas divided by the second area dividing process has reached the end of the initial zoom area.
A method of adjusting the angle of view. An angle of view adjustment method for adjusting an angle of view with respect to a main subject so that the main subject has an appropriate angle of view in an image,
A position specifying step for specifying the center position of the target angle of view;
A first pixel that divides the main subject and the background other than the main subject into regions based on the pixel values in the image and divides the surrounding pixels at the center position into the main subject region. A region dividing process, and a second region dividing process that repeatedly executes a process for increasing the degree of area division for each pixel around the main subject divided into the main subject area until a predetermined end condition is reached. A region dividing step for performing
An image for adjusting the angle of view with respect to the main subject by zooming in so that the circumscribed rectangular frame fits in the image with reference to the circumscribed rectangular frame surrounding the main subject divided by the second region dividing process. Corner adjustment step;
With
As the first region dividing process, while updating the main subject or the region label indicating the background to be given to each pixel in the image, the main subject is changed based on the region label and the pixel value of each pixel. A first cost term that decreases as the value of the first histogram for each pixel value calculated from the image to be displayed is smaller, and the value of the second histogram for each pixel value calculated from the image to indicate the background is A second cost term that decreases as the value increases, and for the surrounding pixels related to the center position, the main subject within the image is identified by the energy function minimization process in which the value of the first cost term is zero. Dividing the background into regions,
A method of adjusting the angle of view. A computer that adjusts the angle of view of the main subject so that the main subject has an appropriate angle of view in the image;
The program for performing each step of Claim 7 or 8.