JP2009159023A - Image processor, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance estimation precision of an interest region in an input image. <P>SOLUTION: An image processor 10 includes an image analysis section 121, a composition determination section 122, a region estimation section 123 and an output control section 129. The image analysis section 121 detects at least one feature pattern by analyzing an input image. The composition determination section 122 calculates the degree of association between the composition and the feature pattern as an evaluation value, and determines the composition of an input image based on the evaluation value. The region estimation section 123 estimate an interest region in an input image by using the composition thus determined. The output control section 129 outputs a local image of the interest region thus estimated. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for estimating a region of interest in one image and displaying an image of this region of interest.

  When a high-resolution image such as a captured image is displayed on a small device mounted display such as a mobile phone with a digital camera, the resolution of the effective display area of the mounted display is low. There was a problem that it took a lot of time and effort to find the interested area of interest. In order to find the region of interest, for example, the user operates a small device to reduce the high-resolution image on the display, designates a rough position in the reduced display image, and displays the local image at the designated position. To zoom in. Thereafter, the user presses the direction key to move (scroll) the display position of the enlarged image in units of several pixels and move the display position to the region of interest. However, the larger the image size, the more operation time is required until the display position reaches the region of interest.

In order to solve such a problem, there is a technique capable of automatically estimating a region of interest in a high-resolution image, displaying a local image of the region of interest on a display, and presenting it to a user. For example, Patent Literature 1 (Japanese Patent Laid-Open No. 2006-277038), Patent Literature 2 (Japanese Patent Laid-Open No. 2006-261711), Patent Literature 3 (Japanese Patent Laid-Open No. 2003-44511), Patent Literature 4 (Japanese Patent Laid-Open No. 11-345340). No.) and Patent Document 5 (Japanese Patent Publication No. 2004-520735) disclose techniques for estimating, searching or classifying a region of interest.
JP 2006-277038 A JP 2006-261711 A JP 2003-44511 A JP-A-11-345340 Japanese translation of PCT publication No. 2004-520735

  However, in the conventional techniques disclosed in Patent Documents 1 to 5, there are cases where the estimated region of interest does not match the region that the user wants to visually confirm, and there is a problem that the estimation accuracy of the region of interest is low.

  For example, in Patent Documents 1 and 2, an image of a face appearing in an image is detected as an object, an importance is set for the detected object, and an object with a high importance is displayed. Technology is disclosed. However, an image area for displaying an object such as a vehicle image is not always related to a region of interest, not limited to a human face. For this reason, when shooting a place such as a stadium where a large number of people gather, it is possible to select a specific object from among a large number of indistinguishable objects in the captured image and set the importance for this. difficult.

  In view of the above, an object of the present invention is to provide an image processing device, an image processing method, and an image processing program capable of estimating a region of interest in an input image such as a captured image with high accuracy.

  According to the present invention, an image analysis unit that analyzes an input image and detects at least one feature pattern from the input image, and evaluates a degree of association between each of a plurality of prepared compositions and the feature pattern A composition determination unit that calculates a value and determines the composition of the input image based on the evaluation value, a region estimation unit that estimates a region of interest in the input image using the determined composition, and the estimation And an output control unit that outputs a local image of the region of interest.

  According to the present invention, the step of analyzing the input image to detect at least one feature pattern from the input image, and the degree of association between each of a plurality of prepared compositions and the feature pattern is used as an evaluation value. Calculating and determining a composition of the input image based on the evaluation value; estimating a region of interest in the input image using the determined composition; and localizing the estimated region of interest An image processing method comprising: outputting an image.

  According to the present invention, an image analysis process that analyzes an input image and detects at least one feature pattern from the input image, and evaluates a degree of association between each of a plurality of prepared compositions and the feature pattern A composition determination process that calculates a composition of the input image based on the evaluation value, a region estimation process that estimates a region of interest in the input image using the determined composition, and the estimation An image processing program for causing a computer to execute an output control process for outputting a local image of the region of interest is provided.

  An image processing apparatus, an image processing method, and an image processing program according to the present invention compare each prepared composition with a feature pattern detected from an input image, and determine the degree of association between each composition and the feature pattern. calculate. Since the composition of the input image is determined based on this degree of association, it is possible to detect a region of interest in the input image with high accuracy using the determined composition.

  Hereinafter, various embodiments of the present invention will be described with reference to the drawings. In all the drawings, components having the same function are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate.

(First embodiment)
FIG. 1 is a functional block diagram showing a schematic configuration of an image processing apparatus 10 according to the first embodiment of the present invention. As illustrated in FIG. 1, the image processing apparatus 10 includes an imaging unit 110, a data processing unit 120, a data storage unit 130, and a display 140. The data processing unit 120 includes an image analysis unit 121, a composition determination unit 122, a region estimation unit 123, and a display control unit (output control unit) 129.

The image analysis unit 121 analyzes the input image has a function of detecting at least one feature pattern g _n from the input image. Composition determining unit 122, a plurality of composition _C 1 prepared in _{advance, C 2, ..., C p} (p is an integer of 3 or more) degree of relevance between each and feature pattern _{g n} (or degree of coincidence) E ₁ , E ₂ ,..., _Ep are calculated as evaluation values, and the composition of the input image is determined based on these evaluation values. The region estimation unit 123 has a function of estimating a region of interest in the input image using the composition determined by the image analysis unit 121. The display control unit 129 has a function of outputting a local image of the region of interest estimated by the region estimation unit 123.

The data storage unit 130 includes a composition storage unit 131 and a composition coefficient storage unit 132. The composition storage unit 131 stores data sets (hereinafter referred to as composition data sets) representing reference compositions C ₁ , C ₂ ,..., C _p . As described below, the composition determining unit 122, evaluation value obtains the composition data set from the composition memory unit _{131 E 1, E 2, ...} , is used to calculate the E _p. Composition coefficient storage unit 132 stores composition _{C 1,} C 2 prepared in advance, ..., composition coefficients lambda ₁ respectively corresponding to the _{C p,} lambda _2, ..., the lambda _p. These Composition coefficients lambda ₁ as described below, λ _2, ..., the lambda _p, the corresponding composition _{C 1, C 2, ...,} a weighting factor for _{C p.} The user operates an input operation unit (not shown) of the image processing apparatus 10 to set the values of the composition coefficients λ ₁ , λ ₂ ,..., Λ _p to arbitrary values within a predetermined numerical range. Can do.

  The “image composition” represents the entire configuration of one image that can give a visually stable aesthetic appearance. For this reason, an object such as a face image that appears in one image does not necessarily represent the entire configuration of the image, and cannot be said to be a composition. The same applies to the second to fourth embodiments described later.

Each composition is a multi-tone pattern corresponding to one screen in which a weight is set in units of a predetermined number of pixels. The composition storage unit 131 stores composition data sets representing the compositions C ₁ , C ₂ ,..., C _p . Basic composition includes, for example, a three-part composition (composition in which weights are concentrated on a reference line that divides the vertical and horizontal directions of the screen into three parts), a Hinomaru composition (composition in which weights are concentrated in the center of the screen), and a triangular composition ( Triangular reference line or composition in which weights are concentrated on the reference body), target composition (composition in which weights are concentrated on the reference line or reference body symmetrically arranged in the horizontal or vertical direction of the screen), curved composition (predetermined curve) Composition with weights concentrated on top), horizontal composition (compositions with weights concentrated on a horizontal reference line arranged in the screen), radiation compositions (radiation composition on the reference line or reference body distributed radially in the screen) Composition) and color plane composition (composition in which weights are set for one or more color planes included in the screen). 2A shows a typical example of a three-part composition, FIG. 2B shows a typical example of a Hinomaru composition, FIG. 2C shows a typical example of a triangular composition, and FIG. 2D shows a typical example of a symmetric composition. FIG. 2E shows a typical example of a curved composition, FIG. 2F shows a typical example of a horizontal composition, and FIG. 2G shows a typical example of a radiation composition. The composition stored in the composition storage unit 131 may be this type of basic composition or a composite composition obtained by superimposing two or more basic compositions.

  The data storage unit 130 includes a recording medium such as a volatile memory or a non-volatile memory (for example, a semiconductor memory or a magnetic recording medium), and a circuit or program for writing and reading data on the recording medium. Can be configured. The composition storage unit 131 and the composition coefficient storage unit 132 may be configured in advance on a predetermined storage area of the recording medium, or may be configured on an appropriate storage area assigned when the image processing apparatus 10 is operated. Also good.

  The image analysis unit 121 can detect the feature pattern of the captured image by analyzing the hue and spatial frequency of the captured image input from the imaging unit 110. Examples of the feature pattern include a multi-tone pattern that indicates the position or distribution of an object such as a face or a vehicle, or a multi-tone pattern that indicates the edge of an object. The imaging unit 110 includes a solid-state imaging device such as a CCD imaging device or a CMOS imaging device, a focus system that collects incident light from a subject on the solid-state imaging device, and a signal processing unit that performs image processing on the output of the solid-state imaging device. And have.

Composition determining unit 122 acquires the composition data set from the composition memory unit 131, the composition _{C 1,} C 2, ..., composition coefficients lambda ₁ respectively corresponding to the _{C p,} lambda _2, ..., composition and lambda _p coefficients Obtained from the storage unit 132. When the image analysis unit 121 detects _m feature patterns g _{1 to} g _m , the composition determination unit 122 uses each of the reference compositions C ₁ , C ₂ ,..., C _p and the feature patterns g _{1 to} g. _The degrees of association E ₁ , E ₂ ,..., Ep with _m are calculated as evaluation values. Specifically, the evaluation value in accordance with a first evaluation formula of the following E _1, E 2, _..., it is possible to calculate the E _p.

Here, C ₁ (i, j), C ₂ (i, j),..., C _p (i, j) are the composition C ₁ , C ₂ , .., C _p , and g ₁ (i, j), g ₂ (i, j),..., G _m (i, j) are respectively at the pixel position of the i-th row and j-th column on the screen. feature pattern g _1, g 2, _..., a pixel value of g _m (e.g., a value corresponding to the luminance value or color difference value of the predetermined color space). In the above evaluation _{formula, C 1 (i, j)} , C 2 (i, j), ..., the C p (i, j), the composition coefficient lambda ₁ respectively corresponding, λ _2, ..., λ _p is the weighting Has been.

Composition determining unit 122, the calculated evaluation value _{E 1,} E 2, ..., to determine the composition of the captured image based on _{E p.} The number of compositions to be determined can be one or more. For example, the calculated evaluation value E _1, E 2, _..., may be determined as the composition of the captured image the composition having the highest value among the E _p, or calculated evaluation value E _1, E _2, ..., two or more of the composition with a threshold value greater than of the E _p may be determined as the composition of the captured image.

The region estimation unit 123 estimates the region of interest in the captured image using the determined composition, and gives the estimation result to the display control unit 129. The region estimation unit 123 can select, as a region of interest, an image region corresponding to a region having a relatively large value in the determined multi-gradation pattern of the composition. More preferably, in order to extract a region of interest, an evaluation value F _s (i, j) may be calculated for each pixel according to the following second evaluation formula.

This evaluation value F _s (i, j) is an evaluation value for the determined composition C _s . The region estimation unit 123 can estimate the region of interest based on the two-dimensional distribution (evaluation pattern) in one screen of the evaluation value F _s (i, j). For example, an image region corresponding to a region having a relatively large value in the evaluation pattern may be estimated as the region of interest.

When the composition determination unit 122 determines a plurality of compositions, the region estimation unit 123 can calculate one overall evaluation value for each pixel based on a plurality of evaluation values respectively corresponding to these compositions. The region estimation unit 123 may estimate the region of interest based on a two-dimensional distribution (evaluation pattern) of the entire evaluation value in one screen. For example, when the composition determination unit 122 determines three compositions C ₁ , C ₂ , and C ₃ as the composition of the captured image, the evaluation values F ₁ (i, j), F ₂ (i, j), and F ₃ ( The added value or average value of i, j) may be used as the overall evaluation value, or one evaluation pattern may be generated by filtering a plurality of evaluation value distributions.

  The display control unit 129 extracts a local image of the region of interest estimated by the region estimation unit 123 and outputs this local image to the display 140. At this time, the display control unit 129 can convert the resolution of the extracted local image into a larger resolution so that the user can easily confirm visually. Alternatively, the display control unit 129 may output a composite image of the local image of the region of interest and another image to the display 140.

  When a plurality of regions of interest are estimated, the display control unit 129 may cause the display 140 to display images obtained by enlarging local images of these regions of interest on the display 140 in order from an image having a high evaluation value. The local image of the region of interest may be converted into a low-resolution thumbnail image, and a list of these thumbnail images may be displayed on the display 140.

  An operation example of the image processing apparatus 10 having the above configuration will be described below with reference to FIG. FIG. 3 is a flowchart schematically showing a processing procedure for extracting the region of interest by the image processing apparatus 10.

  When the user operates an input operation unit (not shown) such as a push button or a switch, the imaging unit 110 captures a subject and outputs a high-resolution captured image to the data processing unit 120 (step S10). ).

As described above, the image analysis unit 121 analyzes the input captured image and detects a plurality of feature patterns g _{1 to} g _m (step S11). Next, the composition determination unit 122 acquires a composition data set from the composition storage unit 131 and also acquires composition coefficients λ _{1 to} λ _p corresponding to the composition data set from the composition coefficient storage unit 132 (step S12). Then, the composition determining unit 122 uses the composition data set and the composition coefficients λ _{1 to} λ _p to relate the composition C ₁ , C ₂ ,..., C _p to the feature patterns g _{1 to} g _m. evaluation value _{E 1,} E 2 representing the degree, ..., determine the composition of the captured image to calculate the _{E p} (step S13).

FIGS. 4A, 4B, and 4C are diagrams schematically illustrating a reference three-part composition C ₁ , a Hinomaru composition C _2, and a triangular composition C ₃ , respectively. In tripartite composition C _1, 4 reference lines BL to each trisection one screen in the vertical direction and the horizontal direction, BL, BL, is set with weights to BL, the intersections CP of these reference lines, A larger weight is set. In Japanese flag Composition C _2, greater weight most is set in the center of the screen at the point A1, than the area A2 two points A1 surrounding the point A1 of the central set is small weight, area A2 to the area A3 surrounding the region A2 A smaller weight is set. In the triangular composition C _3, the weight is set in the reverse V-shaped reference line BL1, which is set smaller weights than the reference line BL1 to the reference line BL2 adjacent to the reference line BL1.

FIG. 5 is a diagram schematically illustrating an example of a captured image. When the image analysis unit 121 analyzes the captured image of FIG. 5, the feature patterns g _{1 to} g ₆ shown in FIGS. 6A to 6F can be detected. The characteristic patterns g ₁ , g ₂ , g ₃ , and g ₄ in FIGS. 6A to 6D are obtained by extracting regions of the same color or the same luminance from the captured image, FIG. 6E. the characteristic pattern g ₅ of those obtained by extracting an edge of an image component, and, FIG. 6 (F) are those obtained by extracting the object flower image.

Composition determining unit 122 according to the first evaluation formula, relevance E between the characteristic pattern _g 1 to g ₆ each and 6 of the composition _C 1 -C ₃ in FIG. 4 (A) ~ (C) it is possible to calculate the ₁ to E _3. By suitably setting the conditions, the evaluation value _{E 1} for tripartite composition _{C 1} 170, the evaluation value _{E 2} for HINOMARU Composition _{C 2} 120, the evaluation value _{E 3} for composition _{C 3} can be calculated respectively as 50 . In this case, the composition determining unit 122 may determine a tripartite composition C ₁ having the highest evaluation value E ₁ as the composition of the captured image (step S13).

After determining the composition of the captured image (step S13), the region estimation unit 123 estimates the region of interest using the determined composition according to the second evaluation formula (step S14). Here, for the composition coefficients λ ₁ , λ ₂ ,..., Λ _p , a large weight coefficient (for example, a value of 0.8) is used at a relatively low frequency for a composition that is used at a relatively high frequency. A small weighting factor (for example, a value of 0.2) can be set for each composition used. Alternatively, a large weight coefficient may be set for a composition having a relatively high importance for the user, and a small weight coefficient may be set for a composition having a relatively low importance for the user.

As described above, the composition determining unit 122, relevance _E 1 to E ₃ between the characteristic pattern _g 1 to g ₆ each and 6 of the composition _C 1 -C ₃ in FIG. 4 (A) ~ (C) If the tripartite composition C ₁ is calculated and determined as the composition of the captured image, evaluation formula used by the region estimation unit 123 is as follows.

However, for this evaluation formula, the image analysis unit 121 detects only the feature patterns g ₁ , g ₂ , and g ₅ in FIGS. 6A, 6B, and 6E, and the other feature patterns g ₃ , g 5. _4, was assumed not detect the _{g 6.} An evaluation pattern can be obtained by calculating the evaluation value F ₁ (i, j) for all values of the variables i and j. For example, the value of the first term C ₁ (i, j) · g ₁ (i, j) on the right side of this evaluation formula is the second term C ₁ on the right side so as to form the two-dimensional distribution of FIG. The values of (i, j) · g ₂ (i, j) are set to the third term C ₁ (i, j) · g ₅ (i, j) on the right side so as to form the two-dimensional distribution of FIG. ) Can be calculated so as to form the two-dimensional distribution of FIG. At this time, the evaluation value F ₁ (i, j) is calculated so as to form the two-dimensional distribution of FIG. 8, that is, the evaluation pattern.

The region estimation unit 123 can estimate an image region corresponding to a region having a relatively large value in the evaluation pattern of FIG. 8 as a region of interest (step S14). Then, the display control unit 129 extracts a local image of the region of interest estimated by the region estimation unit 123 from the captured image, converts the resolution of the local image, and outputs it to the display 140 (step S15). When a plurality of regions of interest are estimated, the region estimation unit 123 can assign a rank corresponding to the sum of the evaluation values F _s (i, j) in each region of interest to the region of interest. Since it is possible to estimate that the interest area has a higher rank, the higher the degree of interest for the user, the display control unit 129 preferentially displays an image of the interest area having a relatively high rank on the display 140. It is desirable to make it.

  Alternatively, the display control unit 129 can display the composite image DS shown in FIG. 9 on the display 140. In the composite image DS, thumbnail images of a plurality of regions of interest are displayed in a list with an order of “1”, “2”, “3”, “4”. Further, an image V1 of the display position confirmation area (that is, a reduced image obtained by converting the resolution of the captured image to a smaller one) V1 is displayed, and the user selects from among a plurality of regions of interest in this image V1. A frame T1 representing the position of the image of the region of interest is shown. Further, an enlarged image corresponding to the frame T1 is displayed as an image V2 in the image quality confirmation area. When the user operates the input operation unit to select any of the regions of interest, the position of the local image of the selected region of interest is indicated by a frame T1 in the image V1 of the display position confirmation region. At the same time, an enlarged image of the region of interest is displayed as an image V2 of the image quality confirmation region.

The effects of the image processing apparatus 10 according to the first embodiment are as follows. The composition determination unit 122 has degrees of association E ₁ , E between each of the compositions C ₁ , C ₂ ,..., C _p stored in advance in the composition storage unit 131 and the detected feature patterns g _{1 to} g _{m. 2} ,..., _Ep are calculated as estimated evaluation values. Composition determining unit 122, they estimated evaluation value E _1, E 2, _..., because it determines the composition of the captured image on the basis of E _p, the region estimating unit 123 in the captured image using the composition which is the determined It is possible to detect the region of interest with high accuracy. That is, it is possible to accurately estimate whether or not a feature pattern (for example, an object) in the captured image corresponds to the region of interest using the determined composition as a criterion.

Further, as shown in the first evaluation expression, the area estimation unit 123, composition coefficients lambda _1, lambda _2, ..., the lambda _p, respectively, the composition _{C 1,} C 2, ..., Te weighting _{C p} evaluation value _{E 1, E 2, ...,} it is possible to calculate the _{E p.} Estimate the region of interest by setting a small weighting factor for the evaluation value for a composition with relatively low importance or usage frequency and setting a large weighting factor for the evaluation value for a composition with relatively high importance or usage frequency The accuracy can be improved.

  Therefore, the user can easily confirm the image of the region of interest at the important position in the composition in the captured image.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. FIG. 10 is a functional block diagram illustrating a schematic configuration of the image processing apparatus 11 according to the second embodiment. The image processing apparatus 11 includes a data processing unit 120A and a data storage unit 130A.

As shown in FIG. 10, the data processing unit 120A includes a coefficient setting unit 124 between the composition determination unit 122A and the region estimation unit 123. The coefficient setting unit 124 is a processing result of the composition determination unit 122A. The composition coefficients λ _{1 to} λ _p stored in the composition coefficient storage unit 132 in the data storage unit 130A are updated.

Data storage unit 130A has a history storage unit 133, the history storage unit 133, composition _{C 1, C 2, ...,} C p of each cumulative use frequency _{N 1,} N 2, ..., a _{N p} I remember it. The composition determining unit 122A has the same function as that of the composition determining unit 122 in FIG. 1, and in addition to this function, the cumulative number of use times N ₁ , N ₂ ,..., N _p stored in the history storage unit 133. It has a function to update. The coefficient setting unit 124 updates the composition coefficients λ _{1 to} λ _p stored in the composition coefficient storage unit 132 based on the updated accumulated number of times N ₁ , N ₂ ,..., N _p .

  An example of the operation of the image processing apparatus 11 will be described below with reference to FIG. FIG. 11 is a flowchart schematically showing a processing procedure performed by the image processing apparatus 11.

Steps S10 to S13 in FIG. 11 are the same as steps S10 to S13 in FIG. That is, the imaging unit 110 images a subject (step S10), the image analysis unit 121 analyzes the input captured image to detect a plurality of feature patterns g _{1 to} g _m (step S11), and the composition determination unit 122A The composition data set is acquired from the composition storage unit 131, and the composition coefficients λ _{1 to} λ _p are acquired from the composition coefficient storage unit 132 (step S12). Then, the composition determination unit 122A uses the composition data set and the composition coefficients λ _{1 to} λ _p to relate the composition C ₁ , C ₂ ,..., C _p to the feature patterns g _{1 to} g _m. evaluation value _{E 1,} E 2 representing the degree, ..., determine the composition of the captured image to calculate the _{E p} (step S13).

Thereafter, the composition determining unit 122A is cumulative use stored in the history storage unit 133 the number _{N 1,} N 2, ..., updates the _{N p} (step S20). For example, when the composition determining unit 122A decides the composition _{C 1} as the composition of the captured image, the cumulative number of uses _{N 1} of the composition _{C 1} is incremented by _{1 (N 1 ← N 1 +1} ).

The coefficient setting unit 124 recalculates the composition coefficients λ _{1 to} λ _p based on the composition determination result by the composition determination unit 122A (step S21), and stores the calculated composition coefficients λ _{1 to} λ _p in the composition coefficient storage. The composition coefficients λ _{1 to} λ _p are updated by being stored in the unit 132 (step S22). At this time, it is desirable that the coefficient setting unit 124 sets the corresponding composition coefficient to a larger value as the cumulative number of uses increases, and sets the corresponding composition coefficient to a smaller value as the cumulative number of uses decreases. For example, the cumulative number of uses _{N r} of a composition _{C r} is the sum of the cumulative number of uses _N 1 to N _p of the total composition _{(= S = N 1 + N} 2 + ... + N p) ratio of the cumulative use frequency _{N r} for (= _Nr / S).

Also, for example, as shown in Tables 1 and 2 below, the cumulative number of uses of composition A, B, C, D (A to D is any one of C ₁ , C ₂ ,..., C _p ) is one. It is assumed that all of the composition coefficients are set to 0.25.

  In this case, when the composition determining unit 122A determines the composition A as the composition of the captured image, the cumulative use number of the composition A is set to 2 as shown in Tables 3 and 4 below, and the composition coefficient of the composition A is 0.4 (= 2 / (2 + 1 + 1 + 1)), and any of the composition coefficients of compositions B to D can be set to 0.2 (= 1 / (2 + 1 + 1 + 1)).

  After step S22, steps S14 and S15 (FIG. 3) are executed in the same manner as in the first embodiment.

  In this embodiment, steps S14 and S15 are executed after steps S20 to S22 are completed. However, the present invention is not limited to this, and steps S14 and S15 may be executed in parallel with steps S20 to S22. Alternatively, steps S14 and S15 may be executed before steps S20 to S22.

The image processing apparatus 11 according to the second embodiment has the same effects as those of the first embodiment, and further has the following effects. That is, in the image processing apparatus 11, the composition determination unit 122A records the usage history of the compositions C _{1 to} C _{p in} the history storage unit 133 (step S20), and the coefficient setting unit 124 uses the composition history λ ₁ based on the usage history. recalculate the to [lambda] _p and updates the storage contents of the composition coefficient storage unit 132 (step S21, S22). Therefore, the image processing apparatus 11 can learn the user's preference and can determine the optimum composition reflecting this preference as the composition of the captured image. Therefore, it is possible to further improve the estimation accuracy of the region of interest.

(Third embodiment)
Next, a third embodiment of the present invention will be described. FIG. 12 is a functional block diagram illustrating a schematic configuration of the image processing apparatus 12 according to the third embodiment. The image processing apparatus 12 includes a data processing unit 120B and a data storage unit 130B.

  As illustrated in FIG. 12, the data processing unit 120 </ b> B includes an object detection unit 125 and an image correction unit 126 between the region estimation unit 123 and the display control unit 129. The region estimation unit 123 provides the object detection unit 125 with position information that identifies the region of interest estimated as described above. The object detection unit 125 refers to the object data storage unit 134 in the data storage unit 130B and detects an object (for example, an image showing a face, a vehicle, a flower, a color configuration) in a local image of the region of interest. have.

  The object data storage unit 134 stores object data representing the characteristics of at least one reference object. The object detection unit 125 reads a local image of the region of interest from a buffer memory (not shown), and acquires object data from the object data storage unit 134. The object detection unit 125 attempts to detect an object by pattern matching the acquired object data and a local image of the region of interest. The detection result is given to the image correction unit 126.

  The image correction unit 126 has a function of referring to the image quality correction data storage unit 135 and executing image quality correction of the local image according to the detected object. The image quality correction data storage unit 135 stores image quality correction data sets respectively corresponding to the reference objects. The image correction unit 126 can acquire the image quality correction data set corresponding to the detected object from the image quality correction data storage unit 135 and use this to correct the image quality of the local image.

  An example of the operation of the image processing apparatus 12 will be described below with reference to FIG. FIG. 13 is a flowchart schematically showing a processing procedure performed by the image processing apparatus 12.

Steps S10 to S14 in FIG. 13 are the same as steps S10 to S14 in FIG. That is, the imaging unit 110 images a subject (step S10), the image analysis unit 121 analyzes the input captured image to detect a plurality of feature patterns g _{1 to} g _m (step S11), and the composition determination unit 122 The composition data set is acquired from the composition storage unit 131, and the composition coefficients λ _{1 to} λ _p are acquired from the composition coefficient storage unit 132 (step S12). In addition, the composition determination unit 122 uses the composition data set and the composition coefficients λ _{1 to} λ _p to relate to each of the compositions C ₁ , C ₂ ,..., C _p and the feature patterns g _{1 to} g _m. evaluation value _{E 1,} E 2 representing the degree, ..., determine the composition of the captured image to calculate the _{E p} (step S13). Then, the region estimation unit 123 estimates a region of interest using the determined composition (step S14).

  Thereafter, the object detection unit 125 refers to the object data storage unit 134 to detect an object in the local image of the estimated region of interest (step S30). Further, the image correction unit 126 appropriately corrects the image quality of the local image according to the detected object with reference to the image quality correction data storage unit 135 (step S31). For example, when the detected object is a face image, the image correction unit 126 acquires the image quality correction data set for the face image from the image quality correction data storage unit 135 and corrects the image quality using this. At this time, the image correcting unit 126 can color-convert the skin color of the face image into a preferable skin color according to the detected skin color of the face image. Alternatively, when the detected object is a sky image, the image correction unit 126 determines that the color of the image is a blue color if the image is a sky blue color close to blue, and if the image is a sky blue color close to red. The color of the image can be converted to a further sunset color.

  After step S31, step S15 (FIG. 3) is executed in the same manner as in the first embodiment. Even when the object detection unit 125 does not detect an object in the local image, step S15 (FIG. 3) is executed as in the first embodiment.

  The image processing apparatus 12 according to the third embodiment has the same effects as those of the first embodiment, and further has the following effects. That is, the image correction unit 126 can appropriately correct the image quality of the region of interest in accordance with the characteristics of the object (subject) in the local image of the region of interest. When the image processing device 12 is a digital camera, the image quality of the region of interest is appropriately and automatically corrected without the user setting the shooting mode in advance. It can be easily confirmed.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. FIG. 14 is a functional block diagram illustrating a schematic configuration of the image processing apparatus 13 according to the fourth embodiment. The image processing apparatus 13 includes a data processing unit 120C and a data storage unit 130C.

  As illustrated in FIG. 14, the data processing unit 120 </ b> C includes a composition generation unit 127 between the composition determination unit 122 and the region estimation unit 123. The composition generation unit 127 is determined by the composition determination unit 122. A function of generating a new composition based on the composition, and a function of newly setting a composition coefficient corresponding to the newly generated composition. The composition storage unit 131 stores the newly generated composition, and the composition coefficient storage unit 132 stores the newly set composition coefficient.

  An example of the operation of the image processing apparatus 13 will be described below with reference to FIG. FIG. 15 is a flowchart schematically showing a processing procedure by the image processing apparatus 13.

Steps S10 to S13 in FIG. 15 are the same as steps S10 to S13 in FIG. That is, the imaging unit 110 images a subject (step S10), the image analysis unit 121 analyzes the input captured image to detect a plurality of feature patterns g _{1 to} g _m (step S11), and the composition determination unit 122 The composition data set is acquired from the composition storage unit 131, and the composition coefficients λ _{1 to} λ _p are acquired from the composition coefficient storage unit 132 (step S12).

Further, the composition determination unit 122 uses the composition data set and the composition coefficients λ _{1 to} λ _p to determine the degree of association between each of the compositions C ₁ , C ₂ ,..., C _p and the feature patterns g _{1 to} g _m. evaluation value representative of _{E 1, E 2, ...,} to determine the composition of the captured image to calculate the _{E p} (step S13). Here, it is assumed that the composition determination unit 122 selects and determines two or more compositions having relatively high evaluation values as the composition of the captured image.

Thereafter, the composition generation unit 127 determines whether or not there is a composition whose evaluation value is equal to or less than a predetermined threshold (threshold value) among the two or more compositions determined in step S13 (step S40). ). For example, when two compositions C ₁ and C ₂ are determined as the composition of the captured image (step S13), the difference absolute value (= | E ₁ −E ₂ |) between the evaluation values E ₁ and E ₂ is a threshold value. It is determined whether or not the following is true (step S40). As shown in Table 5 below, when the four compositions A, B, C, and D have evaluation values of 120, 118, 20, and 10, respectively, if the threshold is 10, between the evaluation values of the compositions A and B The difference absolute value is less than or equal to the threshold value.

  When it is determined that there is no composition in which the absolute difference between the evaluation values is equal to or less than the predetermined threshold (step S40), the composition generation unit 127 shifts the processing to step S14. Thereafter, steps S14 and S15 (FIG. 3) are executed as in the first embodiment.

On the other hand, when it is determined that there is a composition in which the absolute value of the difference between the evaluation values is equal to or less than a predetermined threshold (step S40), the composition generation unit 127 generates a new composition C _{p + 1} based on these compositions ( Step S41). For example, as shown in FIG. 16, a new composition C _x can be generated by adding the three-part composition C ₁ and the Hinomaru composition C ₂ for each pixel. At this time, the weighted compositions C ₁ and C ₂ may be added by weighting each of the three-part composition C ₁ and the Hinomaru composition C ₂ (by multiplying the compositions C ₁ and C ₂ by respective coefficients). .

After that, the composition generation unit 127 stores the newly generated composition C _{p + 1} in the composition storage unit 131 (step S42). Furthermore the composition generating unit 127 recalculates the composition _{C p} Composition coefficients corresponding to _{+ 1 λ p + 1} and the composition coefficient lambda ₁ to [lambda] _p (step S43). Then, the composition generation unit 127 stores the composition coefficients λ _{1 to} λ _{p + 1} in the composition coefficient storage unit 132 and updates the composition coefficient (step S44).

  After step S44, steps S14 and S15 (FIG. 3) are executed in the same manner as in the first embodiment. In step S14, a newly generated composition may be used, or a plurality of compositions determined in step S13 may be used.

  The image processing apparatus 13 according to the fourth embodiment has the same effects as those of the first embodiment, and further has the following effects. That is, in the image processing apparatus 13, the composition generation unit 127 generates a new composition based on a composition having a small absolute difference value between evaluation values, in other words, a composition that is relatively similar to each other (step S41). A composition coefficient corresponding to the new composition is calculated (step S43), and the composition and composition coefficient stored in the composition storage unit 131 and the composition coefficient storage unit 132 are updated (steps S42 and S44). Therefore, since it is possible to generate and use a composition that better reflects the user's preference, it is possible to further improve the estimation accuracy of the region of interest.

  As mentioned above, although embodiment of this invention was described with reference to drawings, these are the illustrations of this invention, Various structures other than the above are also employable. For example, the display control unit 129 outputs a local image to the display 140, but is not limited to this. The display control unit 129 may have a function of transmitting a local image to an external device via a communication network, a communication line, or a connection cable.

  Further, the data processing units 120, 120A, 120B, and 120C are input with images captured by the imaging unit 110. However, the present invention is not limited to this, and an image read from a recording medium may be input. An image received from an external device via a communication network, a communication line, or a connection cable may be input.

  In the first embodiment, the imaging unit 110, the data processing unit 120, the data storage unit 130, and the display 140 may be incorporated in a device such as a mobile phone with a built-in digital camera, or these components. All or a part of 110 to 140 may be distributed and incorporated in a plurality of devices interconnected via a communication network. Examples of the communication network include a packet switching network (for example, the Internet and an intranet) and a small-scale network (for example, a wired or wireless LAN). Alternatively, all or part of the components 110 to 140 may be connected via a connection cable that conforms to a bus standard such as SCSI (Small Computer System Interface). The same applies to the components of the image processing apparatuses 11 to 13 according to the second to fourth embodiments.

  In the first embodiment, all or part of the functional blocks 121 to 123 and 129 of the data processing unit 120 may be realized by hardware such as a semiconductor integrated circuit, or may be a nonvolatile memory or You may implement | achieve with the program or program code recorded on recording media, such as an optical disk. Such a program or program code causes a computer having a processor such as a CPU to execute all or part of the processing of the functional blocks 121 to 123 and 129. The same applies to the components of the image processing apparatuses 11 to 13 according to the second to fourth embodiments.

1 is a functional block diagram illustrating a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. (A)-(G) are figures which show the typical example of a composition. 3 is a flowchart schematically illustrating a processing procedure performed by the image processing apparatus according to the first embodiment. (A), (B), and (C) are diagrams schematically showing a reference three-part composition, a Hinomaru composition, and a triangular composition, respectively. It is a figure which shows an example of a captured image roughly. (A)-(F) is a figure which shows roughly the characteristic pattern detected from the captured image of FIG. (A)-(C) are figures which show the two-dimensional distribution which the value of each term of an evaluation formula forms. It is a figure which shows the evaluation pattern which is a two-dimensional distribution of an evaluation value. It is a figure which shows the synthesized image containing the list of the image of an area of interest. It is a functional block diagram which shows schematic structure of the image processing apparatus which concerns on the 2nd Embodiment of this invention. 10 is a flowchart schematically showing a processing procedure by the image processing apparatus according to the second embodiment. It is a functional block diagram which shows schematic structure of the image processing apparatus which concerns on the 3rd Embodiment of this invention. 10 is a flowchart schematically illustrating a processing procedure performed by an image processing apparatus according to a third embodiment. It is a functional block diagram which shows schematic structure of the image processing apparatus which concerns on the 4th Embodiment of this invention. 14 is a flowchart schematically illustrating a processing procedure performed by an image processing apparatus according to a fourth embodiment. It is a figure for demonstrating an example of a composition production | generation method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 110 Imaging part 10, 11, 12, 13 Image processing apparatus 120,120A, 120B, 120C Data processing part 121 Image analysis part 122,122A Composition determination part 123 Area estimation part 124 Coefficient setting part 125 Object detection part 126 Image correction part 127 Composition generator 129 Display controller (output controller)
130, 130A, 130B, 130C Data storage unit 131 Composition storage unit 132 Composition coefficient storage unit 133 History storage unit 134 Object data storage unit 135 Image quality correction data storage unit 140 Display

Claims (24)

An image analysis unit that analyzes the input image and detects at least one feature pattern from the input image;
Calculating a degree of association between each of a plurality of prepared compositions and the feature pattern as an evaluation value, and determining a composition of the input image based on the evaluation value;
A region estimation unit that estimates a region of interest in the input image using the determined composition;
An output control unit that outputs a local image of the estimated region of interest;
An image processing apparatus comprising: The image processing apparatus according to claim 1, further comprising a composition coefficient storage unit that stores composition coefficients respectively corresponding to the plurality of prepared compositions.
The composition determination unit acquires the composition coefficient from the composition coefficient storage unit, weights the composition coefficient corresponding to each composition, and determines the degree of association between each of the weighted composition and the feature pattern. An image processing apparatus characterized by being calculated as an evaluation value. The image processing apparatus according to claim 2,
A coefficient setting unit for updating the composition coefficient stored in the composition coefficient storage unit;
A history storage unit for storing the cumulative number of times of use of each of the plurality of compositions prepared in advance;
Further comprising
The composition determination unit selects a composition having a relatively high value as the evaluation value from the plurality of compositions prepared in advance as the composition of the input image, and the cumulative use stored in the history storage unit Update the number of times,
The image processing apparatus, wherein the coefficient setting unit updates the composition coefficient stored in the composition coefficient storage unit on the basis of the cumulative use count stored in the history storage unit after the update.   4. The image processing apparatus according to claim 3, wherein the coefficient setting unit sets the corresponding composition coefficient to a larger value as the cumulative number of times of use is larger, and the corresponding composition as the cumulative number of times of use is smaller. An image processing apparatus, wherein the composition coefficient is updated by setting a coefficient to a small value. The image processing apparatus according to any one of claims 1 to 4,
An object detection unit for detecting an object in the region of interest;
An image correction unit that corrects the image quality of the local image according to the detected object;
An image processing apparatus further comprising: 6. The image processing apparatus according to claim 5, further comprising an object data storage unit that stores object data representing characteristics of at least one reference object.
The object detection unit acquires object data from the object data storage unit, and detects the object by pattern-matching the acquired object data and a local image of the region of interest. apparatus. The image processing apparatus according to claim 6, further comprising an image quality correction data storage unit that stores image quality correction data sets respectively corresponding to the reference objects.
The image processing apparatus, wherein the image correction unit acquires the corresponding image quality correction data set from the image quality correction data storage unit and corrects the image quality of the local image using the acquired image quality correction data set. The image processing apparatus according to claim 1,
A composition storage unit for storing data representing the plurality of compositions prepared in advance;
The image determination apparatus, wherein the composition determination unit obtains data representing the composition from the composition storage unit and calculates the evaluation value. The image processing apparatus according to claim 8, wherein
A composition generating unit that newly generates a composition based on the composition determined by the composition determining unit and newly sets a composition coefficient corresponding to the generated composition;
A composition coefficient storage unit that stores composition coefficients respectively corresponding to the plurality of compositions prepared in advance;
Further comprising
The composition storage unit stores data representing a composition newly generated by the composition generation unit, and the composition coefficient storage unit stores a composition coefficient newly set by the composition generation unit. An image processing apparatus. The image processing apparatus according to claim 9,
The composition determination unit selects and determines two or more compositions as the composition of the input image from the plurality of prepared compositions.
The composition generation unit newly generates a composition based on a composition in which a difference absolute value between evaluation values of each of the two or more selected compositions is equal to or less than a predetermined threshold value. .   The image processing apparatus according to claim 1, wherein the output control unit displays the local image on a display.   The image processing apparatus according to claim 11, further comprising an imaging unit that captures an image of a subject and generates the input image, and the display. Analyzing the input image to detect at least one feature pattern from the input image;
Calculating a degree of association between each of a plurality of prepared compositions and the feature pattern as an evaluation value, and determining the composition of the input image based on the evaluation value;
Estimating a region of interest in the input image using the determined composition;
Outputting a local image of the estimated region of interest;
An image processing method comprising: The image processing method according to claim 13, wherein in the step of determining the composition,
Obtaining the composition coefficient from a composition coefficient storage unit that stores composition coefficients respectively corresponding to the plurality of prepared compositions;
Weight each corresponding composition with the corresponding composition coefficient,
Calculating the degree of association between each of the weighted compositions and the feature pattern as the evaluation value;
An image processing method. 15. The image processing method according to claim 14, wherein in the step of determining the composition,
The composition having a relatively high value as the evaluation value is selected as the composition of the input image from the plurality of compositions prepared in advance, and the cumulative use of each of the plurality of compositions stored in the history storage unit Update the number of times,
After the update, the composition coefficient stored in the composition coefficient storage unit is updated based on the cumulative number of times stored in the history storage unit.
An image processing method. The image processing method according to claim 14 or 15,
Detecting an object in the region of interest;
Correcting the image quality of the local image according to the detected object;
An image processing method, further comprising:   17. The image processing method according to claim 14, wherein in the step of determining the composition, the composition storage unit stores data representing the plurality of compositions prepared in advance. An image processing method characterized by obtaining data representing a composition and calculating the evaluation value. The image processing method according to claim 17, comprising:
A step of newly generating a composition based on the determined composition and further setting a composition coefficient corresponding to the generated composition;
An image processing method, wherein data representing the newly generated composition is stored in the composition storage unit, and the newly set composition coefficient is stored in the composition coefficient storage unit. Image analysis processing for analyzing the input image and detecting at least one feature pattern from the input image;
Calculating a degree of association between each of a plurality of prepared compositions and the feature pattern as an evaluation value, and determining a composition of the input image based on the evaluation value;
Region estimation processing for estimating a region of interest in the input image using the determined composition;
Output control processing for outputting a local image of the estimated region of interest;
An image processing program for causing a computer to execute. 20. The image processing program according to claim 19, wherein the composition determination process includes:
Obtaining the composition coefficient from a composition coefficient storage unit that stores composition coefficients respectively corresponding to the plurality of prepared compositions;
Weight each corresponding composition with the corresponding composition coefficient,
An image processing program comprising a process of calculating a degree of association between each weighted composition and the feature pattern as the evaluation value. An image processing program according to claim 20, wherein
In the composition determination process, the composition having a relatively high value as the evaluation value is selected from the plurality of compositions prepared in advance as the composition of the input image, and is stored in the history storage unit. Including the process of updating the cumulative number of uses for each composition of
The coefficient setting process includes a process of updating the composition coefficient stored in the composition coefficient storage unit based on the cumulative number of times stored in the history storage unit after the update.
An image processing program characterized by that. An image processing program according to any one of claims 19 to 21,
An object detection process for detecting an object in the region of interest;
Image correction processing for correcting the image quality of the local image in accordance with the detected object;
Is further executed by a computer.   23. The image processing program according to claim 19, wherein the composition determination process represents the composition from a composition storage unit that stores data representing the plurality of compositions prepared in advance. An image processing program comprising a process of acquiring data and calculating the evaluation value. 24. The image processing program according to claim 23, wherein a composition is newly generated based on the composition determined by the composition determination process, and a composition coefficient corresponding to the generated composition is newly set. Is further executed on the computer,
The composition generation process includes a process of storing data representing the newly generated composition in the composition storage unit and storing the newly set composition coefficient in the composition coefficient storage unit. Image processing program.

Images