JP4586103B2 - Image extraction method and apparatus - Google Patents

Description

  The present invention relates to an image extraction method and apparatus, and more specifically to an image extraction method and apparatus that extract a target of a specific category from an image, and a storage medium.

JP-A-8-7107 JP-A-8-16779 JP 7-334675 A Japanese Patent Publication No. 7-34219 JP-A-3-240884 JP 7-225847 A JP-A-6-251149 JP-A-7-107266 JP-A-8-7075 JP-A-8-77336 Japanese Examined Patent Publication No. 6-4068 Japanese Patent Publication No. 8-20725 Japanese Patent No. 2642368 Japanese Patent Laid-Open No. 9-50532 JP-A-6-168331 JP-A-5-242160

M. Kass, A. Witkin, D. Terzopoulos, Snakes, "Active Contour Models, International Journal of Computer Vision, pp.321-331, 1988 A.De Bimbo and P.Pa1a, "Visua1 Image Retrieval by Elastic Matching of User Sketches", IEEE Trans. On Pattern Analysis and Machine Intelligence, vol. 19, pp. 121-132, 1997 L.H.Steib and J.S.Duncan, "Boundary Finding with Parametrically Deformable Mode1s", IEEE Trans. On Pattern Analysis and Machine Intelligence, vol. 14, pp. 1061-1075, 1992 F. S. Cohen, Z. Huang, Z. Yang, "Invariant Matching and Identification of Curves Using B-Splines curve Representation", IEEE Trans. On Image Processing, vo1. 4, pp. 1-10, 1995 G. C. H. Chuang and C. C. J. Kuo, "Wave1et Descriptor of Planar Curves: Theory and App1ications", IEEE Trans. On Image Processing, vol. 5, pp. 56-70, 1996

  Conventionally, various methods are known as processing for extracting a specific subject or object from an image. The first method is to select a region having a predetermined range of color component values (or shade values) including pixel values of an object specified by the user or a point on the background, or specify the region of the extracted subject. A method of repeating (JP-A-8-7107, JP-A-8-16779, JP-A-7-334675, JP-B-7-34219, etc.), and a rough rough contour including an outline to be extracted A method of specifying a line region or a local region and obtaining and cutting out a target boundary contour line by processing such as thinning or clustering within the specified region (Japanese Patent Laid-Open Nos. 3-240884 and 7-225847, JP-A-6-251149, JP-A-7-107266, JP-A-8-7075, JP-A-8-77336, JP-B-6-4068 and JP-B-8. 20,725 JP, etc.) there is.

  As a second method, a closed curve (or polygonal boundary) is set so as to roughly surround the image portion to be cut out, and a cutout mask image that is almost similar to the shape of the target object is generated using only color component information. There is a method (Japanese Patent No. 2642368).

  As a third method, image dictionary data obtained by mosaicing an input grayscale image by a coarse / fine search using multiple resolutions in a configuration (such as Japanese Patent Laid-Open No. 9-50532) that detects a target object from an image and extracts its contour A dynamic contour method (M. Kass, A. Witkin, D. Terapoulos, Snakes, "Active Control Models, International Journal of Computer" that obtains the position and size of a target object with reference to FIG. There is a method of extracting a contour using Vision, pp.321-331, 1988).

  In addition, as a technique that can be used to determine the presence or absence of a specific object in an image, or to search and extract an image in which a specific object exists from a database, a template prepared in advance is scanned on the image, and each position A method of calculating a matching level in the image, and searching for a position where the matching level is equal to or higher than a predetermined threshold (Japanese Patent Laid-Open No. 6-168331), inputting a region and a component name of a component in an image when creating an image database In addition, a method of searching for an image having a predetermined feature at a high speed (Japanese Patent Laid-Open No. 5-242160), a degree of similarity with an example given by a user (such as a sketch and an image) is obtained, and the degree of similarity is high. Image retrieval method (Reference 1: A. De Bimbo and P. Pa1a, “Visua1 Image Retrieval by Elastic M atching of User Sketches ", IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. Steib and JS Duncan, “Boundary Finding with Parametrically Deformable Mode 1s”, IEEE Trans. On Pattern Analysis and Machine Int.

As a method for modeling a curve parametrically, a method using a spline curve (FS Cohen, Z. Huang, Z. Yang, “Invariant Matching and Identification of B. Spins Elevation E.” Image Processing, vo1. 4, pp . 1-10, 1995), method (Document 2 using the Fourier descriptors), and c methods using E over Brett descriptor (G. C. H. Chuang and C. C. J Kuo, “Wave1 Descriptor of Planar Curves: Theory and Applications,” IEEE Trans. n Image Processing, vol. 5, pp. 56-70, 1996) and the like.

  However, since the image extraction method according to the first method described above requires an operation based on a relatively detailed instruction from the user, the process of searching for and extracting (cutting out) an object having a specific shape from the image is automated. Can not.

  In the second method typified by a method of specifying a closed curve or the like that roughly encloses an image part to be cut out (Japanese Patent No. 2642368), the ratio of the area of the region having the same color component included in the closed curve is used. If there is an area in the background within the closed curve that has the same color as the subject, or if the closed curve area has an area that is twice or more the area to be cut out, erroneous extraction such as extraction of the background portion is likely to occur. There is a problem and lacks versatility.

  In the contour detection apparatus described in Japanese Patent Laid-Open No. 9-50532 showing the third method, the so-called dynamic contour method is used, so that the background portion (portion other than the extraction target) of the input image has a complex texture pattern. In many cases, it is difficult to reflect the complexity of the contour line to be extracted. That is, there is a problem that an unnatural uneven shape deviating from the original contour shape is extracted.

  The contour extraction method described in Document 2 assumes a probabilistic model in which the contour model parameters and the contour model error are assumed to be Gaussian distributions. Therefore, when those distributions are actually non-Gaussian, the correct contours are assumed. Is difficult to extract efficiently, and the tolerance for the difference between the contour model and the shape of the extraction target is low.

  It is an object of the present invention to provide an image extraction method and apparatus that solves such problems.

  It is another object of the present invention to provide an image extraction method and apparatus that can extract a desired image with higher accuracy at a higher speed.

Image extracting method according to the present invention, the first matching processing first matching means obtains a predetermined evaluation function value related to the value of the predetermined feature quantity of a template including the input image and the local graphic element A first matching step to be performed; a local transformation step in which the local transformation means performs local geometric transformation on a region near a predetermined representative point on the template; and a transformation parameter calculation means to obtain a transformation parameter of the local transformation step. A conversion parameter calculation step for obtaining the first matching processing result and at least one of a predetermined deformation mode or a probability distribution, a template obtained by the second matching means by the local conversion step, and the input image a second second matching step of performing matching processing between an image region extracting means is the Characterized by comprising an image region extraction step of extracting an image region including a pre-Symbol contour based on 2 of the matching process result.

Image extracting apparatus according to the present invention, first matching means for performing a first matching processing for obtaining the predetermined evaluation function value related to the value of the predetermined feature quantity of a template including the input image and the local graphic element A local transformation means for performing a local geometric transformation on a region near a predetermined representative point on the template, a transformation parameter of the local transformation means, the first matching processing result, and a predetermined deformation mode. Alternatively, conversion parameter calculation means obtained based on at least one of probability distributions, second matching means for performing a second matching process between the template obtained by the local conversion means and the input image, and the second matching by comprising an image region extracting means for extracting an image region including a pre-Symbol contour based on the result of the process, characterized in.

  According to the present invention, the template data is preliminarily given by executing the template data scanning and the conformity evaluation on the shape, and the deformation (movement range, shape constraint condition, local geometric transformation, etc.) given the constraint condition. It is possible to automatically search for an object that matches the given information from the image. In addition, the tolerance of deformation is high in a suitable range, and contour extraction and image extraction resistant to noise can be performed.

  When deforming a template, an overall set of constraints on deformation (movement of representative points) is given, or a local geometric transformation is estimated that preserves the geometric phase, so global shape information Are always maintained and are not easily affected by local noise. Therefore, even if the background has a complicated pattern, extraction of an unnatural contour is effectively prevented.

It is a schematic block diagram of one Example of this invention. 2 is a schematic block diagram of an image processing device 16. FIG. It is a schematic diagram which shows an example of template data, and a movement range. It is a table | surface which shows the example of template data. It is an operation | movement flowchart of a present Example. 3 is a display screen example of the image display device 14. It is a schematic diagram of the example of description of the edge ratio on a thick line. It is a schematic diagram which shows the mode of the deformation | transformation of the template figure in the update template production | generation process process. It is a schematic diagram which shows the mode of the movement of the representative points P6 and P7 accompanying P5 having moved to P5 '. It is an operation | movement flowchart of the change part of the Example which changed the update template production | generation process (S7 of FIG. 5). It is an example of a human body template model. It is a flowchart which shows the estimation procedure of a geometric (affine) transformation parameter. It is a schematic diagram explaining the outline estimation procedure of the template model which has an indefinite part. It is a flowchart of the template update process in 3rd Example. It is an example of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows a schematic block diagram of an embodiment of the present invention. 10 image input device, 12 is an image memory for storing image data and the template model data, 14 an image display device, 16 is an image processing apparatus, 18 is a pointing selection apparatus comprising a pointing device such as a mouse . The image input device 10 includes an imaging device such as an image scanner or a digital camera, or an image data transfer device connected to an image database.

  In this embodiment, image data including an object to be separated and extracted from the background is input from the image input device 10 and stored in the image storage device 12. The image data stored in the image storage device 12 is displayed in a predetermined format at a predetermined position on the screen of the image display device 14.

  A schematic configuration of the image processing apparatus 16 is shown in FIG. 20 is a feature quantity extraction device that extracts the feature quantity of the input image, 22 is a template model selection apparatus, 24 is a template data generation apparatus that generates template data of multiple resolutions, and 26 is a template scan and a feature of the input image. A template scanning device that executes a first matching evaluation with a quantity; 28, a moving range / condition update setting device that updates and sets the moving range / condition of each representative point on the template; 30, an update template generation device; Is a shape matching degree evaluation apparatus that evaluates the shape matching degree after deformation by evaluating the shape fitting degree after deformation by the second matching process with the image feature amount, and 34 is a spline that smoothly connects the representative points of the update template A boundary line generating device that generates a boundary line of the graphic element by using an interpolation curve, and 36 represents the boundary line of the update template as an image table. Border display device which displays superimposed on the input image on the screen of the device 14, 38 is a mask data generation device for generating mask data indicating a boundary inside region of updating template. Although not shown, a storage device for storing boundary lines and / or mask data of the update template is further provided.

  First, a template model and a parametric curve expression method used in this embodiment will be described.

Template data is extracted category (person, car, fish and the like airplanes) and abstracted graphical elements or representative image (graphic) patterns main components a representative, typically constituting them contour The background including the line data or its outline is given by a solid gray image or color image. The contour line data is not limited to representing the outer shape of the corresponding category, but may include main shape data constituting the category (for example, eyes, nose and mouth constituting a human face). Not too long.

  The contour lines of the graphic elements belonging to each category are composed of curves or line segments that pass between the representative points, and are preliminarily expressed parametrically by spline curves, Fourier descriptors, wavelet descriptors, and the like.

  For example, the entire contour model is composed of J spline curves, and the jth spline curve

ここに、ｊ＝０，１，・・・，Ｊである。また、Ｑ_N，m＋1（ｔ）は、

のように表わされる。Ｃ_Ｎ＝［Ｃ_ｘ，Ｎ，Ｃ_ｙ，Ｎ］は、Ｎ番目の制御点位置を示すベクトルである。 r _j (t) = [x _j (y), y _j (t)]
Is an m _{j -th} order B-spline with n _j +1 control points,

Here, j = 0, 1,..., J. Q _{N, m + 1} (t) is

It is expressed as C _N = [C _{x, N} , C _{y, N} ] is a vector indicating the Nth control point position.

と表現される。但し、

である。ｔは、弧長ｓと全周長Ｓを用いて、ｔ＝２πｓ／Ｓのように表わされる。フーリエ記述子を用いて曲線の一部を表現する場合には、その両端点で When using a Fourier descriptor,

It is expressed. However,

It is. t is expressed as t = 2πs / S using the arc length s and the total circumference S. When expressing a part of a curve using a Fourier descriptor,

  x (t) = x (2π−t)

y (t) = y (2πone t)
Therefore, b _k = d _k = 0.

ここで、

である。 When using wavelet descriptors,

here,

It is.

ここに、ｋ，ｍ，ｎは、０又は正の整数であり、関数φ，ψは、双直交３次スプライン関数等が用いられる。 Each wavelet function has periodicity and is defined as follows. That is,

Here, k, m, and n are 0 or a positive integer, and a bi-orthogonal cubic spline function or the like is used for the functions φ and ψ.

  Contour lines constituting the template do not represent the contour line representing the shape of the extraction target as a single closed curve, but are composed of partial contour lines of the shape representing the extraction target category (for example, the same applicant as the present application). (Japanese Patent Laid-Open No. 7-320086) may be used.

  The density distribution of the representative points and the parameters of these curves (for example, the order of the B-spline function and the number of control points) express and reflect the complexity and smoothness of the shape of the object to be extracted. FIG. 3A shows an example of graphic elements constituting the template. The template data generation device 24 may generate this graphic element by extracting contour lines from the image data.

  As exemplified in FIGS. 3B and 4A and 4B, the template data includes data on parameters (degree and type of curve) related to parameters such as representative point positions and spline curves. Point movement range data or shape constraint conditions described later are given.

  Rather than explicitly representing the movement range at each representative point, the concave or convex shape of the shape in the vicinity of the representative point is simply retained, or the curvature is retained and no new loop is generated. Local shape conditions may be given and given as movement condition data for each representative point. In this case, the movement of the representative point is performed under the two conditions that the local shape condition thus determined is not violated and that the later-described degree of conformity regarding the shape of the updated template after the movement does not satisfy the reference value. The range is gradually expanded to search, and the movement of the representative point is stopped at a position where the fitness level is equal to or higher than the reference value.

とする。（ｘ_ｊ，ｙ_ｊ）は、ｊ番目の代表点位置を示す。数８は、代表点（ｘ_ｊ，ｙ_ｊ）の近傍の形状に関する凹凸及び曲率の保持されるべき範囲の概算値を示す。 As the shape constraint condition, for example, as described in the fourth line of FIG.

And (X _j , y _j ) indicates the j-th representative point position. Expression 8 shows an approximate value of the range in which the unevenness and curvature of the shape in the vicinity of the representative point (x _j , y _j ) should be maintained.

とする。これと形状拘束条件により、（ｘ_ｊ−１，ｙ_ｊ−１），（ｘ_ｊ，ｙ_ｊ）が定まれば、（ｘ_ｊ＋１，ｙ_ｊ＋１）は、数８及び数９を満たす範囲の平均値として求めることが出来る。数９でｄ_ｊは、点（ｘ_ｊ，ｙｊ）と（ｘ_ｊ＋１，ｙ_ｊ＋１）とを結ぶ線分長の基準値、μはその許容する変化幅の基準値をそれぞれ示す。 As a constraint on size, for example,

And If (x _j−1 , y _j−1 ) and (x _j , y _j ) are determined by this and the shape constraint condition, (x _{j + 1} , y _{j + 1} ) is the average of the range satisfying Equation 8 and Equation 9. It can be obtained as a value. In Equation 9, d _j represents a reference value of the length of the line segment connecting the points (x _j , yj) and (x _{j + 1} , y _{j + 1} ), and μ represents a reference value of the allowable change width.

  FIG. 3 shows the j−1, j, and j + 1th representative points, the parametric curves connecting them, and the movement ranges. The movement range data is given in advance as giving a deformation tolerance from the original template shape accompanying the movement of the representative points constituting the template. Each movement range generally differs depending on knowledge about the non-rigidity (elasticity, presence / absence of joints, etc.) and a priori shape of each part to be extracted, and can also be changed appropriately in the process of template deformation and update.

  For example, the head of a person usually has a convex shape approximated by a quadratic curve (elliptical shape) function, but the movement range of the representative points constituting this head does not generate a new concave portion due to the movement. It is determined on the condition. When the contour curve forming the head is expressed as a function of the arc length along the contour line, the occurrence of a recess is detected by the second-order differential value (or an approximate value corresponding thereto) being positive. .

  Therefore, the size and shape of the movement range of a certain representative point is such that when a representative point adjacent to that point is fixed, the curve connecting the representative points is in the vicinity of the representative point after the movement. (Approximate value) is determined to be in a non-positive range. Strictly speaking, this range changes relatively with the movement of the other representative points excluding the representative point. However, as the range in which the deformation amount is not large and the similarity of the shape is maintained, it is previously determined for each representative point. It is fixed and set (the size and shape of the range may be different for each representative point).

  An image extraction procedure in this embodiment will be described. FIG. 5 shows a flowchart of the operation. Image data is input from the image input device 10 such as a photographing unit, is stored in the image storage device 12, and is displayed on the screen of the image display device 14 as illustrated in FIG. 6 (S1). In FIG. 6, 40 is a screen of the image display device 14, 42 is an input image, 44 is template model image data, 46 is a target position confirmation indicator, 48 is an extraction instruction button, and 50 is a template scroll button.

  The feature quantity extraction device 20 obtains a feature quantity distribution (such as edge intensity distribution or information on the local spatial frequency at each point) of the input image, and binarizes it with a predetermined threshold (S2).

The template selection device 22 displays the template model image data 44 on the display screen 40 of the image display device 14. When the user selects template data in the extracted category, information on the selected template data (for example, data illustrated in FIG. 4) is read (S3). By scrolling the display contents of the template model image data 44 from the list displayed in the template model image data 44 with the scroll button 50 to display a desired one and selecting it with the instruction selection device 18, one desired template model is displayed. Image data can be selected. For the extraction category, for example, the name of the extraction category may be verbally input by a voice input device such as a microphone and a voice recognition device. When the desired template data is selected by the user, the template data generating apparatus 22 automatically generates a set of templates plurality each other size from the selected template data different (typically less than 10) (S4).

  The template scanning device 26 raster-scans each template on the image at a predetermined sampling pitch (for example, one tenth of the width of the template figure) according to the size (S5), and the first scanning is performed at each location. The shape matching degree is obtained (S6). Specifically, as a parameter representing the degree of conformity of the shape, an edge on the outline of the graphic element on the template or on the bold line obtained by thickening the outline to a predetermined width (typically about 5 pixels) The sum of the inner products (absolute values) of the gradient vectors indicating the ratio of the points and the direction of the edge intensity gradient at the edge points (or their distance) The ratio of the above points) is obtained.

  As illustrated in FIG. 7, the edge ratio on the thick line refers to the sampling point on the center line of the thick line and the minute point (thick line region cell) including the sampling point when the edge exists. Is the ratio of the number of edge points to the total number of sampling points. The first shape matching degree represents the overall shape similarity between the extraction target in the image and the template.

  As the fitness parameter, as shown in Japanese Patent Laid-Open No. 9-185719 filed by the same applicant as the present application, the reciprocal of an error relating to image data (such as color intensity distribution of color components) in the vicinity of the contour line of the template model. It may be used as a measure of goodness of fit.

  In the process of scanning of multiple templates with different sizes and fitness evaluation, when the fitness parameter of a certain template exceeds a predetermined reference value, scanning of the template is stopped and the target object is confirmed for the user. Indicator 46 (FIG. 6) is displayed. At this stage, information on the position and size of the target is extracted. When the user presses the extraction button 48 to instruct the start of the extraction execution process, the next update template generation process (S7) is executed.

  FIG. 8 shows a state of deformation of the template figure in the update template generation process. In FIG. 8A, the degree of matching between the initial shape of the template figure and the object shape of the input image (such as the edge ratio on the interpolation curve) of the representative points P1, P2,. Is displayed. Representative points other than these are points with relatively good matching, and an optimal destination is detected within each movement range as shown in FIG.

  In the update template generation process (S7), the template graphic element transformation device 30 moves the representative point of a certain graphic element (curve segment) constituting the template graphic within the movement range (or according to the movement conditions such as Equation 8 and Equation 9). ) Moved (S7a), the template graphic element deforming device 30 generates a new curved segment connecting the representative points after the movement (S7b), and the shape conformity evaluation device 32 determines the second shape conformance at each portion. By evaluating (S7c), the representative points are moved so that the second shape conformity is equal to or greater than a predetermined reference value, and finally, the boundary line generating device 34 is a curve ( Or a line segment). In the course of executing these processes, movement conditions are relaxed as necessary (S7d), and finally the entire template is deformed to satisfy the movement conditions (S7e).

  The boundary line display device 36 displays the boundary line superimposed on the input image on the screen of the image display device 14. The update template data is generated as binary mask data indicating the boundary line data or the area.

  Movement and fitness evaluation are not performed while moving each representative point at the same time. Instead, priority is given to moving a representative point that relays a segment with a low edge ratio that is adjacent to a representative point that is an end point of a segment with a high edge ratio. Is efficient. FIG. 8B shows the shape of the update template representing the curve connecting the representative points after moving in such a manner. In FIG. 8B, Pi in FIG. 8A has moved to Pi ′ in FIG. 8B. For example, in FIG. 8A, for a representative point P5 adjacent to the representative point Q that is an end point of a segment with a high edge ratio, a predetermined radius (for example, an arc length from Q to P5) centered on the point Q is used. First, the destination of P5 is searched on the arc.

  In the search for the destination of the representative point position, the above-mentioned movement range is represented by a predetermined probability distribution (particularly, a value having a value of 0 outside the movement range), and the representative point position given by the distribution is stochastically The position where the value is greater than or equal to a predetermined reference value (or large amount) may be estimated by generating and evaluating the shape adaptability of the curve segment including the representative point each time.

  The movement destination of the representative point is determined based on the evaluation result of the second shape suitability. The second shape fitness is defined for each representative point, and the edge ratio on the curve (or line segment) connecting the representative points (usually two) adjacent to the representative point and the curve (or line segment). ) It is represented by a linear sum of the degree of coincidence (the proportion of points that coincide within the allowable range) within the allowable range of the above feature amount gradient (such as the gradient of the edge intensity distribution).

  The movement range / condition update setting device 28 expands the movement range of the representative point according to the second shape suitability value or relaxes the movement condition (S7d), and from the area that has not been searched yet. An optimum representative point position, that is, a representative point position that maximizes the second shape suitability may be searched and moved. That is, by deforming the template graphic element described above based on the second shape conformance, even if the target has a slightly different contour from the contour shape data of the template belonging to the extraction category, As a result of the expansion of the representative point movement range or the relaxation of the movement conditions, the entire contour can be extracted.

  In the example shown in FIG. 8, as P5 moves to P5 ', the movement range of the remaining representative points that are present continuously is updated based on the position of P5'. This is shown in FIG. Specifically, for example, the movement range is set with the constraint condition that at least one of the shape of the curve passing through P5 ′, P6 ′, and P7 ′ is convex and the size of the curve is maintained at the same level. Set and run the search. Which condition or both conditions are used depends on the degree of non-rigidity to be extracted and the presence or absence of a joint, but is defined in the template data in advance.

  In FIG. 9, the sizes of the search ranges of P6 and P6 ′ and P7 and P7 ′ are approximately the same, and the arrangement of each search range is a constraint condition that the shape is convex (for example, in Equations 8 and 9). (Conditions shown) are applied. As shown in FIG. 9 (b), the newly set movement range may be far away from the original movement range.

  When the representative point is moved by giving only the movement constraint condition without explicitly setting the movement range, the rate of change in the tangential direction of the curve passing through P5 ′, P6 ′, and P7 ′ is positive ( (See Equation 8 and Equation 9).

  As the curve connecting the representative points after movement, it is desirable to use the same type of curve as the curve constituting the original template. However, the order of the parameters and the like can be changed to best suit the shape to be extracted. For example, when using a B-spline curve, the order and the number of control points may be changed.

ここで、Ｔは転置行列を示し、Ｂ（ｓ）はＢスプライン行列、ｔは曲線セグメントの長さ、ＱはＢスプラインパラメータをそれぞれ示す。 An example of a method for determining a curve parameter constituting the updated template will be described. It is assumed that a certain curve segment v (s) = (x (s), y (s)) ^T is modeled using an nth-order B-spline function. The optimum value of the parameter after the representative point movement is estimated as follows, for example. That is,

Here, T represents a transposed matrix, B (s) represents a B spline matrix, t represents a length of a curve segment, and Q represents a B spline parameter.

となる。ここで、（σ_ｘ）^２，（σ_ｙ）^２は、Ｅ_ｖ’（ｓ）＝（Ｅ_{ｘ’（ｓ）}，Ｅ_{ｙ’（ｓ）}）に対する分散を示す。 The contour to be extracted corresponding to the curve segment is set to v ′ (s) = (x ′ (s), y ′ (s)) ^T, and the feature value (edge intensity value, etc.) at each point is E _v ′ ( s), and E _v ′ (s) is normally distributed with respect to E _v (s), the probability distribution density function is

It becomes. Here, (σ _x ) ² and (σ _y ) ² indicate variances with respect to E _v ′ (s) = (E _{x ′ (s)} , E _{y ′ (s)} ).

で与えられる。Ｑの階数等は、以下に示す情報量基準ＡＩＣ（Ｑ）が最小となるｎなどのようにして求められる。具体的には、

である。Ｎｕｍｂｅｒ（Ｑ）は、Ｑの要素数である。 In this embodiment, in order to simplify the explanation and make it easy to understand, E _v (s) is binary (0 or 1), and the contour model data of the template figure is E _v (s) = 1. It shall be given as a set. At this time, the most likely estimate is

Given in. The rank of Q and the like are obtained in the following manner, such as n that minimizes the information criterion AIC (Q) shown below. In particular,

It is. Number (Q) is the number of elements of Q.

  The expansion of the movement range typically involves simply expanding the shape of the original movement range at a predetermined magnification. However, it may be an anisotropic expansion in which the tangent direction of the curve including the representative point of the template graphic element is increased. At this time, it goes without saying that the search is performed in a range satisfying the above-mentioned movement condition.

  If the number of representative points whose second shape conformance is less than a predetermined reference value (for example, 0.3) is less than or equal to a certain number (for example, a majority) of the template even after the restriction on movement is relaxed, The above-described processing may be executed by stopping the deformation and switching to another template having a different size.

  When the edge ratio is very high (for example, 0.9 or more) on the curve segment of the template connecting the representative points in the vicinity of the representative point, it is not necessary to search for the movement destination of the representative point as described above. Good.

  As shown in FIG. 8C, when there is a non-edge point of the feature amount extracted previously on the curve (or line segment) connecting the representative points, the non-edge point is added as an additional representative point. (If such non-edge points are continuous, the midpoint on the continuous non-edge line segment is set as a new representative point), and the movement range is similarly set (or moved) for the representative point. It is also possible to perform a movement search so as to satisfy the condition) and move to a position satisfying the constraint condition to generate a curve connecting the representative point adjacent to the representative point. The addition of the representative point can be automatically performed, but may be executed according to a user instruction. If an appropriate representative point position is not detected, the movement range is expanded or the movement condition is relaxed in the same manner.

  Finally, the updated template contour line is displayed (S8), mask data is generated from the contour line data (S9), and the contour line or mask data is recorded in the storage device as necessary (S10), and the processing is performed. Exit.

Each of the above processes may be formed in a predetermined program format that can be executed by a computer, and each part of the processing may be performed by hardware such as a predetermined gate array (FPGA, ASLC, etc.) or a predetermined computer program. it is obvious that can be realized in a form partially hardware module that implements part of the elements shown in 1 are mixed. When a hardware module is included, each component does not necessarily have the same configuration as that illustrated in FIG. 1, and the function is substantially the same, or one component is a plurality of components in FIG. 1. Needless to say, those having functions are included in the technical scope of the present invention. This point is the same in the following embodiments.

  An embodiment in which the update template generation process (S7 in FIG. 5) is changed will be described. In this modified embodiment (second embodiment), the template scan and the fitness evaluation are executed, the position of the extraction target and an appropriate template model are selected, and then the update template generation process specific to this embodiment is executed. To do. A flowchart of the specific processing procedure is shown in FIG.

  First, representative points including a segment having a low shape matching degree are automatically extracted from the template data before update (S11), and the template data is divided into local regions so as to include the representative points (S12). In the case of using a template that has been divided into regions for each shape element that has been collected in advance, this region division processing is unnecessary.

  In the local region including the representative point, estimation and conversion of geometric conversion parameters as described below are executed (S13). In the present embodiment, the template is updated by local geometric deformation (such as local affine transformation, local bilinear transformation, and local perspective transformation) in the vicinity region of a predetermined representative point. The geometric transformation parameter is obtained as a parameter that best fits the object in the input image based on the Bayesian estimation method of the stochastic process regarding the curve parameter.

  Further, in order to finely adjust the contour model shape as necessary, a curved segment connecting the converted representative points is generated (S14), and the second shape conformance is evaluated, as in the above-described embodiment. (S15) The template shape is deformed and updated (S16).

  The significance of performing local geometric deformation and fitness evaluation in this embodiment is that the relative positional relationship between representative points (ie, geometric phase and concave or convex shape) is generally maintained, By converting the positions of a plurality of representative points in a specific region so as to match a given deformation mode, it is possible to improve the efficiency of searching for a contour model that matches the extraction target.

ｘ’＝ｘｗ’、ｙ’＝ｙｗ’、［ｕｖ］は変換前の座標、［ｘｙ］は変換後の座標、｛ａ_ｉｊ｝｛ａ_ｉ｝｛ｂ_ｉ｝及びｗは変換パラメータである。 The affine transformation, perspective transformation, and bilinear transformation are each given by the following equations. That is,

x ′ = xw ′, y ′ = yw ′, [uv] is a coordinate before conversion, [xy] is a coordinate after conversion, {a _ij } {a _i } {b _i }, and w are conversion parameters.

  The neighborhood area may be a rectangular area having a predetermined size including a plurality of representative points, or the template may be divided in advance for each predetermined block, and the deformation may be performed for each block. For example, in a template representing a human body, a region divided into blocks for each meaningful shape, such as a head, a torso, a chest, and a leg, may be used as a neighborhood region. FIG. 11 shows the result of dividing each part of the human body in advance and performing a predetermined geometric transformation on each part of the original data FIG. 11A. FIG. 11B, FIG. 11C, and FIG. Is illustrated. Needless to say, the contour data actually used is the outermost contour of each figure in FIG.

上式の最右辺の対数をとり、またＰ（ｂ）が定数であることを考慮すると、結局、

となる。 The template is deformed by moving the position of each point (or representative point) in the vicinity region by the above-described geometric transformation. Taking affine transformation as an example, a procedure for obtaining an optimum transformation parameter will be described. When the prior probability of the transformation parameter set {a _ij } is P (a), the prior probability of the image data is P (b), and a transformation parameter is given, the template figure element obtained by the transformation is the feature quantity of the input image. And P (b | a), which is a feature quantity b of the image, and P (a | b), the posterior probability that the conversion parameter set of the corresponding template is {a _ij } Let the conversion parameter to be obtained be {a _opt }. {A _opt } is given by the following equation as a parameter that gives the maximum posterior probability max [P (a | b)] according to Bayes' law. That is,

Taking the logarithm of the rightmost side of the above equation and considering that P (b) is a constant, after all,

It becomes.

Next, a procedure for estimating a geometric (affine) transformation parameter will be specifically described with reference to FIG. The prior probability distribution P (a) of the geometric transformation parameter and the posterior probability P (b | a) that the template figure element obtained by the transformation matches the feature quantity of the input image are obtained as shown in the following equation (S21). These are substituted into the right side of Equation 18 to obtain the maximum posterior probability evaluation function (S22). A parameter {a _opt } that maximizes the value of the obtained function is obtained by an optimization method such as a continuous gradient hill climbing method (S23).

ここで、σは分散、ａ（ｕ，ｖ）はアフィン変換により点（ｕ，ｖ）が移される座標、ｂは入力画像の２値のエッジマップ（エッジ強度分布を所定の閾値で２値化したもの）をそれぞれ示す。テンプレートデータが濃淡画像の場合、Ｎは当該代表点を含む局所領域又は予め分割された当該代表点を含む領域上の点の集合となり、ｂはエッジ強度分布を示す。 Assume that the template data consists of line figures, and affine transformation into binary template data t (u, v) in a set N of points on a local line segment (curve element) constituting a graphic element of a template including a representative point Assume that the error resulting from performing {a} is Gaussian. Under this assumption, the posterior probability P (b | a) is given by the following equation. That is,

Where σ is the variance, a (u, v) is the coordinate to which the point (u, v) is moved by affine transformation, b is the binary edge map of the input image (the edge intensity distribution is binarized with a predetermined threshold value) Each). When the template data is a grayscale image, N is a set of points on a local region including the representative point or a region including the representative point divided in advance, and b indicates an edge intensity distribution.

ここで、ｍ_ｉｊ，σ_ｉｊはそれぞれａ_ｉｊの平均及び分散を示し、

である。ｍｏｄｅ＝｛回転_１、回転_２、・・・、倍率_１、倍率_２、・・・、シフト_１、・・・、その他｝のように表わされ、各変形モードごとに、ｍ_ｉｊ及びσ_ｉｊのパラメータ値が与えられる。‘その他’のモードは、特定の名称が与えられない任意の変換モードである。 The prior probability distribution P (a) is assumed to take a high value in a deformation mode that gives scaling, rotation or shift in the horizontal or vertical direction,

Here, m _ij and σ _ij indicate the mean and variance of a _ij , respectively.

It is. mode = {rotation ₁ , rotation ₂ ,..., magnification ₁ , magnification _2, ..., shift ₁ ,..., etc.}, and m _ij and σ _ij for each deformation mode. Parameter values are given. 'Other' mode is any conversion mode that is not given a specific name.

The optimum transformation parameters are obtained by substituting Equations 19, 20, 21, and 22 into Equation 18 and using a continuous gradient hill climbing method or the Powel method for V _mode and {a _ij }. It is done. The above is an example of a method for obtaining an optimum conversion parameter, and it goes without saying that other optimization methods (EM algorithm, dynamic programming, genetic algorithm, etc.) may be used.

  Using the template data converted by the conversion parameter obtained by the method described above as an initial value, generation of a curve connecting the movement of the representative points and the representative points based on the second shape fitness evaluation shown in the first embodiment Processing may be performed.

  When the contour line (closed curve) to be extracted is obtained by the above-described method, the closed curve data is displayed on the screen of the image display device 14, and if necessary, mask data is generated as area data inside the closed curve. Is recorded in a predetermined storage means.

  You may use the variable shape template which consists of a graphic element which has an indefinite part in part. The indeterminate part is a part with a large degree of freedom of deformation, and it is a constraint that the new loop shape does not occur with the deformation and that the shape corresponding to the indeterminate part is known in advance as concave or convex. A part where no other specific constraint is imposed.

  As an extraction target, for example, portions having joints such as a person's chest and legs are known in advance to have a greater degree of freedom of deformation than other portions. Define it above.

  As in the first embodiment, template scanning and fitness evaluation are executed, and the position to be extracted and an appropriate template model are selected. Next, template update is executed by deformation of the graphic elements constituting the template and shape suitability evaluation. At that time, first, the template is deformed by the method shown in the first embodiment or the second embodiment except for the indefinite portion, and then the contour of the indefinite portion is estimated.

  Two methods will be described as methods for estimating the contour of the indefinite portion. In the first method, contour tracking with one end point of the indefinite portion as the start point and the other end point is executed for each indefinite portion. A curve that approximates the contour of the indefinite portion obtained as a result may be generated. FIGS. 13A and 13B show an example of the contour shape extraction process of the indefinite portion by the first method.

In the second method, as shown in FIG. 13 (c), the probe circle whose diameter is a line segment connecting both end points of the indeterminate part is moved outward if the indeterminate part is convex, and inward if it is concave. However, the radius and the position of the circle are changed little by little at each place, and the position and radius of the circle where the contour to be extracted is in contact with the circle at two locations (p _j , q _j ) are obtained. Thereby, the coordinates of the contour line are determined sequentially. This method is suitable for extracting a contour line having no branch. After extracting the approximate shape by this method, the extracted shape may be used as the initial contour model, and the detailed shape may be extracted by the method described in the first embodiment or the second embodiment.

  FIG. 14 shows a flowchart of the template update process in the third embodiment. The process shown in FIG. 14 corresponds to S7 in FIG.

  First, a curve segment pair having end points is selected (S31), and the shape of the template graphic element other than the indefinite portion is updated by the method described in the previous embodiment (S32). Next, the start point and the end point are set from the opposite end points of the adjacent graphic elements after the update (S33), and the above-described method, that is, the contour tracking connecting the start point and the end point (first method), or the contour by the probe circle Position determination (second method) is executed (S34). The above processing is repeated until all the end points are eliminated (S35). When one closed curve is formed by the graphic elements of the template (S35), it is output as updated template data (S36).

  With such a configuration, it is possible to stably extract the contour of the extraction target that is known in advance to have high shape uncertainty (degree of freedom) for a specific portion. That is, even if there is an object in the image that has a part that differs greatly from the shape of the given template, the most probable curve that complements the remaining defined part shape gap for the outline of the different part (e.g. , A curve obtained by contour tracing as a condition satisfying the continuity of the shape at the end points or the continuity of the image feature amount.

  As a fourth embodiment, as a process corresponding to the template update process (S7 in FIG. 5), a curve along a graphic element (or line segment) constituting a template passing between representative points (or a center line of the curve). A thick line having a predetermined width may be generated as shown in FIG. 15, and the boundary position inside the thick line may be estimated based on the feature values (color component values, local spatial frequency representative values, etc.) on both sides of the thick line. . In this case, the same process as in the previous embodiment is performed before the template update process.

  Specifically, as shown in FIG. 15, a nucleus for region growth is set on at least one of the contour line 1 or the contour line 2 of the thickened region. In the region growing process, a region near the nucleus whose image feature value difference at the nucleus is smaller than a predetermined threshold is merged with the nucleus, and the resulting growth result from the contour line 1 and the growth result from the contour line 2 are obtained. Integrate. Alternatively, one boundary line is estimated based on the growth result from any one of the contour lines.

  The boundary position is, for example, a region based on the maximum edge strength position of the feature amount on a line substantially orthogonal to the direction of the thick line, or the similarity of feature amounts (for example, color components) from predetermined sampling points on both sides of the thick line. It is determined as the boundary position by growth.

  In this embodiment, the contour line finally extracted does not necessarily have to be expressed by a parametric curve.

10: image input device 12: image storage device 14: image display device 16: image processing device 18: instruction selection device 20: feature quantity extraction device 22: template model selection device 24: template data generation device 26: template scanning device 28: Movement range / condition update setting device 30: update template generation device 32: shape conformity evaluation device 34: boundary line generation device 36: boundary line display device 38: mask data generation device 40: screen 42 of image display device 14: input image 44: Template model image data 46: Target position confirmation indicator 48: Extraction instruction button 50: Template scroll button

Claims (2)

A first matching step of the first matching means performs a first matching processing for obtaining the predetermined evaluation function value related to the value of the predetermined feature quantity of a template including the input image and the local graphic element,
A local transformation step in which a local transformation means performs a local geometric transformation on a region near a predetermined representative point on the template;
A conversion parameter calculation step in which a conversion parameter calculation means obtains a conversion parameter of the local conversion step based on the first matching processing result and at least one of a predetermined deformation mode and a probability distribution;
A second matching step in which a second matching means performs a second matching process between the template obtained by the local conversion step and the input image ;
Image extraction method, wherein the image region extracting means comprises an image region extraction step of extracting an image region including a pre-Symbol contour based on the result of said second matching processing. A first matching means for performing a first matching processing for obtaining the predetermined evaluation function value related to the value of the predetermined feature quantity of a template including the input image and the local graphic element,
Local transformation means for performing a local geometric transformation on a region near a predetermined representative point on the template;
Conversion parameter calculation means for obtaining a conversion parameter of the local conversion means based on the first matching processing result and at least one of a predetermined deformation mode and a probability distribution;
A second matching means for performing a second matching process between the template obtained by the local conversion means and the input image ;
Image extracting apparatus characterized by comprising an image region extracting means for extracting an image region including a pre-Symbol contour based on the result of said second matching processing.

Images