JP2000048212A - Device and method for picture processing and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to cut out a specific subject precisely and easily from within a picture. SOLUTION: This device has a means 501 for setting a closed initial outline of an arbitrary shape, a means 507 for setting a specific reference point on the basis of the initial outline, a means 508 for setting plural belt-like areas of a specified shape including a line segment connecting the specified point on the initial outline and the reference point, a means 502 for splitting each belt-like area into plural areas on the basis of picture data in the plural belt-like areas, a means for 503 for updating the outline on the basis of each split point position after area split, and a means 504 for obtaining an outline shape of a specific picture area to be extracted on the basis of the outline after update. It is possible to almost automatically obtain the outline of an extraction object after setting and to surely obtain a correct outline of the extraction object only by roughly setting the initial outline in a way in which the extraction object is surrounded.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing method and apparatus, and a recording medium, and more particularly to image editing such as cutting out a specific area where a desired object is present from an entire image and combining it with another image. Also, it is suitable for use in software or an apparatus having a function of performing a processing process such as generation of an image mask that limits an extraction region or a region other than the extraction region as a range to which image processing such as filtering is applied. is there.

[0002]

2. Description of the Related Art Conventionally, a process of cutting out an object such as a specific subject from an image is performed while selecting a region having a predetermined range of color component values including a pixel value of an object or a point on a background designated by a user. There has been a method of repeating the removal of the background or the designation of the area of the extracted subject.

In addition, a method in which a user specifies a rough rough contour region or a local region including a contour to be extracted, and obtains and cuts out a boundary contour of the target by processing such as thinning or clustering in the designated region. There was also.

[0004]

However, of the above-mentioned prior arts, the method of designating an area to be extracted while selecting an area having a color component value within a predetermined range involves a method in which a subject locally has many color components. In the case where the image is composed of a region having the image or the boundary between the background and the object is low in contrast and it is difficult to distinguish only by the pixel value, it takes a lot of trouble (color component) to accurately extract the desired object. (For example, selection of a selection range and specification of a point in an extraction target).

In the method of specifying a rough outline including the outline to be extracted, when the width of the outline is small, the coarse outline is taken with great care so as not to protrude from the true outline. There is a problem that the contour must be set. On the other hand, when the width of the rough outline is large, even if the outline of the target is obtained by thinning or clustering, if the boundary of the target has a low contrast, or if the outline of the target in the rough outline is a complex shape, There is a problem that it is difficult to extract a true outline when the image has the contour.

The present invention has been made to solve such a problem, and an object of the present invention is to enable a specific object to be easily and accurately cut out from an image. Another object of the present invention is to enable such simple and accurate cutout processing to be performed on an object having any complicated shape.

[0007]

An image processing method according to the present invention comprises an initial contour for setting a closed contour of an arbitrary shape so as to include an object to be extracted inside the object to be extracted, or within the object to be extracted. A setting step; a reference point setting step of setting a predetermined reference point based on the initial contour; and a plurality of band-shaped areas of a predetermined shape including a line segment connecting the predetermined point on the initial contour and the reference point. A band-shaped region setting step of dividing the plurality of band-shaped regions into a plurality of regions based on image data in the plurality of band-shaped regions, and a contour based on each division point position after the region division. The method is characterized by comprising a contour updating step of updating a line, and a contour determining step of obtaining a contour shape of an object to be extracted based on the updated contour line. According to the present invention configured as described above, for example, the outline of the extraction target can be obtained in a stepwise and accurate manner only by roughly specifying the region so as to roughly surround the target to be extracted. That is, after designating the initial contour, the target contour can be obtained almost automatically only by instructing the update of the contour.

In another aspect of the present invention, the reference point is set at the position of the center of gravity of the initial contour or at a plurality of positions within the initial contour. Thus, in most cases, the convergence target of the initial contour can be automatically set inside the target to be extracted. Alternatively, by setting reference points at a plurality of positions arbitrarily or automatically by the user,
It is possible to almost automatically determine the contour of an object to be extracted from an elongated or complicated shape. In this case, as in the invention according to claim 6, in the band-shaped area setting step, the reference point closest to the point on the initial contour may be selected from the plurality of reference points to set the band-shaped area. . This makes it possible to automatically obtain precise contour data of a complicated shape with high accuracy.

In another aspect of the present invention, in the band-shaped area setting step, points on the initial contour are sampled at predetermined intervals, and a plurality of band-shaped areas each including a line segment connecting each of the sampling points and the reference point. Is set. In the contour updating step,
An update section on the contour is set, and the contour is updated based on a division point position in a band-like area including a point on the initial contour in the set update section. In this case, it may be possible to select whether to update the entire contour or to update by setting an update section. As described above, since the contour is updated using the result obtained by dividing the local area defined by the band-shaped area in the predetermined direction, the background is extracted using the information of the entire area including the extraction target and the background surrounding the extraction target. It is possible to determine the contour of the extraction target step by step with much less computation than other general texture segmentation without the difficult process of separating the extraction target from the extraction target. it can.

In other aspect of the present invention, in the contour determining step, and obtains the outline shape of the object by performing contour tracing for the image data of the thick line area including an outline of the updated. As the contour tracing, for example, a point that is at or near the maximum point of the edge strength and in which the shape continuity of the boundary obtained by the tracing is maintained may be tracked. In this case, since the edge search range for tracking is limited to the thick line area, the updated contour can be converged to the extraction target with higher accuracy and at high speed. In addition, by performing the outline update process and the outline tracking process in two stages, it is possible to obtain the outline of the extraction target with high accuracy and almost automatically even by simply setting an area roughly surrounding the extraction target. Can be.

In another aspect of the present invention, in the contour determination step, a predetermined evaluation function value is obtained for the updated contour line, and each point on the contour line is moved to a position where the evaluation function value becomes a minimum. The contour to be extracted is determined based on the active contour processing to be performed. The dynamic contour processing is to automatically converge the updated contour to the contour to be extracted with higher accuracy. By performing the contour updating processing and the dynamic contour processing in two stages, the extraction contour is determined. By simply setting a roughly surrounding area, the contour to be extracted can be obtained with high accuracy and almost automatically. This is the same as the case where the contour update processing and the contour tracking processing are performed in two stages.

[0012] Here, as for the evaluation function, as described in claim 12, the image feature quantity relating to the sampling point on the updated contour line and the image feature quantity obtained for the outer neighboring area corresponding to the sampling point are determined. A first feature value deviation, and first and second feature values obtained from a second feature value deviation between the image feature value related to the sampling point and the image feature value obtained for the inner vicinity area corresponding to the sampling point. A term for evaluating the difference between deviations may be included. In addition, a first probability that the sampling point on the updated contour is in an area near the outside of the contour and a second probability that the sampling point is in an area near the inside of the contour are evaluated. The first obtained
And a second term for evaluating the difference between probabilities may be included. As described above, the direction of the external force acting on the active contour processing is determined based on a predetermined evaluation function (or probability model) as to whether a point on the contour belongs to the inner area or the outer area of the thickened contour area. Since evaluation is performed, it is possible to converge more reliably on the contour of the target to be extracted.

According to another aspect of the present invention, in the contour updating step, among the division point positions in the plurality of strip-shaped areas, the division points furthest from the reference point are connected by a predetermined method to form a new contour line. Is generated.
Thereby, it is possible to gradually converge to the contour to be extracted with a small amount of calculation.

In the present invention, a direction passing through a division point defined in the strip-shaped region and substantially perpendicular to a line connecting the point on the contour line and the reference point is provided. A predetermined feature value of two neighboring regions that are in contact with the dividing boundary line having the following formula, and a feature evaluation value of the two neighboring regions that are in contact with the dividing boundary line that are similarly obtained for the other band-like regions. A new contour may be generated by connecting the division point positions having the highest evaluation value by a predetermined method. As described above, when selecting the division point positions of a plurality of band-shaped regions and setting the outline connecting them as the update outline, the image feature amount of the neighboring region along the outline inside the extraction target is continuous. For example, by selecting a division point having a high evaluation value of the feature amount consistency as the division point position, the number of steps is smaller than that of the division point position in order from the outside, and the number of steps is smaller than that on the extraction target contour line or in the vicinity thereof. Can be converged. Further, in the present invention, the image feature value used in the image dividing step may be used as the image feature amount used in the outline updating step.
By doing so, it is not necessary to obtain the image feature amount redundantly.

In another aspect of the present invention, the contour updating step includes a dividing boundary having a direction passing through a dividing point defined in the strip-like area and substantially orthogonal to a line segment connecting a point on the contour and the reference point. Evaluate the continuity of the line and the dividing boundary line passing through the dividing point already set for the band-like region adjacent to the band-like region, and position the dividing point so that the continuity becomes higher than a predetermined reference value. It may be determined. As described above, by updating the contour using the continuity of the updated contour shape as a constraint condition, the contour line can be placed on or near the contour line to be extracted in a smaller number of steps than in the case where the division point positions are sequentially determined from the outside. Can be converged.

In another aspect of the present invention, the contour updating step includes a pixel having an image characteristic amount similar to an image characteristic amount of a sampling point on a contour line in a set update section and a neighboring area thereof. A selection step of selecting, as a background pixel, a pixel that includes a sampling point on the contour line and forms a connected area as an area having a high degree of similarity of the image feature amount; a background pixel obtained after the selection step and an object to be extracted And a step of updating a contour line in an update section for performing a process of setting a boundary line with the contour line as an updated contour line in the update section. In this way, by combining the contour updating step and the so-called selection processing for selecting a region to which a pixel in a predetermined color range belongs, for example, the contour updating is performed for a relatively gentle shape to be extracted. Convergence to the target contour by step and subsequent contour tracking processing or active contour processing, and for the contour shape of the complex shape part to be extracted, apply the selection processing to reliably extract the contour line Is possible. Also, once the contour update processing of the present invention is applied (or a plurality of times), the background area is partially selected and deleted by a so-called selection processing, resulting in overall processing efficiency and extraction accuracy. Can be improved compared to performing each alone.

In another aspect of the present invention, in the area dividing step, the resolution of the area division is increased as the number of applications of the contour updating step increases, and in the band-shaped area setting step, the number of applications of the contour updating step is increased. The sampling interval on the contour line is made smaller as the number increases. As a result, the outline is initially updated coarsely, and the outline is updated with higher precision and resolution as the object approaches the target outline, thereby improving processing efficiency.

In another aspect of the present invention, an image input step of inputting an image including an object to be extracted, a contour setting step of setting a closed contour of an arbitrary shape, and a method of determining the shape of the contour by a predetermined method A contour update step to update with
An update result display step of superimposing and displaying the updated outline on the input image, an instruction input step of instructing to stop or start updating the outline, and a contour output step of outputting the updated outline shape. It is characterized by becoming. Here, the contour updating step includes, for example, an updating section setting step for specifying a section on the contour line to be updated or a non-updating section setting step for specifying a section on the contour line not to be updated. And a partial contour updating step of updating by the method of (1). Thus, after setting the contour so as to surround the object to be extracted (or so as to be present inside the area of the object), the user sees the contour that is sequentially updated and deformed, and determines an appropriate one. At this stage, it is only necessary to give a simple instruction such as giving an instruction to stop or start (or restart) the updating of the contour line, so that the labor required for cutting out the extraction target can be greatly reduced. In addition, by specifying and updating only the contour of the undetermined section excluding the section that has converged first on the true contour of the extraction target, the contour of the extraction target is completed by stepwise stacking. In this regard, steady and accurate clipping is achieved.

The image processing apparatus according to the present invention further comprises: an initial contour setting means for setting a closed contour having an arbitrary shape; a reference point setting means for setting a predetermined reference point based on the initial contour; Band-shaped region setting means for setting a plurality of band-shaped regions of a predetermined shape including a line segment connecting a predetermined point on the contour and the reference point; and forming the plurality of band-shaped regions based on image data in the plurality of band-shaped regions. An area dividing means for dividing the object into a plurality of areas, an outline updating means for updating the outline based on the positions of the division points after the area division, and an object to be extracted based on the updated outline. And a contour determining means for obtaining a contour shape.

In another aspect of the present invention, image input means for inputting an image including an object to be extracted, contour setting means for setting a closed contour having an arbitrary shape, and a method for determining the shape of the contour by a predetermined method Contour updating means for updating in the above, an update result display means for displaying the updated contour line superimposed on the input image, an instruction input means for instructing stop or start of updating of the contour line, And a contour output means for outputting a contour shape.

Further, the computer-readable recording medium of the present invention comprises: an initial contour setting means for setting a closed contour having an arbitrary shape; a reference point setting means for setting a predetermined reference point based on the initial contour; A band-shaped region setting means for setting a plurality of band-shaped regions of a predetermined shape including a line segment connecting a predetermined point on the initial contour and the reference point, the plurality of band-shaped regions based on image data in the plurality of band-shaped regions. Area dividing means for dividing each into a plurality of areas, contour updating means for updating a contour line based on each division point position after the area division,
And a program for causing a computer to function as contour determination means for obtaining a contour shape of an object to be extracted based on the updated contour line.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Embodiment) FIG. 1 is a block diagram of a main part of an image processing apparatus for performing a subject cutout process according to a first embodiment of the present invention. In FIG.
1 is an image data input unit, 2 is an image data storage unit, 3 is an image display unit, 4 is an instruction selection unit (such as a mouse), and 5 is an image processing operation unit.

In the present embodiment, an image including a subject to be cut out is first read from the image data input unit 1, stored in the image data storage unit 2, and displayed on the image display unit 3. The image data input unit 1 includes a scanner, a digital image pickup unit, an image data transfer unit connected to an image database or the like. Then, in accordance with an instruction from the user using the instruction selecting unit 4, a process of cutting out a specific subject is performed by the image processing operation unit 5.

As shown in FIG. 2, the image processing operation unit 5
Initial contour setting means 501, area dividing means 502, contour updating means 503, contour determining means 504, primary storage means 50
5, a processing result output means 506, a reference point setting means 507, and a band-shaped area setting means 508.

Hereinafter, based on FIGS. 1 and 2 showing the configuration of the image processing apparatus of this embodiment, FIG. 3 showing the processing flow, and FIG.
The following description focuses on the processing contents of each unit in the figure.

Referring to FIG. 3, first, the user selects FIG.
As shown in (a), an instruction selecting means 4 such as a mouse (hereinafter referred to as a mouse) is used to surround an object to be cut out (a car in this case) in an image displayed on the image display means 3. A closed initial contour of an arbitrary shape is set (step S1). Specifically, while checking the position of a cursor (not shown) displayed on the image display means 3, the mouse 4
Press the button to draw the trajectory of the curve surrounding the cutout target.

Then, the initial contour setting means 501 of the image processing operation unit 5 samples points on the locus at predetermined intervals, generates a smooth curve connecting the sampling points, and sets the curve as the initial contour line 6. (Step S2). Note that the user may click the mouse 4 multiple times at an arbitrary point position outside the cutout target, and the initial contour setting means 501 may generate the initial contour line 6 by connecting the clicked points with a straight line in response to the click. It may be.

Next, the reference point setting means 507 calculates the position of the center of gravity of the initial contour line 6 as the position of the reference point 7 (step S3). Further, the belt-shaped area setting means 508 samples points on the initial contour line 6 at predetermined intervals so that the lengths along the contour line (path length) are equal (step S).
4) A rectangular band-like area 9 having a predetermined width (for example, about 10 pixels) centering on a line connecting the sampled point 8 and the reference point 7 is set (step S5).

In the process of step S4, polar coordinates centering on the position of the center of gravity (reference point 7) may be set, and sampling may be performed so that the intervals between the angle components are equal. For example, 12 points on the initial contour line 6 may be sampled so that the angle width becomes an interval of 30 degrees in the polar coordinate system. It is assumed that the strip-shaped area 9 corresponds to each sampling point 8 one by one and has the same number as the number of sampling points.

Next, the area dividing unit 502
Then, the longitudinal direction (the direction of the line connecting the reference point 7 and the sampling point 8) of the band-like area 9 is divided based on the image data (step S7). FIG. 4A shows a boundary line obtained as a result of region division performed on the rectangular band-like region 9.

The shape of the band-like region 9 may be a sector having a predetermined apex angle (for example, about 10 degrees) in addition to a rectangle. In this case, a boundary (a plurality of boundaries can be set) that passes through a certain point in the region and mainly has a circumferential direction component is determined. FIG. 4B shows an example of area division of image data of a fan-shaped area. In the case of a rectangle and a sector, a boundary line is typically a straight line or an arc-shaped curve, but the shape of the boundary line is not particularly limited thereto, as long as the main direction component is substantially orthogonal to the longitudinal direction of the region. . At other points, the shape of the boundary may be set according to the image data.

Further, the belt-like region 9 does not necessarily need to include the reference point 7 therein, and may be set in the vicinity of the sampling point 8.

Here, the area division in this embodiment will be described in detail with reference to the processing flow shown in FIG.
First, the longitudinal direction of the band-like region 9 is divided into blocks having a predetermined width (step S21), and a histogram of the edge density and the local color component (such as hue) in each block is obtained (step S22).

Based on the obtained edge density and local hue histogram, each block is divided into a texture dominant region (a region where the edge density is larger than a predetermined threshold) and a plain dominant region (a region where the edge density is less than the threshold). And labeling is performed (step S23). Further, when regions having the same label (texture superiority or solid color superiority) are adjacent to each other, a merging process is performed to integrate them together (step S24).

Next, based on whether or not the label of each area is a solid dominant area (step S25), the solid dominant area is set to a boundary position where the amount of change in the local color component is large or an edge existing position. Further division is performed as a position (step S26). Further, the texture dominant region is further divided using, for example, an existing method described below (step S27). Finally, the resulting boundary line in the belt-shaped region 9 is output (step S2).
8).

As the above-mentioned existing texture division method, for example, a method based on the calculation of a co-occurrence matrix and a division / merging algorithm (Segmentation by Texture Using a Co-Oc
urrence Matrix and a Split-and-Merge Algorithm, P.
C. Chen and T. Pavlidis, Computer Graphics and Image
Processing, vol. 10, pp. 172-182, 1979). Also,
This may be performed by defining the degree of separation when the band-shaped region 9 is divided by a predetermined boundary (Japanese Patent Laid-Open No. 7-93561), and determining a boundary (a plurality of boundaries can be set) where the degree of separation is equal to or larger than a threshold. Further, the strip-like area 9 may be consistently divided by the texture division algorithm alone as in the above example.

Next, the outline updating means 503 updates the outline by selecting the outermost boundary position among the division boundaries of each band-shaped area 9 and connecting the selected boundary point positions with a straight line ( Step S8). The contour may be updated by using a B-spline function or the like to generate an interpolation curve that smoothly connects the representative points of each boundary line.

The updated outline is thickened to a predetermined width and displayed on the image display means 3 so as to overlap the input image (step S9). The user determines whether or not to update the outline further by checking whether or not the true outline of the subject to be cut out is included in the updated thickened outline (step S10). Are not on the true contour line, it is determined that the entire contour line is further updated (step S11), and an instruction to update the whole is sent by pressing a return key or the like (step S1).
2).

Note that the entire contour line is automatically updated, and
After confirming the progress on the image display means 3, only when the user shifts to the partial update mode (to be described later) at a certain point in time or stops the outline update, the user stops the outline update processing mode or An instruction to shift the processing mode may be sent.

If a part of the thickened outline updated in step S9 is on the true outline and the other part is not on the true outline, the process moves to the partial update mode and the outline to be updated is updated. A start point and an end point of the part are specified (step S13).
For example, if you select the start point or end point from the icon menu and click the corresponding point in the image,
Each coordinate is registered and displayed. Here, a format in which the start point and the end point of the partial outline that is not updated may be specified. In this case, the non-designated outline portion is updated in a later process.

FIGS. 6B and 6C show examples of the result of setting the positions of the update section start point 10 and the update section end point 11. However, the update section start point 10 to the update section end point 11
Is determined to be counterclockwise, or the direction is determined by a user's instruction (such as selecting a direction closer to the direction drawn with a mouse).

The setting of the contour update section in step S13 can be automatically performed when the background area is almost solid. In this case, points on the updated contour line that are not on the edge (the edge strength is very low) may be registered as the start point and end point of the update section. Whether or not the background region is almost plain is determined by determining the edge density (the edge obtained by binarizing the edge intensity data) on the local region obtained by thickening the initial outline 6 when the initial outline 6 is set. Can be determined from the magnitude of the existence density of

Next, as shown in FIG. 6D, when all the true outlines are included in the updated thickened outline, the mode is shifted to the outline determination processing mode in accordance with a user's instruction. Here, the contour determining means 504 performs a process of extracting a contour to be cut out using the image data in the thickened contour line (step S14). Specifically, a process such as tracking a point at which the edge intensity is maximized while satisfying the continuity of the contour (for example, JP-A-7-225847) may be performed.

Note that this processing (step S14) is unnecessary if the center line of the thickened contour line coincides with the true contour, and if the cutout (region division extraction) processing is terminated by the user's instruction, Needless to say, it's good. As another method, a local search area is set inside a thickened outline, and a boundary point having the highest degree of separation in the width direction of the thick line is selected by a method such as clustering (Japanese Patent Laid-Open No. 6-251).
149) while the local search area is automatically moved along the center line of the bold line to extract the contour of the target. Further, when the sampling points on the updated thickened outline are on the true outline, contour tracking (edge tracking or edge connection processing) may be performed so as to sequentially pass those sampling points.

At the stage where a part of the updated thickened outline is on the true outline, the outline determination processing in step S14 described above may be performed in the corresponding section. In the above description, processing is performed such that the initially set initial contour line 6 is sequentially contracted and converged on the contour line to be cut out, but the initial contour line expands in the same manner. Accordingly, the processing procedure may be set so as to converge on the contour line on the cutout target.

That is, after a closed curve having an arbitrary shape is set as an initial contour inside the cutout target, a predetermined band-like area including a sampling point on the initial contour and a reference point inside the closed curve is set. However, it is assumed that the tip position of the band-shaped area is always far from the sampling point when viewed from the reference point. Next, the region division of the set band-shaped region is performed in the same manner as described above, and a division point position that is the innermost side from the reference point and that is outside the sampling point is selected as a sampling point position of the updated contour line. Then, the process of connecting the update points and generating an updated contour line may be sequentially repeated. Other processes are the same as in the above-described embodiment.

As described above, according to the present embodiment, if the user first sets the curve surrounding the target to be cut out, then the user sets whether or not to update the contour or where to update. With only the semi-automatic processing, it is possible to reliably and accurately extract a target contour with a small number of steps and cut out an image regardless of the complexity of the background image pattern and the complexity of the shape of the cut target object. Further, the processing time can be greatly reduced in that the result of region division in a more limited local region is used as compared with the existing method of dividing the entire image.

(Second Embodiment) In the second embodiment,
When determining the update position of the sampling point on the contour line in a certain band-like region, based on the distribution of predetermined feature amounts obtained by region division of other band-like regions, the consistency (to be described later) is evaluated. Select the division position with the highest evaluation value.
Hereinafter, the processing in the contour updating means 503 will be described with reference to the processing flow shown in FIG. Other processes are the same as those in the first embodiment.

Referring to FIG. 7, first, feature amounts in the vicinity area inside and outside each division point position in each band-like area 9 are obtained (step S31). For example, a hue value that gives the maximum frequency of an edge density and a local hue histogram in a corresponding neighboring area may be used as the feature amount. However, if these features are the same as those used in the region division processing, there is no need to calculate again,
The result may be used.

Next, an evaluation value of the consistency of the feature amount at each division point position in each band-like area 9 is calculated (step S3).
2). Hereinafter, the m-th feature amount in the outer neighboring region at the j-th division point position from the outermost in the k-th band region is F ^OUT (k, j, m), and the m-th feature amount in the inner neighboring region is ^Let F ^IN (k, j, m) be the feature amount of the parameter, Q be the number of parameters of the feature amount, and J _k be the number of divisions in the k-th band-like region. The quantity consistency C (k, j) is represented by the following equation (1).

[0051]

(Equation 1)

However, in the above equation (1), ε is a minute constant and a value such as 0.01 is given, and k is 2 ≦ k ≦ K−1
(K is the total number of band-shaped areas). When the value of k is 1, the denominator of the first term of the above equation (1) is | F ^r (K, p, q) −F ^r (k, j,
m) | + ε, and when the value of k is K, the second
The denominator of the term is given by | F ^r (1, p, q) −F ^r (k, j, m) | + ε.

The feature consistency C (k, j) defined in this way
The value increases as the difference between the inside / outside feature amount at the j-th division point position of a certain k-th band-like region and the inside / outside feature amount at the division point position of two other adjacent band-like regions decreases. In addition, as the definition of the feature quantity consistency C (k, J), the following equation (2) or (3) may be used.

[0054]

(Equation 2)

In the above equation (2), when the division point position is determined in order, for example, when evaluating the characteristic amount consistency C (k, j), only the characteristic amount data regarding the immediately preceding adjacent band-shaped region is used. Indicates that In Equation (3), N _k indicating the range of i indicates a set of index values of neighboring band-like regions to be considered for a certain band-like region, and elements in N _k take values other than k. Note that when N _k = {k-1, k + 1}, equation (1) is the same as equation (2), and equation (2) is similar to equation (3).
_This is a special case where _k = {k-1}.

Next, based on the evaluation value of the feature quantity consistency C (k, j) obtained as described above, the optimum division point position in each band-like region 9 is set as the update point position in the band-like region 9. It is determined (step S33). Specifically, the optimum division position in the k-th band-like region 9 j _{ou t, k,} the feature quantity consistency C (k,
j that maximizes the evaluation value of j), that is, the following expression
Choose j as in (4).

[0057]

(Equation 3)

Finally, by connecting the thus obtained dividing point positions, a center line of the updated contour is obtained (step S34), and the center line is made thick (step S35). Ask for an updated outline.

By selecting the division point positions and updating the contour as in the second embodiment, the object can be obtained more quickly and reliably than in the first embodiment in which the outermost division point positions are sequentially selected. Can be converged to the contour of the extraction region. In particular, when a part of the thickened outline after the update is located on the true outline, in the equation (3), for the section of the outline to be updated in the next step, the division point where N _k is determined Each time a contour line is updated so that it is a set of the index of the band-shaped region having the position and the nearest index, N _k
Updating the value is also more effective in terms of processing speed and reliability.

In the above equation, the evaluation value of the feature amount consistency C (k, j) takes into account both the image feature amounts inside and outside the boundary line of the division point position. For example, the definition may be made so as to consider only the continuity of only the inside.

When the position of the dividing point (each point S _i shown in FIG. 8) on the boundary line of the band-like region 9 is included as the feature amount, the evaluation value of the feature amount coherence C (k, j) becomes It can be formulated so as to evaluate the continuity of the shape of the updated contour line together with the continuity of the feature amount. That is, in this case, the following equation (5) is used instead of the equation (3).

[0062]

(Equation 4)

In this case, the feature value F (i, p, q) does not distinguish between the inside and outside of the boundary, and the index q indicates various types of feature values. As the neighborhood N _k , an element corresponding to Expression (1) or Expression (2) may be used. It is needless to say that the scale for evaluating the consistency of the feature amount between the strip-shaped regions for selecting the division point position is not limited to the above-described definition formula.

(Third Embodiment) In the third embodiment,
The resolution in the region division processing is initially set to be coarse, and the resolution is increased as the number of updates of the contour line is increased. Also, increase the number of sampling points on the contour line every time
Reduce the sampling interval. The processing from the setting of the reference point to the update of the contour according to the present embodiment will be described below with reference to the processing flow shown in FIG.

In FIG. 9, first, the reference point setting means 50
7 calculates the position of the center of gravity of the initial contour line 6 as the position of the reference point 7 (step S41), and sets the band-shaped area setting means 508.
Samples the points on the contour at predetermined intervals according to the number of updates of the contour so that the lengths along the contour (path length) are equal (step S42).

Specifically, assuming that the initial number of division points is N, the number of division points when the number of updates is M is set to WN (M-1) or the like (W is typically a constant of "2 Is used.) Further, assuming that the perimeter of the contour is S, the sampling interval is S / (WN (M-1)). When a polar coordinate system based on the position of the center of gravity is set and sampling is performed so that the angular interval is constant,
The interval is 360 / (WN (M-1)) degrees.

Next, the band-shaped area setting means 508 inputs image data of a predetermined resolution in accordance with the number of updates of the contour line (so that the higher the number of updates, the higher the resolution) (step S43). For example, image data having different resolutions may be generated in advance from an input image by a thinning-out interpolation process or the like, and image data having a corresponding resolution may be selected and input from the image data. Then, a band-like area 9 centering on a line connecting each sampling point 8 and the reference point 7 is set (step S44).

Next, the area dividing means 502 obtains image data in the band-like area 9 (step S45),
The longitudinal direction of the band-like region 9 (the direction of the line connecting the reference point 7 and the sampling point 8) is divided based on the image data of the given resolution (step S46). The method of area division may be the same as in the first embodiment.

Thereafter, the contour updating means 503 updates the contour in the same manner as in the first or second embodiment (step S47), and if the update is to be continued (step S48), returns to step S42. To repeat the above processing. In step S47, the contour updating unit 503 selects, for example, the optimum division point position in the same manner as in the second embodiment.

In this embodiment, the section in which the updated thickened outline does not include the true outline is specified in the same manner as in the first embodiment, but the update of the outline itself is always performed in all the sections. You may do it. This is because the image data of the new resolution is input each time the contour is updated. Therefore, even if a true contour exists within the thickened contour at the previous resolution, the new resolution is used. This is because the accuracy of approximating the true contour with the center line of the thickened contour can be increased by performing the region division with.

When setting a boundary line having a direction component substantially perpendicular to the direction of the line segment connecting the sampling point and the reference point for improving the accuracy of the position of the contour line, the range within the width of the thick line or the range within the range May be divided in a range in which is multiplied by several times.

(Fourth Embodiment) In the fourth embodiment,
It is assumed that a plurality of reference points 7 can be arbitrarily set. If there are a plurality of reference points 7 when setting the band-like area 9, a shorter distance from the sampling point 8 is automatically selected. In particular, when the subject to be extracted has an elongated shape or a complicated shape, if the reference point 7 is additionally set at an appropriate position, the correct shape of the extraction region can be automatically obtained even for a complicated shape.

The reference point setting process by the reference point setting means 507 will be described below with reference to FIG. In FIG. 10, first, when setting the initial contour line 6, a minimum rectangular area which is in contact with the contour is considered, and it is determined from the aspect ratio whether the shape of the area surrounded by the initial contour is portrait or landscape. S51). For example, if the aspect ratio is 1 or more, it is determined to be landscape.

Next, it is determined whether the above-mentioned aspect ratio is out of the predetermined range (for example, 2 or more and 0.5 or less) (step S52). The center line of the region is obtained in the vertical direction or the horizontal direction depending on whether it is horizontally long (step S53). For example, in the case of a vertically long region, the center line curve is obtained by obtaining the midpoint of the boundary points (two points) of the contour at each position in the vertical direction. Further, the points on the obtained center line that are equally divided by N are obtained as the reference points 7 (step S54), and the reference point setting process ends.

After the user sets the initial contour 6, the reference point 7 may be set at an arbitrary position inside the contour that is considered appropriate. In this case, in particular, the reference point 7 is superimposed on the input image so that the reference point 7 is uniquely displayed so that the position of the set reference point 7 can be recognized. Or brightness).

(Fifth Embodiment) In the fifth embodiment,
In the thickened outline area after the outline update processing, the outline of the cutout area is determined by the active outline processing (described below). The process up to the outline updating process is the same as that of any of the above-described embodiments, and thus the description is omitted.

The active contour method (Snakes: Active Contour Mod)
els, M. Kass, A. Witkin, D. Terzopoulos, Internationala
l Journal of Computer Vision, pp321-331, 1988)
This is a method of extracting the contour of an object from the edge information, by deforming the contour model so that the energy evaluation function that expresses that the contour is smooth and on the edge as a constraint is minimized. , Converge on the contour on the object. The length s along the contour from a predetermined reference point on the center line of the thickened contour is used as a parameter,
The coordinates of the point v (s) on the center line of the thickened outline are (x (s), y (s))
, The energy function is given by, for example, the following equation (6).

[0078]

(Equation 5)

[0079] Here, s _i represents the value of the circumferential length along the outline that corresponds to the i-th sampling point position on the contour. E _int is the internal energy at which the contour model tries to be smooth, and is given by, for example, the following equation (7).

[0080]

(Equation 6)

E _image is energy representing the force attracted to the edge, and is given by the following equation (8), for example.

[0082]

(Equation 7)

Here, I (v (s)) represents the pixel value on the point v (s), and the values of α (s), β (s) and w are set by the user. Further, E _ext (s) corresponds to an external force and is appropriately set. In the present embodiment, the inside of the bolded outline is a cutout target area, and the outside is a background area. Is defined as

[0084]

(Equation 8)

Here, γ is a positive constant, and P _in (I (v
(s)) and P _out (I (v (s)) are the average values μ _{in, Ns} , μ _out, of the pixel values of the local image in the inner and outer neighboring regions of the thickened outline _{. Ns} and variance σ _{in, Ns} , σ
_{Using out, Ns} , it is expressed as the following equation (10). Here, N _s represents a neighborhood area for which an average value and a variance are obtained corresponding to the point v (s).

[0086]

(Equation 9)

From equation (10), equation (9) indicates that if there is a high probability that the sampling points on the contour are on the image in the region near the inside of the thickened contour, an external force that expands outward acts on the contour. Conversely, if the probability that the region near the outside is on the image is high, it means that an external force in the direction of contracting inward acts. Equation (10) evaluates a difference between an image feature amount (such as a pixel value) at a certain sampling point v (s) and an image feature amount (such as an average pixel value) in an inner or outer neighboring region. It can be regarded as a function, and in that sense, another evaluation scale of the feature amount difference regarding the sampling point and the neighboring area may be used. Although the probability model is Gaussian in equation (10), another model may be used.

Hereinafter, a method of setting the inside / outside vicinity area corresponding to the point v (s) will be described with reference to FIG.
In the present embodiment, as shown in FIG. 11A, as a neighboring area corresponding to the point v (s) on the center line, about 10 pixels are arranged outside the thickened contour area in the normal direction of the thickened contour. And a local area having a width of about 10 pixels in the direction of the outline of the thick line around the intersections v _in and v _out of the normal line of the center line passing through the point v (s) and the boundary of the thickened outline region Set. Then, it is assumed that the average value and the variance of the pixel values are obtained for the image in the defined local region.

However, the pixel values (hue,
Brightness) histogram has multiple peaks,
As shown in FIG. 11B, a boundary is determined so as to be divided into two regions, a region having a small difference (below a predetermined threshold) from the pixel value giving the maximum peak, and a region having a large difference. The local region is defined as a nearby region. in this case,
If the length of the neighboring region in the direction of the contour is very short, the local region is enlarged in the direction of the contour so as to have a length of, for example, 10 pixels. The same division as shown in FIG. 11 (b) may be performed.

As described above, as a result of the outline updating process based on the band-shaped region dividing process, it is determined that the inside of the obtained thickened contour line is a cutout region and the outside is a background region, By performing the dynamic contour processing (Equations (9), (10)) in which the difference between the image feature amounts on the outside and the outside is reflected as an external force, the contour to be extracted can be automatically obtained with high accuracy.

(Sixth Embodiment) In the sixth embodiment,
After performing the contour update processing based on the band-shaped area division processing, when a contour section to be updated is designated, the area is grown or expanded in the inner direction of the closed contour centering on each point on the contour of the section. Perform selection processing.

Here, the selection processing means that the pixel values (RGB values or hues, This means that, among a plurality of regions obtained by extracting pixels in a closed contour having a luminance value) as a selected region, those in a region connected to a sampling point are registered as background pixels. The boundary between the background and the subject obtained in this manner is considered to be sufficiently close to the contour of the region to be cut out.

Here, the region growth means a pixel value (or image) of a local region (such as a bold line region) centered on the contour specified on the contour sampling point (or its vicinity) or in the update section. This is a process of sequentially increasing the combined length of a region near a sampling point having a pixel value or a feature value similar to the (feature value) toward the inside of the closed contour.

FIG. 12 shows a processing flow of the outline update processing in this embodiment. In the present embodiment, in particular, local image data of a region obtained by bolding is obtained between sampling points on the contour (step S6).
1) A histogram of feature amounts (hue, luminance value, etc.) relating to the acquired local image data is obtained (step S6).
2) A pixel value selection range is set centering on the pixel value giving the peak frequency (if there is more than one, half the maximum frequency or the pixel value giving the next frequency value) (step S63).

Next, an area having a pixel value within the set selection range is extracted as a background pixel, and the above-described selection processing is performed (step S64). At this time, if island-shaped isolated areas occur in the selected area, they are converted as background areas (step S65). Thereafter, a new outline is obtained by removing the outline of the update section from the boundary line of the area composed of the background area including the sampling points on the outline and the update section of the outline (step S66).

As a result, even if the contour of the section to be updated is a region having a texture pattern that is not plain, a correct contour to be cut can be obtained with a small amount of calculation.

When the pixel value histogram is similarly obtained for the region near the reference point, and the result of similarly obtaining the selected pixel value range based on the histogram and the above-described pixel value range substantially coincides with each other, It is desirable not to perform the selected processing. This is because the boundary between the region to be extracted and the background is often ambiguous and has low contrast. As described above, by combining the outline update processing based on the area division of the belt-shaped area with the selection or area growth processing, a complicated target shape can be easily and automatically extracted.

(Other Embodiment of the Present Invention) Even if the present invention is applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), one device (for example, a personal computer) ) May be applied.

Further, in order to operate various devices to realize the functions of the above-described embodiment, an apparatus connected to the various devices or a computer in a system is required to realize the functions of the above-described embodiment. The present invention also includes a software program code supplied and implemented by operating the various devices according to a program stored in a computer (CPU or MPU) of the system or the apparatus.

In this case, the program code of the software implements the functions of the above-described embodiment, and the program code itself and means for supplying the program code to a computer are provided.
For example, a recording medium storing such a program code constitutes the present invention. As a recording medium for storing such a program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM,
A magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the supplied program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) or other operating system running on the computer. Needless to say, even when the functions of the above-described embodiments are realized in cooperation with application software or the like, such program codes are included in the embodiments of the present invention.

Further, after the supplied program code is stored in a memory provided in a function expansion board of a computer or a function expansion unit connected to the computer, the function expansion board or the function expansion unit is specified based on the instruction of the program code. It is needless to say that the present invention also includes a case where the CPU or the like provided in the first embodiment performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0103]

As described above, according to the present invention, for example, the initial outline is roughly set so as to surround the extraction target or to exist inside the region of the target object.
Since the process of updating a limited number of sampling points on the contour line to the boundary point position obtained by region division performed in the band-shaped region set in the reference point direction in the contour line is repeated, any complicated shape A contour line approximating the contour of the extraction target can be almost automatically obtained with a small amount of calculation, and the contour line after the update processing can be surely converged on the correct contour line of the extraction target object. Further, after setting the contour so as to surround the object to be extracted or to exist inside the area of the object, the user sees the contour that is sequentially updated and deformed, and stops updating, Since it is only necessary to give a simple instruction such as giving a start (or restart) instruction, it is possible to greatly reduce the time and effort required for extracting the extraction target.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a main configuration of an image processing apparatus according to an embodiment.

FIG. 2 is a block diagram illustrating a detailed configuration of an image processing operation unit illustrated in FIG. 1;

FIG. 3 is a flowchart showing a flow of basic processing of the present invention.

FIG. 4 is a diagram illustrating an example of a band-shaped region and a division result thereof.

FIG. 5 is a flowchart illustrating a flow of a region dividing process.

FIG. 6 is a diagram illustrating an intermediate result of a processing process of the image processing apparatus according to the present embodiment.

FIG. 7 is a flowchart illustrating a flow of a contour update process according to the second embodiment.

FIG. 8 is a diagram showing positions of division points on a boundary line of a belt-shaped region.

FIG. 9 is a flowchart illustrating a flow of processing from setting a reference point to updating a contour according to a third embodiment.

FIG. 10 is a flowchart illustrating a flow of a reference point setting process according to a fourth embodiment.

FIG. 11 is an explanatory diagram of inner / outer vicinity regions corresponding to sampling points on a contour line according to a fifth embodiment.

FIG. 12 is a flowchart illustrating a flow of an outline update process according to a sixth embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Image data input part 2 Image data storage part 3 Image display means 4 Instruction selection means 5 Image processing calculation part 6 Initial contour line 7 Reference point 8 Sampling point 9 Strip area 10 Update section start point 11 Update section end point 501 Initial contour setting means 502 Area dividing means 503 contour updating means 504 contour determining means 505 primary storage means 506 processing result output means 507 reference point setting means 508 band-shaped area setting means

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Fumiaki Takahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo F-term in Canon Inc. (Reference) 5L096 AA02 CA24 DA01 EA04 EA35 FA06 FA15 FA35 FA41 FA60 GA19 GA34 JA11

Claims (24)

[Claims] 1. An image processing method for extracting a specific object from an image, comprising: an initial contour setting step of setting a closed contour of an arbitrary shape; and a predetermined reference based on the set initial contour. A reference point setting step of setting a point; a band region setting step of setting a plurality of band regions of a predetermined shape including a line segment connecting the predetermined point on the initial contour and the reference point; An area dividing step of dividing each of the plurality of band areas into a plurality of areas based on image data in the band area; a contour updating step of updating a contour line based on each division point position after the area division; A contour determining step of determining a contour shape of an object to be extracted based on the updated contour line. 2. The image processing method according to claim 1, wherein in the initial contour setting step, a closed contour having an arbitrary shape is set so as to include the object to be extracted. 3. The image processing method according to claim 1, wherein in the initial contour setting step, a closed contour having an arbitrary shape is set in the object to be extracted. 4. The image processing method according to claim 1, wherein in the reference point setting step, the reference point is set at a center of gravity of the initial contour. 5. The image processing method according to claim 1, wherein in the reference point setting step, the reference points are set at a plurality of positions in the initial contour. 6. The strip-shaped area setting step, wherein the reference point closest to a point on the initial contour is selected from the plurality of reference points to set the strip-shaped area. Image processing method. 7. The step of setting a band-like area includes sampling points on the initial contour at predetermined intervals, and setting a plurality of band-like areas each including a line segment connecting each of the sampling points and the reference point. The image processing method according to claim 1, wherein: 8. The contour updating step includes setting an updating section on the contour line, based on the position of the division point in the strip-shaped area including a point on the contour line in the set updating section. 2. The image processing method according to claim 1, further comprising an updating step of updating the outline. 9. The image processing method according to claim 8, wherein in the outline updating step, it is possible to select whether to update the entire outline or to update by setting the update section. 10. In the contour determining step, a contour shape of the object is obtained by performing contour tracking on image data in the thick line area including the updated contour line. The image processing method according to claim 1. 11. The contour determining step includes a step of obtaining a predetermined evaluation function value for the updated contour line, and performing a dynamic contour process for moving each point on the contour line to a position at which the evaluation function value is minimized. 2. The image processing method according to claim 1, wherein a contour shape of the object is obtained based on the object. 12. The evaluation function includes a first feature amount deviation between an image feature amount related to a sampling point on the contour line after the update and an image feature amount obtained for an outer neighboring area corresponding to the sampling point; A term for evaluating a difference between the first and second feature value deviations obtained from the second feature value deviation between the image feature value related to the sampling point and the image feature value obtained for the inner vicinity area corresponding to the sampling point is included. The image processing method according to claim 11, wherein: 13. The evaluation function evaluates a first probability that a sampling point on the updated contour line is in a region near the outside of the contour and a second probability that the sampling point is in a region near the inside of the contour. 12. The image processing method according to claim 11, further comprising a term for evaluating a difference between the first and second probabilities obtained by the calculation. 14. In the contour updating step, a new contour line is generated by connecting division points farthest from the reference point among the division point positions in the plurality of strip-shaped areas by a predetermined method. The image processing method according to claim 1, wherein: 15. The outline updating step is in contact with a division boundary line passing through a division point defined in the strip-shaped area and having a direction substantially orthogonal to a line segment connecting a point on the outline and the reference point. A consistency evaluation value between a predetermined feature value of the two neighboring regions and a feature value of the two neighboring regions adjacent to the dividing boundary line similarly obtained for the other band-like regions is determined, and the division value having the highest evaluation value is determined. 2. The image processing method according to claim 1, wherein the point positions are connected to each other by a predetermined method to generate a new contour line. 16. A division having a direction substantially perpendicular to a line segment passing through the division point and connecting a point on the contour line and the reference point when the division point position is determined in the strip-shaped region in the region division step. 16. The image processing method according to claim 15, wherein predetermined feature amounts are obtained for two neighboring regions that are in contact with the boundary line, and the obtained feature amounts are used in the contour updating step. 17. The outline updating step includes: a division boundary line having a direction passing through a division point defined in the band-shaped region and substantially orthogonal to a line segment connecting a point on the outline line and the reference point; Evaluating the continuity of a strip-shaped area adjacent to the strip-shaped area with a division boundary line that has already been set and passing through the division point, and determining the position of the division point so that the continuity is higher than a predetermined reference value. Claim 1 characterized by the following:
The image processing method according to 1. 18. The method according to claim 18, wherein the step of updating the contour includes pixels having an image feature amount similar to that of a sampling point on the contour line in the set update section and an image feature amount of an area near the sampling point. A selection step of selecting, as a background pixel, a pixel forming a connected area as a region having a high degree of similarity of the image feature amount; and a boundary line between the background pixel obtained after the selection step and the object to be extracted. 9. An image processing method according to claim 8, further comprising the step of: updating a contour line in an update section to perform the process of setting an updated contour line in the update section. 19. The region dividing step, wherein the resolution of the region division is increased as the number of applications of the contour updating step is increased, and in the band-shaped region setting step, as the number of applications of the contour updating step is increased, 2. The image processing method according to claim 1, wherein the sampling interval is reduced. 20. An image inputting step of inputting an image including an object to be extracted, a contour setting step of setting a closed contour of an arbitrary shape, and a contour updating for updating the shape of the contour by a predetermined method. An update result display step of displaying the updated outline superimposed on the input image; an instruction input step of instructing to stop or start updating the outline; and outputting the updated outline shape. An image processing method comprising: 21. An updating section setting step for specifying a section on a contour line to be updated or a non-updating section setting step for specifying a section on a contour line not to be updated, wherein the contour shape of the updating section is determined. 3. A partial contour updating step of updating by the method of claim 2.
0. The image processing method according to 0. 22. An image processing apparatus for extracting a specific object from an image, comprising: an initial contour setting means for setting a closed contour having an arbitrary shape; and setting a predetermined reference point based on the initial contour. Reference point setting means for setting, band-shaped area setting means for setting a plurality of band-shaped areas having a predetermined shape including a line segment connecting a predetermined point on the initial contour and the reference point, and image data in the plurality of band-shaped areas Area dividing means for dividing each of the plurality of band-shaped areas into a plurality of areas based on the following; contour updating means for updating a contour line based on each division point position after the area division; An image processing apparatus, comprising: a contour determining unit for determining a contour shape of an object to be extracted based on the contour shape. 23. Image input means for inputting an image including an object to be extracted, contour setting means for setting a closed contour having an arbitrary shape, and contour updating means for updating the shape of the contour by a predetermined method. Update result display means for displaying the updated outline superimposed on the input image; instruction input means for instructing to stop or start updating the outline; and outputting the updated outline shape. An image processing apparatus comprising: a contour output unit. 24. Initial contour setting means for setting a closed contour of an arbitrary shape, reference point setting means for setting a predetermined reference point based on the initial contour, a predetermined point on the initial contour and the reference point A band-shaped region setting means for setting a plurality of band-shaped regions having a predetermined shape including a line segment connecting the plurality of band-shaped regions; a region dividing means for dividing the plurality of band-shaped regions into a plurality of regions based on image data in the plurality of band-shaped regions, A computer functions as contour updating means for updating a contour line based on each division point position after the region division, and contour determining means for obtaining a contour shape of an object to be extracted based on the updated contour line. A computer-readable recording medium on which a program for recording is recorded.