JP2008102946A - Contour extraction method for image, object extraction method from image and image transmission system using the object extraction method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To speed up processing of an object area extraction system using self-similar mapping. <P>SOLUTION: This contour extraction method comprises: a first step for setting a search reference block to an contour portion of shape data; a second step for finding a similar block having a similar pattern of image data and having a block size larger than the search reference block from the same image in each the search reference block; and a third step for replacing data obtained by binarizing the image inside each the search reference block with the shape data of the corresponding similar block. In contour extraction processing for bringing an contour of the shape data into line with a contour of an object by repeating the third step prescribed times, the contour extraction processing is repeatedly performed, and the image data, the shape data and the search reference block are reduced in the beginning of the repetition to perform contour position correction processing. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

    The present invention relates to an image contour extraction method for extracting a region of a contour extraction target object from an image, an object extraction method from an image, and an image transmission system using the object extraction method.

  A technique for extracting a target object region from an image is useful for processing such as replacing the background with another image, for example. At that time, a high-quality processed image cannot be obtained unless the region of the extraction target object that is the target subject is correctly obtained.

  Therefore, there is a demand for a technique for obtaining an object region with higher accuracy based on the approximate shape when the approximate shape of the object is obtained by some method. Proposes what is disclosed in the literature: “Fitting contours by self-similar mapping” (Ida, Mimotosugi, Proceedings of the 5th Symposium on Image Sensing, C-15, pp.115-120, June 1999). did. This technique is a technique using a self-similar mapping, and its outline will be described with reference to FIGS.

  In this technique, as shown in FIG. 5, the image portion of the extraction target object (correct object region) 1 in the frame image to be processed is represented by diagonal lines, and the schematic shape 2 of the extraction target object is represented by thick lines. The outline shape 2 of the extraction target object is, for example, an outline drawn roughly manually by the operator along the outline of the extraction target object on the frame image displayed on the screen. There is a gap.

  Therefore, the pixel value “255” is substituted for the inner pixel of the schematic shape 2 of the extraction target object, and the pixel value “0” is substituted for the outer pixel. That is, an image in which all the pixels inside the outline shape 2 of the extraction target object are filled with the pixel value “255” and all the backgrounds are filled with the pixel value “0” (that is, a binary image) is obtained. An image (binary image) obtained by performing such processing is called shape data or an alpha map.

  According to the method using the self-similar mapping, the tentatively given outline 2 of the extraction target object, that is, the tentatively set outline of the shape data is the extraction target object 1 that is the correct outline of the object to be obtained. It can be made to correspond to the outline 1a.

  For this purpose, first, search reference blocks B1, B2,..., Bn−1, Bn are arranged along the outline of the temporary shape data (provisional alpha map) (schematic shape 2 of the extraction target object).

  This scans the image of provisional shape data (provisional alpha map) from the upper left pixel (coordinate position 0, 0) in the right direction and sequentially from the upper line to the lower line, that is, from the reference coordinate position. The pixel is examined by XY scanning, and when the pixel value is different from the pixel on the left or upper side and is not included in the block set so far, a predetermined size ( This is done by arranging blocks of block size b). Thereby, as shown by B1, B2,..., Bn in FIG. 5, a search reference block set while overlapping the blocks in a daisy chain is obtained.

  Next, each search reference block B1,..., Bn is positioned on the corresponding coordinate position in the frame image to be processed. As a result, each search reference block B1,..., Bn includes a partial region of the extraction target object 1 partially including the contour position of the extraction target object (correct object region) 1. Find similar blocks with similar pixel conditions.

  Similar blocks are trial-and-error set within the range of the corresponding search reference block, and the error between the reduced image in the similar candidate block and the image in the search reference block is minimized. Find what will be and find it as a similar block. For example, FIG. 6 shows a similar block Bs1 of the search reference block B1, which corresponds to this diagram.

In this way, similar blocks of the extraction target object (correct object region) 1 for each search reference block B1,..., Bn are obtained.
As described above, a similar block is a block that is larger than the corresponding search reference block and has a pixel value that is substantially equal to the image data in the search reference block when the image data is reduced.

  A specific method for finding similar blocks is as follows. For example, in order to find a similar block of the search reference block B1, a search area Fs1 having an appropriate size is set around the search reference block B1, as shown in FIG. Then, while setting various similar candidate blocks Bc in the search area Fs1, the error evaluation between the block size reduction process and the search reference block B1 is performed each time, and the similar candidate block whose error is minimized Is determined as the similar block Bcd.

  In the example of FIG. 7, the error evaluation is performed while shifting the vertical direction and the horizontal direction of the search reference block B1 by one pixel within the range of w pixels on the basis of the similarity candidate block Bc whose center is the same position.

  6 shows only the similar block Bs1 of the search reference block B1, but of course, similar blocks are obtained for all the search reference blocks B1,... Bn shown in FIG.

  Next, with respect to the shape data (alpha map), the data corresponding to the position of each search reference block B1,... Bn is the shape data (alpha map) obtained from the image in the similar block corresponding to the search reference block. It corrects by the technique of replacing. Since the similar block is larger in size than the search reference block, the size is naturally reduced and matched (size normalization).

  In this way, correction is performed by a technique of replacing with normalized shape data (alpha map) obtained from the image in the similar block corresponding to the search reference block. If this is performed once in all the search reference blocks B1,... Bn, the outline of the shape data (the outline shape 2 of the extraction target object) approaches the outline 1a of the correct object region 1. Then, by repeating the replacement recursively, the contour of the shape data (schematic shape 2 of the extraction target object) converges near the contour 1a of the correct object region 1 as shown in FIG.

In this method, the search reference blocks B1,... Bn need to include the correct contour line of the object region 1. Therefore, when the difference between the search reference blocks B1,... Bn with respect to the contour of the correct object region 1 is large, first, by performing the above-described processing using a large block, the contour of the shape data (the outline of the object to be extracted) The shape 2) is brought close to the contour 1a of the correct object region 1, and then the above-described processing is performed on a small block, thereby matching fine irregularities.
As a result, the contour can be extracted with high accuracy even when the contour shift is large. The flowchart is shown in FIG. The process will be described with reference to the flowchart of FIG.
[Process of Step S1]: If the block size of the search reference block is b, this block size b is set to A.

  [Processing of Step S2]: Using this block size b, the contour position correcting process described above with reference to FIGS.

  [Process of Step S3]: Next, the block size b of the search reference block is set to a half value of the previous time.

  [Process of Step S4]: As a result, if the block size b of the search reference block is smaller than Z (<A), the process is terminated, and if not, the process returns to the process of Step S2.

  By such processing, even if the contour of the provisionally given shape data (provisionally given alpha map) is largely different from the contour of the target extraction target, the contour is matched with the target extraction target contour. Therefore, the contour of the target extraction target image can be extracted from the image with high accuracy by using the shape data (alpha map) corrected by the contour position correction processing. become.

  As a method of extracting a target image portion from image data, when a rough shape of an object is obtained by some method, a technique for obtaining a region of an extraction target object based on the rough shape is required. One method is to use a self-similar map.

  In this technique, an operator roughly draws a rough shape of a target image portion in an image by manual operation to obtain a rough shape 2 of the extraction target object, and a pixel inside the rough shape 2 of the extraction target object is a pixel. By substituting the value “255” and substituting the pixel value “0” for the outer pixel (that is, binarization), shape data is obtained.

  Then, search reference blocks B1, B2,..., Bn−1, Bn are arranged along the outline of the shape data (schematic shape 2 of the extraction target object).

  Next, similar blocks are obtained for the respective search reference blocks B1,..., Bn.

  Next, with respect to the shape data, the data at the position of each search reference block B1,... Bn is replaced with the data cut out by the similar block corresponding to the search reference block and reduced. If this is performed once in all the search reference blocks B1,... Bn, the outline of the shape data (the outline shape 2 of the extraction target object) approaches the outline 1a of the correct object region 1. Then, by repeating the replacement recursively, the contour of the shape data (schematic shape 2 of the extraction target object) converges near the contour 1a of the correct object region 1 as shown in FIG. This is because each similar block is larger than the corresponding search reference block, and the pixel value when the image data is reduced is almost equal to the image data in the search reference block.

  In this method, the search reference blocks B1,... Bn need to include the correct contour line of the object region 1. Therefore, when the difference between the search reference blocks B1,... Bn with respect to the contour of the correct object region 1 is large, first, the above-described processing is performed using a large block, so that the contour of the provisionally given shape data ( By bringing the outline shape 2) of the extraction target object close to the outline 1a of the correct object region 1 and then performing the above-described processing with a small block, it is possible to correct even fine irregularities. As a result, the contour can be extracted with high accuracy even when the contour shift is large.

  According to such a method using the self-similar mapping, the outline shape 2 of the provisional object to be provisionally given, that is, the outline of the shape data provisionally given coincides with the outline 1 a of the correct object region 1. Can be made.

  However, in this method, there is a problem that it is difficult to extract a target image portion from an image at a high speed because the calculation amount of the similar block search is large. Therefore, further improvement is necessary to apply to moving images.

  Accordingly, an object of the present invention is to provide an image contour extraction method capable of reducing the amount of processing and extracting a target image portion from an image at high speed.

  In order to achieve the above object, the present invention is configured as follows.

[1] First, image data obtained by capturing the contour extraction target object and shape data that is a binary image of a provisional contour shape representing the region of the contour extraction target object in the image data are input. , A first step of setting a plurality of search reference blocks of a predetermined size with their respective center positions placed on the contour portion of the shape data, and for each search reference block within each block, A second step of searching for a similar block in the same image with a similar block size of the image data and a block size larger than the search reference block; and among the shape data, the shape in each search reference block By replacing the data with the correction-corrected shape data obtained by the reduction process obtained from each of the similar blocks, the shape is corrected. And a third step of the correction processing of the data, by repeating the number of times the third step a given, in the contour extraction processing method for matching the contour of the shape data in the contour of the object,
The contour extraction process is repeated a predetermined number of times,
At the beginning of the iteration, the image data and the shape data are reduced and contour extraction processing is performed.

  In the present invention, a method using self-mapping is employed to correct the temporary shape data (alpha map) so as to match the contour of the contour extraction target object. For this purpose, image data obtained by capturing the contour extraction target object is used. And a search for a predetermined size by using the shape data which is a binarized image of a temporary contour shape representing the region of the contour extraction target object in the image data, and placing the center position on each contour portion of the shape data. A plurality of reference blocks are set by shifting their positions. Then, for each search reference block, a search is made in the same image for a similar block having a similar design of the image data in the block and a block size larger than the search reference block. Then, the shape data correction processing is performed by replacing the shape data in each search reference block among the shape data with the shape data for correction that has been size-adjusted by the reduction processing obtained from each of the similar blocks.

  In the present invention, the contour extraction process is repeated a predetermined number of times, and at the beginning of the repetition, the image data and shape data are reduced to perform the contour extraction process.

  That is, in the contour extraction process that is repeated a plurality of times, the image data and the shape data are reduced at the beginning, and the contour extraction process is performed. However, if the image size to be processed is reduced, the contour shifts accordingly. Therefore, the fact that the contour of the shape data can be brought close to the correct position without making the block size of the search reference block so large is utilized. Therefore, according to the present invention, in the first round of contour extraction processing, the image data is reduced and the contour extraction processing is performed using the shape data and the search reference block also reduced.

  Since the processing can be performed with a block having a smaller size than the conventional processing, the amount of calculation can be reduced accordingly. As the number of contour extraction processes increases, the contour extraction process is performed with the original image size, so that the shape data is corrected so that the fine contours of the contour extraction target object are finally reflected. By using this corrected shape data, the contour of the contour extraction target object can be extracted with high accuracy.

  Therefore, according to the present invention, it is possible to provide an image contour extraction method capable of reducing the amount of processing for contour position correction of shape data (alpha map image) and extracting a target image portion from an image at high speed.

[2] Further, according to the present invention, image data obtained by capturing the contour extraction target object and shape data that is a binary image of a provisional contour shape representing the region of the contour extraction target object in the image data are input. And a first step of setting a plurality of search reference blocks of a predetermined size with their respective center positions positioned in the contour portion of the shape data, and for each search reference block within each block A second step of searching the same image for a similar block having a block size larger than that of the search reference block, and the shape of the image data within each search reference block. By replacing the shape data with the correction-corrected shape data obtained by the reduction process obtained from each of the similar blocks And a third step of the correction process Eipudeta, by repeating the number of times the third step a given, in the contour extraction processing method for matching the contour of the shape data in the contour of the object,
The processing in the second step is characterized in that the search range of the similar block is limited to a direction perpendicular to the direction of the contour of the shape data in the search reference block.

  In the present invention, the shape data (alpha map) is corrected so as to match the contour of the contour extraction target object by adopting a method using self-mapping, for that purpose, image data obtained by capturing the contour extraction target object, A search reference block having a predetermined size with a center position on each contour portion of the shape data using shape data which is a binarized image of a temporary contour shape representing the region of the contour extraction target object in the image data Are set by shifting the position of each other. For each search reference block, a search is made in the same image for a similar block having a similar design of the image data in the block and having a block size larger than that of the search reference block. Of the shape data, the shape data in each search reference block is replaced with the shape data for correction that has been size-adjusted by the reduction processing obtained from the respective similar blocks, thereby correcting the shape data.

  In the present invention, the processing in the second step, that is, for each search reference block, the image data in the block is similar, and the similar block has a larger block size than the search reference block. In the similar block search process of searching for the same image from the same image, the search range of the similar block is limited to a direction perpendicular to the direction of the contour of the shape data in the search reference block.

  In other words, according to the conventional method, the similar block search process for a certain search reference block is to search for a block having a similar pixel pattern by extending the block size within a predetermined range up, down, left, and right around the search reference block. However, in the present invention, the search range of the similar blocks is limited to a direction perpendicular to the contour direction of the shape data in the search reference block. This reduces the amount of calculation.

  Of course, the contour of the contour extraction target object is unknown, and it is naturally unknown when searching for similar blocks whether the contour of the shape data given provisionally is moved to approach the contour of the contour extraction target object. Since the direction of the contour of the shape data and the direction of the contour of the contour extraction target object are empirically almost the same, it is most reasonable to search for the direction perpendicular to the contour direction of the shape data. It is.

  Therefore, according to the present invention, it is possible to provide an image contour extraction method capable of reducing the amount of processing for contour position correction of shape data (alpha map) and extracting a target image portion from an image at high speed. .

  Furthermore, the present invention compares each part of the image data in which the extraction target object is imaged with another part of the same image data or another image data, so that the provisional region of the extraction target object in this image data is obtained. A second step of matching the contour of the shape data with the contour of the object to be extracted using the first step of generating shape data that is an image representing the image, and the shape data generated provisionally. An object extraction method from an image is provided.

  According to the present invention, an object can be appropriately extracted without fixing the object at a predetermined position.

The present invention receives, as input, image data obtained by capturing an extraction target object and shape data that is an image representing a provisional area of the extraction target object in the image data. A first step of setting a plurality of search reference blocks having a predetermined size by shifting their positions, and for each search reference block, the design of the image data in the block is similar, and A second step of searching for a similar block having a block size larger than the search reference block from the same image; and a reduction process of obtaining the shape data in each search reference block among the shape data from the similar block The third step of correcting the shape data by replacing the corrected shape data with the size of In the object extraction method comprising the flop,
The shape data has different pixel values in different object regions and background regions, and in the reduction process in the third step, the pixel value of any pixel around the sampling point of the shape data is obtained. Provided is a method for extracting an object from an image, characterized by using a sampling value.

The present invention provides a transmission system composed of a server and a client that mutually transmit and receive transmission / reception data, wherein the client extracts an object from image data obtained by capturing an extraction target object and obtains extracted image data; and When the extracted image data is compressed as it is or is compressed and sent to the server as upstream transmission / reception data, and the downstream transmission / reception data sent from the server is received, and the transmission / reception data is not compressed It is composed of client receiving means for reproducing the image data when compressed as image data as it is, and means for displaying the image data.
The server receives the upstream transmission / reception data, and when the transmission / reception data is not compressed, it is used as extracted image data as it is, and when it is compressed, server reception means for reproducing the extracted plane image data; Transmission comprising: combining means for combining extracted image data into one composite image data; and server transmission means for transmitting the composite image data as it is or after compressing the composite image data as the downlink transmission / reception data Provide a system.

  According to the present invention, the process of bringing the contour of the provisionally given shape data (alpha map) closer to the contour of the contour extraction target object can be executed with a greatly reduced amount of computation. Therefore, it is possible to provide an image contour extraction method that enables high-speed contour extraction accordingly.

  Also, according to the present invention, initial shape data can be automatically obtained wherever an object to be extracted is on the screen, and an object can be extracted with very few operations whether or not a user operation is required.

  Embodiments of the present invention will be described below with reference to the drawings. In the present invention, first, the shape data (alpha map) used for the contour extraction of the contour extraction target object in the image using the image data, the provisional shape data, and the search reference block is used as the contour of the contour extraction target object. The contour correction processing for correcting the image data to match the above is repeated, and at the beginning of the iteration, the image data, the shape data, and the search reference block are reduced to perform the contour correction processing.

  That is, in the first invention, the contour correction processing is performed with the image size being reduced in the first round, so that the number of pixels of the contour shift is also reduced accordingly. The fact that the contour of the shape data (alpha map) can be brought close to the correct position without making the block size of the block so large is utilized. As described above, since the processing can be performed with the search reference block having a smaller block size than the conventional one, the calculation amount can be reduced. Finally, by performing contour correction processing with the original image size, it is possible to obtain shape data (alpha map) fitted to irregularities with fine contours.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Basic processing contents used in the present invention)
Also in the present invention, the basic processing content of the contour correction processing follows the method described in the conventional example.

[Basic method]
That is, the basic method is as follows. Now, as shown in FIG. 5, there is an image of the object to be extracted (correct object region) 1 in the frame image (original image) in which the image to be extracted is picked up, and it is considered that the contour is extracted. .

[Process 1]
<Outline shape setting of extraction target object>
The operator displays the original image on the display screen, and sets the schematic shape 2 of the extraction target object in the image of the extraction target object (correct object region) 1 on the screen. The coordinate position of the frame image (original image) and the data of the approximate shape 2 of the object to be extracted is shared by both, but even if they can be overlaid such as by separate layers, the data of both will not be mixed To manage.

  The outline shape 2 of the extraction target object is, for example, an outline drawn roughly by a manual operation so that the operator follows the outline of the extraction target object on the frame image (original image) displayed on the screen. There is a deviation from the outline 1a.

Therefore, a process for matching the outline of the schematic shape 2 of the extraction target object with the outline 1a of the extraction target object 1 which is the correct outline of the object to be obtained is performed by the self-similar mapping method.
[Process 2]
The self-similar mapping method is as follows.

[Process 2-1]
<Generate shape data (alpha map)>
First, for the data of the outline shape 2 of the extraction target object, the pixel value “255” is assigned to the pixel inside the outline shape 2 of the extraction target object, and the pixel value “0” is assigned to the outside pixel. Apply the substitution process. As a result, all the pixels inside the outline shape 2 of the extraction target object are filled with the pixel value “255”, and the background is all filled with the pixel value “0” (that is, a binary image). Shape data (alpha map) is obtained.

[Process 2-2]
<Location of search reference block>
Next, search reference blocks B1, B2,..., Bn−1, Bn are arranged along the outline of the shape data (alpha map) (schematic shape 2 of the extraction target object).

  This is done by scanning from the upper left pixel (coordinate position 0, 0) of the alpha map in the right direction, sequentially from the upper line to the lower line, that is, by XY scanning from the reference coordinate position and examining the pixels. Alternatively, when the pixel value is different from that of the upper adjacent pixel and the pixel is not included in the previously set block, a block having a predetermined size (block size b) is arranged around the pixel. . As a result, search reference blocks arranged with their own position coordinates are obtained as indicated by B1, B2,..., Bn in FIG. In this example, the search reference blocks B1, B2,..., Bn are set while overlapping in a daisy chain.

[Processing 2-3]
<Acquiring similar blocks>
If search reference blocks B1, B2,..., Bn each having unique position coordinates are obtained, then the search reference blocks B1,..., Bn are coordinated on the frame image (original image) to be processed. Align and position.
As a result, each search reference block B1,..., Bn is the position of the extraction target object 1 partially including the contour position of the extraction target object (correct object region) 1 within the range of the block size that the search reference block B1,. Since the partial area is specified, a similar block having similar pixel conditions in each block is obtained using this.

  Here, the similar block is a block that is larger than the corresponding search reference block and has a pixel value that is substantially equal to the image data in the search reference block when the image data is reduced.

  Similar blocks are obtained by setting each similar candidate block by trial and error in a range in which the corresponding search reference block is enlarged, reducing the image within the block range to the block size of the search reference block, and the corresponding search criteria. Check the degree of whether or not the pixel status of the image in the block (simply speaking, the design in the block) is similar by error evaluation, find the one with the smallest error (best evaluation result), and compare it Ask as a block.

  In this way, similar blocks of the extraction target object (correct object region) 1 for each search reference block B1,..., Bn are obtained.

A specific method for finding similar blocks is as follows.
For example, in order to find a similar block of the search reference block B1, a search area Fs1 having an appropriate size is set around the search reference block B1, as shown in FIG. Then, various similar candidate blocks Bc are set within the search area Fs1, and each time, the reduction process to the block size of the search reference block B1 and the similar candidate block and the search reference block B1 after the reduction process are performed. Error evaluation for evaluating the similarity of the pixel distribution is performed, and the similarity candidate block having the smallest error (the evaluation result is the best) is determined as the similarity block Bcd.

  In the example of FIG. 7, the similarity candidate block Bc whose vertical and horizontal dimensions are the same as the search reference block B1 and whose center coordinate position is the same position is used as a reference, and this is shifted by one pixel in the range of w pixels vertically and horizontally. An error evaluation is performed on the pixel distribution of the search reference block B1 using the obtained new similarity candidate block.

  6 shows only the similarity block Bs1 of the search reference block B1, but of course, final similarity blocks Bcd1,... Bcdn are obtained for all the search reference blocks B1,.

[Process 2-4]
<Outline position correction processing>
If similar blocks Bcd1,... Bcdn are obtained for all the search reference blocks B1,... Bn, next, the portions in the search reference blocks B1,. Then, the process of replacing with the correction shape data (alpha map) is performed. The correction shape data (alpha map) is obtained by cutting out from the shape data (alpha map) with the similar blocks Bcd1,... Bcdn corresponding to the search reference block, and reducing the block size of the search reference block.

  If this is performed once in all the search reference blocks B1,... Bn, the outline of the shape data (the outline shape 2 of the extraction target object) approaches the outline 1a of the correct object region 1. Then, by repeating such replacement processing recursively, as shown in FIG. 8, the contour of the shape data (schematic shape 2 of the extraction target object) converges near the contour 1 a of the correct object region 1. .

  By such processing, the contour of the shape data (alpha map) can be matched with the contour of the correct object region 1 up to fine irregularities.

  However, in this method, “when the search reference blocks B1,... Bn are arranged on the frame image (original image), the contour lines of the correct object region 1 are included in each search reference block B1,. Is necessary to be satisfied. Therefore, when the difference between the search reference blocks B1,... Bn with respect to the contour of the correct object region 1 is large, first, by performing the above-described processing using a large block, the contour of the shape data (the outline of the object to be extracted) The process of bringing the shape 2) close to the contour 1a of the correct object region 1 is performed to adjust the necessary conditions, and then the above processing is recursively performed with small blocks, so that the contour of the correct object region 1 is finally obtained. To match.

The contour of the corrected shape data (alpha map) obtained by such processing can be matched with the contour 1a of the contour extraction target object (correct object region 1) to fine irregularities.
As a result, the contour can be extracted with high accuracy even when the contour shift is large.

  The above is the contour position correction processing of the shape data (alpha map). In the present invention, the content of the basic processing follows the conventional self-similar mapping method. However, in the present invention, in order to reduce the burden of calculation processing, the block size of the search reference block used in the contour position correction processing is made smaller than that of the conventional method. Details of the method will be described.

(First embodiment)
That is, in the conventional method, as shown in FIG. 9, the shape data (alpha map), the original image (the frame image being picked up), and the search reference block have the original size. As-is (not reduced) was used. However, in the present invention, in order to reduce the arithmetic processing, the size is reduced to half the current size in the vertical and horizontal directions.

  That is, conventionally, when the block size per side of the search reference block is b, this b is set as A. Then, the search reference block having the block size b of A is used, and the original image (the frame image being picked up as an extraction target) and the shape data (alpha map) are also used in the original size, and FIG. The contour position correction process described with reference to FIG. 8 was performed. Then, the block size b of A is reduced to half of the value, and if the block size b is smaller than Z (<A), the process ends. Otherwise, the block size is further reduced to half. The process of performing the contour position correction process was repeated.

  By such processing, the contour can be extracted with high accuracy even when the contour shift is large.

  On the other hand, in the present invention, the contour position correction process is a two-stage process of a rough adjustment stage and a main adjustment stage. In the contour position correction process in the coarse adjustment stage, provisional shape data (alpha map) from the beginning. ) Is reduced to 1/2 in the vertical and horizontal directions, and the original image is also reduced to 1/2 in the vertical and horizontal directions, and the search reference block having a block size of b / 2 is used accordingly. A position correction process is performed. As a result, the arithmetic processing is reduced.

  FIG. 1 is a flowchart showing an embodiment of the present invention, and the present invention will be described with reference to this flowchart. In the present invention, the shape data (alpha map), the original image, and the search reference block are reduced in size as the first stage to perform contour position correction processing, thereby extracting provisional shape data (alpha map). This coarse adjustment is performed in order to approximate the correct contour shape of the target object (coarse adjustment stage), and then to fit the coarsely adjusted provisional shape data (alpha map) to the correct contour shape of the target object to be extracted. The contour shape correction processing is performed using the original shape data (alpha map) and the original image at the original size, and the search reference block using the small size used at the coarse adjustment stage (this book). (Adjustment stage), thereby making it possible to implement contour extraction processing with a greatly reduced amount of computation.

The main part of the present invention will be described.
If shape data (alpha map) in the first stage, that is, provisional shape data (provisional alpha map) is obtained, first, in the coarse adjustment stage, the process in [Process 2-2] described above is performed. Prior to the processing of “place search reference blocks B1, B2,..., Bn−1, Bn along the outline of the shape data (alpha map) (schematic shape 2 of the extraction target object)”, the search reference block Determine the block size.

  [Step S11]: As in the existing technology, the initial value of the block size b of the search reference block is A.

  [Step S12]: The provisional shape data (provisional alpha map), which is the shape data (alpha map) in the first stage, is reduced to 1/2 in the vertical and horizontal directions. This reduction can be obtained by skipping one pixel from the original provisional shape data (provisional alpha map), or by performing filtering using a majority filter of four pixels in the vicinity of the sampling point. .

[Step S13]: The frame image (original image) is reduced to 1/2 in the vertical and horizontal directions, and the block size of the search reference block is reduced to 1/2 in the vertical and horizontal directions.
Then, using the reduced frame image (original image), the search reference block, and the shape data (alpha map), the contour position correction process after [Process 2-2] is performed.

[Step S14]: Next, the block size of the search reference block used in the above-described contour position correction process is further reduced to 1/2 in the vertical and horizontal directions.
[Step S15]: Then, it is checked whether or not the block size b of the search reference block reduced in S14 is smaller than Y. Y is a desired value set in advance and has a relationship of Y <A. As a result of the size comparison between b and Y, if b is smaller than Y, the process proceeds to step S16. Otherwise, the process returns to step S13. This is a measure for avoiding that the block size of the search reference block used for the contour position correction process in the coarse adjustment is excessively reduced and the image in the search reference block becomes too small.

  By repeating this process until the block size reaches a predetermined size in this way, provisional shape data (tentatively an alpha map) can be approximated to the correct contour shape of the extraction target object ( Coarse adjustment stage).

  If b is smaller than Y as a result of comparing the magnitudes of b and Y, the rough adjustment stage is completed, and the process is shifted to the main adjustment stage. This adjustment stage corresponds to the processing after step S16.

  [Step S16]: In this adjustment stage, first, the shape data (alpha map) reduced and used in the coarse adjustment stage is returned to the original size. That is, the size of the coarsely adjusted shape data (roughly adjusted alpha map), which is the coarsely adjusted shape data (alpha map) used in the coarse adjustment stage, is enlarged two times vertically and horizontally. Thereby, it becomes the same size as the original original image.

  [Step S17]: The search reference block of the block size b used in the original processing of the original size and used in the final processing of the coarse adjustment stage and the coarsely adjusted shape data (roughly adjusted) returned to the original size The contour position correction process is performed using the (alpha map).

  [Step S18]: Next, the value of the block size b of the search reference block is halved.

  [Step S19]: Check whether the block size b of the search reference block is smaller than Z. Z is a desired value set in advance and has a relationship of Z <Y. As a result of comparing the size of b and Z, if b is smaller than Z, the process is terminated, and if b is larger than Z, the process returns to step S17.

  This is a measure for avoiding that the block size of the search reference block used for the contour position correction processing in the main adjustment is excessively reduced and the image in the search reference block becomes too small. As the size of the search reference block is gradually reduced, it becomes possible to fit even fine irregularities.

  By repeating this process until the search reference block size reaches a predetermined size in this way, the shape data (alpha map) can be brought closer to the correct contour shape of the extraction target object.

<Outline position correction processing flow>
Here, FIG. 2 shows a detailed flowchart of the contour position correction process which is the process in steps S13 and S17. Note that S13 is a process with an image size that is ½ of the original image, and S17 is a process with the same image size as the original image.

  [Step S21]: A block is set in the contour portion of the provisional shape data (provisional alpha map) as shown in FIG. Here, in S13, the size of one side of the search reference block is b / 2, and in S17, the size of one side of the search reference block is b.

  [Step S22]: A similar block of the search reference block is obtained using the image data.

  [Step S23]: If similar blocks have been obtained for all the search reference blocks set in Step S21, the process proceeds to Step S24. Otherwise, the process returns to step S22 to obtain a similar block of other search reference blocks.

  [Step S24]: Replace the shape data of the search reference block with the reduced shape data of the similar block.

  [Step S25]: If replacement has been completed for all search reference blocks, the process proceeds to step S26. Otherwise, the process returns to step S24 in order to replace another search reference block.

  [Step S26]: When the number of replacements reaches a predetermined number of times, the process ends. Otherwise, the process returns to step S24.

  In the present invention, this contour position correction processing is as shown in the following table when the size of the frame image (original image) is 320 × 240 pixels, A = 32, Y = 16, and Z = 4, for example. Such values of A, Y, and Z are values set as appropriate. A is the original size, Y is used as a threshold value at which the contour position correction process as the coarse adjustment is terminated in the coarse adjustment stage, and Z is the position at which the contour position correction process in the present adjustment stage is terminated. Use as a threshold.

table:
---------------------------------------------
[Times] [b] [image size] [block size]
---------------------------------------------
i-th time 32 160 × 120 16
ii 16th 160 × 120 8
iii 8 8 320 × 240 8
iv 4th 4 320 × 240 4
---------------------------------------------
That is, assuming that the original block size b = A of the search reference block is “32”, a frame image (original image) having a configuration of “320 × 240 pixels” is processed for the first time (i in the table) in the coarse adjustment stage. In the second), a “160 × 120 pixel” configuration with a half size is used, and the search reference block at that time has a block size of 16 pixels in the vertical and horizontal directions. Then, since b is “32” at the stage when the first processing is completed, this is halved to obtain “16”, which is compared with the value of Y. Since the value “16” is not smaller than the Y value set to “16”, the process proceeds to the second process (ii-th in the table) in the coarse adjustment stage. In the second processing in the coarse adjustment stage, “160 × 120 pixels, which is half the size of the frame image (original image) having the“ 320 × 240 pixels ”configuration, similar to that used in the first processing. A reduced-size frame image (original image) having a configuration is used, and the search reference block at that time has a block size of 8 pixels each in the vertical and horizontal directions, which is the half size of the first time.

  Then, since b is “16” at the stage when the second processing is completed, “8” is obtained by halving it, and this is compared with the value of Y. Since the value “8” is smaller than the Y value set to “16”, the coarse adjustment stage ends, and the process proceeds to the main adjustment stage.

  In this adjustment stage, b follows the value in the coarse adjustment stage final process, and the size of the search reference block follows the value in the coarse adjustment stage final process, but the frame image (original image) is the original size. Use one. That is, in this example, a frame image (original image) having a configuration of “320 × 240 pixels” is used, the size of the search reference block is “8”, and b is “8”.

  In this adjustment stage, a full size frame image (original image) having a configuration of “320 × 240 pixels” is used for each processing. In the first process (the iiith process in the table) in this adjustment stage, the search reference block has a block size of 8 pixels in the vertical and horizontal directions. Then, at the stage where the first processing (the iiith time in the table) is finished, b is “8”, so this is halved to obtain “4”, and this is compared with the value of Z . Since the value “4” is not smaller than the value of Z set to “4”, the process proceeds to the second (iv) in the adjustment stage. A frame image of “320 × 240 pixels” is used in the second processing (iv time in the table) in the main adjustment stage in the same manner as that used in the first processing (iii time in the table) in the main adjustment stage. The full-size frame image (original image) of (original image) is used, and the search reference block at that time is a half size in the first processing in the main adjustment stage, and the block size is 4 pixels each in vertical and horizontal directions It has become.

  Since b is “4” at the stage when the second processing (iv time in the table) is completed, this is halved to obtain “2”, which is compared with the value of Z. . Since the value “2” is smaller than the value of Z set to “8”, the end condition is also satisfied in this adjustment stage, and the process ends and the process is completed.

  As described above, according to the present invention, the contour position correction process is performed when the size of the frame image (original image) is 320 × 240 pixels, A = 32, Y = 16, and Z = 4. The adjustment process is completed twice, twice, a total of four times. In rough adjustment, the number of pixels used in the calculation is halved each time. In this adjustment stage, the frame image (original image) and shape data ( (Alpha map) is restored to the full size, but the search reference block is resized to 1/2 each processing step from the coarse adjustment stage, greatly reducing the total computational burden. Is done. In addition, the size of the search reference block can be reduced by ½ for each process by continuing from the coarse adjustment stage, and the shape data (alpha map) is brought close to the outline of the contour extraction target object in the coarse adjustment stage. Because it is possible.

  Therefore, when the contour extraction is performed using the method using the self-similar mapping, the outline shape of the extraction target object given first can be matched with the contour of the contour extraction target object with a small amount of arithmetic processing. A contour extraction method is obtained in which the contour of the shape data provisionally given can be matched with the contour of the contour extraction target object at high speed.

  In the above example, the block size b of the search reference block does not necessarily have to be reduced by 1/2. For example, “32”, “8”, “4” may be used, and in this way, the processing amount can be further reduced.

  As described above, the contour position adjustment processing of the shape data (alpha map) is performed in two stages of the coarse adjustment stage and the main adjustment stage. In the coarse adjustment stage, both the original image and the search reference block are shaped data (alpha map). If the block size to be reduced reaches a predetermined condition, the contour position adjustment process is performed by further reducing the size of the block in the previous contour position adjustment process. This adjustment stage uses the original image and the shape data (alpha map) of the original size, and the search reference block has the size used at the end of the coarse adjustment stage. After that, until the block size reaches a predetermined condition, the contour position adjustment process is performed using the search reference block that has been reduced from the previous time each time. Carried out, whereby is obtained with aligning the contour shape of the extraction target shape of the shape data given the desired (alpha map).

  In this contour position adjustment process, when searching for similar blocks, similar candidates are expanded in the vertical and horizontal directions of the search reference block for a while to search for the corresponding ones, but always expand in the vertical and horizontal directions. Instead of this, it is more reasonable if the direction is expanded corresponding to the direction according to the direction in which the contour of the shape data in the search reference block exists. An example of this will now be described as a second embodiment.

(Second Embodiment)
An embodiment in which the search range of similar blocks is switched according to the direction of the contour of the shape data in the search reference block will be described.

  As described with reference to FIG. 7, conventionally, the search range of the similar block is determined by the relative position with respect to the search reference block, and is not controlled by the screen location, shape data, or image data.

However, for example, when the outline of the shape data crosses the inside thereof as in the search reference block B1 shown in FIG. 3, the horizontal direction is omitted, and the range Bs1 ′ moved by W pixels in the vertical direction. The extraction performance is hardly deteriorated even by searching with.
This is because in the case of the similar block Bs1, the outline of the shape data is not moved until it is replaced, and the correction effect is obtained only in this case. Therefore, in this example, the contour is moved vertically. This is because we want to be able to select similar blocks shifted in the vertical direction.

  The error when shifting to the left or right may be minimal due to the influence of the background or the fine design of the object, but even if the error is slightly larger than that, it is more correct to move the contour of the shape data vertically Get closer to.

  The contour 1a of the contour extraction target object (correct object region) 1 is, of course, unknown, and the contour extraction target object (correct object) should be moved to which of the provisionally given shape data contour (schematic shape 2 of the extraction target object). It is obvious that the approach to the contour 1a of the region 1) is not possible when searching for similar blocks, but the direction of the contour of the shape data (schematic shape 2 of the extraction target object) and the contour of the contour extraction target object (correct object region) 1 Since the direction of 1a is almost the same as empirically, it is most reasonable to move the similarity candidate block in a direction perpendicular to the direction of the outline of the shape data (schematic shape 2 of the extraction target object). .

For example, the upper left, upper right, lower left, and lower right four pixel values of the shape data block are compared. If the upper left and upper right are equal, and the lower left and lower right are equal, it is determined as a horizontal contour, and the similarity candidate block Moves the range only in the vertical direction and searches for similar blocks.
Similarly, when the upper left and the lower left are equal and the upper right and the lower right are equal, it is determined as a vertical contour, and the similarity candidate block is changed in the horizontal direction to search for a similar block.

If none of them is found, the search is performed in all directions as in the conventional method.
By doing so, the processing amount for searching for similar blocks can be reduced without degrading the extraction accuracy.

  Here, when searching for similar blocks, instead of determining the similar candidate block with the smallest error within the search range as the similar block there, when evaluating the error by sequentially changing the similar candidate block, a predetermined tolerance is used. However, it is preferable to stop the search when a similar candidate block with a smaller error is found and determine the similar candidate block as a similar block.

  By doing so, the processing amount can be further reduced without degrading the extraction accuracy.

  In some cases, it is effective to switch the search range depending on the position of the block in the screen. For example, as proposed in Japanese Patent Application No. 11-186537 “Image Contour Extraction Device”, it is assumed that the extraction target is a face image portion including the head. Prepare in advance, display this head contour image on the screen, and position and image the head to fit within the frame as the head contour image displayed on the screen, There is a technique for extracting the head from the image along the contour using the frame line as the initial state of the shape data (provisional shape data (provisional alpha map)). If the user always adjusts the position of the chin to the lower limit of the frame line, the deviation of the contour of the chin portion becomes smaller than the other portions.

  In such a case, in the portion below the frame line, the probability of erroneous extraction is lower when processing with a large block size is omitted. Alternatively, the probability of erroneous extraction becomes lower as the search range is narrowed. Omitting processing in a large block and narrowing the search range also reduce the amount of processing. In addition, when the initial shape is known as described above, the arrangement of the search reference block is also uniquely determined. Therefore, the arrangement is stored, and in step S21 for arranging the search reference block, only the reading is performed. For example, the process of detecting the contour of the shape data can be omitted.

  In addition, as shown in FIG. 5, instead of obtaining similar blocks for all of the search reference blocks B1,. The processing of the search can be largely omitted by processing the similar block of the block as being in the middle of the similar blocks of the blocks at both ends.

  There is also a method of arranging the search reference blocks so as not to overlap each other. In this way, since the total number of search reference blocks is reduced, the processing amount is naturally reduced.

  Also, instead of performing a reduction process every time a similar candidate block is set, a reduced version of the entire screen is prepared in advance, and when a similar candidate block is set, the corresponding portion is extracted from the reduced image. There is also a method for comparing errors with the image in the search reference block. This also makes it possible to reduce the overall processing amount.

  In this case, for example, when the size of the search reference block and the similarity block is 1: 2, and the similarity candidate block is set while being shifted by one pixel, the original images A, B, C, and D as shown by circles in FIG. A reduced screen having a half size in the vertical and horizontal directions is created using the pixels sampled at the A pixel position. Further, a reduced image of each image when the sampling point is shifted to the pixel positions of the B, C, and D points, that is, the phase is shifted, is created. As a result, a total of four reduced screens are created. At this time, when the positions of the similarity candidate blocks are set to even numbers for both the ordinate and abscissa, for example, when searching while shifting by two pixels within the search range, one of A, B, C, and D is selected. It is sufficient to make only one in advance, and in this case, the process of making the other three sheets can be omitted.

  If the shift by one pixel is shifted by two pixels, the number of similar candidate blocks for similarity search is reduced accordingly, and the search process is reduced. However, the method using a reduced screen can further greatly reduce the process of creating a reduced screen.

  In addition, when the initial shape data is small, it is not necessary to create the entire reduced screen. If a part that does not fall within the search range of similar blocks is known in advance, the process of creating a reduced screen of that part may be omitted.

In the above, the generation of the reduced screen when searching for similar blocks has been described. However, when the image of the similar block found is replaced with shape data (alpha map) and converted, the first reduced shape screen is also displayed. The shape data of each search reference block can be extracted and replaced from the reduced shape screen.
In this way, the reduction process for each replacement can be omitted, and the burden on the calculation process can be further reduced.

  As described above, according to the second embodiment, when searching for similar blocks, the search range is limited to the direction perpendicular to the contour direction of the shape data (alpha map) in the search reference block. Further, it is possible to rationally perform the process of bringing the contour of the shape data (alpha map) close to the contour of the contour extraction target object, and the processing amount can be greatly reduced. In addition, the search processing blocks arranged on the outline of the shape data (alpha map) are not overlapped with each other, so that the calculation processing amount can be greatly reduced. Also, instead of performing a reduction process every time a similar candidate block is set, a reduced version of the entire screen is prepared in advance, and when a similar candidate block is set, the corresponding portion is extracted from the reduced image. Thus, by comparing the error with the image in the search reference block, the overall calculation processing amount can be reduced.

  By the way, when extracting the region of the subject from the image, the user draws the approximate shape of the object by manual operation or the user looks at the screen and puts his head on the contour line displayed there. Together, the approximate shape data of the object is obtained, and then the contour of that shape data is corrected to the position of the contour of the object in the image data, which allows accurate extraction in most situations Is done. In this case, shape data representing the elliptical area in the center of the screen is generated, and the user aligns his / her head with the outline of the ellipse displayed on the screen. The head is approximated enough to extract the region of. Using the ellipse shape data as the initial shape, the contour position of the shape data is corrected to the position of the head contour in the image data.

  However, in order to obtain initial shape data, it is inconvenient for the user to fix the head at a predetermined position in the screen. In situations where the object can be extracted relatively easily, such as when the background is relatively flat or the background does not move, it is desirable that the initial shape data be automatically obtained.

  Therefore, an embodiment according to this demand will be described below.

(Third embodiment)
FIG. 10 is a flowchart illustrating an object extraction method from an image according to the third embodiment. In this embodiment, a statistical index called a degree of separation is used. The degree of separation S is expressed by the following equation (Fukui “Object contour extraction based on the degree of separation between regions”, IEICE Transactions, D-II, Vol. J80-D-II, No. 6, pp. 1406-1414, 1997). That is,
S = X / Y
X = Nb * (Ab−A) * (Ab−A) + Nf * (Af−A) * (Af−A)
Nb: Number of pixels in the background area Nf: Number of pixels in the object area A: Average value of all pixel values Ab: Average value of pixel values in the background area Af: Average value of pixel values in the object area
Y: Sum of squares of all pixel values and the square of the difference between A As is clear from the steam equation, the larger the difference between the average values of the object region and the background region, the closer to 1, and the smaller the difference between the average values Approaching zero. Usually, since the statistical properties such as the average value of the pixel values are different between the object and the background, it can be estimated that the greater the degree of separation, the more correct the object region.

  Therefore, as shown in FIG. 11 in advance, the shape data candidate areas are prepared as 10 pieces of shapes 1 to 10, the degree of separation is obtained for each of these, and the shape data having the largest value is searched, In the present embodiment, this is used as initial shape data. This process will be described with reference to the flowchart of FIG.

  First, zero is substituted into a variable M that holds the maximum value of the degree of separation (S31). Initial shape data candidates are sequentially set in numerical order (S32). A separation degree S in the current input image data when using the shape data is obtained (S33). If S is larger than M, the process proceeds to step S35, and if not, the process proceeds to step S36.

  In step S35, the shape data for which the degree of separation has been obtained is stored as initial shape data. If there is any data previously stored as initial shape data, the previous data is discarded. Further, it is assumed that M = S.

  It is determined whether the processing for all initial shape data candidates has been completed (S36). If the process is finished, the process proceeds to step S17, and if not, the process returns to step S32. In step S37, the contour line of the shape data is matched with the contour line of the object in the image data using the initial shape data and the current input image data.

  If it is known that the movement of the object is relatively gentle, the initial shape data is not always searched on the entire screen, but for example, the initial shape data candidate selected in each frame is stored, and the previous If the search is performed only near the initial shape data selected in this frame, the amount of processing can be reduced. For example, if the shape 11 in FIG. 12 is selected in the previous frame, as shown in the figure, if the search is performed with four of the top, bottom, left, and right and five of them, the search processing amount is larger than the case of searching with ten. Can be cut in half. Alternatively, if 10 shape data candidates are set in the vicinity of the shape 11 while being slightly shifted from each other, initial shape data with higher accuracy can be obtained with the same processing amount. In this way, when the initial shape data search range is set around the shape of the previous frame to extract a person's head, the search range is made wider in the left-right direction than in the vertical direction of the screen. This is because the head is more likely to swing left and right than to move up and down.

  Further, the area for obtaining the degree of separation does not have to be exactly the same as the shape data. If the rectangle can save more processing than the ellipse, the degree of separation is obtained using the rectangular area as shown in FIG. 13A, and the corner of the rectangle where the degree of separation is maximized is selected. There is also a method in which an octagon as shown in FIG. 13B is used as initial shape data. The reason for putting the corner is to bring it closer to the shape of the person's head.

  Further, the head of a person is roughly divided into two regions: the upper hair and the face below the center. Therefore, as shown in FIG. 13C, the rectangle is further divided into two regions, and the degree of separation S is obtained by the following equation.

S = X / Y
X = Nb * (Ab-A) * (Ab-A) + Nf1 * (Af1-A) * (Af1-A) + Nf2 * (Af2-A) * (Af2-A)
Nb: Number of pixels in the background area Ni1: Number of pixels in the first object area Nf2: Number of pixels in the second object area A: Average value of all pixel values Ab: Average value of pixel values in the background area Af1: First Average value of pixel values of object region Af2: Average value of pixel values of second object region Y: Sum of squares of difference between each pixel value and A This makes it possible to detect a human head more accurately. There is.

  The contour position correction process in step S37 is performed according to the flowchart of FIG. That is, first, a block is set in the contour portion of the shape data (S41). Next, a block similar block is obtained using the image data (S42). It is determined whether similar blocks have been obtained for all the blocks set in step 41 (S43). If completed, the process proceeds to step S44. Otherwise, the process returns to step S42 to obtain a similar block of another block.

  In step S44, the shape data of the block is replaced with the reduced shape data of the similar block. It is determined whether or not replacement has been completed for all blocks (S45), and if completed, the process proceeds to step S46. Otherwise, the process returns to step S44 to replace another block. If it is determined in step S46 that the number of replacements reaches a predetermined number, the process ends. Otherwise, the process returns to step S44.

  If there is an object that matches any of the initial shape data candidates in the background, the degree of separation in this case may be higher than the degree of separation in the head. However, when the background is relatively flat, this embodiment can provide a good initial shape regardless of where the head is on the screen.

(Fourth embodiment)
Next, a fourth embodiment will be described with reference to FIGS. 15 and 16. In this embodiment, a method of generating initial shape data by acquiring a background image before starting extraction, and then determining an area where the difference between the input image and the background image is a certain value or more as an object area It is.

  First, an image (FIG. 16A) in which an object to be extracted is not within the shooting range is acquired as a background image while being confirmed by the user (S51). Next, when an object (person) enters the screen as shown in FIG. 16B, at this time, the difference between the current input frame and the background image is obtained, and the pixel whose absolute value is equal to or greater than a certain value is determined as the object pixel. Then, initial shape data (FIG. 16 (c))) is generated using the other pixels as background pixels (S52).

  However, if this processing is simply performed in units of pixels, the region is often divided into granular shapes. Therefore, the background image is divided into blocks, and a block average value of pixel values is obtained for each block. Similarly, an average block value is obtained for the input image, and initial shape data is generated using a block whose absolute value of the difference between the average block values of the input image and the background image is larger than a predetermined value as an object region and a block other than that as a background region. In such a block unit, only shape data having a step in the contour can be obtained, but there is no problem because the contour correction is performed in step S53. Further, the sum of the absolute values of the differences obtained in units of pixels described above in the block is obtained, and when the sum in the block is larger than a predetermined value, the block is also added to the object region. Accordingly, it is possible to avoid erroneous detection that an object region whose design is different from the background is close to the block average value of the background due to the block average value, and is not determined as an object.

  In step S53, using the initial shape data and the current input image data, the contour line of the shape data is matched with the contour line of the object in the image data, and the process proceeds to the next frame (S54). When the background is stationary, according to the present embodiment, an object of any shape or a plurality of objects can be extracted with high accuracy wherever they are in the screen.

(Fifth embodiment)
Next, a fifth embodiment for generating initial shape data by motion compensation with a reference frame will be described with reference to FIG.

  First, the outline of the head is extracted while the user matches the head to the screen display by a method using fixed initial shape data. In this contour extraction, the user performs an image acquisition operation while confirming that the contour is correctly extracted. The image and final shape data at that time are set as reference image data and reference shape data, respectively (S61). Next, motion detection from the reference image data to the image data of the current input frame as shown in FIG. 18 is performed. Using the motion detection result, motion compensation is performed on the reference shape data to generate initial shape data of the current input frame (S62). For example, the image data of the object area of the reference frame is cut out, and matching is performed while moving the image data in the current input frame. The initial shape data is generated by translating the object region of the reference shape data to the portion where the matching error is minimized. Thereafter, using the initial shape data and the current human power image data, the contour line of the shape data is matched with the contour line of the object in the image data, that is, the contour position is corrected (S63). Next, the process proceeds to the next frame (S64).

  According to the present embodiment, the user must first adjust the position of the head while checking the screen, but after acquiring the reference image, the same surface of the object faces the camera by motion compensation. As long as it is, it will be extracted anywhere in the screen. Motion detection is not limited to two parameters of translation, but the amount of processing increases as the number of parameters increases, such as three parameters including rotation, four parameters including enlargement / reduction, and six parameters of affine transformation. improves.

  If it is known in advance that the object does not move much from its initial position, the reference frame remains the first and does not need to be changed. However, if the object moves gradually from the initial position, the reference frame is updated as appropriate, for example, updated every frame, and if the motion compensation is always performed with the immediately preceding frame, the object takes time. Even when moving greatly after that, it can be extracted correctly.

  Next, an embodiment in which a plurality of objects are extracted while being distinguished will be described. Assuming that the object A and the object B are extracted separately in a certain frame as shown in FIG. 19, it is assumed that tracking and extraction are performed while distinguishing these in subsequent frames. In this case, it is a method that can be realized easily by extracting objects according to the third embodiment and the fourth embodiment, and determining that objects having a large number of overlapping pixels between frames are the same object. However, in these methods, what was in the state of FIG. 19A changes to 19 (b), (c), and (d), and when the objects partially overlap in the middle, FIG. ) And (c), the object regions are merged. As a result, in FIG. 19D, it is not possible to know which is the object A and which is the object B. In order to deal with such a case, the fifth embodiment is used for each of two objects, and those subjected to motion compensation are distinguished from each other as the same object.

  When the initial shape data is obtained by motion compensation as shown in FIG. 19B, the contour position is corrected by using the self-similar mapping method described in Japanese Patent Application Nos. 11-001891 and 11-301415. An example of performing the above will be described with reference to FIGS. 20 and 21. FIG. Here, in the shape data, the pixel value 1 is substituted for the pixel set as the area of the object A, the pixel value 2 is substituted for the pixel of the object B, and the pixel value is set for the pixel that is neither the object A nor the object B. Assume that 0 is assigned. The contour of such shape data, that is, the boundary between 0 and 1, the boundary between 0 and 2, and the boundary between 1 and 2 are the contours of the object in the image data, that is, the boundary between the background and the object A, and the boundary between the background and the object B. The purpose is to correct the boundary between the object A and the object B, respectively. In the prior application, the shape data (same as alpha map, alpha data, and alpha mask) was a binary image, but here it is noted that it is a ternary image of {0, 1, 2}.

  First, as shown in FIG. 21, blocks are arranged along the contour of the shape data. For this purpose, the shape data is raster-scanned, and blocks are sequentially set around pixels that have different pixel values from those of adjacent pixels and are not included in the previously set blocks. In this method, the blocks are arranged in a daisy chain while overlapping each other. Alternatively, as shown in Japanese Patent Application No. 11-001891, there is a method of arranging so that the blocks do not overlap. Next, a similar block is searched for each block using image data.

  Finally, the block shape data is replaced with a reduced version of the similar block shape data. For example, as shown in FIG. 20, when the block is 2 × 2 pixels and the similar block is 4 × 4 pixels, the sampling point for the pixel 12 in the similar block is point 13. Therefore, the pixel values of the four pixels around the point 13 are checked, and the pixel value (either 0, 1 or 2) with the largest number is used as a sampling value, and the pixel 12 is replaced with this value. The pixel 14 is similarly replaced by using the tetrahedron around the point 15. By repeating the replacement of the pixel values of all blocks in this manner a plurality of times, the contour of the shape data approaches the contour of the object in the image data and converges in a state consistent with it.

  According to this method, an object can be tracked while being distinguished, and extraction along a contour can be performed. When the number of objects is three or more, the type of pixel value, that is, the label is increased accordingly. Even if the number of labels increases, the sampling value is determined by majority vote.

  Further, in a portion where the blocks overlap, a plurality of sampling points correspond to one pixel. In that case, the last replacement value in the replacement for each block described above is valid. Alternatively, the value to be replaced is determined by a search around a plurality of sampling points, for example, a majority decision using a total of 12 pixels around three sampling points.

  The contour extraction correction method using three or more labels used in this embodiment can also be used for image segmentation. At this time, it is assumed that the arrangement of the blocks is obtained by dividing the entire screen as shown in FIG. Next, obtaining similar blocks for each block using this image data is the same as step S42 in FIG. As the initial shape data, for example, data in which a label is randomly arranged on each pixel is prepared. The number of labels is determined in advance. Alternatively, the screen is divided into square blocks and a label is attached to each block. This block division may be the same as or different from that in FIG. Alternatively, the pixel value of the image data is quantized and a label is assigned to each quantum level.

  When the replacement from the similar block to the block is repeated with respect to the initial shape data thus created, the initial shape data converges to certain shape data. The segmentation image is obtained as shown in FIG. 22B as an image color-coded for each label of the converged shape data. In setting the reference frame in the fifth embodiment, this segmentation image may be displayed, and the object area of the reference frame may be set by the user selecting areas belonging to the object one by one.

(Sixth embodiment)
Next, a surface image transmission system using the object extraction method of the present invention will be described as a sixth embodiment with reference to FIG. 23, FIG. 24 and FIG.

  In FIG. 23, for example, an image transmission / reception terminal X, which is a mobile phone, a personal computer, or a game machine, for example, includes a camera 16 attached to the terminal, an image segmenter 18, an encoder 20, a decoder 36, and a display 38. Yes. The other image transmitting / receiving terminals Y and Z are configured in the same manner as the terminal X.

  The image distribution center A connected to these image transmission / reception terminals X, Y, and Z includes decoders 24, 25, and 26 connected to the terminals X, Y, and Z, a synthesizer 30 connected to these decoders, and a background. A memory 31 and an encoder 34 are provided.

  In such a configuration, an image of the user of the terminal X is taken by the camera 16 attached to the terminal, and the image data 17 is sent to the clipping unit 18. In the clipper 18, the face image of the user is cut out by a method such as the third, fourth, and fifth embodiments, and the face image data 19 is sent to the encoder 20. The face image data is a combination of the same image data 17 and shape data (alpha map, alpha data, alpha mask) representing the face area. In the encoder 20, for example, face image data is object-encoded by the MPEG-4 system, which is an international standard for moving image encoding, and the compressed data 21 is sent to the image distribution center A via a communication line.

  Similarly, the face images of the respective users are compressed from the terminals Y and Z located away from the terminal X, and sent to the center A as compressed data 22 and 23, respectively. In the center A, the received compressed data 21, 22, and 23 are reproduced as face image data 27, 28, and 29 by the decoders 24, 25, and 26, and sent to the image synthesizer 30.

  A background image 32 is also input from the background memory 31 to the synthesizer 30. In the synthesizer 30, the face image data 27, 28, and 29 are synthesized with the background image 32 as shown in FIG. 24, and this synthesized image 33 is sent to the encoder 34. In the encoder 34, the composite image 33 is compressed as a normal rectangular image by the MPEG-4 method or the like, and the compressed data 35 is sent to each of the terminals X, Y, and Z.

  In the terminal X, the received compressed data 35 is sent to the decoder 36. The decoder 36 reproduces the synthesized image, and the synthesized image 37 is sent to the display unit 38 attached to the terminal, and the same image as that shown in FIG. 24 is displayed. The compressed data 35 is also sent to the terminals Y and Z, and similar composite images are displayed on the terminals Y and Z.

  If voice is also transmitted separately, this system can realize a fun real-time chat system while sharing cyber space while looking at each other's faces.

  The procedure in the terminal of this system and the image distribution center is shown in FIG. In FIG. 25, the left flow shows the procedure at the terminal, and the right flow shows the procedure at the image distribution center.

  In this procedure, first, an image is captured (S71). Next, a face image is cut out (S72). Thereafter, the face image data is compressed, and the compressed data is transmitted to the center (S73).

  The center receives the compressed data from the terminal and reproduces the face image data (S74). The reproduced face image data is combined to create a combined image (S75). The composite image is compressed, and the compressed data is transmitted to the terminal (S76).

  The terminal receives the compressed data from the center and reproduces it into a composite image (S77). This reproduced composite image is displayed on the display of the terminal (S78). After this, the flow returns to step S71.

  In S74 at the image distribution center, a plurality of face images are synthesized by receiving a plurality of compressed data.

(Seventh embodiment)
Another image transmission system according to the seventh embodiment that realizes the same function without going through the area image distribution center will be described with reference to FIG. In this system, since the face image data 19 is generated in the terminal X and the compressed data 21 is generated via the encoder 20, the description is omitted because it is the same as the previous system.

  The compressed data 21 is sent to terminals Y and Z via communication lines. Similarly, the compressed data 22 and 23 are sent from the terminals Y and Z and sent to the terminal X. Also, the compressed data is transmitted between the terminals Y and Z. The compressed data 22 and 23 received by the terminal X are decoded by the decoders 39 and 40, respectively, and the face image data 41 and 42 are sent to the synthesizer 43. The face image data 19 of the user of the terminal X is also input to the synthesizer 43, and these are combined with the background surface image 45 sent from the background memory 44. The composite image 46 is sent to the display unit 47 and displayed. The terminals Y and Z also transmit and receive data in the same manner as described above, and the composite image is displayed on each display unit.

  This system requires more processing at each terminal, and it is necessary to communicate with multiple partners at the same time, but there is no need for an image distribution center, so that each terminal likes the arrangement of background images and face images. There are benefits that can be determined.

  The procedure at the terminal of the system of FIG. 26 is shown in FIG. According to this, first, an image is captured (S81). Next, an object is extracted. That is, the face image is cut out (S82). The clipped image, for example, own face image data is compressed, and the compressed data is transmitted to another terminal (S83). At the same time, the compressed data is received from another terminal and reproduced as the other party's face image data. The own face image data and the other party's face image data are combined to create a combined image (S84). When this composite image is displayed (S85), the flow returns to step S81.

  Although the procedure at another terminal is shown on the right side of FIG. 27, this procedure is the same as the procedure on the left side.

  Also, when generating a composite image, it is automatically determined whether or not the face image is correctly obtained, that is, whether or not the face has been correctly extracted, and if a face image is obtained, it is combined with the background for extraction. If the face image is not obtained due to the failure, if it is not combined, unnecessary things other than the face will not be mistakenly combined. In this determination, for example, the number of skin color pixels in the object region is counted, and if the ratio of the number of skin color pixels to the entire object region is equal to or greater than a predetermined value, the face image is determined. Here, for example, in the case of an image in which the color of each pixel is three colors Y, U, and V and each is represented by a value of 0 to 255, U = 110 and V = 160 are used as skin color standards. A pixel in which the absolute value of the difference between U and 110 of the image data is smaller than a predetermined value and the absolute value of the difference between V and 160 is smaller than the predetermined value is determined as skin color.

  One method is to determine whether or not the image is a face image by the synthesizers 30 and 43. However, if the determination is performed by the encoder and the image is not a face image, the amount of communication can be reduced.

  The display units 38 and 47 may use a television other than the terminal, for example. At this time, the transmission of the image to the television can be made without the hassle of wiring by using wireless communication such as Bluetooth. In addition, the background images 32 and 45 may be used by receiving broadcast video from a television or the like instead of being read from the background memories 31 and 44. Then, people in remote locations can enjoy the same TV program together while watching each other.

  In the delivery center A, a face image such as a talent is prepared in advance as an image to be combined, and the face of the talent is combined with the user's face and transmitted in response to a request from the terminal user. Is also possible. Similarly, a background of the character is prepared for the background image, and the background is selected according to the user's desire.

  The present invention is not limited to the above-described embodiment, and can be implemented with various modifications. Further, the method described in the embodiment is a program that can be read and executed by a computer, such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a semiconductor memory, etc. It can be stored in a medium and distributed.

The flowchart for demonstrating the image outline extraction method according to the 1st Embodiment of this invention. The flowchart which shows the detailed processing content of the outline position correction process which is a process of step S13 in FIG. 1, and step S17. The figure for demonstrating the example of the search range of the similar block in the 2nd Embodiment of this invention. The figure for demonstrating the sampling position of the reduction | decrease screen in the 2nd Embodiment of this invention. An arrangement example of search reference blocks on shape data (alpha map) used in contour position correction processing by a method using a self-similar map. The example of the search reference | standard block used by the outline position correction process by the method using a self-similar map and a similar block. The example of the search range of the conventional similar block in the outline position correction process by the method using a self-similar map. The figure which shows the correction | amendment result of the shape data (alpha map) in the outline position correction process by the method using a self-similar map. The flowchart which shows the content of the conventional outline position correction process by the method using a self-similar map. The flowchart for demonstrating the image outline extraction method according to the 3rd Embodiment of this invention. The figure which shows the some initial shape data candidate produced | generated by 3rd Embodiment. The figure which shows another initial shape data candidate. (A) The area | region which calculates | requires a degree of separation, (b) Initial shape data, (c) The figure which shows the area | region which calculates | requires a degree of separation. The flowchart explaining an outline position correction process. The flowchart for demonstrating the image outline extraction method according to the 4th Embodiment of this invention. The figure for demonstrating the image outline extraction method of 4th Embodiment. The flowchart for demonstrating the image outline extraction method according to the 5th Embodiment of this invention. The screen figure for demonstrating the image outline extraction method of 5th Embodiment. The figure for demonstrating a mode that an object cross | intersects. The figure for demonstrating sampling of a pixel value. The figure explaining arrangement | positioning of the block which identifies a some object. Figure showing a block-divided screen and segmentation image The block diagram of the image transmission system according to 6th Embodiment. The figure which shows the synthesized image formed in 6th Embodiment. 10 is a flowchart showing a procedure in the transmission system of the sixth embodiment. The block diagram of the image transmission system of 7th Embodiment. The flowchart which shows the procedure in the transmission system of 7th Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Contour extraction target object (correct object area), 1a ... Contour of contour extraction target object (contour of correct object area 1), 2 ... Shape data outline (general shape of extraction target object), 16 ... Camera attached to terminal , 18 ... Image segmenter, 20, 34 ... Encoder, 24, 25, 26, 36 ... Decoder, 30 ... Synthesizer, 31 ... Background memory, 38 ... Display, B1, ... Bn ... Search reference block, Bs1 ˜Bsn: similarity block of search reference blocks B1 to Bn, Fs1: search area of appropriate size around search reference block B1, Bc: similarity candidate block, Bcd: result of error evaluation with search reference block B1 , A similarity candidate block determined as a similarity block with a minimum error, X, Y, Z...

Claims (5)

An object extraction method from an image using an image contour extraction device,
The first processing means of the image contour extracting device compares each part of the image data in which the extraction target object is imaged with another part of the same image data or another image data, thereby extracting the extraction target in the image data. A first step of generating shape data that is an image representing a provisional region of the object;
A second step in which the second processing means of the image contour extracting apparatus matches the contour of the shape data with the contour of the extraction target object using the image data and the shape data provisionally generated;
A method for extracting an object from an image, comprising:   In the first step, when a plurality of candidate regions are determined in advance and these candidate regions are selectively set in the image data, the statistical properties of the pixel values inside and outside the candidate region are the most. The method for extracting an object from an image according to claim 1, further comprising: setting candidate areas that are separated from each other as a provisional area of the object to be extracted.   In the first step, the current input image data is compared with the background image data in which the extraction target object that is acquired in advance is not captured, and the areas where the values of the current input image data and the background image data are different are extracted. The method for extracting an object from an image according to claim 1, further comprising setting as a provisional area of the target object. An object extraction method from an image using an image contour extraction device,
The first processing means of the image contour extraction apparatus receives image data obtained by capturing an extraction target object and shape data that is an image representing a provisional region of the extraction target object in the image data, and the shape data A first step of setting a plurality of search reference blocks of a predetermined size with their respective positions being shifted from each other in the contour portion,
The second processing means of the image contour extracting apparatus uses the same image for each search reference block in which the blocks of the image data in the block are similar and have a block size larger than the search reference block. A second step of searching from within,
The third processing means of the image contour extracting device replaces the shape data in each search reference block of the shape data with the corrected shape data for which size correction has been performed by the reduction processing obtained from the similar block. And a third step of correcting the shape data,
The shape data has different pixel values in different object regions and background regions, and in the reduction process in the third step, the pixel value of any pixel around the sampling point of the shape data is obtained. A method for extracting an object from an image, characterized by using a sampling value. A transmission system comprising a plurality of communication terminals that transmit and receive data to and from each other,
The communication terminal is
Object extraction means for extracting an object from image data obtained by capturing an extraction target object and obtaining extracted image data;
Transmitting means that directly or compresses the extracted image data and sends it to another communication terminal as communication terminal transmission / reception data;
The inter-communication terminal transmission / reception data sent from another communication terminal is received, and when the transmission / reception data is not compressed, it is used as extracted image data as it is, and when it is compressed, the extracted image data is reproduced. Receiving means;
Synthesizing means for synthesizing the extracted image data into one synthesized image data;
And a means for displaying the composite image data.

Images