JP2005191964A - Image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method in which a plurality of participants can communicate sharing one composite image. <P>SOLUTION: The image processing method includes a step for inputting self-portraits taken by cameras 14 of self terminals, an external terminal image inputting step for inputting each of the external terminal images taken by a plurality of external terminals via a network, an image inverting step for performing mirror transposition of each of images input from the self terminal belonging to a first terminal group containing the self terminal and from the other terminals, and a combining step for combining each of the first terminal group's images that have undergone the mirror transposition and the external terminal images of a second terminal group not belonging to the first terminal group to produce a composite image. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing method for allowing a plurality of participants to communicate while sharing one composite image, for example.

  In order to support communication between users in remote locations, systems such as video conferencing and video chat have been proposed, but since these systems cannot feel a sense of familiarity and unity, they will be widely used. Yes, not. Therefore, with the aim of a system where a sense of familiarity and sense of unity can be felt, a system has been studied in which participants' images are combined on one screen and communication is performed while sharing the combined image.

  In Non-Patent Document 1, a camera (S1206, S1208), a mirror image inverting device (S1204, S1207), a chroma key device (S1202, S1203), and a screen (S1201, S1205) are connected as shown in FIG. Shooting with multiple cameras, sending and receiving shot images via the network, the background part of the foreground shot video is chroma key method (preparing a monochrome background such as blue or green in the camera shooting range in advance Sometimes, a portion different from the background color from the photographed image is regarded as a subject and is removed by an image processing method (compositing on another image) and synthesized as one image, and the resulting synthesized image is combined with a large screen screen or projector device. Use to present to each participant in a size close to life size. At this time, the number of cameras is limited to two, as shown in the flowchart of FIG. 18, mirror image reversal is performed for the captured video (S1202), and the other captured video is synthesized without performing mirror image reversal (S1203). ) By presenting the composite image (S1204), it is possible to construct a communication system that feels natural and unity like a mirror. In constructing this system, it is necessary to prepare a special environment with a monochrome background for the chroma key method.

  On the other hand, in Non-Patent Document 2, instead of the chroma key method, the background difference method (photographing range of the camera is captured and stored in advance, and a portion different from the stored background from the captured image is regarded as a subject during operation, and is displayed on another image. The image processing method) to be combined is used to cut out and combine the subject and to construct a system for sharing the combined image. If the background subtraction method is used, it is not necessary to prepare a single-color background, so that the same communication can be performed without preparing a special environment.

  However, it is difficult to accurately extract the subject only with the background difference. For this reason, in order to accurately extract the contour of the subject, accurate contour extraction is performed by the fractal contour extraction method of Non-Patent Document 3 using the shape of the subject extracted by the background difference.

  This fractal contour extraction method will be described based on the flowchart of FIG.

  First, information to be input is image data and rough shape data which is shape data that is not accurate but close to the contour. Note that the output information is more accurate shape data.

S1309: The number M of mappings that can be determined from the block size and can be accurately extracted is set. Note that M is a value such that a value obtained by multiplying the number X of vertical and horizontal pixels by the M power of the block magnification A is 1 or less. That is, it is a value satisfying X × A ^M = <1.

  S1302: While referring to the schematic shape data, a plurality of processing blocks (for example, squares of 32 pixels each in length and width) are arranged on the outline.

  S1303: With reference to the image data, each processing block has a size obtained by multiplying the processing block by A (for example, A = 2.0), and the error (for example, the sum of error absolute values for each pixel) is the largest. Search for parent blocks that are small blocks. This search is a full search in which the search step is one pixel, or a search in which a search is first performed in a large search step and a search is performed in a small search step only in the vicinity of the solution several times until the search step becomes 1. In the search.

  S1304: Next, in each block, the shape data is mapped from the parent block to the processing block.

  S1305: The process of S1304 is repeated until M mappings are completed.

  S1306 to S1307: If the size of the processing block is sufficiently small (for example, a square of 4 pixels each in vertical and horizontal directions), shape data is output and the image processing is terminated. Otherwise, the processing block is changed to a smaller one (for example, a block having half the length and width), and the process returns to S1302.

  As shown in FIG. 17, the fractal contour extraction method has such characteristics that the larger the processing block size, the larger the applicable range, the more calculation amount, and the lower the accuracy of the obtained contour. Therefore, in the processing of S1306 to S1307, in order to increase the applicable range and increase the accuracy of the obtained contour, the contour extraction processing is first performed with a large processing block size, and then the previous contour extraction processing is performed. Using the result as the next input, the contour extraction process is repeated while gradually reducing the applicable range. For this reason, if the applicable range is increased, the amount of calculation increases.

By performing such image processing, shape data closer to the true shape than before processing can be obtained.
Osamu Morikawa, “Supermirror: Toward an Attractive Video Dialogue”, Transactions of Information Processing Society of Japan, Vol.41-3, pp.815-822, 2000. Hidenori Takeshima, Takashi Ida, Osamu Hori, “Development of a chat system for sharing participants' cut-out images in real time”, Proceedings of the 2002 IEICE General Conference, SD-3-10, pp.391- 392, Waseda University, Mar. 2002. T. Ida and Y. Sambonsugi, `` Self-affine mapping system and its application to object contour extraction, '' IEEE Trans. Image Processing., Vol.9, no.11, pp.1926-1936, Nov. 2000.

  Using the technology described in the background art, if you limit the number of cameras to two and perform mirror image reversal of the self-portrait and cut out and combine the subject using the background subtraction method, a special environment of a single color background is prepared. Even without it, it is possible to build a communication system that feels as if you are looking at a mirror and feel a sense of unity.

  However, there is a problem that this system cannot communicate at three or more points.

  Therefore, the present invention provides an image processing method in which three or more participants can communicate while sharing one composite image.

  Further, the fractal contour extraction method of Non-Patent Document 3 described above is an image processing that requires calculation cost. For example, when the system is configured by the method of Non-Patent Document 2, the delay time in LAN connection is about 0.5 seconds. It becomes.

  Therefore, the present invention also provides an image processing method capable of high-speed processing when post-processing is performed using a fractal contour extraction method.

  The invention according to claim 1 is a system that includes three or more terminals connected via a network, and that inputs and outputs images captured by each imaging means of the plurality of terminals via the network. In the image processing method in a constituting terminal, the plurality of terminals are divided into a first terminal group to which the own terminal belongs and a second terminal group to which the own terminal does not belong, and the own terminal belonging to the first terminal group is divided. An image inversion step for mirror-inverting all images input from the terminal and one or a plurality of other terminals, each image of the first terminal group that has been mirror-inverted, and the second terminal group And a synthesis step of generating a synthesized image by superimposing and synthesizing images input from one or a plurality of other terminals to which it belongs. That is an image processing method.

  According to a second aspect of the present invention, there is provided a self-image input step of inputting a self-portrait photographed by the photographing means of the self-terminal, and an external terminal image input step of inputting each external terminal image photographed at a plurality of external terminals via a network An image inversion step of mirror-inverting each image input from the own terminal and other external terminals belonging to the first terminal group including the own terminal, and each of the first terminal group that has been mirror-inverted An image processing method comprising: a combining step of combining an image and an external terminal image of a second terminal group that does not belong to the first terminal group, and generating a combined image. .

  The invention according to claim 3 is the image processing method according to claim 2, wherein all of the own terminal and the external terminal belong to the first terminal group.

  According to a fourth aspect of the present invention, there is provided a system that includes a plurality of terminals connected via a network, and that inputs and outputs images captured by each imaging unit of the plurality of terminals via the network. In the image processing method in the terminal, an image inversion step for mirror-inverting at least one of the images, and an image in which only the inversion processing exclusion area specified in the mirror-inverted image is not mirror-inverted An image replacement step for replacing the image with the image, and a combining step for generating a composite image by superimposing and synthesizing one image obtained by replacing only the inversion processing exclusion area with an image that has not been mirror-image inverted and another image. An image processing method characterized by this.

  The invention according to claim 5 is constituted by a plurality of terminals connected via a network, and constitutes a system for inputting and outputting images taken by the respective photographing means of the plurality of terminals to each other via the network. In the image processing method in the terminal, a first subject step for calculating a subject that is a subject region of one image in each image, and a second subject for calculating a subject that is a subject region of an image other than the one image And a synthesis step of superimposing and synthesizing the subject of the one image and the subject of an image other than the one image on the one image in accordance with a predetermined superposition sequence, and obtaining a composite image. An image processing method characterized by:

  In the invention according to claim 6, the superposition order in the synthesis step is determined based on the comparison result by calculating the size of the subject from the subjects of the images and comparing the calculated sizes of the subjects. 6. The image processing method according to claim 5, wherein:

  The invention according to claim 7 is composed of one or a plurality of external terminals connected to the own terminal via a network, and images captured by each imaging means of the plurality of terminals are mutually connected via the network. In the image processing method of the own terminal in the input / output system, an external terminal image input by an external terminal image captured by one external terminal among the external terminals is input, and the external terminal image is captured by the one external terminal. A reference image input step for inputting a reference image that is an image and an image to which information distinguishable from the external terminal image is added; and a storage step for storing the input reference image as a storage reference image; When the external terminal image is input, a different part of the stored reference image and the external terminal image is detected, and this different A subject area calculating step for calculating the minutes as a subject area of the external terminal image, an external terminal image taken by an external terminal other than the one external terminal, or a self-image taken by the own terminal, and the calculated subject An image processing method comprising: a combining step of superimposing regions and combining to obtain a combined image.

  According to an eighth aspect of the present invention, there is provided a first step of inputting N-dimensional image data composed of a set of N-dimensional (where N is a natural number of 2 or more) unit pixel data; A second step of inputting N-dimensional shape data indicating a rough shape of the predetermined shape, and an N-dimensional processing block having a predetermined size and shape at the boundary portion of the N-dimensional shape data A third step, search processing for searching an N-dimensional parent block most similar to the N-dimensional processing block in the N-dimensional image data based on a predetermined conversion method, and the search in the N-dimensional shape data The fourth step of repeating the replacement process for replacing the content of the N-dimensional processing block with the content of the N-dimensional parent block is repeated M times, and obtained by repeating the M times. A fifth step of outputting the N-dimensional shape data as the data of the predetermined shape, wherein in the fourth step, the size of the N-dimensional processing block of the (k + 1) th (where M> k ≧ 1). The image processing method is characterized in that the search is performed more coarsely as the size of the (k + 1) th N-dimensional processing block is larger in the search process. is there.

  The invention according to claim 9 is a self-image input means for inputting a self-portrait photographed by a photographing means of the self-terminal, and an external terminal image input means for inputting each external terminal image photographed at a plurality of external terminals via a network. And image reversing means for mirror-inverting each image inputted from the own terminal belonging to the first terminal group including the own terminal and another external terminal, and each of the first terminal group having the mirror-inverted image An image processing apparatus comprising: a combining unit configured to superimpose and combine an image and an external terminal image of a second terminal group not belonging to the first terminal group to generate a combined image .

  The invention according to claim 10 comprises group storage means for storing in advance the external terminals belonging to the first terminal group and the external terminals belonging to the second terminal group. 9. The image processing apparatus according to 9.

  In the inventions according to claims 1, 2, 3, and 9, it is possible to realize natural image sharing for the user even at three or more points.

  In the invention according to claim 4, mirror image reversal of characters in the reversal exclusion region can be avoided.

  In the invention according to claim 8, since the fractal contour extraction method which is post-processing can be performed at high speed, the delay time until the composite image is displayed can be reduced, and smooth communication between the users is enabled.

(Embodiment 1)
The image processing method according to the first embodiment of the present invention will be described below. This embodiment relates to the connection of three or more terminals 10 (for example, four terminals 10), and will be described with reference to FIGS.

  FIG. 1 is a flowchart for realizing the image processing method of the present embodiment. FIG. 2 is a flow of image data when image input, subject extraction, mirror image inversion, and image composition are performed. FIG. FIG. 4 is an example of the internal configuration of the terminal 10 in the system. FIG. 5 is an explanatory diagram showing a mirror image inversion state.

(1) System Overview The system shown in FIG. 3 is a kind of multipoint video conference. As shown in FIG. 4, each of the plurality of terminals 10 connected to the network first sequentially captures images from the camera, that is, captures and captures images at predetermined time intervals, and simultaneously connects to other terminals connected via the network. The camera image taken by the terminal 10 (external terminal 10) is captured.

  Next, the plurality of captured images are superimposed and combined and displayed on a screen (for example, a terminal screen, a television screen, or a projector).

  Here, in order to make the displayed self-portrait a natural image, it is necessary to perform mirror image inversion processing. This is because it is difficult to use because it appears to move in the opposite direction on the screen displayed when the user moves left and right. In Non-Patent Document 1, it is possible to synthesize a self-portrait between two points after mirror-inversion, but mirror-image inversion processing cannot be performed in communication at three or more points.

  Therefore, in this embodiment, the four terminals 10 belong to one of the two terminal groups, and in the image processing of each terminal 10, as shown in FIG. Hereinafter, an image of “own terminal group”) is a mirror image inversion target, and an image of a terminal group to which the terminal 10 does not belong (hereinafter referred to as “partner group”) is not a mirror image inversion target. The self-portrait is displayed with the mirror image reversed at the terminal 10 in the self-terminal group, and is displayed without being mirror-inverted at the terminal 10 in the partner group.

(2) Configuration of Terminal 10 The configuration of the terminal 10 will be described with reference to FIGS.

  The terminal 10 is connected to a monitor 12 and a camera 14 and is connected to other three terminals via a network.

  The terminal 10 includes an image composition unit 16, three cutout units 18, 18, 18, two mirror image inversion units 20, 20, and a capture unit 22. In FIG. 4, the direction of the arrow indicates the flow of image data.

  The image captured by the camera 14 (that is, the self-portrait) is input to the mirror image inversion unit 20 via the capture unit 22 and the left and right sides of the image are inverted, and the mirror image inverted image is input to the clipping unit 18 and only the subject from the background image Is cut out. This cutting method uses the technique described in the background art. An image obtained by cutting out only this subject is input to the image composition unit 16. In addition, images captured via the network are transmitted to the other three external terminals 10.

  The first external terminal 10 belongs to its own terminal group, and an image transmitted from the first external terminal 10 is an image obtained by mirror-inverting the image through the mirror image reversing unit 20 and the clipping unit 18 and cutting out only the subject. Is input to the image composition unit 16.

  The second external terminal 10 belongs to the partner group, and the image transmitted from the second external terminal 10 is an image in which only the subject is cut out by the cutout unit 18 without performing mirror image reversal. Is input.

  The third external terminal 10 belongs to the partner group, and the image transmitted from the third external terminal 10 is not mirror-inverted and is not cut out from the background image. Input to the image composition unit 16.

  The image composition unit 16 subjects the subject terminal 10 of the subject terminal group and the first external terminal to the cropped subject that is mirror-inverted, the subject that is the second external terminal of the counterpart group, and the subject that is not subject to mirror-inverted, and the third external terminal. The image including the background image is synthesized and displayed on the monitor 12.

  This grouping is set in advance or is set when the system is operated.

(2) Processing Method A flowchart of the image processing method of this embodiment is shown in FIG.

  Each terminal 10 is managed by attaching a terminal number k. The processing of S102 to S107, which will be described below, is performed on each image of terminal number k from 1 to the number of terminals (in the above case, the number of terminals is 4), and image synthesis is performed using the obtained plurality of output images. (S108), the information is presented by display means (for example, a display or a projector) (S109). In addition, since each mirror image inversion process is independent for each image, it is also possible to perform parallel processing.

  Here, in S104 and S105, if the terminal group belongs to the own terminal, the input image is mirror-inverted to be an output image. Otherwise, the input image is directly used as the output image.

  As a specific example, consider a case where a teacher teaches a student while using a whiteboard.

  In this case, the image photographed by the teacher's terminal 10 includes characters written on the whiteboard. Therefore, it is desirable that the mirror image is not reversed at the student terminal 10. Therefore, taking advantage of the two-group configuration of this embodiment, teachers are assigned to the first group and students are assigned to the second group. In the configuration of FIG. 4, it is desirable that the third external terminal 10 is a teacher's terminal.

  Thereby, the image of the teacher including the white board is not mirror-inverted on the student terminal 10. In the teacher's terminal 10, since the teacher's own image is a mirror image inversion target, the characters are also mirror-inverted. A solution to this problem will be described in the second embodiment.

  As described above, in the present embodiment, natural image sharing for the user can be realized even at three or more locations.

(3) Modification Example When characters do not exist in the image, all terminals 10 may be regarded as one group without performing grouping. In this case, a mirror image is displayed on all terminals 10.

(Embodiment 2)
The image processing method according to the second embodiment of the present invention will be described below. In this embodiment, a part of the area in the image is excluded from the mirror image inversion area, and will be described with reference to FIGS.

  When the image is synthesized on the screen without mirror image reversal, the image is as shown by 501 in FIG. 6, and the characters are correctly displayed.

  However, as described in the first embodiment, the display of the mirror image is as indicated by 601 in FIG. 7, and characters cannot be recognized.

  Therefore, conventionally, for example, an electronic virtual whiteboard (electronic whiteboard) is prepared, and characters and figures written on the whiteboard are shared via a communication channel and superimposed on a display image. It corresponded with. However, users who are unfamiliar with electronic devices often have a resistance to an electronic whiteboard. In addition, since the mirror image inversion is performed unconditionally on the image, the character cannot be shared except for the electronic whiteboard. For example, even if the user wants to see the character printed on paper, the portion is a mirror image. It remains inverted.

  Therefore, the present embodiment solves this problem, and the image processing method will be described below based on the flowcharts of FIGS. 8 and 9.

  A method is prepared for allowing the user to select a symmetric region in the mirrored image. After sequentially inputting images from at least two terminals, some of the images are used as image composition inputs, and some images are subjected to mirror image inversion before being used as image composition inputs.

  If an area in the image has been selected by the user for the mirror-inverted image (S802) (S803), only the designated area is mirror-inverted for the image after mirror-inversion (S804: Re-run). Instead of performing mirror image reversal, an image before mirror image reversal may be pasted only in a designated area), and input to the image composition unit 16.

  When the area is not selected, only normal mirror image inversion is performed.

  Thereafter, using the obtained mirror image and image synthesis (S805) and image presentation (S806) as shown in FIG.

  As a result, as long as there is no user in the selected area, the self-portrait is mirror-inverted, and the mirror image inversion of characters can be avoided.

  This embodiment is applicable to two terminals or a multi-terminal configuration.

  Further, when using a configuration divided into two groups, if a mirror image inversion area is shared via a communication line, a user operation for designating the mirror image inversion area can be performed at one place.

  According to the present embodiment, even in a mirror image such as a self-portrait, it is possible to naturally present a region such as a character that should not be a mirror image to the user.

(Embodiment 3)
The image processing method according to the third embodiment of the present invention will be described below. In the present embodiment, a subject area is calculated and superimposed in a terminal that captures an image including a background of a composite image (hereinafter referred to as “background side terminal”), which will be described with reference to FIG. In FIG. 4, the third external terminal 10 corresponds to the background side terminal.

  Conventionally, the user of the background side terminal is always overlapped by the users of other terminals. However, for example, when a teacher is used to guide a student, it is often inconvenient if the teacher disappears by overlapping. Therefore, it is useful to provide a function of superimposing the user of the background side terminal on the basis of the user's instruction.

  Here, when the system is configured using the chroma key method as in Non-Patent Document 1, the background-side terminal usually does not prepare a monochrome background, and thus the subject cannot be cut out. This is because it is not so much favored by the user to synthesize a monochrome background.

  However, when the system is configured using the background subtraction method, the subject can be cut out even if the background is not a single color, and thus the subject of the background side terminal can be cut out. Therefore, the subject of the background side terminal can be displayed over the subject photographed by another terminal. An example of a method for providing a function of superimposing the user of the background side terminal on the basis of the user's instruction using this will be described according to the flowchart shown in FIG.

  First, an order specified separately is set as an overlapping order (S902). For example, the images of terminal numbers 2, 4, and 3 are sequentially superimposed on the image of terminal number 1.

  Next, the subject is cut out from the image captured by the k-th terminal (S904, S905). This is performed for each terminal number k from 1 to the number of terminals (S903 to S907). Note that the processes in S903 to S907 are sequential processes, but the extraction of the subject for each image is an independent process, and parallel processes can also be performed.

  Next, the cut-out subject image is superimposed on the background image using the superposition sequence (S908). At this time, by superimposing the images shot on the background side terminal, the subject on the background side terminal can be superimposed on the subject shot on the other terminal, and the previous situation can be met. Can do.

The vertical relationship between the subject of the background side terminal set in step S902 and other subjects can be determined statically by separately prepared settings or button operations. Alternatively, the size of the subject (size is, for example, area or bounding box) obtained from the subject area (shape data) may be determined for each frame. For example, if the overlay order is determined so that the subject is larger as the subject size is larger, the user is superimposed higher as the user gets closer to the camera. Therefore, even if a two-dimensional image having no depth information is used. Depth can be realized in a pseudo manner, contributing to the naturalness of the communication space provided by the system (Embodiment 4).
The image processing method according to the fourth embodiment of the present invention will be described below. In the present embodiment, a method for performing the background subtraction method using a separately captured reference image will be described with reference to FIGS. Here, the “reference image” refers to an image showing only the background that is the basis of the difference when the subject is cut out from the subject image taken together with the background using the background difference method.

  FIG. 11 shows a flowchart of a terminal (own terminal) that inputs an image from an external terminal and performs the background difference method and image synthesis, and FIG. 12 shows a flowchart of the external terminal that transmits the input image.

(1) Processing of own terminal The own terminal has a function of storing a reference image (reference image memory) and sequentially performs the image processing shown in the flowchart of FIG.

  In the image processing, first, an image and image additional information from an external terminal are received (S1002). This image additional information includes information for identifying whether the image is a normal captured image or a reference image (for example, binary information in which 0 is a captured image and 1 is a reference image).

  The own terminal refers to the received image additional information and checks whether it is an input image or a reference image (S1003).

  If the image is a reference image, it is stored in the reference image memory (S1007). If the reference image is already stored in the memory, it is updated (S1007).

  When the image is an input image, a difference area between the reference image memory and the input image is detected (S1004), and synthesis with another image is performed (S1005).

(2) Processing of External Terminal Next, the flow of processing in the external terminal will be described with reference to FIG.

  The external terminal has means for capturing an image and means for instructing the user to update the reference image, and sequentially inputs images (S1102).

  Immediately after the start of communication or when there is a reference image input instruction (S1103) by a user instruction, information indicating that the image is a reference image is added to the image additional information (S1106), and the image and the image additional information are transmitted.

  In other cases, information indicating that the image is an input image is added to the image additional information (S1104), and the image and the image additional information are transmitted.

(3) Effects of Embodiment 4 A general video conference system does not have means for transmitting a reference image for the background subtraction method, but the background subtraction method requires updating of the reference image every time the background changes. There are many cases. Therefore, the external terminal can update the reference image at any time in such a configuration, so that when the phenomenon that adversely affects the background subtraction method such as camera movement or significant change in lighting conditions occurs, the reference image Can be updated and communication can be continued.

(4) Modification Example Note that if an image before subject extraction can be viewed at any time, it is useful for adjusting a parameter in the background subtraction method (for example, a threshold value for determining the foreground).

  For example, if the terminal status change such as capturing a reference image is transmitted and the status of the external terminal is displayed on the receiving side, the subject of the external terminal is not suddenly synthesized for capturing the reference image. The problem that the user is confused can be avoided.

  In addition, when device adjustment is performed at the start of image transmission / reception, the reference image device can be adjusted smoothly by transmitting / receiving voice communication or pre-combination images before performing communication for sharing a composite image. .

(Embodiment 5)
The image processing method according to the fifth embodiment of the present invention will be described below.

  A fractal contour extraction method that performs accurate contour extraction using the shape of the subject extracted by background difference will be described with reference to FIGS. 13 to 17.

(1) Content of Fractal Contour Extraction Method of the Present Embodiment Hereinafter, the content of the fractal contour extraction method of the present embodiment will be described based on FIGS.

  14 and 15, X1702 is a true contour line, and X1703 is a given rough contour line. In the contour extraction process, a plurality of processing blocks are arranged based on a rough contour line (that is, a rough contour line). X1701 and X1801 represent one of the processing blocks.

  If a true contour line is included in the processing block as in X1701 in FIG. 14, the contour extraction processing is performed correctly. On the other hand, if the true contour line is not included in the processing block as in X1801 of FIG. 15, the contour extraction process fails. That is, the degree of deviation allowed for a given rough contour is such that the true contour is included in the processing block when the processing block is arranged.

  In the conventional fractal contour extraction method, contour extraction processing has been performed with high calculation accuracy each time. However, since the contour extraction processing is performed again using the result of the previous contour extraction processing as the next input, the next contour extraction processing in the previous contour extraction processing (corresponding range is smaller than the previous one, but the accuracy is higher than the previous one. High calculation accuracy is not required as long as it is within the corresponding range. Only the final contour extraction processing needs to be performed with high calculation accuracy. However, considering the characteristics of the fractal contour extraction method (Fig. 17), the processing blocks used in the final contour extraction processing are small, so it is not well understood. Even if the calculation is done with, the calculation amount is small.

  Using this, the fractal contour extraction method becomes faster.

  First, it is assumed that a rough outline X1703 is given by the method shown in FIG. 14 as shown in FIG. Assume that a processing block X1901 for the next contour extraction processing is arranged based on the next rough contour line X1902 that is the result of the contour extraction processing performed at the current stage. In this case, it is only necessary that the true contour line is included in the processing block X1901. If the calculation amount of the current contour extraction processing is reduced so that the processing block X1901 includes the true contour line, the processing speed is increased.

(3) Description of Flowchart of Fractal Contour Extraction Method Processing The processing of the fractal contour extraction method of this embodiment will be described based on the flowchart of FIG.

  In step S1301, the search step and the number of mappings M are appropriately set according to the block size. Then, the search for block matching (S1303) is performed for each S pixel (S is a natural number), and the number of reduced mappings (S1304) is reduced, thereby increasing the speed.

  However, if the processing of S1302 to S1305 is the last processing, in S1301, M is set such that the error is less than 1 in the full search (step = 1) (for example, M = 3 if the vertical and horizontal pixels are 4 pixels). Set.

(3-1) Search for Block Matching First, the high speed search for block matching in S1303 will be described.

  Since the processing block is centered on the given rough contour, the deviation between the given rough contour and the true contour should not exceed 1/2 of the size of the processing block.

  The ratio of the size of the current processing block to the next processing block (the length of each axis of the processing block) is ½. In order to include the true contour line in the next processing block, it is only necessary that the deviation of the contour line obtained by the contour extraction process at the current stage is ¼ or less of the processing block at the current stage. That is, when the size of the processing block at the current stage is B, the allowable error in the contour extraction process at the current stage is B / 4.

  When the ratio of the size of the parent block to the processing block is r (where r> 1), if the position of the parent block is shifted by one pixel, the contour line obtained is approximately (1 / (r−1)). Pixel shift. The value of this shift is obtained because the fixed point of mapping from the parent block to the processing block is substantially equal to the shift of the fixed point when the parent block is shifted by one pixel. As described in the background art, the “parent block” has a size obtained by multiplying each processing block by A (for example, A = 2.0) and has the most error (for example, for each pixel). A block having a small sum of absolute errors.

  When searching for each S pixel, it is assumed that the parent block closest to the parent block when searching for each pixel is always obtained. That is, it is assumed that the deviation of the parent block is a maximum of S / 2 pixels.

  Then, S may be determined so that (1 / (r−1)) × (S / 2) does not exceed an allowable deviation.

(3-2) Reduced Mapping Processing Next, the reduced mapping processing in S1304 will be described.

  In this embodiment, unlike the nonpatent literature 3, it does not carry out until a reduction | restoration mapping converges, but it carries out at high speed by truncating by M times.

  The contour error becomes (1 / r) every time the reduced mapping is performed. Therefore, when the mapping is performed M times, the contour error is a value obtained by multiplying the Mth power of (1 / r) by the initial allowable error B / 2. Become.

  Therefore, as long as the error caused by performing the search every S pixels and the error caused by terminating the reduced mapping in M times are B / 4 or less, the contour is extracted at high speed without affecting the contour extraction result. Extraction can be performed. This condition is satisfied if, for example, S = B / 4 and M = 3.

(4) Effect From the above, it can be seen that M does not need to be increased even if the size B of the processing block is large, and that S can be increased as the size of the processing block increases.

  Further, as described above, the larger the processing block size B is, the more time is required for the contour extraction processing. Therefore, if the time required when the processing block size B is large can be shortened according to this embodiment, the fractal can be reduced. The contour extraction method can be speeded up.

  In addition, according to the present embodiment, the fractal contour extraction method, which is time-consuming post-processing, can be performed at high speed, and smooth communication hindrance due to the introduction of processing with a larger amount of calculation than chroma key can be solved. it can.

(5) Modification Example In the above description, the speed-up method of the two-dimensional fractal contour extraction method has been described. This method is, for example, a three-dimensional space-time that is a three-dimensional image obtained by MRI or a two-dimensional image according to a time series. The present invention can also be applied to an image of three or more dimensions such as a four-dimensional spatiotemporal image that is a three-dimensional image according to an image or time series.

  For example, Non-Patent Document 4 (Hidenori Takeshima, Takashi Ida, Osamu Hori, Nobuyuki Matsumoto, “Extraction of object contour using self-similarity of spatio-temporal image”, IEICE Technical Report (Science Technical Report), Vol.103, No.325, IE2003-62, Sept.2003.) It is applicable to the spatio-temporal three-dimensional fractal contour extraction method.

  That is, in Non-Patent Document 4, the size of the current processing block may be the same as the size of the next processing block in the time direction. In this case, if the same consideration is made, the contour extraction result will be affected as long as the error caused by performing the search every S pixels and the error caused by cutting the reduced map M times are equal to or less than B / 2. It can be seen that contour extraction can be performed at high speed without giving.

  The present invention is suitable, for example, for an image processing method in which a plurality of participants communicate while sharing one composite image.

It is a flowchart of the method of performing mirror image inversion only with respect to the own terminal group of Embodiment 1 of this invention. It is an example of the image data in the case of performing image processing on images from four terminals. It is an example of a system configuration using this embodiment. It is a structural example of the terminal of the system using this embodiment. It is explanatory drawing which shows the relationship of mirror image inversion. It is an example of the screen which does not invert the mirror image of Embodiment 2. It is an example of the screen which reversed the whole image. It is an example of the screen which excluded only the character area from the object of mirror image reversal. 10 is a flowchart of a method of removing a part of a region from a target for mirror image inversion according to the third embodiment. It is a flowchart of the image processing method which can superimpose the image of the own terminal of Embodiment 4 on top. In this method, a reference image and a target image are separated and processed using additional information. In this method, information for distinguishing between a reference image and a target image is added and transmitted. 10 is a flowchart of a high-speed fractal contour extraction method according to the fifth embodiment. It is explanatory drawing when a true outline is contained in the processing block. It is explanatory drawing when a true outline is not contained in a processing block. It is explanatory drawing by which the processing block of the next outline extraction process was arrange | positioned. It is explanatory drawing which shows the characteristic that the response | compatibility range is large, the amount of calculations is large, and the precision of the acquired outline becomes low, so that the size of a process block is large. It is a flowchart of the image processing method in the conventional system. It is a structural example of the conventional system. It is a flowchart of the conventional fractal outline extraction method.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Terminal 12 Monitor 14 Camera 16 Image composition part 18 Clipping part 20 Mirror image inversion part 22 Capture part

Claims (9)

An image processing method for a terminal that is composed of three or more terminals connected via a network and that inputs and outputs images taken by the respective photographing means of the plurality of terminals via the network. In the image processing method in the terminal constituting the system,
The plurality of terminals are divided into a first terminal group to which the own terminal belongs and a second terminal group to which the own terminal does not belong, and the own terminal belonging to the first terminal group and one or more other An image inversion step for mirror inversion of all images input from the terminal of
A composite image is generated by superimposing and synthesizing each image of the first terminal group that has been mirror-inverted and an image input from one or more other terminals belonging to the second terminal group. A synthesis step to
An image processing method comprising: A self-portrait input step for inputting a self-portrait photographed by the photographing means of the self-terminal;
An external terminal image input step of inputting each external terminal image captured by a plurality of external terminals via a network;
An image inversion step of mirror-inverting each image input from the own terminal belonging to the first terminal group including the own terminal and other external terminals;
A synthesis step of superposing and synthesizing each image of the first terminal group that has been mirror-inverted and an external terminal image of a second terminal group that does not belong to the first terminal group, and generating a synthesized image;
An image processing method comprising: The image processing method according to claim 2, wherein all of the own terminal and the external terminal belong to the first terminal group. In an image processing method in a terminal comprising a plurality of terminals connected via a network, and constituting a system for inputting and outputting images taken by the respective photographing means of the plurality of terminals to each other via the network,
An image reversing step of mirror inverting at least one of the images;
An image replacement step of replacing only the inversion processing exclusion area designated in the mirror image inverted image with an image not mirror-inverted;
A synthesis step of superimposing and synthesizing one image in which only the inversion processing exclusion area is replaced with an image that is not mirror-inverted and another image, and generating a synthesized image;
An image processing method comprising: In an image processing method in a terminal comprising a plurality of terminals connected via a network, and constituting a system for inputting and outputting images taken by the respective photographing means of the plurality of terminals to each other via the network,
A first subject step for calculating a subject that is a subject area of one of the images;
A second subject step for calculating a subject that is a subject region of an image other than the one image;
A composition step of superimposing and synthesizing the subject of the one image and the subject of an image other than the one image on the one image in accordance with a predetermined superposition sequence;
An image processing method comprising: The superposition order in the synthesis step is:
Calculate the size of the subject from the subject of each image,
Compare the calculated size of each subject,
The image processing method according to claim 5, wherein the determination is made based on a result of the comparison. The own terminal in a system that is composed of one or a plurality of external terminals connected to the own terminal via a network and that inputs and outputs images taken by the respective photographing means of the plurality of terminals via the network In the image processing method of
An external terminal image input by an external terminal image captured by one external terminal among the external terminals; and
A reference image input step for inputting a reference image that is an image taken by the one external terminal and to which information that can be distinguished from the external terminal image is added;
A storage step of storing the input reference image as a storage reference image;
When the external terminal image is input, a subject area calculation step of detecting different parts of the stored reference image and the external terminal image and calculating the different parts as subject areas of the external terminal image;
An external terminal image captured by an external terminal other than the one external terminal, or a composite image obtained by superimposing the calculated subject area on the self-portrait captured by the self-terminal, and obtaining a composite image;
An image processing method comprising: A first step of inputting N-dimensional image data composed of a set of N-dimensional (where N is a natural number of 2 or more) unit pixel data;
A second step of inputting N-dimensional shape data indicating a rough shape of the predetermined shape represented in the N-dimensional image data;
A third step of arranging an N-dimensional processing block having a predetermined size and shape at a boundary portion of the N-dimensional shape data;
Search processing for searching for an N-dimensional parent block most similar to the N-dimensional processing block in the N-dimensional image data based on a predetermined conversion method, and the searched N-dimensional parent block in the N-dimensional shape data A fourth step of repeating the replacement process of replacing the contents of the N-dimensional processing block with the contents of the M times, respectively,
A fifth step of outputting N-dimensional shape data obtained by repeating the M times as data of the predetermined shape;
Comprising
In the fourth step,
The size of the (k + 1) -th (where M> k ≧ 1) N-dimensional processing block is reduced from the size of the k-th N-dimensional processing block;
In the search process, the larger the (k + 1) -th N-dimensional processing block size, the rougher the search is performed. Self-portrait input means for inputting a self-portrait photographed by the photographing means of the own terminal;
An external terminal image input means for inputting each external terminal image captured by a plurality of external terminals via a network;
Image inverting means for mirror-inverting each image input from the own terminal belonging to the first terminal group including the own terminal and other external terminals;
Combining means for superimposing and synthesizing each image of the first terminal group that is mirror-inverted and an external terminal image of a second terminal group that does not belong to the first terminal group, and generating a synthesized image;
An image processing apparatus comprising:

Images