JP2010020581A - Image synthesizing system eliminating unnecessary objects - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a synthesizing system which, in photographing using a camera and the like, when shooting a subject, acquires images where images taken so as to include the subject and unnecessary objects obstructing the subject, and a portion of the image obstructing the subject, which is separately taken, acquires a part of the subject to be obstructed by the unnecessary object from the portion obstructing the subject, having been separately taken, and pastes it to change the portion. <P>SOLUTION: The system acquires images with reference to a plurality of shooting positions, acquires the position of an object, identifies a hidden part, acquires the hidden part from another image, synthesizes the images, and thereby complements a lacking area. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for identifying an object and removing unnecessary objects that obstruct the object in an image of an object including an object acquired in advance.

When photographing a subject in photography using a camera or the like, unnecessary objects that block the subject are often reflected in the image. For example, it is a photograph of the apartment itself described in the apartment sale advertisement. In order to take a picture of the apartment itself, it is necessary to shoot from a certain distance from the size, and if it is taken from a long distance, unnecessary objects that block the apartment that is the subject often appear. Therefore, a technique such as that described in Non-Patent Document 1 is provided that removes unnecessary objects that block the subject from the image. Specifically, it is a technique that compensates for deficiency from surrounding information that is not missing by using only one image partially missing due to an unnecessary object or the like by an eigenspace method and an image interpolation method using BPLP.
Toshiyuki Amano "Image interpolation by BPLP using eigenspace method" IEICE Transactions D-II, Vol.J85-D-II, No.3, pp.457-465,2002

  However, the technique described in Non-Patent Document 1 has a problem that, in order to complement a missing area to be complemented from a single image, the missing area and the other area must be similar images. In other words, the area of the missing area is large, for example, an image where an unnecessary object that obstructs a building with a characteristic shape is captured, an image where the missing area is not similar to an area that is not, or a large proportion of unwanted objects that obstruct the building In an image or the like, it is difficult to remove unnecessary materials.

  Accordingly, the present invention provides a technique for acquiring a missing region of a reference image from a plurality of other complementary images from one reference image and solving the missing region of the reference image in order to solve such a problem. Specifically, the following image composition system is used.

  In the first invention, an image acquisition unit that acquires an image including a subject that can be identified by a relative position from a photographing position and an unnecessary object that can be identified by a relative position from the photographing position, for each of the plurality of photographing positions; Using the relative position information obtained from the position of the object in each image and the information indicating the positional relationship between the object and the shooting position for each image, the object is identified as a subject and an unnecessary object. In addition, the image composition includes a hidden part identifying unit that identifies a hidden part of a subject in which an unnecessary object other than the subject overlaps the subject in one image, and a hidden part obtaining unit that obtains the hidden part from another image Provide a system.

  In the second invention, based on the first invention, a correction unit that corrects the acquired hidden part into a hidden part region shape in one image using relative position information, and a hidden part of one image using the correction result. An image composition system is provided that includes a pasting unit for pasting a part to an image of a subject.

  In the third invention, an image acquisition step of acquiring an image including a subject identifiable by a relative position from a photographing position and an unnecessary object identifiable by a relative position from the photographing position for each of a plurality of photographing positions; Using the relative position information obtained from the position of the object in each image and the information indicating the positional relationship between the object and the shooting position for each image, the object is identified as a subject and an unnecessary object. And a hidden part identifying step for identifying a hidden part of a subject in which an unnecessary object other than the subject overlaps the subject in one image, and a hidden part acquiring step for obtaining the hidden part from another image. Provide a method of operating the system.

  In the fourth invention, based on the third invention, a correction step of correcting the acquired hidden part into a hidden part region shape in one image using relative position information, and the hidden result of one image using the correction result An image synthesizing system operating method comprising: a step of replacing a portion with an image of a subject.

  According to the invention as described above, an image in which an object and an unnecessary object are captured so as to block the subject and an image obtained by separately capturing the portion that blocks the object are acquired, and the portion where the subject is blocked by the unnecessary object is blocked. It is possible to acquire a part from an image taken separately and paste it.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to these embodiments, and can be carried out in various modes without departing from the scope of the invention.

  The first embodiment will mainly describe claims 1 and 3, and the second embodiment will mainly describe claims 2 and 4.

  Example 1

    <Overview>

  FIG. 1 is a diagram for explaining functions of the image composition system of the present embodiment. FIG. 1 shows the positional relationship between the image A (0101) and the image B (0102) and the respective photographing positions (0106, 0107) to the object (0103, 0104, 0105). The image A (0101) is an image obtained by shooting the object (0103, 0104, 0105) from the shooting position of the image A (hereinafter referred to as shooting position A), and the object is located between the object C (0105) and the shooting position. Object A (0103) and object B (0104) are reflected so as to block C (0105). The image B is an image obtained by shooting the object (0103, 0104, 0105) from the shooting position of the image B (hereinafter referred to as shooting position B), and the object A is located between the object C (0105) and the shooting position B. (0103) and the object B (0104) are shown so as to block the object C (0105) differently from the image A (0103). Here, it is assumed that the objects A, B, and C photographed in the images A (0101) and B (0102) correspond to each other (0103, 0104, and 0105). In addition, a portion of the image A (0101) where the objects A and B (0103, 0104) block the object C (0105) is captured in the image B (0102).

  In this embodiment, first, in the images A (0101) and B (0102), the respective shooting positions (0106, 0104, 0105, and 0105) are determined based on the positional relationship of the objects A, B, C (0103, 0104, 0105) captured in the respective images. The relative position is determined from 0107).

  Once the relative position is determined, the subject is identified from the objects A, B, C (0103, 0104, 0105) in the reference image (for example, image A (0101)) (for example, the object C (0105)). )). When the subject is identified, an unnecessary object in the foreground of the subject is specified (objects A and B (0103, 0104) are specified). When an unnecessary object is specified, a part of the reference image (image A (0101)) that overlaps the subject due to the unnecessary object is specified and acquired from a complementary image (here, image B (0102)).

    <Functional configuration>

  The functional blocks of the image composition system described below can be realized as hardware, software, or both hardware and software. Specifically, if a computer is used, a CPU, a main memory, a bus or a secondary storage device (hard disk or nonvolatile memory), a driver program for controlling those hardware, and other application programs are included. can give.

  In addition, these hardware and software process the program developed on the main memory with the CPU, process and store data stored on the memory and hard disk, data input via the interface, etc. This is used for output processing or for controlling each hardware component. The present invention can be realized not only as an apparatus but also as a method. A part of the invention can be configured as software. Furthermore, a software product used for causing a computer to execute such software and a recording medium on which the product is fixed are naturally included in the technical scope of the present invention. (Same throughout this specification)

  The configuration requirements of the “image composition system” of this embodiment will be described below with reference to FIG.

  The “image composition system” of this embodiment includes at least an “image acquisition unit” (0201), a “hidden part specifying unit” (0202), and a “hidden part acquisition unit” (0203).

  The “image acquisition unit” (0201) has a function of acquiring, for each of a plurality of shooting positions, an image including a subject that can be identified by a relative position from the shooting position and an unnecessary object that can be identified by a relative position from the shooting position. Have. Specifically, first, a reference image in which a subject to be processed is captured by being blocked by an unnecessary object and a complementary image in which at least a portion of the reference image where the subject is blocked are captured are acquired. Here, the number of acquired complementary images may be one or a plurality of images.

  Next, when images are acquired, feature points in the respective images (reference image and complementary image) are extracted. For feature point extraction, pre-processing that facilitates feature point extraction, such as noise removal, edge extraction, contrast enhancement, and binarization, was performed on the acquired image when necessary. In addition, feature points are extracted using the SIFT method, the Harris operator method, the tracking method, or the like. Here, the feature point is an arbitrary point on each image, and is a point characterized by information around the point (for example, RGB information and brightness of the surrounding point). For example, it is a point that becomes a boundary between the object and the background. A specific extraction method is described in, for example, “JP 2003-242162 A”.

  When feature points are extracted from the respective images, the feature points extracted from the respective images are associated with each other. For example, the surrounding information at an arbitrary feature point in the reference image is compared with the surrounding information at all the feature points acquired in the complementary image, and the feature point with the matching surrounding information is searched. Here, it is sufficient that at least eight or more feature points are associated with each other, and it is not necessary to associate all the feature points (feature points that cannot be associated are excluded).

  Once the feature points are associated, a basic matrix is obtained for each subject and unnecessary object from the plurality of associated feature points.

Here, the basic matrix is a matrix indicating the relationship (mapping relationship) on the coordinates of corresponding points of the reference image and the complementary image (FIG. 3). Specifically, two points that are considered to be corresponding points of two images are p1 and p2, and a basic matrix is F based on the principle of epipolar geometry. Then, F when the relationship of p1 ^T * F * p2 = 0 is established is a basic matrix. Here, the feature points p1 and p2 are matrices indicating the three-dimensional position information configured by 1 × 3 shown in FIG. 3, and F can be expressed by a matrix configured by 3 × 3. Thereby, a plurality of corresponding feature point pairs (p1, p2), (p3, p4), (p5, p6), (p7, p8), (p9, p10), (p11, p12) from the two images. , (P13, p14), (p15, p16)... And a plurality of equations are established and solved to obtain a basic matrix. For example, “Kenichi Kanaya, Mishima et al.“ 3D reconstruction from unconstructed camera and its reliability evaluation ”, Information Processing Society of Japan: Computer Vision and Image Media, Vol.42, No. .SIG 6 (CVIM 2), 2001 "and the like.

  When the basic matrix is derived, the depth is obtained for each feature point based on the derived basic matrix F (that is, the distance from the photographing position is derived). Here, a specific calculation method for obtaining the depth is also described in the paper described above.

  When the depth of each feature point is obtained for each image, the feature points are grouped for each depth to determine which object the feature points scattered in the image are. When the feature point for each object is determined, the outline information of the object connecting the determined feature points for each object is acquired for each object.

The “hidden part specifying unit” (0202) calculates relative position information obtained from the position of the object in each of the plurality of images and information indicating the positional relationship between the object and the shooting position for each image. It has a function of identifying an object as an object and an unnecessary object, and identifying a hidden portion of the object in which an unnecessary object other than the object overlaps the object in one image. Specifically, from the object captured in the image based on the depth information from each imaging position derived from a plurality of images to each object and the contour information connecting the feature points acquired for each object, the object and Identify unwanted items. When the subject and the unwanted object are identified from the target object, the part hidden by the unwanted object that overlaps the identified subject is identified based on the contour information.
Here, when identifying the subject, the “hidden part identifying unit” of the present embodiment is selected by receiving any input from a person and selecting one of the objects captured in the reference image output to the PC or the like. (0202) may identify the selected object as a subject, specify a hidden part that hides the subject, and perform classification according to the distance of each object from the photographing position (for example, photographing position) The distance from the object to the farthest object is divided into equal parts), and the object within the specified range may be identified as the subject. Furthermore, the subject can be identified from a plurality of objects using different information according to the distance obtained from the image (for example, a luminance histogram expressed with different luminance depending on the distance). The hidden part to be hidden may be specified.

  Hereinafter, an example of a method for identifying a subject and an unnecessary object and identifying a hidden part based on the depth from the photographing position actually acquired to each object will be described with reference to FIGS. 4A and 4B. FIG. 4A is a diagram showing the positional relationship of the objects in the image A that is the reference image and the distance of each object from the shooting position (for example, “object C” is the background of the objects A and B). The distance in the X-axis direction from the shooting position is 5a). FIG. 4B is a diagram illustrating the positional relationship of the objects in the image B that is a complementary image and the distances of the objects from the shooting position.

  Here, as shown in FIG. 4A, classification is performed at an arbitrary distance from the photographing position of the image A in the X-axis direction (in this example, classification is performed at a distance “a”). When division is performed, an arbitrary range in which the subject is considered to be photographed is designated. When the range is specified, the object belonging to the specified range is searched (for example, if the range of distance “5a to 6a” is specified, “object C” is searched). When the object is searched, the searched object is identified as a subject (objects not to be searched are identified as unnecessary objects). Here, in this example, the classification is performed at an arbitrary distance. However, a certain depth may be determined in advance, and it may be divided by an arbitrary number. Furthermore, as a more specific method, any one of a fixed threshold discrimination, a cumulative histogram, a discriminant analysis method, or a plurality of methods may be selected. In searching for an object, an object belonging to a specified range is searched. However, if an object cannot be searched, an object closest to the specified range may be searched.

  When the subject and the unwanted object are identified, the unwanted object existing in the foreground from the identified subject is specified (in this example, the objects A and B correspond). When an unnecessary object is specified, a hidden part in the image A is specified from the image B which is a complementary image using the basic matrix.

  The “hidden part acquisition unit” (0203) has a function of acquiring a hidden part from another image. Specifically, first, it is assumed that a hidden part that hides the subject is specified in the “hidden part specifying unit” (0202). When the hidden portion is specified, the basic matrix of the object identified as the subject obtained in the “image acquisition unit” (0201) is used to acquire the portion corresponding to the hidden portion in the reference image from the complementary image. To do.

<Hardware configuration>
FIG. 5 is a schematic diagram illustrating an example of a hardware configuration of the “image composition system” of the present embodiment. The operation of each hardware component in the image composition system of the present embodiment will be described using this figure.

  The “image composition system” of this embodiment includes a “CPU” (0501) that performs various arithmetic processes and controls, and a “main memory” (0502). The “image composition system” also includes a “storage device” (0503) for storing acquired images and the like, and an “I / O” (0504) for acquiring image data. In addition, a “screen” (0505) for outputting a subject or the like and a “UI” (0506) such as a mouse for selecting the subject or the like are also provided. These are connected to each other via a data communication path such as a “system bus”, and perform transmission / reception and processing of information.

  Here, it is assumed that the “image composition system” has acquired the reference image and the complementary image via “I / O” (0504). When the image is acquired, the “CPU” (0501) records the acquired image in the “storage device” (0503) and is expanded in the work area of the “main memory” (0502). Call "Pre-processing program" (0507). The called “pre-processing program” (0507) performs necessary pre-processing on each acquired image and delivers each processed image to the “feature point detection program” (0511). Here, preprocessing refers to a noise removal program (0508), image contrast enhancement (0509), image binarization (0510), which are prepared in advance, as necessary for extracting feature points of the acquired image. ) And the like.

  When the “feature point detection program” (0511) acquires the preprocessed image, the feature point detection program detects the feature point in each image. When the feature point is detected, the feature point is matched between the images. Feature points that do not match are removed as defective points. When each feature point is matched or removed, information of each feature point matched for each object is delivered to the “basic matrix calculation program” (0512).

  The “basic matrix calculation program” (0512) receives a feature point matched for each object, and calculates a basic matrix for each object based on the feature point information. When the basic matrix is derived, the basic matrix is transferred to the “distance calculation program” (0513).

  The “distance calculation program” (0513) obtains each depth using the basic matrix calculated for each object delivered from the “basic matrix calculation program” (0512).

  Next, the “CPU” (0501) calls the “hidden part specifying program” (0514) developed in the work area of the “main memory” (0502). The “hidden part identification program” (0514) identifies a subject from a plurality of objects photographed in the reference image. Here, as a method of recognizing the subject, the program may be automatically recognized, or the user may designate the subject using the UI and recognize the subject as necessary. Next, when the subject is identified, an unnecessary object that is a foreground is acquired from the subject whose distance calculated in the “distance calculation program” (0513) is identified. Thereby, the hidden part which is hiding the subject is specified. When the hidden part is specified, the “hidden part acquisition program” (0515) is called.

  The “hidden part acquisition program” (0515) acquires a part that complements the hidden part specified in the “hidden part specifying program” (0514) from the complement image using the basic matrix.

    <Process flow>

  FIG. 6 is a flowchart illustrating an example of a flow from image acquisition to hidden part acquisition in the image composition system according to the present exemplary embodiment. The steps shown below may be processing steps that are recorded on a medium and constitute a program for controlling the computer.

  Here, it is assumed that the image composition system has acquired a reference image and a complementary image (S0601, S0604). When the reference image and the complementary image are acquired, preprocessing before feature point extraction is performed (S0602, S0605). When the preprocessing is completed, feature points are extracted from each of the reference image and the complementary image (S0603, S0606). When the feature point is extracted, the feature point is matched between the reference image and the complementary image (S0607). When the matching process is completed, a basic matrix is calculated (S0608). Once calculated, the depth (distance) is calculated from the basic matrix (S0609). When the depth is calculated, the subject is identified from a plurality of objects (S0610, S0611). When the subject is identified, the hidden portion of the subject is specified in the reference image (S0612). When the hidden part is specified, the hidden part is acquired from the complementary image (S0613).

  <Brief description of effect>

  By acquiring the reference image and a plurality of or a single complementary image by the image composition system of the present embodiment, it is possible to specify a portion that blocks the subject imaged in the reference image.

    << Example 2 >>

    <Overview>

  The present embodiment is characterized in that, in the image composition system based on the first embodiment, the hidden portion acquired from the complementary image can be corrected according to the reference image, and the corrected hidden portion can be replaced with the reference image. .

    <Functional configuration>

  FIG. 7 is a diagram illustrating an example of a functional configuration in the image composition system of the present embodiment. As shown in this figure, the present embodiment is characterized by newly having a “correction unit” (0704) and a “replacement unit” (0705) based on the image composition system of the first embodiment.

  In the present embodiment, the detailed description of the configuration described in the first embodiment is omitted, and only characteristic portions in the present embodiment are described.

  The “correction unit” (0704) has a function of correcting the acquired hidden part into a hidden part region shape in one image using relative position information. Specifically, the “hidden part acquisition unit” (0703) corrects the part to be complemented with the reference image acquired from the complemented image according to the reference image. Here, the specific correction means that there is no sense of incongruity when the balance between the reference image and the complementary image is corrected using a method such as color correction, luminance correction, and shape correction, and the balance between the reference image and the complementary image is different. Like that.

  The “replacement unit” (0705) has a function of reusing the hidden part of one image with the image of the subject using the correction result. Specifically, the hidden portion that should be pasted to the reference image acquired from the complementary image, corrected in the “correction unit” (0704), is actually pasted to the reference image.

<Hardware configuration>
FIG. 8 is a schematic diagram illustrating an example of a hardware configuration of the “image composition system” of the present embodiment. The processing of the image composition system of this embodiment will be described using these figures.

  Here, it is assumed that the “image composition system” has acquired the reference image and the complementary image via “I / O” (0804). When the image is acquired, the “CPU” (0801) records the acquired image in the “storage device” (0803) and is expanded in the work area of the “main memory” (0802). “Pre-processing program” (0807) is called. The called “pre-processing program” (0807) performs necessary pre-processing for each acquired image, and delivers each processed image to the “feature point detection program” (0811). Here, pre-processing refers to a noise removal program (0808) prepared in advance, image contrast enhancement (0809), and image binarization (0810) as necessary for extracting feature points of the acquired image. ) And the like.

  When the “feature point detection program” (0811) acquires a preprocessed image, it detects a feature point in each image. When the feature point is detected, the feature point is matched between the images. Feature points that do not match are removed as defective points. When each feature point is matched or removed, information on each feature point matched for each object is delivered to the “basic matrix calculation program” (0812).

  The “basic matrix calculation program” (0812) receives a feature point matched for each object and calculates a basic matrix for each object based on the feature point information. When the basic matrix is derived, the basic matrix is transferred to the “distance calculation program” (0813).

  The “distance calculation program” (0813) obtains each depth using the basic matrix calculated for each object delivered from the “basic matrix calculation program” (0812).

  Next, the “CPU” (0801) calls the “hidden part specifying program” (0814) developed in the work area of the “main memory” (0802). The “hidden part identification program” (0814) identifies a subject from a plurality of objects photographed in the reference image. Here, as a method of recognizing the subject, it may be automatically recognized by a program, or the user may recognize the subject using a UI if necessary. Next, when the subject is identified, an unnecessary object that is a foreground is acquired from the subject whose distance calculated in the “distance calculation program” (0813) is identified. Thereby, the hidden part which is hiding the subject is specified. When the hidden part is specified, the “hidden part acquisition program” (0815) is called.

  The “hidden part acquisition program” (0815) acquires the hidden part from the complement image using the basic matrix for the part that complements the hidden part specified in the “hidden part identification program” (0814).

  When the hidden part is acquired, various image correction programs are called as necessary, and correction for pasting the acquired hidden part image on the reference image is performed. Here, the various image correction programs are “color correction program” (0817), “brightness correction program” (0818), “shape correction program” (0819), and the like.

  When the image correction is performed, the “image replacement program” (0820) replaces the corrected hidden portion with the reference image and combines the images.

    <Process flow>

  FIG. 9 is a flowchart illustrating an example of a flow from the acquisition of an image to the replacement of a hidden portion on a reference image in the image composition system of the present embodiment. Note that the steps occupying the following may be processing steps that are recorded on the medium and constitute a program for controlling the computer.

  Here, it is assumed that the image composition system has acquired a reference image and a complementary image (S0901 and S0904). When the reference image and the complementary image are acquired, pre-processing before feature point extraction is performed (S0902, S0905). When the preprocessing is completed, feature points are extracted from each of the reference image and the complementary image (S0903, S0906). When the feature point is extracted, the feature point is matched between the reference image and the complementary image (S0907). When the matching process is completed, a basic matrix is calculated (S0908). Once calculated, the depth (distance) is calculated from the basic matrix (S0909). When the depth is calculated, the subject is identified (S0910, S0911). The hidden part of the subject is specified in one image (S0912). When the hidden part is specified, the hidden part is acquired from another image (S0913). When the hidden part is acquired, the acquired image of the hidden part is corrected (S0914). When the hidden portion is corrected, the hidden portion is pasted on the reference image (S0915).

  <Brief description of effect>

  With the image composition system of the present embodiment, it is possible to correct the hidden part acquired from the complementary image in accordance with the reference image, and actually paste the hidden part on the reference image.

FIG. 3 is a diagram for explaining functions in the image composition system according to the first embodiment. FIG. 3 is a diagram illustrating an example of a functional configuration in the image composition system according to the first embodiment. The figure for demonstrating the basic matrix in the image composition system of Example 1. FIG. The figure which showed the positional relationship of the target object of the image A which is a reference | standard image in the image composition system of Example 1, and the distance of each target object from an imaging position. The figure which showed the positional relationship of the target object of the image B which is a reference | standard image in the image composition system of Example 1, and the distance of each target object from an imaging position. 1 is a diagram illustrating an example of a hardware configuration in an image composition system according to a first embodiment. 7 is a flowchart illustrating an example of a flow from image acquisition to hidden portion acquisition in the image composition system according to the first embodiment. The figure showing an example of a functional structure in the image composition system of Example 2. FIG. 10 is a diagram illustrating an example of a hardware configuration in the image composition system according to the second embodiment. The flowchart showing an example of the flow from the acquisition of the image to the reference image to the replacement of the hidden portion in the image composition system of the second embodiment.

Explanation of symbols

0201 Image acquisition unit 0202 Hidden part identification unit 0203 Hidden part acquisition unit 0704 Correction unit 0705 Rewriting unit

Claims (4)

An image acquisition unit that acquires an image including an object that can be identified by a relative position from a shooting position and an unnecessary object that can be identified by a relative position from the shooting position;
Using the relative position information obtained from the position of the object in each of the plurality of images and information indicating the positional relationship between the object and the shooting position for each image, the object is classified as an object and an unnecessary object. A hidden part identifying unit that identifies a hidden part of a subject in which an unnecessary object other than the subject overlaps the subject in one image,
A hidden part acquisition unit that acquires a hidden part from another image;
An image synthesizing system. A correction unit that corrects the acquired hidden part to a hidden part region shape in one image using relative position information;
A rewriting unit that replaces the hidden part of one image with the image of the subject using the correction result;
The image synthesizing system according to claim 1. An image acquisition step for acquiring, for each of a plurality of shooting positions, an image including an object that can be identified by a relative position from the shooting position and an unnecessary object that can be identified by a relative position from the shooting position;
Using the relative position information obtained from the position of the object in each of the plurality of images, information indicating the positional relationship between the object and the shooting position for each image, the object is converted into an object and an unnecessary object. A hidden part identifying step that identifies and identifies a hidden part of a subject in which an unnecessary object other than the subject overlaps the subject in one image;
A hidden part acquisition step of acquiring a hidden part from another image;
A method for operating an image composition system. A correction step of correcting the acquired hidden part into a hidden part region shape in one image using relative position information,
A pasting step of pasting the hidden part of one image into the subject image using the correction result;
The operation method of the image composition system according to claim 3 having

Images