JP6694760B2 - Image synthesizing device and image synthesizing program - Google Patents

Description

  The present invention relates to an image synthesizing device, and more particularly to an image synthesizing device that synthesizes a first image and a second image.

  There is an image synthesizing device for chrominically synthesizing a first image and a second image. First, the apparatus extracts a color of a background portion from a first image to generate a mask image, and performs an AND process (logical product) between the first image and a reverse image of the mask image to obtain a target from the first image. An object part image in which the object part is extracted is generated. Secondly, the apparatus generates a background part image in which the background part is extracted from the second image by ANDing the second image and the mask image. Thirdly, the apparatus generates a composite image by OR processing (logical sum) of the object partial image and the background partial image.

  A technique for performing automatic chroma-key composition that automatically combines a first image shot with a background of a substantially single color on a second image is disclosed (for example, Patent Document 1).

JP, 2014-72697, A

  In the chroma key composition, a process of extracting an object part different from the background part from the first image is performed by using the color (chroma) of the background part of the first image as a key. Therefore, when the color of the object portion of the first image is similar to the color (blue or green) of the background portion of the first image, the object portion and the background portion of the first image are separated from each other. Indistinguishable, the object part is recognized as the background part. In this case, an appropriate composite image is not generated.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image synthesizing apparatus capable of generating an appropriate synthetic image regardless of the color of the object portion of the first image.

  In order to solve the above-mentioned problems, an image synthesizing apparatus according to an embodiment of the present invention, for each pixel of a first image captured with a checkerboard pattern as a background, has a color pattern determined by a peripheral pixel row around the pixel. And a color pattern based on the checkerboard pattern to determine whether each pixel is a foreground pixel corresponding to a foreground portion or a background pixel corresponding to a background portion, and the foreground. A mask image composed of pixels and the background pixels is generated, and an inversion image of the mask image and an image based on the first image are combined with a second image different from the first image and an image based on the mask image. And generating means for generating a composite image.

  According to the image composition device of the present invention, an appropriate composite image can be generated regardless of the color of the object portion of the first image.

FIG. 3 is a block diagram showing a configuration example of an image synthesizing apparatus according to the embodiment. The figure which shows as an example of an operation | movement of the image synthesizing | combining apparatus which concerns on embodiment as a flowchart. (A) is a diagram showing a shooting scene for generating the first image, (B) is a diagram showing an example of the first image, (C) is an enlarged view of the first image of (B), D) is a diagram showing an example of a second image. 3A is an enlarged view showing an example of the region of FIG. 3C, and FIGS. 3B and 3C are views showing a method of setting a peripheral pixel column of each pixel. (A) is a figure which shows the case where a noticed pixel has a pixel value which shows another color, (B)-(L) is a figure which shows a case where a noticed pixel is a pixel value which shows green or yellow. (A)-(G) is a figure which shows the case where a pixel of interest has a pixel value which shows green or yellow. FIG. 6 is a diagram showing a table for classifying a target pixel into some sections according to a color pattern of a peripheral pixel column. (A) is a diagram showing an example of an inverted image of a mask image, (B) is a diagram showing an example of an object partial image, (C) is a diagram showing an example of a mask image, and (D) is a background. The figure which shows an example of a partial image, (E) is a figure which shows an example of a synthetic image.

  Embodiments according to the present invention will be described with reference to the accompanying drawings.

1. Configuration FIG. 1 is a block diagram showing a configuration example of an image synthesizing apparatus according to an embodiment.

  FIG. 1 shows an image synthesizing apparatus 1 according to the embodiment. The image composition device 1 includes a control unit 11, a storage unit 12, an input unit 13, and a display unit 14. The image synthesizing device 1 is configured based on a computer, and is capable of mutual communication with an external device via a network such as a LAN (Local Area Network).

  The control unit 11 is interconnected to each hardware component configuring the image synthesizing device 1 via a bus serving as a common signal transmission path. The control unit 11 may be configured by dedicated hardware or may be configured to realize various functions by software processing by a built-in processor.

  The processor in the above description includes a dedicated or general-purpose CPU (Central Processing Unit) and GPU (Graphics Processing Unit). The dedicated hardware includes circuits such as an application-specific integrated circuit (ASIC), a programmable logic device, and a field programmable gate array (FPGA). Examples of the programmable logic device include a simple programmable logic device (SPLD: Simple Programmable Logic Device) and a complex programmable logic device (CPLD: Complex Programmable Logic Device).

  Here, as an example, a case where the control unit 11 realizes various functions by software processing by the processor will be described. The control unit 11 realizes each function by reading and executing the program stored in the storage unit 12 or the program directly installed in the processor of the control unit 11.

  The control unit 11 may be composed of a single processor or a combination of a plurality of independent processors. In the latter case, a plurality of storage units 12 respectively corresponding to a plurality of processors may be provided, and a program executed by each processor may be stored in a storage circuit corresponding to the processor. As another example, one storage unit 12 may collectively store programs corresponding to respective functions of a plurality of processors.

  The storage unit 12 is configured by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory (Flash Memory), a hard disk, an optical disk, or the like. The storage unit 12 may be configured by a removable medium such as a USB (Universal Serial Bus) memory and a portable medium such as a DVD (Digital Video Disk). The storage unit 12 stores various programs executed by the control unit 11 (including an OS (Operating System) in addition to application programs), character information necessary for executing the programs, and image data. Further, the storage unit 12 may store various commands for controlling the OS and a GUI (Graphical User Interface) program that supports input from the input unit 13.

  The input unit 13 includes, for example, a keyboard, a numeric keypad, a mouse, and the like. The input unit 13 outputs the input information input by the user who is the operator to the control unit 11.

  The display unit 14 is a display device such as a liquid crystal display panel, a plasma display panel, and an organic EL (Electro Luminescence) panel. The display unit 14 displays characters, images, etc. under the control of the control unit 11.

  As shown in FIG. 1, the control unit 11 includes a determination unit 41 and a generation unit 42. The functions of the determination unit 41 and the generation unit 42 are realized by the processor of the control unit 11 executing the programs stored in the storage unit 12.

  The determination unit 41 compares, for each pixel of the first image captured with the checkerboard pattern as a background, the color pattern determined by the peripheral pixel row around the pixel with the color pattern based on the checkered pattern, and each pixel has the foreground It is determined whether the pixel is a foreground pixel corresponding to a portion or a background pixel corresponding to a background portion. The checkered pattern is a kind of lattice pattern and is a pattern in which squares (or rectangles) of two colors are alternately arranged. Here, a case where the first color is green and the second color is yellow will be exemplified. The first image may be captured with a periodic pattern as a background, and is not limited to an image captured with a checkered pattern as an example of the periodic pattern as a background. The periodic pattern is, for example, a pattern in which two-color diamonds are alternately arranged or a pattern in which two-color triangles are alternately arranged. The periodic pattern may be a pattern in which three or more colors are alternately arranged.

  The generation unit 42 generates a mask image (illustrated in FIG. 8C) including the foreground pixels and the background pixels determined by the determination unit 41. Further, the generation unit 42, the inverted image of the mask image and the image based on the first image (object partial image shown in FIG. 8B), the second image different from the first image and the image based on the mask image ( A synthetic image (a synthetic image shown in FIG. 8E) is generated by performing a synthetic process with the background partial image shown in FIG. 8D.

  The specific operations of the determination unit 41 and the generation unit 42 will be described later.

2. Operation A specific operation of the determination unit 41 and the generation unit 42 will be described with reference to FIGS. 1 and 2.

  FIG. 2 is a flowchart showing an example of the operation of the image composition device 1 according to the embodiment.

  The determination unit 41 reads the first image captured with the checkered pattern as the background from the storage unit 12 (step ST1). The determination unit 41 reads the first image sent from the camera that captures the first image and stored in the storage unit 12.

  FIG. 3A is a diagram showing a shooting landscape for generating the first image. FIG. 3B is a diagram showing an example of the first image. FIG. 3C is a diagram showing an enlarged image of the first image shown in FIG. 3B. FIG. 3D is a diagram showing an example of the second image.

FIG. 3A shows a shooting scene in which an object such as a person is shot by a camera against a curtain or screen with a checkered pattern as a background. By this photographing, the first image shown in FIG. 3B is generated. The first image is sent from the camera to the image synthesizing device 1 and stored in the storage unit 12. The first image may be an independent still image or an image of each frame forming a moving image.

  As shown in FIGS. 3 (B) and 3 (C), the first image is a pixel having information (pixel value) of colors showing green and yellow forming a checkered pattern and pixels showing green forming an object. A pixel having a value and a pixel having a pixel value indicating a color that is neither green nor yellow that constitutes the object. The pixel value indicating green may be one pixel value indicating green or a plurality of pixel values indicating green (for example, a pixel value indicating dark green or a pixel value indicating light green). Good.

  Returning to the description of FIG. 2, the determination unit 41 extracts the target pixel from the first image read in step ST1 (step ST2). The determination unit 41 determines whether or not the pixel of interest extracted in step ST2 has a pixel value indicating green or yellow (step ST3).

  If the result of determination in step ST3 is YES, that is, if the pixel of interest has a pixel value indicating green or yellow, the pixel of interest extracted in step ST2 may be a foreground pixel corresponding to the foreground portion. There is also a possibility of a background pixel corresponding to the background portion. Therefore, when the determination in step ST3 is YES, that is, when the pixel of interest has a pixel value indicating green or yellow, the determination unit 41 determines the peripheral pixel row of the pixel of interest extracted in step ST2. The determined color pattern is compared with the checkerboard-based color pattern to determine whether or not the pixel of interest is a foreground pixel (step ST4).

  If the determination in step ST4 is YES, that is, if the pixel of interest is determined to be the foreground pixel, and if the determination in step ST3 is NO, that is, if the pixel of interest indicates a pixel value other than green and yellow, If it is determined that the target pixel is present, a pixel value indicating black is added to the target pixel (step ST5). If NO in the determination in step ST4, that is, if the target pixel is determined to be the background pixel, a pixel value indicating white is added to the target pixel (step ST6). The pixel values indicating black and white provided in steps ST5 and ST6 are the basis of the mask image.

  Here, steps ST3 to ST6 will be specifically described with reference to FIGS.

  FIG. 4A is an enlarged view showing an example of the region R of FIG.

  The enlarged view shown in FIG. 4A shows a plurality of pixels having pixel values. The plurality of pixels on the upper right (regions of other colors in the figure) indicate the object portion of the first image and have pixel values indicating other colors that are neither green nor yellow. The plurality of pixels on the left side (green and yellow checkered areas in the figure) indicate the background portion of the first image, and have pixel values indicating green or yellow. A plurality of pixels at the lower right (one-color region of green in the figure) indicate an object portion of the first image and have a pixel value indicating green.

  The thick line in FIG. 4A is an ideal edge between the background portion and the foreground portion. The determination unit 41 determines whether each pixel is a foreground pixel or a background pixel, and the boundary between the foreground pixel and the background pixel approaches the ideal edge indicated by the thick line. .. For that purpose, first, the determination unit 41 sets a peripheral pixel row for the pixel of interest.

  4B and 4C are diagrams showing a method of setting a peripheral pixel column of each pixel.

  As shown in FIG. 4C, the peripheral pixel row is set within the range of the distance s (illustrated in FIG. 4B) from the target pixel. Then, the color pattern determined by the peripheral pixel row is compared with the color pattern based on the checkered pattern. Alternatively, as shown in FIG. 4C, a peripheral pixel row within a range of a distance s from the target pixel and a range of a distance t (illustrated in FIG. 4B) from the target pixel inside the peripheral pixel row. Circular pixel rows and are set. The rest of the peripheral pixel column excluding the inner peripheral pixel column becomes the outer peripheral pixel column. Then, the color pattern determined by the inner peripheral pixel row is compared with the color pattern based on the checkered pattern, and the color pattern determined by the outer peripheral pixel row is compared with the color pattern based on the checkered pattern. Hereinafter, a case will be described in which a peripheral pixel column within a range of a distance s from the target pixel and an inner peripheral pixel column within a range of a distance t from the target pixel are set.

  Here, the distances s and t are set arbitrarily. In the case shown in FIG. 4 (A), one pixel in a checkerboard pattern is represented by two pixels in each of the left, right, up and down directions. In order to include both checkered colors in the inner peripheral pixel row and the outer peripheral pixel row, the distances s and t are preferably set such that the number of pixels in the inner peripheral pixel row and the outer peripheral pixel row are both 3 or more. Is. In this way, the distances s and t are changed depending on how many pixels represent one element of the checkered pattern. The distances s and t may be changed according to zoom-in and zoom-out of the camera that captures the first image.

  The checkered pattern is given to the background curtain or the like when the first image is taken, but the pattern on the first image is distorted due to the installation condition of the curtain, for example, the wrinkles of the curtain. Sometimes. Further, when the checkered pattern is applied to the curtain or the like on the floor when the first image is taken, the perspective may distort the pattern on the first image. Therefore, when one element of the checkerboard pattern is expressed by two pixels in each of the left and right and the up and down directions, the distances s and t are set so that the number of pixels in the inner peripheral pixel row and the outer peripheral pixel row are both four or more. It may be set. This is because one color of the checkered pattern may appear continuously in three adjacent pixels.

  FIG. 4C is within a range of a distance s from the target pixel, and is within a range of a distance t from the target pixel, and a peripheral pixel row including six peripheral pixels in each of the four vertical and horizontal directions. And an inner peripheral pixel row including three inner peripheral pixels in each of the four directions of up, down, left and right. Hereinafter, in the case where the peripheral pixel rows are four directions in the vertical and horizontal directions and each includes six peripheral pixels, the inner peripheral pixel row is the four directions in the vertical and horizontal directions, and the three inner peripheral pixels are respectively formed. The case of including will be described as an example. The peripheral pixel row and the inner peripheral pixel row may be set in an oblique direction.

  Next, the determination of the foreground pixel or the background pixel of the pixel of interest will be described.

  FIG. 5A is a diagram showing a case where the pixel of interest has a pixel value indicating another color.

  Consider a case where the pixel of interest is the pixel P0 shown in FIG. 4A as shown in FIG. 5A. Since the pixel P0 is a pixel having a pixel value indicating another color, it is determined to be a foreground pixel without using the peripheral pixel row or the inner peripheral pixel row.

  FIGS. 5B to 5L and FIGS. 6A to 6G are diagrams showing a case where the pixel of interest has a pixel value indicating green or yellow.

  Further, FIG. 7 is a diagram showing a table for classifying the target pixel into some sections according to the color pattern of the peripheral pixel row. If the pixel of interest has a pixel value indicating green or yellow, it is first determined to which category the pixel of interest is classified, and then, based on the classification, whether the pixel of interest is a background pixel, or , Determine whether it is a foreground pixel.

  Consider a case where the target pixel is the pixel P1 shown in FIG. 4A as shown in FIG. 5B. Since the pixel P1 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P1 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P1. If the color patterns of the peripheral pixel rows in two or more directions around the target pixel correspond to the classification A1 (or classification A) shown in FIG. 7, it can be considered that there are many pixels showing a checkered pattern in the periphery. In this case, the pixel of interest is determined to be a pixel corresponding to a checkerboard pattern, that is, a background pixel. In the case of the pixel P1, the color patterns of all peripheral pixel rows in four directions correspond to the classification A1 shown in FIG. Therefore, the pixel P1 is determined to be the background pixel.

  Hereinafter, the target pixels P2 to P10, P17 are determined to be background pixels, and the target pixels P11 to P16, P18 are determined to be foreground pixels.

  Consider a case where the target pixel is the pixel P2 shown in FIG. 4A as shown in FIG. 5C. Since the pixel P2 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P2 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P2. The color pattern of the peripheral pixel row in three directions of the pixel P2, that is, in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P2 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5C, it can be inferred that the pixel P2 is relatively close to the foreground pixels showing other colors on the right side thereof.

  Consider a case where the target pixel is the pixel P3 shown in FIG. 4A as shown in FIG. 5D. Since the pixel P3 is a pixel having a pixel value indicating green or yellow, whether the pixel P3 is a foreground pixel or a background pixel is determined from the pixel values of the peripheral pixel row of the pixel P3. The color pattern of the peripheral pixel row in three directions of the pixel P3, that is, in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P3 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5D, it can be inferred that the pixel P3 is a pixel on the right side of the pixel P3 that is relatively close to the foreground pixels indicating other colors.

  Consider a case where the pixel of interest is the pixel P4 shown in FIG. 4A as shown in FIG. 5E. Since the pixel P4 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P4 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P4. The color pattern of the pixel P4 in three directions, that is, the color patterns of the peripheral pixel rows in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P4 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5 (E), it can be inferred that the pixel P4 on the right side thereof is a pixel relatively close to the foreground pixels indicating other colors.

  Consider a case where the target pixel is the pixel P5 shown in FIG. 4A as shown in FIG. Since the pixel P5 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P5 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P5. The color pattern of the peripheral pixel row in three directions of the pixel P5, that is, two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P5 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5 (F), it can be inferred that the pixel P5 is a pixel on the right side thereof which is relatively close to the foreground pixels indicating other colors.

  Consider a case where the pixel of interest is the pixel P6 shown in FIG. 4A as shown in FIG. Since the pixel P6 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P6 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P6. The color pattern of the pixel P6 in three directions, that is, the color patterns of the peripheral pixel rows in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P6 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5G, it can be inferred that the pixel P6 is a pixel on the right side of the boundary with the foreground pixel indicating another color.

  Consider a case where the pixel of interest is the pixel P7 shown in FIG. 4A as shown in FIG. 5H. Since the pixel P7 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P7 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P7. The color patterns of the pixel P7 in two directions, that is, the peripheral pixel rows in two or more directions correspond to the classification A1 shown in FIG. In this case, the pixel P7 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5 (H), it can be inferred that the pixel P7 is a pixel on the right side and the upper side of the pixel P7 at the boundary with the foreground pixels indicating other colors.

  Consider a case where the pixel of interest is the pixel P8 shown in FIG. 4A as shown in FIG. Since the pixel P8 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P8 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P8. The color pattern of the pixel P8 in three directions, that is, in the peripheral pixel rows in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P8 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5 (I), it can be inferred that the pixel P8 on the right side thereof is a pixel relatively close to the foreground pixel showing green or yellow.

  Consider a case where the pixel of interest is the pixel P9 shown in FIG. 4A as shown in FIG. Since the pixel P9 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P9 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P9. The color pattern of the pixel P9 in two directions, that is, the color patterns of the peripheral pixel rows in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P9 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5 (J), it can be inferred that the pixel P9 is a pixel on the right side thereof which is relatively close to the foreground pixel showing green or yellow.

  Consider a case where the pixel of interest is the pixel P10 shown in FIG. 4A as shown in FIG. 5K. Since the pixel P10 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P10 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P10. The color pattern of the pixel P10 in two directions, that is, the peripheral pixel rows in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P10 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 5K, it can be inferred that the pixel P10 is a pixel on the right side of the boundary with the foreground pixel showing green or yellow.

  Consider a case where the pixel of interest is the pixel P11 shown in FIG. 4A as shown in FIG. Since the pixel P11 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P11 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P11. If the color pattern of the peripheral pixel rows in two or more directions around the pixel of interest corresponds to the category E5 (or the category E) shown in FIG. 7, it can be considered that there are many pixels around the target pixel such as a person. In this case, the pixel of interest is determined to be a pixel corresponding to the target object, that is, a foreground pixel.

  Further, as shown in FIG. 5L, it can be inferred that the pixel P11 is a pixel on the left side of the pixel P11 at the boundary with the background pixel.

  Consider a case where the pixel of interest is the pixel P12 shown in FIG. 4A as shown in FIG. Since the pixel P12 has a pixel value indicating green or yellow, it is determined whether the pixel P12 is a foreground pixel or a background pixel based on the pixel values of the peripheral pixel row of the pixel P12. The color patterns of the pixel P12 in all four directions, that is, the color patterns of the peripheral pixel rows in two or more directions correspond to the classification E5 shown in FIG. 7. In this case, the pixel P12 is determined to be a pixel corresponding to the target object, that is, a foreground pixel.

  Further, as shown in FIG. 6A, it can be inferred that the pixel P12 is a pixel on the upper side of the pixel P12 which is relatively close to the foreground pixels indicating other colors.

  Consider a case where the pixel of interest is the pixel P13 shown in FIG. 4A as shown in FIG. 6B. Since the pixel P13 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P13 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P13. The color pattern of the peripheral pixel row in three directions of the pixel P13, that is, two or more directions corresponds to the classification E5 shown in FIG. 7. In this case, the pixel P13 is determined to be a pixel corresponding to the target object, that is, a foreground pixel.

  Further, as shown in FIG. 6B, it can be inferred that the pixel P13 is a pixel on the upper side thereof which is relatively close to the foreground pixels indicating other colors.

  Consider a case where the target pixel is the pixel P14 shown in FIG. 4A as shown in FIG. 6C. Since the pixel P14 is a pixel having a pixel value indicating green or yellow, whether the pixel P14 is a foreground pixel or a background pixel is determined from the pixel values of the peripheral pixel row of the pixel P14. The color pattern of the pixel P14 in three directions, that is, in the peripheral pixel rows in two or more directions corresponds to the category E5 shown in FIG. In this case, the pixel P14 is determined to be a pixel corresponding to the target object, that is, a foreground pixel.

  Further, as shown in FIG. 6C, it can be inferred that the pixel P14 is a pixel on the upper side thereof which is relatively close to the foreground pixels indicating other colors.

  Consider a case where the pixel of interest is the pixel P15 shown in FIG. 4A as shown in FIG. 6D. Since the pixel P15 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P15 is a foreground pixel or a background pixel based on the pixel values of the peripheral pixel row of the pixel P15. The color pattern of the pixel P15 in three directions, that is, the color patterns of the peripheral pixel rows in two or more directions corresponds to the classification E5 illustrated in FIG. 7. In this case, the pixel P15 is determined to be a pixel corresponding to the target object, that is, a foreground pixel.

  Further, as shown in FIG. 6D, it can be inferred that the pixel P15 is a pixel on the upper side thereof which is relatively close to the foreground pixels indicating other colors.

  Consider a case where the pixel of interest is the pixel P16 shown in FIG. 4A as shown in FIG. 6E. Since the pixel P16 is a pixel having a pixel value indicating green or yellow, whether the pixel P16 is a foreground pixel or a background pixel is determined from the pixel values of the peripheral pixel row of the pixel P16. The color pattern of the pixel P16 in three directions, that is, the color patterns of the peripheral pixel rows in two or more directions corresponds to the classification E5 shown in FIG. 7. In this case, the pixel P16 is determined to be a pixel corresponding to the target object, that is, a foreground pixel.

  Further, as shown in FIG. 6E, it can be inferred that the pixel P16 is a pixel on the upper side of the pixel P16 which is a boundary with the foreground pixel indicating another color.

  Consider a case where the pixel of interest is the pixel P17 shown in FIG. 4A as shown in FIG. 6F. Since the pixel P17 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P17 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P17. The color pattern of the pixel P17 in two directions, that is, the color patterns of peripheral pixel rows in two or more directions corresponds to the classification A1 shown in FIG. In this case, the pixel P17 is determined to be a pixel corresponding to a checkered pattern, that is, a background pixel.

  Further, as shown in FIG. 6 (F), it can be inferred that the pixel P17 is a pixel on the right side of the boundary with the foreground pixel showing green or yellow.

  Consider a case where the target pixel is the pixel P18 shown in FIG. 4A as shown in FIG. 6G. Since the pixel P18 is a pixel having a pixel value indicating green or yellow, it is determined whether the pixel P18 is a foreground pixel or a background pixel from the pixel values of the peripheral pixel row of the pixel P18. The color pattern of the pixel P18 in two directions, that is, the peripheral pixel row in two or more directions corresponds to the classification E5 illustrated in FIG. 7. In this case, the pixel P17 is determined to be a pixel corresponding to the target object, that is, a foreground pixel.

  Further, as shown in FIG. 6G, it can be inferred that the pixel P18 is a pixel on the left side of the pixel P18 at the boundary with the background pixel. It can also be inferred that the pixel P18 is a pixel on the upper side of the boundary with the foreground pixels of other colors.

  Returning to the description of FIG. 2, the determination unit 41 determines whether or not it has been determined whether all the pixels of the first image read in step ST1 are background pixels or foreground pixels (step ST7). .. If YES in step ST7, that is, if it is determined that all pixels of the first image are background pixels or foreground pixels, the generation unit 42 determines in step ST5 and ST6 pixel values indicating black and white. To generate mask images (step ST8). The generation unit 42 performs an AND process (logical product) of the first image read out in step ST1 and the reverse image of the mask image generated in step ST8 to obtain the object from which the object portion is extracted. An object part image is generated (step ST9). FIG. 8B shows a partial image of the object based on the first image shown in FIG. 3B and the reverse image shown in FIG. 8A.

  The generation unit 42 reads the second image shown in FIG. 3D from the storage unit 12, and performs the AND process on the second image and the mask image generated in step ST8 to extract the background portion from the second image. The background portion image is generated (step ST10). The second image shown in FIG. 3D is an image acquired mainly by photographing a landscape. Similar to the first image, the second image may be an independent still image or an image of each frame forming a moving image. FIG. 8D shows a background partial image by AND processing of the second image shown in FIG. 3D and the mask image shown in FIG. 8C.

  The production | generation part 42 produces | generates a synthetic | combination image by the OR process (OR) of the target object partial image produced | generated by step ST9, and the background partial image produced | generated by step ST10 (step ST11). FIG. 8E shows a composite image based on the object partial image shown in FIG. 8B and the background partial image shown in FIG. 8D.

  By applying the operations of steps ST1 to ST11 to one first image corresponding to a still image, one combined image with the still image as the foreground and the still image as the background can be generated. Alternatively, by applying the operations of steps ST1 to ST11 to a plurality of first images corresponding to a plurality of frames of a moving image, a plurality of composite images with the moving image as the foreground and the moving image or the still image as the background ( A moving image) can be generated.

3. Effect According to the embodiment described above, an appropriate composite image can be generated regardless of the color of the object portion of the first image.

  It should be noted that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements within a range not departing from the gist of the invention in an implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiment.

1 ... Image synthesizer 11 ... Control unit 12 ... Storage unit 41 ... Judgment unit 42 ... Generation unit

Claims (2)

For each pixel of the first image photographed with the checkerboard pattern as a background, the color pattern determined by the peripheral pixel row around the first image is compared with the color pattern based on the checkerboard pattern, and each pixel corresponds to the foreground portion. A foreground pixel or a background pixel corresponding to the background portion,
A mask image composed of the foreground pixels and the background pixels is generated, and an inverted image of the mask image and an image based on the first image, and a second image different from the first image and an image based on the mask image are generated. Generating means for generating a combined image by performing a combining process;
Have a,
For each pixel of the first image, the determination means compares an inner peripheral color pattern determined by an inner peripheral pixel row included in the peripheral pixel row with a color pattern based on the checkerboard pattern, Comparing an outer peripheral color pattern determined by an outer peripheral pixel row included in the pixel row and a color pattern based on the checkerboard pattern,
Image synthesizer.   On the computer,
  For each pixel of the first image photographed with the checkerboard pattern as the background, the color pattern determined by the peripheral pixel row in the periphery is compared with the color pattern based on the checkerboard pattern, and each pixel corresponds to the foreground portion. And a determination function of determining whether the pixel is a foreground pixel or a background pixel corresponding to a background portion,
  A mask image composed of the foreground pixels and the background pixels is generated, and an inverted image of the mask image and an image based on the first image, and a second image different from the first image and an image based on the mask image are generated. A generation function that generates a composite image by performing composition processing,
  Is realized,
  The determination function compares, for each pixel of the first image, an inner peripheral color pattern determined by an inner peripheral pixel row included in the peripheral pixel row with a color pattern based on the checkered pattern, and A peripheral color pattern determined by a peripheral pixel row included in the pixel row, and a function of comparing the color pattern based on the checkered pattern,
  Image synthesis program.