JP4303696B2 - Background replacement device, background replacement program, background replacement program storage medium, and background replacement method - Google Patents

Description

  The present invention relates to a background replacement device that creates a background replacement image by replacing a background in a captured image with another background, a background replacement program that causes a computer to operate as the background replacement device, and a background that stores the background replacement program The present invention relates to a replacement program storage medium and a background replacement method for generating a background replacement image by replacing a background in a captured image with another background.

  Conventionally, photography using a background curtain has been performed in a photo studio or the like. However, in such shooting, a large backdrop is used to create a sufficient background, and a large shooting space is required. In addition, it takes time to replace the background curtain, and as a result, the photographing cost increases.

  Therefore, there is a demand for the development of a compact photographing system that does not require a large background screen and a wide photographing space as in the past so that photographing can be performed more easily. In such a small shooting system, shooting is performed in front of some temporary background instead of the background curtain as described above. It is conceivable to apply a background replacement process in which the background is replaced with a background desired by the customer.

  As an example of the technology related to such background replacement processing, for example, a background panel painted in blue or the like is placed behind the subject to perform shooting, and in the captured image obtained by the shooting, the color of the background panel There is a process called chroma key processing that distinguishes the same color part and the different color part as a background area and a subject area in the photographed image and replaces the part distinguished in the background area in the photographed image with a desired background. Are known.

  In this chroma key process, if a shadow of a subject appears on the background panel in the captured image, the shadow may interfere with the distinction between the subject region and the background region. Therefore, when shooting a photographic image that is subject to chroma key processing, a background panel that emits light is used as the background, and the shadow of the subject on the background panel is canceled with the light emitted from the background panel. The technique of performing is proposed (for example, refer to Patent Document 1). Further, in Patent Document 1, in the chroma key process, when a portion distinguished as a subject region and a desired background are combined, the color tone of the portion distinguished as the subject region is set according to the color tone of the desired background. There has also been proposed a process of creating a composite image in which a subject appears to be naturally blended into the desired background by adjusting the color tone.

In addition, for example, in a place where the back of the subject is open, such as outdoors, shooting is performed under two illumination conditions, that is, a state where the illumination directed to the subject is turned on and a state where the illumination is turned off. A photographed image is acquired, and in these two photographed images, a part having a different luminance and a part having the same brightness are distinguished from each other as a subject area and a background area in the photographed image. There is also known a technique of replacing a distinguished portion with a desired background (see, for example, Patent Document 2).
JP 2000-224410 A (page 3-10, FIG. 3) Japanese Patent Laid-Open No. 10-210340 (page 2-3, FIG. 1)

  By the way, when background replacement processing is performed by the above-described Patent Document 1, Patent Document 2, or general chroma key processing in a photo studio or the like, in many cases, a background replacement image created by the processing is printed. Prior to this, a background replacement image is displayed on a monitor screen or the like and presented to the customer. Depending on the customer's request, the background replacement process is performed several times while changing the replacement background, and finally the background replacement image that the customer likes most is printed. In such operations, it is desirable that the background replacement image can be presented quickly without waiting for the customer too much, but on the other hand, the background that is finally provided to the customer by means such as printing, etc. If the image quality of the replacement image deteriorates, it will fall to the end. As described above, the background replacement process has an excellent operability in that the background replacement image can be quickly presented to the customer, and finally, the background replacement image with good image quality can be provided to the customer. It is desired.

  In view of the above circumstances, the present invention provides a background replacement device excellent in operability, a background replacement program that causes a computer to operate as such a background replacement device, a background replacement program storage medium in which the background replacement program is stored, and An object of the present invention is to provide a background replacement method with excellent operability.

The background replacement device of the present invention that achieves the above object includes an image acquisition unit that acquires a plurality of photographed images obtained by photographing a common subject under each of a plurality of photographing conditions;
A determination method selection unit that determines a mixing ratio between the subject color and the background color, and selects any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected by the determination method selection unit for a captured image captured under a predetermined capturing condition among a plurality of captured images acquired by the image acquisition unit, and the determined mixing And a background replacement unit that replaces the background of the captured image with another background using the ratio.

  Here, “selecting a determination method” may be selection according to a selection operation by the user, or may be automatic selection according to a selection rule.

  In the operation of presenting the background replacement image to the customer several times on the monitor screen etc., and finally providing the customer with the background replacement image that the customer likes by means such as printing, etc. Immediateness is required, and high quality is required for the background replacement image that is finally provided to the customer. As described above, the background replacement image created in the operation as described above has different characteristics depending on the purpose of use. Here, the mixing ratio greatly affects the image quality after replacement, and the determination time of the mixing ratio occupies a considerable part of the processing time of the background replacement processing. According to the background replacement apparatus of the present invention described above, one determination method is selected from a plurality of determination methods with different mixing ratio determination times, and background replacement processing is performed using the selected determination method. It is. Thereby, for example, when creating a background replacement image for presentation to the customer, the determination method of the background replacement process is shortened by selecting a determination method with a short determination time of the mixing ratio by the determination method selection unit, The background replacement process can be quickly presented to the customer. In addition, when creating a background replacement image that is finally provided to a customer, the determination method selection unit selects a determination method with a long determination time of the mixing ratio, thereby obtaining high image quality in the background replacement processing. The mixing ratio can be determined over time to provide the customer with high quality background replacement processing. That is, according to the background replacement device of the present invention, the background replacement image can be quickly presented to the customer, and finally, the background replacement image with good image quality can be provided to the customer. Excellent operability.

Further, in the background replacement device of the present invention, “decision information indicating the decision method selected by the decision method selection unit in the photographed image photographed under the predetermined photographing condition and the mixture determined by the decision method” It has an image storage unit that adds and stores ratios,
The background replacement unit also executes background replacement processing for the captured image stored by the image storage unit again, and is added to the captured image when the background replacement processing is performed again. When the determination method of the determination time equal to or less than the determination time of the mixture ratio in the determination method indicated by the determination information is selected by the determination method selection unit, the background replacement is performed using the mixture ratio added to the photographed image. When the determination method selection unit that selects a determination time longer than the determination time of the mixture ratio in the determination method indicated by the determination information added to the captured image is selected, the selected method is selected. In the preferred embodiment, the mixing ratio is determined by the determination method described above, and the background replacement process is executed using the determined mixing ratio.

  As described above, the background replacement process may be repeated several times while changing the replacement background according to the customer's request. Further, in the reordering of the background replacement image provided to the customer in the past, a replacement process may be performed in which the background in the background replacement image is replaced with another background according to the customer's request. According to the preferred embodiment of the background replacement process, when the background replacement process as described above is repeated or re-replaced, the captured image stored by the image storage unit and the mixing ratio added to the captured image are added. Is reused, and the processing efficiency is improved. Also, regarding the reuse of the mixture ratio, the mixture ratio is reused only when the determination method corresponding to the determination time equal to or less than the determination time when the mixture ratio is determined is selected by the determination method selection unit. By providing such a restriction, it is possible to avoid that the image quality of the background replacement image obtained by the background replacement process again is lower than the image quality of the background replacement image obtained in the past.

  Further, in the background replacement device of the present invention, “one of the plurality of determination methods is that the color in the captured image is a mixed color in which the true subject color and the background color are mixed at the above-mentioned mixing ratio. A form of “determining a true subject color and a mixing ratio on the premise” is also a preferable form.

  According to this preferred embodiment of the background replacement apparatus, the determination method selection unit selects a determination method for determining the true subject color and the mixing ratio on the assumption as described above, and uses the selected determination method. By performing the background replacement process, a background replacement image with good image quality can be created.

Further, in the background replacement device of the present invention, “one determination method among the plurality of determination methods is to determine the mixture ratio using a reduced image obtained by reducing the captured image acquired by the image acquisition unit”. `` It is a method ''
“A region distinguishing unit that distinguishes a subject region, a background region, and a boundary region in a captured image based on the captured image acquired by the image acquiring unit,
One of the plurality of determination methods includes a mixing ratio representing a pure background color, a mixing ratio representing a pure subject color, and a mixing ratio representing a pure subject color, respectively, with respect to the background region, the subject region, and the boundary region. A form of “a method of determining a mixing ratio by assigning a certain mixing ratio” is also a preferable form.

  According to these preferred embodiments of the background replacement device, the determination method selection unit selects each method as described above, and performs the background replacement processing using the selected determination method, thereby performing background replacement processing. Therefore, the background replacement process can be quickly presented to the customer. In the determination method using the reduced image, it is possible to reduce the amount of image data to be handled when determining the mixture ratio, so that the determination time of the mixture ratio can be shortened and the processing time of the background replacement process can be shortened. . In addition, in the determination method for assigning the mixture ratio to the region, the determination of the mixture ratio is a simple process of assigning a prepared mixture ratio to each region identified by the region distinguishing unit. The time can be shortened, and the processing time of the background replacement process can be shortened.

Further, the background replacement program of the present invention that achieves the above object is incorporated in a computer, and on the computer,
An image acquisition unit for acquiring a plurality of photographed images obtained by photographing a common subject under each of a plurality of photographing conditions;
A determination method selection unit that determines a mixing ratio between the subject color and the background color, and selects any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected by the determination method selection unit for a captured image captured under a predetermined capturing condition among a plurality of captured images acquired by the image acquisition unit, and the determined mixing A background replacement unit that replaces the background of the captured image with another background using the ratio is constructed.

  According to the background replacement program of the present invention, the above-described background replacement device with excellent operability can be easily realized.

Further, the background replacement program storage medium of the present invention that achieves the above object is incorporated in a computer, and on the computer,
An image acquisition unit for acquiring a plurality of photographed images obtained by photographing a common subject under each of a plurality of photographing conditions;
A determination method selection unit that determines a mixing ratio between the subject color and the background color, and selects any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected by the determination method selection unit for a captured image captured under a predetermined capturing condition among a plurality of captured images acquired by the image acquisition unit, and the determined mixing A background replacement program for storing a background replacement unit that replaces the background of a captured image with another background using a ratio is stored.

  The recording medium of the present invention may be a hard disk, a memory chip, or the like in addition to a storage medium such as an FD (flexible disk), a CD-R, a CD-RW, an MO, and a DVD.

Further, the background replacement method of the present invention that achieves the above object includes an image acquisition process for acquiring a plurality of captured images obtained by capturing a common subject under each of a plurality of shooting conditions;
A determination method selection process for determining a mixing ratio between the subject color and the background color and selecting any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected in the determination method selection process for a captured image captured under a predetermined imaging condition among a plurality of captured images acquired in the image acquisition process, and the determined mixture And a background replacement step of replacing the background of the photographed image with another background using the ratio.

  According to the background replacement method of the present invention, the background replacement image can be quickly presented to the customer, and finally, the background replacement image with good image quality can be provided to the customer. Excellent in properties.

  Note that the background replacement program of the present invention, the background replacement program storage medium of the present invention, and the background replacement method of the present invention are shown only in its basic form here, but this is merely to avoid duplication. The background replacement program, the background replacement program storage medium, and the background replacement method according to the present invention include various forms corresponding to the above-described forms of the background replacement device.

  As described above, according to the present invention, the background replacement device with excellent operability, the background replacement program that causes the computer to operate as such a background replacement device, and the background replacement program storage in which the background replacement program is stored A medium and a background replacement method excellent in operability can be provided.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram showing an imaging system to which an embodiment of the background replacement device of the present invention is applied.

  The photographing system 1 shown in FIG. 1 distinguishes between a photographing studio 10 for photographing a subject and obtaining a photographed image, and the subject and background in the photographed image, and replaces the background in the photographed image with a desired background. A personal computer 20 that operates as an embodiment of the background replacement device of the present invention that creates a background replacement image, a printer 30, and a server 40 are provided. The photographing studio 10 includes a digital camera 11, a flash light emitting device 12, an EL (electroluminescence) panel 130, and a power source 14 for the EL panel.

  The digital camera 11 has a continuous shooting function. When the cameraman presses the shutter button once, the digital camera 11 automatically captures the subject P twice in a short time. A photographed image obtained by each photographing is temporarily stored in a memory in the digital camera 11, but in the photographing system 1, the digital camera 11 is connected to the personal computer 20, and two photographings are performed. When the processing is completed, the captured image obtained by the two capturing operations is immediately sent to the personal computer 20. The digital camera 11 is also connected to a flash light emitting device 12, and the digital camera 11 sends an instruction signal for instructing the flash light emitting device 12 to emit a flash light every time the flash light emitting device 12 is photographed. send.

  Here, in this embodiment, space saving is achieved for the photographing studio 10, and the digital camera 11 photographs the subject slightly looking down from a short distance. As a result, there is a high possibility that the captured image captured by the digital camera 11 has a so-called trapezoidal distortion in which the width of the image decreases as it goes downward. In the present embodiment, this trapezoidal distortion is corrected in the personal computer 20.

The flash light emitting device 12 receives an instruction signal from the digital camera 11 and emits a flash toward the subject P. Here, the flash light emitting device 12 needs to be charged every time it emits light, and once it emits light, it takes time until the next light can be emitted. From the digital camera 11, the instruction signal is transmitted every time the above-described shooting is performed. However, the flash light emitting device 12 emits a flash only at the first shooting, and at the second shooting. Since the charging is not in time, the flash light emitting device 12 remains off. Further, the flash light emitted from the flash light emitting device 12 is high-intensity light having a light emission luminance of about 2000 cd / m ² .

  The EL panel 130 includes a housing 131 in which a placement surface 131 a on which the subject P is placed and a surface 131 b behind the subject P are transparent, and a distributed EL element 132 housed in the housing 131. The power source 14 is a power source that applies a driving voltage to the distributed EL element 132. Further, four markers 131c indicating the range of the surface 131b are arranged at the four corners of the surface 131b behind the subject P of the casing 131. These four markers 131c appear in a photographed image together with the subject P at the time of photographing, and are used for correcting the trapezoidal distortion for the photographed image.

  The dispersion type EL element 132 is a sheet in which phosphor powder is dispersed in a binder having a high dielectric constant and is sandwiched between two sheet-shaped electrodes having a flexible plastic substrate. Shaped surface emitting light source. The distributed EL element 132 emits light when an AC voltage is applied from the power source 14 between the two electrodes.

This distributed EL element 132 is a very thin and light light source having a thickness of several hundreds μm to 1 mm, and can be easily installed in a place where there is no sufficient thickness, such as the inside of the housing 131. In addition, the dispersion type EL element 132 has a slight heat generation at the time of light emission of about 2 ° C., has a fast response speed from the start of light emission until it reaches the maximum luminance, and has a constant light emission life and is planned. There are various advantages in terms of performance, such as replacement, local light emission, resistance to shock and vibration, and low power consumption of 50 W / m ² (when AC power with a frequency of 50 Hz is applied). Further, the dispersion type EL element 132 can be manufactured by a simple manufacturing process, and thus has an economic advantage that the manufacturing cost is low.

  In addition, a general dispersion-type EL element can adjust the emission color to various colors including white by further mixing a plurality of phosphor powders having different emission colors at the time of manufacture. In the dispersion type EL element 132 in FIG. 1, the phosphor powder for color adjustment is not mixed. As a result, the dispersion-type EL element 132 has two or more emission peaks between emission wavelengths of 400 nm to 530 nm. This light is light of a color between blue and green, which is inherent to the dispersion-type EL element, and the dispersion-type EL element 132 in the present embodiment emits blue-green light composed of the above two or more emission peaks. To do. Here, since blue-green is a complementary color of the skin color of a person, according to the photographing system 1 of the present embodiment, when photographing a person as a subject, the subject and background in the photographed image are highly accurate. A captured image with a blue-green background can be obtained. Further, the phosphor powder for color adjustment described above tends to reduce the luminance of the emission color, but the dispersion type EL element 132 in the present embodiment is a dispersion type because such phosphor powder is not mixed. The EL element emits light with the original luminance.

The power source 14 can adjust the AC voltage applied to the distributed EL element 132 between 50 Hz to 10 kHz with respect to the frequency and between 40 V to 300 V with respect to the magnitude. The luminance of light emitted from the dispersion type EL element 132 is substantially proportional to the magnitude of the applied AC voltage. In the photography studio 10 of the present embodiment, the AC voltage of the power supply 14 is any frequency of 1.2 kHz to 1.5 kHz with respect to the frequency and any magnitude of 130 V to 200 V with respect to the magnitude. being adjusted to a result, the luminance of light emission of the dispersion-type EL element 132 is adjusted to one of the luminance of ^{500cd / m 2 ~600cd / m 2} .

  Here, in the shooting studio 10 of the present embodiment, shooting by the digital camera 11 is performed in a state where the EL panel 130 is always lit. Further, as described above, the first shooting is performed with the flash light emitting device 12 emitting a flash, and the second shooting is performed with the flash light emitting device 12 turned off. Furthermore, in this photography studio 10, the brightness of the flash emitted by the flash light emitting device 12 is considerably higher than the brightness when the EL panel 130 is turned on. As a result, in the first shooting, a follow-up photographed image in which the subject P is clearly photographed in a follow-up state in which a flash having a brightness higher than that of the EL panel 130 illuminates the subject P from the digital camera 11 side. Is acquired. In the second shooting performed continuously after the first shooting, the image of the subject P is biased to the shadow side by shooting in a backlight state in which only the light from the EL panel 130 illuminates the subject P from behind. Then, a backlight image in which the image of the background panel 130 is biased toward the highlight side is acquired.

  As described above, the personal computer 20 operates as an embodiment of the background replacement device of the present invention. The personal computer 20 performs image correction processing including correction on the trapezoidal distortion on the two captured images passed from the digital camera 11. This image correction process will be described later.

  Further, the personal computer 20 creates a background replacement image in which the background is replaced with another background desired by the customer based on the corrected captured image. In this embodiment, the corrected captured image is used. For the image, the following three types of background replacement images are created.

  First, although the image quality is inferior to other background replacement images, it can be created in the shortest time, and a display replacement image that is presented to the customer immediately after shooting is created. This display replacement image is repeatedly created and presented to the customer until a replacement background that the customer likes is found.

  Once the replacement background has been determined, it is presented to the customer for final confirmation that can be created in a time that does not place too much burden on the customer, with image quality that does not make the customer feel uncomfortable. A confirmation background replacement image is created.

  With the process so far, the exchange with the customer is completed, and then the background determined in this exchange is used, and the replacement image for output with the best image quality is created, although it takes the most time to create. The The output replacement image is output and provided to the customer.

  Here, a series of background replacement processes from the generation of the display replacement image to the generation of the output replacement image will be described later.

  This personal computer 20 has a flexible disk (hereinafter referred to as FD) and a CD-ROM loading port in terms of appearance and has a function of accessing the FD and CD-ROM loaded in the loading port. 210, an image display device 220 that displays an image on the display screen 220a in accordance with an instruction from the main device 210, a keyboard 230 that inputs various types of information according to key operations to the main device 210, and an arbitrary on the display screen 220a Is designated, for example, a mouse 240 for inputting an instruction corresponding to an icon or the like displayed at that position, and a small storage medium used for storing a photographed image by a digital camera or the like are loaded. A media drive 250 for accessing storage media is provided.

  The printer 30 prints an image sent from the personal computer 20, and the photographing system 1 plays a role of printing a background replacement image created by the personal computer 20 by the background replacement process.

  The server 40 stores a plurality of types of backgrounds used in the background replacement process executed by the personal computer 20. The background in the server 40 is presented to the customer by being displayed on the display screen 220a. In addition to the background stored in the server 40, the imaging system 1 also provides a background provided via some input storage medium such as the CD-ROM or the small storage medium. It is presented to the customer through 220a. The customer selects a desired background from the backgrounds displayed on the display screen 220a.

  Next, a flow of work to which the embodiment of the background replacement method of the present invention executed in the photographing system 1 shown in FIG. 1 is applied will be described. In the following description, the elements shown in FIG. 1 are referred to by using the reference numerals in FIG.

  FIG. 2 is a flowchart showing a work flow to which the embodiment of the background replacement method of the present invention is applied, which is executed in the photographing system of FIG.

  The operations shown in the flowchart of FIG. 2 are performed on the assumption that the subject P, the digital camera 11, and the flash light emitting device 12 are arranged at appropriate positions in the photographing studio 10.

  First, a voltage is applied from the power supply 14 to the distributed EL element 132, and the EL panel 130 is turned on (step S101). Next, when the cameraman presses the shutter button of the digital camera 11 after adjusting the focus and exposure, the subject P is shot twice in succession. First, in the first shooting, the flash light emitting device 12 emits a flash with a luminance higher than that of the illumination of the EL panel 130 in response to an instruction signal transmitted from the digital camera 11, and the subject P is shot in a forward light state by the flash. (Step S102). Subsequently, the second shooting is performed in a state where the flash light emitting device 12 is turned off, that is, in a backlight state by illumination of only the EL panel 130 (step S103). Two captured images, a follow light image and a back light image, obtained in the processes of steps S102 and S103, respectively, are temporarily stored in the memory in the digital camera 11.

  The processes in steps S101 to S103 described above are manual image capturing processes performed by the image studio 10 such as a clerk in a photo studio.

  Next, processing after step S104 in this flowchart will be described. Of the processing described below, processing from step S104 to step S110 corresponds to an embodiment of the background replacement method of the present invention.

  The two photographed images obtained by the above photographing process are transferred to the personal computer 20 immediately after the photographing process is completed. Further, a background desired by the customer is read out from a plurality of backgrounds stored in the server 40 (step S104). The processing in step S104 corresponds to an example of an image acquisition process in the background replacement method of the present invention.

  Subsequently, an image correction process described later including correction for the trapezoidal distortion is performed on the two captured images (step S105).

  Next, a determination method for determining the composition ratio between the subject color and the background color is selected from three determination methods having different determination times (step S106). The synthesis ratio here corresponds to an example of “mixing ratio” in the present invention. The processing in step S106 corresponds to an example of a determination method selection process in the background replacement method of the present invention.

  Each background replacement image based on the composition ratio determined by each of the three determination methods has a higher image quality as the determination time is longer. Therefore, in the present embodiment, a determination method is selected according to the purpose of use of the background replacement image. That is, when creating the display replacement image, the first determination method that determines the composition ratio in the shortest of the three determination methods is selected, and when the confirmation background replacement image is created. In this case, the second determination method, which requires a slightly longer time for determining the composition ratio than the first determination method, is selected, and when the above output replacement image is created, the composition ratio is determined accurately. In addition, the third determination method having the longest determination time is selected from the three types of determination methods.

  Here, in the process in step S106, when the determination method is selected following the image correction process in step S105, the first determination method is selected from the above three types of determination methods. Is done.

  Then, the background replacement process for determining the composition ratio using the determination method selected in the process of step S106 and replacing the background using the composition ratio is performed by correcting the corrected image out of the two corrected photographed images. It is applied to the follow light image (step S107). The processing in step S107 corresponds to an example of a background replacement process in the background replacement method of the present invention.

  Further, the created image data representing the background replacement image is transferred to the image display device 220 (step S108) and displayed on the display screen 220a (step S109).

  In addition, in the process of step S107, the correction information indicating the method for determining the composition ratio used in the background replacement process is determined for the corrected follow-light photographed image that is the target of the background replacement process, and determined in the background replacement process. The storage ratio is created by adding the synthesized ratio, and the storage data is stored (step S110).

  When the process up to step S110 is completed, the operator is asked whether or not to end the background replacement process (step S111). At this time, for example, when the background replacement process is necessary again because the customer does not like the background of the currently displayed background replacement image, the operator continues the background replacement process. (No determination in step S111). In this case, a background different from the background in the current background replacement image is read from the server 40 according to the customer's request (step S112). Then, the processing from step S107 to step S110 described above is executed again. However, in this re-processing, the composition ratio added to the follow light photographed image stored at the first processing is used in the background replacement process in step S107, and the composition ratio determination process in this background replacement process is performed. Omitted. Further, in this re-processing, the above-described determination information, composition ratio, and corrected follow-light photographic image that is the source of background replacement are determination information that forms storage data stored in the first processing, Since it is the same as the composite ratio and the corrected follow light captured image, the storage process in step S110 is omitted.

  The processing from step S107 to step S112 is repeated until the customer likes the background of the replacement image for display and the operator informs that the background replacement processing is finished (Yes determination in step S111).

  When the operator reports that the background replacement process is to be terminated, it is determined whether or not the confirmation background replacement image has already been created (step S113). Here, at the stage where the determination of the replacement background by the creation and display of the display replacement image has been completed, the confirmation background replacement image has not yet been generated. At this stage, in step S113, the confirmation background replacement image is created. Is determined not to have been created (No determination in step S113). In this case, the process returns to step S106, and the determination method is selected again. However, when the process returns from step S113 to step S106 to select the determination method, the second determination method is selected in step S106. In step S107, a background replacement process using the second determination method is executed to create a confirmation background replacement image, which is displayed on the display screen 220a in step S109. Further, in the storage process of the storage data in step S110, the determination information, the combination ratio, and the determination information, the combination ratio, and the corrected follow-light photographed image that form the storage data stored in the first process are determined. Are rewritten with determination information indicating the second determination method and the combination ratio determined by the determination method.

  At the stage where the creation of the confirmation background replacement image is completed, in step S111, the end of the replacement process is determined according to the operation performed in the past by the operator (Yes determination in step S111), and further, the confirmation is performed in step S113. It is determined that there is a background replacement image for use (Yes determination in step S113). Then, it is next determined whether or not there is the output replacement image to be finally provided to the customer (step S114). Here, at the stage where the creation of the confirmation background replacement image has been completed, the output replacement image has not yet been generated. At this stage, it is determined in step S114 that there is no output replacement image (No in step S114). Judgment). In this case, the process returns to step S106, and the determination method is selected again. However, when the process returns from step S114 to step S106 and the determination method is selected, the third determination method is selected. In step S107, the background replacement process using the third determination method is performed. Is executed to create an output replacement image. Then, after the output replacement image is displayed on the display screen 220a in step S109, in step S110, the already stored determination information is rewritten with the determination information indicating the third determination method, and further, for output. The final composition ratio determined when the replacement image is created is additionally stored.

  When the output replacement image creation process described above is completed, it is determined in step S111 that the replacement process is complete (Yes in step S111), and in step S113, it is determined that there is a confirmation background replacement image (in step S113). Further, in step S114, it is determined that there is an output replacement image (Yes determination in step S114).

  Then, the created image data representing the output replacement image is transferred to at least one of the printer 30 and the output storage medium desired by the customer (step S115). When the image data representing the output replacement image is transferred to the printer 30, the printer 30 prints the output replacement image based on the image data (step S116).

  Here, the determination method selection process in step S106 corresponds to an example of the determination method selection process in the background replacement method of the present invention, and the background replacement process in step S107 is the background replacement in the background replacement method of the present invention. It corresponds to an example of the process.

  The processes in steps S104 to S115 including an embodiment of the background replacement method of the present invention are processes executed in the personal computer 20. Hereinafter, the personal computer 20 that executes these processes and the processes of those processes will be described. This will be described with a focus on details.

  First, the internal configuration of the personal computer 20 will be described.

  FIG. 3 is a hardware configuration diagram of the personal computer shown in FIG.

  As shown in FIG. 3, inside the main body device 210, a CPU 211 that executes various programs, a main memory 212 that is read out by a program stored in the hard disk device 213 and developed for execution by the CPU 211, A hard disk device 213 in which programs and data are stored, an FD 520 are loaded, an FD drive 214 that accesses the loaded FD 520, a CD-ROM drive 215 that accesses a CD-ROM 510, the server 40 and the digital camera 11 in FIG. And an input interface 216 for receiving a replacement background, a photographed image and the like from these devices, and an output interface 217 for connecting the printer 30 of FIG. 1 and outputting the background replacement image and the like. And the various elements shown in Figure 1 Image display device 220, a keyboard 230, a mouse 240 and a media drive 250 for accessing a small storage medium 530 are connected to each other via a bus 260.

  Here, the CD-ROM 510 described above stores one embodiment of the background replacement program of the present invention for causing the personal computer 20 to operate as one embodiment of the background replacement device of the present invention. This corresponds to an embodiment of the background replacement program storage medium.

  When the CD-ROM 510 is loaded into the CD-ROM drive 215, the program stored in the CD-ROM 510 is uploaded to the personal computer 20 and written to the hard disk device 213. Thereby, the personal computer 20 operates as a background replacement device.

  Next, an embodiment of the background replacement program of the present invention and an embodiment of the background replacement program storage medium of the present invention will be described.

  FIG. 4 is a conceptual diagram showing an embodiment of the background replacement program storage medium of the present invention in which an embodiment of the background replacement program of the present invention is stored.

  A CD-ROM 510 shown in FIG. 4 stores a background replacement program 600, which is an embodiment of the background replacement program of the present invention, and corresponds to an embodiment of the background replacement program storage medium of the present invention.

  The background replacement program 600 includes an image acquisition unit 610, an image correction unit 620, a determination method selection unit 630, a replacement processing unit 640, an image storage unit 650, an image data transfer unit 660, and an image display unit 670.

  The details of each part of the background replacement program 600 will be described together with the operation of each part of the embodiment of the background replacement device of the present invention. In the following description, the elements shown in FIGS. 1, 3, and 4 are referred to by using the reference numerals in the respective drawings without particularly referring to the figure numbers.

  FIG. 5 is a functional block diagram showing functions when the background replacement program shown in FIG. 4 is installed in the personal computer shown in FIGS. 1 and 3 and this personal computer operates as an embodiment of the background replacement device of the present invention. It is. Further, in FIG. 5, the photographing studio 10 including the digital camera 11, the flash light emitting device 12, and the EL panel 130, the printer 30, the server 40, and the hard disk device 213 also shown in FIG. Is also shown.

  As described above, the background replacement device 700 shown in FIG. 5 acquires the follow-up photographed image and the backlight photographed image taken by the digital camera 11 of the photography studio 10 under the backlight and backlight, respectively. Of the two captured images is used to perform background replacement processing in which the subject and the background in the backlight image are distinguished based on the backlight image and the background is replaced with a desired background. A unit 710, an image correction unit 720, a determination method selection unit 730, a replacement processing unit 740, an image storage unit 750, an image data transfer unit 760, and an image display unit 770.

  The image acquisition unit 710 first receives two captured images captured by the digital camera 11 and delivered from the digital camera 11. The image acquisition unit 710 reads a background desired by the customer from the server 40 or from an input storage medium 540 such as a CD-ROM or a small storage medium. In this image acquisition unit 710, the CPU 211 of the personal computer 20 substantially controls the input interface 216, the FD drive 214, the CD-ROM drive 215, and the media drive 250 in accordance with the image acquisition unit 610 of the background replacement program 600. It is composed by doing. The image acquisition unit 710 corresponds to an example of an image acquisition unit in the background replacement device of the present invention, and is an example of an image acquisition process in the background replacement method of the present invention by the image acquisition unit 710 of FIG. The image acquisition process of step S104 in the flowchart is executed.

  The image correction unit 720 performs an image correction process including correction of trapezoidal distortion on each of the two captured images received by the image acquisition unit 710. In practice, the CPU 211 of the personal computer 20 The operation is performed according to the image correction unit 620 of the background replacement program 600. The image correction unit 720 executes the image correction process in step S105 in the flowchart of FIG. Details of the image correction unit 720 will be described later with reference to the accompanying drawings together with details of the image correction processing.

  The determination method selection unit 730 includes a first determination method for generating the display replacement image, a second determination method for generating the confirmation background replacement image, and the output replacement image. One of the three determination methods is selected from among the three determination methods for generating the first and second determination methods. Here, which one of the three determination methods the determination method selection unit 730 selects is determined by the progress of the process described with reference to the flowchart of FIG. 2, and the first method is determined according to the progress of the process. The determination method, the second determination method, and the third determination method are sequentially selected. The determination method selection unit 730 is substantially configured by the CPU 211 of the personal computer 20 operating according to the determination method selection unit 630 of the background replacement program 600, and the determination method in the background replacement device of the present invention. This corresponds to an example of a selection unit. Further, the determination method selection unit 730 executes determination method selection processing in step S106 in the flowchart of FIG. 2, which is an example of a determination method selection process in the background replacement method of the present invention.

  The replacement processing unit 740 uses the determination method selected by the determination method selection unit 730 for the corrected follow light captured image out of the two captured images that have undergone the image correction processing by the image correction unit 720. The CPU 211 of the personal computer 20 operates substantially according to the replacement processing unit 640 of the background replacement program 600. The replacement processing unit 740 corresponds to an example of a background replacement unit in the background replacement device of the present invention, and the replacement processing unit 740 corresponds to an example of a background replacement process in the background replacement method of the present invention. The background replacement process of step S107 in the flowchart of 2 is executed. Details of the replacement processing unit 740 will be described later with reference to the accompanying drawings together with details of the above three determination methods.

  The image storage unit 750 adds the storage information by adding determination information indicating the determination method selected by the determination method selection unit 730 and the composition ratio determined in the background replacement process to the corrected follow light image. The data for storage is created and stored in the hard disk device 213. The image storage unit 750 is substantially configured by the CPU 211 of the personal computer 20 operating in accordance with the image storage unit 650 of the background replacement program 600, and corresponds to an example of the image storage unit in the background replacement device of the present invention. To do. The image storage unit 750 executes the image storage process in step S110 in the flowchart of FIG. Details of the image storage processing unit 750 will be described later with reference to another drawing.

  The image data transfer unit 760 transfers the image data representing the background replacement image created by the replacement processing unit 740 to the image display unit 770. Among the background replacement images, the output replacement image The image data representing the image is transferred to at least one of the printer 30 and the output storage medium 550 desired by the customer. In the image data transfer unit 760, the CPU 211 of the personal computer 20 is substantially controlled by the output interface 217, the FD drive 214, the CD-ROM drive 215, and the media drive 250 in accordance with the image data transfer unit 640 of the background replacement program 600. It is configured by controlling. Further, the image data transfer unit 760 executes the image data transfer process of step S115 in the flowchart of FIG.

  The image display unit 770 displays an image represented by each image data based on the image data transferred from the image data transfer unit 760. In practice, the CPU 211 of the personal computer 20 has the background replacement program 600. The image display device 220 is controlled according to the image display unit 650. Further, the image display unit 770 executes the image display process of step S109 in the flowchart of FIG.

  When image data representing the replacement image is output from the image data transfer unit 760 to the printer 30, the printer 30 prints the replacement image based on the image data, and transfers the image data. When the image data representing the replacement image is output from the unit 760 to any output storage medium 550, the image data is written to the output storage medium 550. Then, both or one of the print paper on which the replacement image is printed and the output storage medium 550 on which the image data representing the replacement image is written are provided according to the customer's request.

  Next, details of the image correction unit 720 and the replacement processing unit 740 shown in FIG. 5 each as one block will be described below.

  FIG. 6 is a diagram showing details of the image correction unit shown by one block in FIG. In the following description, the elements shown in FIG. 5 will be referred to by using the reference numerals in the figure without any special reference.

  As described above, the image correction unit 720 illustrated in FIG. 6 performs image correction processing including correction for trapezoidal distortion on the follow-up photographed image and the backlight photographed image received by the image acquisition unit 710. And a marker detection unit 721, a trapezoidal correction processing unit 722, an effective image cutout unit 723, and a color correction unit 724.

  In the following, the operation of each element of the image correction unit 720 will be described with reference to an example of a follow-light photographed image in which the trapezoidal distortion has occurred.

  FIG. 7 is a schematic diagram illustrating an example of a follow-light photographed image in which trapezoidal distortion has occurred.

  FIG. 7 schematically shows a captured image in which a trapezoidal distortion is generated such that the width of the image becomes narrower as it goes downward.

  First, the marker detection unit 721 detects four markers 131c shown in the captured image. Originally, when these four markers 131b are connected, a rectangle should be formed. However, in the example of FIG. 7, when the four markers 131b are connected, a trapezoid is formed due to trapezoidal distortion.

  The trapezoidal correction processing unit 722 corrects the trapezoidal distortion by appropriately deforming the entire captured image so that the figure formed by connecting the four markers 131c detected by the marker detection unit 721 becomes an original rectangle.

  FIG. 8 is a schematic diagram illustrating a state in which the trapezoidal distortion in the follow light photographed image of FIG.

  The photographed image used for the background replacement process is desirably only the light emitting surface of the EL panel 130 in terms of ease of distinguishing between the subject and the background. However, in this embodiment, since the space is saved for the photographing studio 10, the size of the EL panel 130 is limited, and the photographed image has an EL panel as shown in FIGS. Unnecessary portions other than the light emitting surface 130 are reflected as a background.

  The effective image creation unit 723 creates an effective image in which images outside the effective range E1 determined by the four markers 131c are deleted from the captured image that has undergone the correction processing by the trapezoidal correction processing unit 722.

  FIG. 9 is a schematic diagram showing an effective image created based on the follow-light photographed image corrected for trapezoidal distortion shown in FIG.

  When the effective image creation unit 723 creates an effective image in which the background of the subject P is only the light emitting surface of the EL panel 130 as shown in FIG. 9, the color correction unit 724 then creates the effective image. For example, color correction processing such as red-eye correction for a person's eyes or correction of a color tone of the entire image to a preferable color tone is performed.

  The image correction processing performed by the image correction unit 720 has been described above with reference to the backlight photographed images in FIGS. 7 to 9. However, in the present embodiment, the backlight photographed image photographed together with the backlight photographed image is also used. The same process as the image correction process applied to the follow light photographed image is performed. However, in this backlight photographed image, the subject appears to be blacked out, so the processing in the color correction unit 724 is excluded for this backlight photographed image.

  FIG. 10 is a schematic diagram illustrating a corrected backlight image that has been subjected to image correction processing by the image correction unit of FIG.

  As shown in FIG. 10, the corrected backlight image is an image in which the area of the subject P is biased toward the shadow side and the area where the EL panel 130 is reflected as the background is biased toward the highlight side. 7 to 9 described above, since the luminance of light emitted from the EL panel 130 is lower than the luminance of the flash emitted from the flash light emitting device 12, the light emitting surface of the EL panel 130 is the actual emission color. In contrast, in the backlight image shown in FIG. 10, the image appears in blue-green, which is the actual emission color of the EL panel 130.

  As described above, in the corrected backlight image shown in FIG. 10, the brightness of the subject P region and the background region are greatly different, so that both can be easily distinguished. In the corrected backlight image, the position and contour of the subject P in the image are substantially the same as the position and contour of the subject in the corrected backlight image. Therefore, in this embodiment, in the replacement processing unit 740 of FIG. 5, first, the corrected backlight image is distinguished from the subject P and the background, and the distinction is directly applied to the corrected backlight image. Thus, the subject P and the background in the follow light photographed image are distinguished. Then, the replacement processing unit 740 generates a background replacement image by replacing the background in the follow light photographed image with the background desired by the customer.

  Next, the details of the replacement processing unit 740 will be described by focusing on the background replacement processing that leads to the creation of the background replacement image from the distinction between the subject and the background in the corrected backlight image.

  Here, in the replacement processing unit 740, as described above, the first determination method for generating the display replacement image and the second determination method for generating the confirmation background replacement image described above. And three background replacement processes using the three determination methods, ie, the third determination method for creating the output replacement image, are sequentially executed. In the following, among these three background replacement processes, first, the background using the third determination method for creating the above-described output replacement image with high image quality by accurately determining the above-described composition ratio over time. The replacement process will be described. Since the first determination method and the second determination method both reduce the processing time by simplifying the third determination method, the background replacement process using these determination methods. Will be described after the background replacement process using the third determination method.

  In the following description, the following images will be referred to as examples of the corrected follow-light photographed image and the corrected backlight photographed image.

  FIG. 11 is a diagram illustrating an example of a corrected backlight image, and FIG. 12 is a diagram illustrating an example of a corrected backlight image.

  11 and 12 show a person's subject P '. However, since the photographed image shown in FIG. 12 was photographed by backlighting, the area of the subject P ′ is biased toward the shadow side, and the area where the EL panel 130 is reflected as the background is biased toward the highlight side. It has become. Here, the situation around the boundary between the subject P ′ and the background is shown in another figure for the area A1 in FIG. 11 and the area A2 in FIG. 12 corresponding to the area A1 in FIG.

  13 is an enlarged view of area A1 in FIG. 11, and FIG. 14 is an enlarged view of area A2 in FIG.

  FIG. 13 and FIG. 14 show a state in which the background EL panel 130 is visible over the hair of the subject P ′. In general, a portion where the background is visible over a part of the subject is often seen in the photographed image, but such a portion is separated from the subject in the photographed image as shown in the examples of FIGS. It exists around the boundary with the background.

  Here, FIG. 13 and FIG. 14 are shown in an ideal state where details are shown up to one hair of a person for convenience of explanation. However, in an actual captured image, the pixels constituting the image are larger than a fine subject such as a single hair because of resolution limitations. Therefore, for example, in the image of the portion where the background EL panel 130 can be seen over the hair shown in FIGS. 13 and 14, the color of the fine subject called the hair and the background color visible behind it are synthesized. Thus, the pixels having the combined color are assembled. Therefore, in such a part, it is impossible to simply distinguish between the subject and the background.

  The replacement processing unit 740 of FIG. 5 first replaces the background of the corrected follow-light photographed image as shown in FIG. 11, for example, with the background desired by the customer. Are classified into a subject area, a background area, and a boundary area sandwiched between these two areas. The replacement processing unit 740 leaves the subject area in the corrected follow-light photographed image as it is, and replaces the background area with another background desired by the customer.

  At this time, as shown in FIG. 13, the portion where the background is seen over a part of the subject, that is, the portion constituted by the pixels having the composite color obtained by synthesizing the fine subject color and the background color, As described above, since it is impossible to simply distinguish between the subject and the background, the replacement processing unit 740 distinguishes the boundary region.

  The replacement processing unit 740 performs color synthesis of each pixel on the premise that all the pixels in the boundary region in the corrected follow light captured image have the composite color as described above. The original subject color and the background color are estimated, and the combination ratio of both is estimated. However, the actual boundary region includes pixels having only the background color and pixels having only the subject color in addition to the pixel having the composite color. When the pixel to be processed is a pixel having only the background color, the replacement processing unit 740 estimates a composition ratio at which the ratio of the subject color to the color of the pixel is “0”, and the pixel to be processed However, if the pixel has only the subject color, the composition ratio is estimated such that the ratio of the subject color to the pixel color is “1”.

  Then, the replacement processing unit 740 combines the color of each pixel in the boundary area with the estimated subject color and another background color desired by the customer at the estimated synthesis ratio. Replace with composite color. In this process, the color of the pixel having only the background color in the boundary region is replaced with the other background color described above, the color of the pixel having only the subject color is maintained as it is, and the background over a part of the subject is further maintained. The color of the pixel in the portion where is visible is replaced with the above synthesized color. As a result of this processing, even if the boundary area includes a portion where the background is visible over a part of the subject, another background desired by the customer can be seen over a part of the subject. A natural background replacement image is created.

  Details of the replacement processing unit 740 will be described below.

  FIG. 15 is a diagram showing details of the replacement processing unit shown by one block in FIG. In the following description, the elements shown in FIG. 5 are referred to by using the reference numerals in FIG.

  The replacement processing unit 740 executes the above-described background replacement processing, and includes a mask creation unit 741, a color estimation unit 742, and a background replacement image creation unit 743.

  First, an outline of each of these components will be described.

  The mask creating unit 741 defines a subject mask that defines a range in which only subject-color pixels exist in the backlight photographed image and a range in which only background-color pixels exist in the backlight photographed image. Create a background mask. Here, these two masks are spaced from each other so that the pixels defined by the respective masks do not include the pixels of the portion where the background is visible over the subject as shown in FIG. 14, for example. Created.

  The color estimation unit 742 applies the subject mask and the background mask created by the mask creation unit 741 to the corrected follow-light photographed image, and first, in the follow-light photograph image, between these two masks. Open gaps, i.e. boundary regions, are recognized. Based on the premise that all the pixels in this boundary region have a composite color, the subject color and background color from which the color of each pixel is combined are estimated, and both Estimate the synthesis ratio.

  First, the background replacement image creation unit 743 leaves the area defined by the subject mask as it is in the corrected follow light photographed image, and sets the area defined by the background mask as another background desired by the customer. Replace. Next, for each pixel in the boundary region, the background color desired by the customer and the subject color estimated by the color estimation unit 742 are combined at a combination ratio estimated by the color estimation unit 742 and combined. Find the color and replace the color of each pixel with its composite color. As a result, a background replacement image is created in which the background of the corrected follow light image is naturally replaced with another background desired by the customer.

  The image data representing the background replacement image created in this way is transferred from the image data transfer unit 740 in FIG. 5 to each output device or output storage medium 550 (see FIG. 5).

  Next, details of each component of the replacement processing unit 740 will be described.

  First, details of the mask creation unit 741 will be described.

  FIG. 16 is a diagram showing details of the mask creating unit shown by one block in FIG.

  The mask creation unit 741 includes a histogram creation unit 741a, a threshold calculation unit 741b, a background mask setting unit 741c, a subject mask setting unit 741d, a background mask reduction unit 741e, and a subject mask reduction unit 741f.

  The histogram creation unit 741a creates a histogram for the brightness of each pixel constituting the corrected backlight image.

  FIG. 17 is a diagram illustrating various examples of histograms regarding the brightness of each pixel constituting a corrected backlight image.

  In part (a) of FIG. 17, a luminance histogram H <b> 1 showing the frequency of how many pixels are allocated to each luminance range and the luminance of each pixel constituting the backlight image is allocated to a number of luminance ranges. In this luminance histogram H1, the horizontal axis H1_1 represents the luminance, and the vertical axis H1_2 represents the frequency.

  Here, in the present embodiment, the above-described backlight image and backlight image are coordinates in the RGB color space that defines colors with three colors of R (red), G (green), and B (blue), that is, The color of each pixel is represented by the R value, G value, and B value in the RGB color space. Since the R value, G value, and B value representing the color of each pixel reflect the brightness of that pixel, a histogram of the brightness of each pixel that constitutes the backlit photographed image is represented by these R values. , G value, and B value can be used.

  In part (b) of FIG. 17, the R value of each pixel constituting the backlight image is allocated to a large range of R values, and the R value indicating how many pixels are allocated for each range. A histogram H2 is shown, and in part (c) of FIG. 17, the G value of each pixel constituting the backlight image is allocated to a large range of G values, and how many pixels are allocated to each range. A G-value histogram H3 showing the frequency is shown. Part (d) of FIG. 17 distributes the B value of each pixel constituting the backlight image into a large range of B values. A B-value histogram H4 showing how many pixels have been distributed with respect to frequency is shown. In the R value histogram H2 shown in part (b) of FIG. 17, the R value is taken on the horizontal axis H2_1, and in the G value histogram H3 shown in part (c) of FIG. 17, the G value is taken on the horizontal axis H3_1. In the B value histogram H4 shown in Part (d) of FIG. 17, the B value is taken on the horizontal axis H4_1. In addition, frequencies are taken on the vertical axes H2_2, H3_2, and H4_2 of these three histograms.

  In the present embodiment, in the backlight image, the background of the subject appears in blue-green, which is the emission color of the EL panel 130. Therefore, the brightness of each pixel of the backlight image is the R value, G value, and B value. Of the values, it is best reflected in the G value. Therefore, the histogram creation unit 741a in FIG. 16 is configured to create a G-value histogram H3 shown in part (c) of FIG. 17 among the four histograms shown in FIG.

  Here, in the G value histogram H3, two peaks Pk1 and Pk2 appear on the shadow side and the highlight side as in the other three histograms. The shadow-side peak Pk1 appears after the pixels of the subject area in the corrected backlight image are distributed, and the highlight-side peak Pk2 appears when the pixels of the background area in the backlight image are distributed. It is.

  The threshold calculation unit 741b illustrated in FIG. 16 calculates the following two thresholds based on the two peaks Pk1 and Pk2 in the G value histogram created by the histogram creation unit 741a. As shown in part (c) of FIG. 17, from the shadow side peak Pk1, a first threshold value Sr1 that is the upper limit of the G value at which the frequency is equal to or higher than a predetermined frequency is calculated at the peak Pk1. Further, from the highlight-side peak Pk2, the second threshold value Sr2 that is the lower limit of the G value at which the frequency is equal to or higher than the predetermined frequency is calculated at the peak Pk2.

  Then, the subject mask setting unit 741c shown in FIG. 16 uses, as an initial subject mask, an area composed of pixels having a G value equal to or smaller than the first threshold value Sr1 among the plurality of pixels constituting the corrected backlight image. Then, the background mask setting unit 741d sets, as an initial background mask, an area including pixels whose G value is equal to or greater than the second threshold value Sr2.

  FIG. 18 is a diagram illustrating an area set as the initial subject mask and an area set as the initial background mask in the enlarged view of the backlight image shown in FIG.

  The hatched portion on the right side in FIG. 18 is a portion in which the hair of the subject P ′ is gathered so that the background cannot be seen in the enlarged view of FIG. The G value of the pixel in the portion is lower than the first threshold value Sr1. As a result, the hatched portion on the right side in FIG. 18 is set as the initial subject mask M1 by the subject mask setting unit 741c.

  Further, the hatched portion on the left side in FIG. 18 is a portion that is largely biased toward the highlight side, in which only the background EL panel 130 is shown in the enlarged view of FIG. The threshold value Sr2 of 2 is exceeded. As a result, the left hatched portion in FIG. 18 is set as the initial background mask M2 by the background mask setting unit 741d.

  On the other hand, in FIG. 18, the portion that does not belong to either the left or right hatched portion is a portion where the background is visible over the hair, and the G value of the portion exceeds the first threshold value Sr1 and Below a threshold Sr2 of 2. As a result, the portion that does not belong to either the left or right hatched portion in FIG. 18 is left between the initial subject mask M1 and the initial background mask M2, and is left as a boundary region M3 that is outside both masks.

  FIG. 19 is a diagram showing the initial subject mask and the initial background mask shown in FIG. 18, and FIG. 20 is an overlay of the initial subject mask and the initial background mask on the entire corrected sequential light image shown in FIG. FIG.

  As shown in FIGS. 19 and 20, the initial subject mask M1 and the initial background mask M2 are very close to each other, and the boundary region M3 is very narrow. Here, as described above, in the corrected backlight image, the position and contour of the subject in the image are substantially the same as the position and contour of the subject in the corrected backlight image. However, since the shooting timing is slightly shifted, the position and contour of the subject may be slightly shifted between the two shot images. In such a case, if the boundary region M3 is narrow, the portion of the corrected follow-light photographed image where the background is visible through a part of the subject as shown in FIG. 13 is completely in the boundary region M3. May not be included, and may be included in one of the two masks. When the background replacement is performed based on the initial subject mask M1 and the initial background mask M2, an old background remains in a portion where the background is visible through a part of the subject, or There is a possibility that such a part may be replaced with a new background and a part of the subject may be lost.

  In order to avoid such a problem, in the present embodiment, the initial subject mask M1 is reduced in the direction away from the boundary region M3 by the subject mask reduction unit 741e shown in FIG. 16, and the initial background mask M2 is reduced by the background mask reduction unit 741f. Is reduced in a direction away from the boundary region M3. As a result, the boundary region M3 is widened, and a portion where the background is visible through a part of the subject is completely included in the boundary region M3.

  FIG. 21 is a diagram illustrating a state in which the initial subject mask and the initial background mask illustrated in FIG. 19 are each reduced in a direction away from the boundary region.

  As shown in FIG. 21, the initial subject mask M1 is reduced by moving the contour L1 of the initial subject mask M1 by a predetermined number of pixels in a direction away from the boundary region M3. Similarly, the initial background mask M2 The initial background mask M2 is reduced by moving the outline L2 by a predetermined number of pixels in a direction away from the boundary region M3. Here, in the present embodiment, 5 pixels are employed as the movement amount of the contours of the two masks. After such reduction of the mask, the final subject mask M1 'and the background mask M2' are completed.

  FIG. 22 is a diagram showing the finally completed subject mask and background mask superimposed on the entire corrected follow-light photographed image shown in FIG.

  The final mask between the subject mask M1 ′ and the background mask M2 ′, that is, the final boundary region M3 ′, is 10 pixels wider than the boundary region M3 shown in FIG. As shown in FIG. 5, the final boundary region M3 ′ sufficiently includes a portion where the background is visible through a part of the subject.

  The description of the mask creation unit 741 included in the replacement processing unit 740 in FIG. 15 is finished. Next, details of the color estimation unit 742 included in the replacement processing unit 740 will be described.

  FIG. 23 is a diagram showing details of the color estimation unit shown by one block in FIG.

  The color estimation unit 742 includes a background color estimation unit 742a, a subject color estimation unit 742b, and a synthesis ratio estimation unit 742c.

  First, the background color estimation unit 742a will be described.

  FIG. 24 is a diagram showing details of the background color estimation unit shown by one block in FIG.

  The background color estimation unit 742a estimates the background color constituting the color of each pixel in the final boundary area in the corrected follow light captured image, and the search range A setting unit 742a_1, a background region confirmation unit 742a_2, and a background color calculation unit 742a_3 are provided.

  When estimating the background color for a certain pixel in the boundary area, the background color estimation unit 742a selects a plurality of pixels located near the pixel to some extent from the pixels in the area defined by the background mask. The average color of the searched pixels is obtained as the background color for the pixel.

  The search range setting unit 742a_1 sets a square search range centered on a pixel whose background color is estimated as follows.

  FIG. 25 is a diagram illustrating how a search range is set around a pixel when a background color is estimated for the pixel.

  In setting the search range for a certain pixel G in the boundary region, the search range setting unit 742a_1 first starts from a predetermined square initial range T1 centered on the pixel G and gradually expands the search range. . Then, the search range setting unit 742a_1 stops the expansion when the pixels of the region defined by the background mask M2 ′ exceed the predetermined number and are included in the search range, and the search range T1 ′ at that time is Set as search range for background color estimation.

  When the search range is set, the background region confirmation unit 742a_2 illustrated in FIG. 24 confirms the next region in the corrected follow light image as the region used for estimating the background color for the pixel G.

  FIG. 26 is a diagram illustrating an example of a region used for estimating a background color for a certain pixel.

  As shown in FIG. 26, the region used to estimate the background color for the pixel G is a region defined by the background mask in the corrected follow light captured image, and is set by the search range setting unit 742a_1. A region B1 that is within the search range T1 ′ is confirmed.

  Thus, when the area B1 used for estimating the background color for the pixel G is confirmed, the background color calculation unit 742a_3 shown in FIG. 24 performs an average color of a plurality of pixels in the area B1, that is, a plurality of colors. The average value of each of the R value, the G value, and the B value of the pixels is calculated. Then, the color represented by the calculation result is adopted as the background color for the pixel G.

  The background color estimation unit 742a described above executes the above-described processing from the setting of the search range to the calculation of the average color for all pixels in the boundary region in the corrected follow-light photographed image.

  Next, the subject color estimation unit 742b shown in FIG. 23 will be described.

  FIG. 27 is a diagram showing details of the subject color estimation unit shown by one block in FIG.

  The subject color estimation unit 742b estimates the subject color that constitutes the color of each pixel in the final boundary region in the corrected continuous light captured image, and the search range. A setting unit 742b_1, a subject region confirmation unit 742b_2, a pixel color confirmation unit 742b_3, and a subject color search unit 742b_4 are provided.

  When the subject color estimation unit 742b estimates the subject color for a certain pixel in the boundary region, the subject color estimation unit 742b selects a plurality of pixels located near the pixel to some extent from the pixels in the region defined by the subject mask. The subject color is searched for by the search method described later from the color of the searched plurality of pixels.

  The search range setting unit 742b_1 sets a square search range centered on a pixel for estimating the subject color as follows.

  FIG. 28 is a diagram illustrating a state in which a search range is set around a pixel when the subject color is estimated for the pixel.

  In setting a search range for a certain pixel G in the boundary region, the search range setting unit 742b_1 first starts from a predetermined square initial range T2 centered on the pixel G and gradually expands the search range. . Then, the search range setting unit 742b_1 stops the expansion when the pixels in the region defined by the subject mask M1 ′ exceed the predetermined number and are included in the search range, and the search range T2 ′ at that time is Set as the search range for subject color estimation.

  When the search range is set, the subject region confirmation unit 742b_2 shown in FIG. 27 confirms the next region in the corrected continuous-light image as the region used for estimating the subject color for the pixel G.

  FIG. 29 is a diagram illustrating an example of an area used for estimating a subject color for a certain pixel.

  As shown in FIG. 29, the region used for estimating the subject color for the pixel G is a region defined by the subject mask in the corrected follow light captured image, and is set by the search range setting unit 742b_1. A region B2 that is within the search range T2 ′ is confirmed.

  Thus, when the region B2 used for estimating the subject color for the pixel G is confirmed, first, the pixel color confirmation unit 742b_3 shown in FIG. 27 confirms the color of the pixel G, and then The subject color search unit 742b_4 shown in FIG. 27 searches for the subject color for the pixel G from the colors of the plurality of pixels in the region B2 by the following search method.

  FIG. 30 is a diagram illustrating a search method when searching for a subject color for a certain pixel in a boundary region in a corrected continuous light captured image.

  In FIG. 30, a certain pixel in the boundary region in the corrected continuous light image is confirmed by the background color C1 estimated by the background color estimation unit 742a of FIG. 23 and the pixel color confirmation unit 742b_3 of FIG. A straight line CL1 connecting the color C2 of the pixel in the RGB color space is shown.

  As described above, the color of the pixel in the boundary area in the corrected continuous-light image is a composite color in which the background is seen through a part of the subject at the time of shooting and the background color and the subject color are combined. There is a possibility. If the color of the pixel for which the subject color is to be searched for is a composite color, the color C2 of the pixel that is the composite color and the background color C1 that is the origin of the color C2 The subject color is arranged on a straight line in the RGB color space.

  Therefore, first, the subject color search unit 742b_4 shown in FIG. 27 searches for the color C3 arranged on the extended line of the straight line CL1 from the colors of the plurality of pixels in the region B2.

  When there are a plurality of such colors C3, it is most likely that the color closest to the color C2 of the pixel on the extension line of the straight line CL1 is the origin of the color C2 of the pixel together with the background color C1. . Therefore, when there are a plurality of colors C3 arranged on the extension line of the straight line CL1, the subject color search unit 742b_4 selects a color closest to the pixel color C2 from these colors C3. Adopted as subject color C4.

  The subject color estimation unit 742b described above executes the above-described processing from the setting of the search range to the search for the subject color for all pixels in the boundary region in the corrected continuous light captured image.

  This is the end of the description of the subject color estimation unit 742b shown in FIG. 23. Next, the synthesis ratio estimation unit 742c shown in FIG. 23 will be described. Note that FIG. 30 is referred to in the description of the synthesis ratio estimation unit 742c.

  As shown in FIG. 30, the composition ratio estimation unit 742c connects the background color C1 estimated by the background color estimation unit 742a and the subject color C4 estimated by the subject color estimation unit 742b in the RGB color space. The ratio of the length of the line segment connecting the background color C1 and the color C2 of the pixel to the length of the line segment is the composition ratio α when the color C2 of the pixel is synthesized with the background color C1 and the subject color C4. Calculate as In addition, the synthesis ratio estimation unit 742c executes such calculation of the synthesis ratio α for all the pixels in the boundary region in the corrected follow light captured image.

  Here, as can be seen from FIG. 22 and the like, in the boundary region in the corrected continuous light image, a portion occupied only by the subject, in addition to a portion where the background is visible over a part of the subject, Also included is the part occupied only by the background. The color of the pixel in the portion occupied only by the subject is the color of the subject itself, and even if the background is replaced with another background, the color of the pixel needs to be maintained. Also, the color of the pixel that is occupied only by the background is the color of the background itself, and when the background is replaced with another background, the color of that pixel must be completely replaced with the other background color. There is.

  When the pixel for which the composition ratio is to be estimated is a part of the pixel occupied only by the subject, the composition ratio estimation unit 742c sets the composition ratio α to the same color as the subject. Estimated as “1”. On the other hand, if the pixel for which the background color, the subject color, or the composition ratio is to be estimated is a pixel of a portion occupied only by the background, the color of the pixel is the same color as the background color, so the composition ratio α Is estimated to be “0”.

  When the composite ratio is estimated for all the pixels in the boundary region in the corrected continuous light photographed image in this way, in the present embodiment, the composite ratio estimation unit 742c first selects the subject in the followlight captured image. A composite ratio α of “1” is assigned to all the pixels in the area defined by the mask M1 ′, and “0” is assigned to all the pixels in the area defined by the background mask M2 ′ in the follow light image. Is assigned a composite ratio α. Then, the composition ratio α for all the pixels in these two regions and the composition ratio α for all the pixels in the boundary region are combined to form a composition composed of the composition ratio α for all the pixels in the corrected follow light image. A ratio group is created. This combination ratio group is used for processing in the background replacement image creation unit 743 described below.

  Above, the description about the detail of the color estimation part 742 with which the replacement process part 740 of FIG. 15 is provided is complete | finished, Next, the detail of the background replacement image preparation part 743 with which this replacement process part 740 is provided is demonstrated.

  The background replacement image creation unit 743 performs the following combining process based on the above-described combination ratio group for all the pixels constituting the corrected follow-light photographed image.

  First, among all of these pixels, for the pixel to which the synthesis ratio α of “1” is assigned in the synthesis ratio group, the color of the pixel is maintained as it is. As a result, the image defined by the subject mask M1 'and the image of the portion occupied only by the subject within the boundary region are left as they are. In addition, among all the above-described pixels, for a pixel to which a composite ratio α of “0” is assigned in the composite ratio group, the color of the pixel is the color of the pixel at the corresponding position in the image representing the replacement background. Is completely replaced. As a result, the area defined by the background mask M2 'and the image of the portion occupied only by the background in the boundary area are replaced with the image representing the replacement background.

  Next, out of all the pixels described above, a pixel to which a synthesis ratio α other than “0” and “1” is assigned in the synthesis ratio group, that is, other than “0” and “1” in the boundary region. For the pixels for which the composition ratio α is estimated, the color estimation unit 742 estimates the subject color estimated by the color estimation unit 742 and the color of the pixel at the corresponding position in the image representing the replacement background. A synthesized color is obtained by synthesizing at a synthesis ratio.

  FIG. 31 is a diagram illustrating a state in which a composite color is obtained for a certain pixel in a boundary region in a corrected continuous light captured image.

  FIG. 31 shows a straight line CL1 shown in FIG. 30 that connects the estimated background color C1, the color C2 of the pixel, and the estimated subject color C4 in the RGB color space. . Here, it can be said that the pixel color C2 is a point that divides a straight line connecting the background color C1 and the subject color C4 at a composition ratio α: (1−composition ratio α). Therefore, as shown in FIG. 31, the background replacement image creating unit 743 generates a straight line CL2 connecting another background color C5 and the estimated subject color C4 at a composition ratio α: (1−composition ratio α). A point to be divided is adopted as the composite color C6.

  With the method described with reference to FIG. 31, among all the pixels described above, the composite color is obtained for all the pixels to which the composite ratio α other than “0” and “1” is assigned in the composite ratio group. The background replacement image creation unit 743 replaces the color of these pixels with the composite color obtained for each pixel, so that the background of the corrected follow-light photographed image is replaced with another background desired by the customer. The completed background replacement image is completed.

  FIG. 32 is a diagram illustrating an example of the background replacement image.

  FIG. 32 is an enlarged view in which the same portion as the area A1 in FIG. 11 is enlarged in the background replacement image created by replacing the background of the corrected follow light image shown in FIG. 11 with another background. It is shown.

  The background replacement image created by the background replacement image creation unit 743 is an image in which the color of the pixel in the boundary region is replaced with the above composite color. From this background replacement image, the viewer shows this in FIG. Thus, a state in which another replaced background TB is seen through the hair of the subject P ′ is recognized.

  The background replacement process described above is a background replacement process using the above third determination method in which the composition ratio is accurately determined for all the pixels constituting the corrected follow light captured image. In the present embodiment, a high-quality background replacement image created by the background replacement process using the third determination method is adopted as the output replacement image provided to the customer.

  Next, background replacement processing using the first determination method and the second determination method in which the third determination method is simplified and the processing time is shortened will be described.

  First, the background replacement process using the first determination method will be described.

  In this background replacement process, first, the corrected follow light captured image is reduced to a predetermined display size to create a reduced image, and the reduced image is subjected to the following processing. First, the subject mask M1 ′ and the background mask M2 ′ are created, and a composite ratio of “1” is assigned to all the pixels in the area defined by the former mask, and within the area defined by the latter mask. The process up to assigning a composite ratio of “0” to all the pixels is performed in the same manner as the background replacement process using the third determination method. Thereafter, in the background replacement process using the first determination method, the estimation process of the subject color and the synthesis ratio for the pixels in the boundary area is omitted, and “0.5” is applied to all the pixels in the area. The composition ratio of values is assigned uniformly.

  In the background replacement process using the third determination method, the time required for the subject color and synthesis ratio estimation process occupies most of the processing time of the entire background replacement process. In the background replacement process using the third determination method, the processing time of the entire background replacement process is long in order to accurately perform the estimation process. The background replacement process using the first determination method is performed by reducing the amount of image data to be handled by reducing the corrected follow-light photographed image, and further omitting the above estimation process. This is a time-saving process. On the other hand, the color of the pixels in the boundary area is uniformly replaced with a composite color based on a composite ratio of “0.5”, so that the background replacement image has an image quality that gives the viewer some discomfort. Lower image. In the present embodiment, as described with reference to FIG. 2, this first processing is performed as a process for creating a display replacement image that is presented to the customer immediately after shooting and requires immediacy rather than image quality. Background replacement processing using a determination method is employed.

  Next, background replacement processing using the second determination method will be described.

  In this background replacement process, as in the background replacement process using the first determination method, first, the corrected follow-light photographed image is reduced to a predetermined display size. In the background replacement process using the second determination method, the same process as the background replacement process using the third determination method is performed on the reduced image. In other words, the background replacement process using the second determination method is a process in which the processing time is shortened compared to the background replacement process using the third determination method by reducing the amount of image data to be handled. It is. Further, the image quality of the background replacement image created by this processing is not particularly uncomfortable when viewed in the display size. Therefore, in the present embodiment, before the background replacement image is finally output by the printer 30 or the like, the above confirmation for allowing the customer to confirm the image quality on the display screen 220a (see FIG. 1). As a process for creating a replacement image, a background replacement process using the second determination method is employed.

  When the background replacement process according to each of the three determination methods described above is performed, the storage process by the image storage unit 750 shown in FIG. 5 is executed as described above. In this storage process, first, determination information indicating a method for determining a composition ratio in the background replacement process and a composition ratio determined in the replacement process are applied to the corrected follow-light photographed image that has been subjected to the background replacement process. Are added to create storage data, and the storage data is stored in the hard disk device 213. Hereinafter, the storage data created by the image storage unit 750 will be described.

  FIG. 33 is a schematic diagram showing storage data created by the image storage unit shown in FIG.

  The storage data D shown in FIG. 33 includes a header part D1, a display / confirmation composite ratio group storage part D2, an output composite ratio group storage part D3, and a captured image storage part D4.

  In the header part D1, first, size information indicating the image size of the created background replacement image is described. That is, when the display replacement image or the confirmation replacement image is created, the size information indicating the display size is described. When the output replacement image is created, the size information indicating the output size is displayed. be written. By the way, in the above, this output size is not particularly mentioned, but in the present embodiment, the size of the photographed image is directly adopted as the output size. In addition, in the header part D1, information indicating the first determination method when the display replacement image is generated is described as determination information indicating the determination method of the composite ratio, and the confirmation replacement image is generated. Information indicating the second determination method is described, and information indicating the third determination method is described when the output replacement image is created.

  The display / confirmation composite ratio group storage unit D2 stores the composite ratio group determined when the display replacement image or the confirmation replacement image is created.

  Further, the above-described combination ratio group determined when the output replacement image is created is stored in the output combination ratio group storage unit D3.

  The captured image storage unit D4 stores a corrected follow-light captured image that is a source of the background replacement image.

  In the present embodiment, as shown in FIG. 5, such saved data is saved in the hard disk device 213. For example, when the creation of the replacement image for display is repeated while changing the background for replacement, the saved data created and saved during the first process is used during the second and subsequent processes.

  When there is a reorder regarding background replacement, first, the decision information in the saved data saved in the past is confirmed, and it is determined whether or not the decision information is information indicating the third decision method. The When the determination information is information indicating the third determination method, the output combination ratio group and the follow light image in the stored data are used. When the determination information is information indicating the first or second determination method, it is determined that the past processing has been interrupted until the creation of the display replacement image or the confirmation replacement image. The processing is performed from the determination of the composite ratio group for output.

  As described above, according to the background replacement device 700 described with reference to FIGS. 1 to 33, the background replacement image is quickly presented to the customer, and finally, the background replacement image with good image quality is provided to the customer. It is easy to operate.

  In the above description, as an example of the background replacement unit according to the present invention, the background replacement process using the first determination method for creating a display replacement image and the second determination method for generating a confirmation replacement image are used. The replacement processing unit 740 that executes the three background replacement processes of the background replacement process and the background replacement process using the third determination method for creating the output replacement image has been described. It is not limited. The background replacement unit of the present invention, for example, generates a high-quality background replacement image such as the background replacement process using the third determination method described above, and the determination time is shorter than the background replacement process. Two background replacement processes, that is, a background replacement process using a method for determining a composition ratio, may be executed.

  In the above description, as an example of the background replacement unit according to the present invention, the replacement processing unit 740 that creates a background replacement image for presentation to the customer by reducing the corrected follow light image is described. However, the present invention is not limited to this. The background replacement unit of the present invention, for example, creates the above-described display replacement image among the background replacement images for presentation to the customer using a direct-light image having the original size. May be.

It is a figure which shows the imaging | photography system with which one Embodiment of the background replacement apparatus of this invention was applied. It is a flowchart which shows the flow of the operation | work to which one Embodiment of the background replacement method of this invention performed with the imaging | photography system of FIG. 1 was applied. It is a hardware block diagram of the personal computer shown in FIG. It is a conceptual diagram which shows one Embodiment of the background replacement program storage medium of this invention in which one Embodiment of the background replacement program of this invention was memorize | stored. FIG. 4 is a functional block diagram showing functions when the background replacement program shown in FIG. 4 is installed in the personal computer shown in FIGS. 1 and 3 and this personal computer operates as an embodiment of the background replacement device according to the present invention. It is a figure which shows the detail of the image correction part shown by one block in FIG. It is a schematic diagram which shows an example of the follow light imaging | photography image which the trapezoid distortion produced. It is a schematic diagram which shows the state by which the trapezoid distortion in the follow light photography image of FIG. 7 was correct | amended. It is a schematic diagram which shows the effective image produced based on the follow light imaging | photography image by which the trapezoid distortion was correct | amended shown in FIG. FIG. 7 is a schematic diagram illustrating a corrected backlight image that has been subjected to image correction processing by the image correction unit of FIG. 6. It is a figure which shows an example of the corrected follow light imaging | photography image. It is a figure which shows an example of the backlit image after correction | amendment. It is an enlarged view of area A1 in FIG. It is an enlarged view of area A2 in FIG. It is a figure which shows the detail of the replacement process part shown by one block in FIG. It is a figure which shows the detail of the mask preparation part shown by one block in FIG. It is a figure which shows the various examples of the histogram about the brightness of each pixel which comprises the corrected backlight photo image. FIG. 15 is a diagram illustrating an area set as an initial subject mask and an area set as an initial background mask in the enlarged view of the backlight image shown in FIG. 14. FIG. 19 is a diagram showing an initial subject mask and an initial background mask shown in FIG. 18. FIG. 12 is a diagram showing an initial subject mask and an initial background mask superimposed on the entire corrected follow-light photographed image shown in FIG. 11. FIG. 20 is a diagram illustrating how the initial subject mask and the initial background mask illustrated in FIG. 19 are reduced in a direction away from the boundary region. FIG. 12 is a diagram showing the finally completed subject mask and background mask superimposed on the entire corrected follow-light photographed image shown in FIG. 11. It is a figure which shows the detail of the color estimation part shown by one block in FIG. It is a figure which shows the detail of the background color estimation part shown with one block in FIG. It is a figure which shows a mode that a search range is set centering on the pixel when estimating a background color about a certain pixel. It is a figure which shows an example of the area | region used in order to estimate a background color about a certain pixel. It is a figure which shows the detail of the to-be-photographed color estimation part shown with one block in FIG. It is a figure which shows a mode that a search range is set centering on the pixel, when a subject color is estimated about a certain pixel. It is a figure which shows an example of the area | region used in order to estimate a to-be-photographed object about a certain pixel. It is a figure which shows the search method at the time of searching a to-be-photographed object about a certain pixel in the boundary area | region in the corrected follow light image. It is a figure which shows a mode that a synthetic | combination color is calculated | required about a certain pixel in the boundary area | region in the corrected follow light image. It is a figure which shows an example of a background replacement image. It is a schematic diagram which shows the data for a preservation | save produced by the image preservation | save part shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Shooting system 10 Shooting studio 11 Digital camera 12 Flash light emitting device 130 EL panel 131 Case 131a Mounting surface 131b Back surface 131c Marker 132 Dispersion type EL element 14 Power supply 20 Personal computer 210 Main body device 211 CPU
212 Main memory 213 Hard disk device 213a Background replacement image storage folder 213b Captured image storage folder 213b_1, 213b_2, 213b_3 Subfolder 214 FD drive 215 CD-ROM drive 216 Input interface 217 Output interface 220 Image display device 220a Display screen 220a_1 Scroll button 220a_2 Start replacement Instruction button 220a_3 Approve button 230 Keyboard 240 Mouse 250 Media drive 260 Bus 30 Printer 40 Server 510 CD-ROM
520 FD
530 Small storage medium 540 Input storage medium 550 Output storage medium 600 Background replacement program 610 Image acquisition unit 620 Image correction unit 630 Determination method selection unit 640 Replacement processing unit 650 Image storage unit 660 Image data transfer unit 670 Image display unit 700 Background Replacement device 710 Image acquisition unit 720 Image correction unit 721 Marker detection unit 722 Trapezoid correction processing unit 723 Effective image cropping unit 724 Color correction unit 730 Determination method selection unit 740 Replacement processing unit 741 Mask generation unit 741a Histogram generation unit 741b Threshold calculation unit 741c Background mask setting unit 741d Subject mask setting unit 741e Background mask reduction unit 741f Subject mask reduction unit 742 Color estimation unit 742a Background color estimation unit 742a_1 Search range setting unit 742a_2 Background region confirmation unit 742a_ Background color calculation unit 742b Subject color estimation unit 742b_1 Search range setting unit 742b_2 Subject region confirmation unit 742b_3 Pixel color confirmation unit 742b_4 Subject color search unit 742c Composition ratio estimation unit 743 Background replacement image creation unit 750 Image storage unit 760 Image data transfer unit 770 Image display

Claims (8)

An image acquisition unit for acquiring a plurality of photographed images obtained by photographing a common subject under each of a plurality of photographing conditions;
A determination method selection unit that determines a mixing ratio between the subject color and the background color, and selects any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected by the determination method selection unit for a captured image captured under a predetermined capturing condition among a plurality of captured images acquired by the image acquisition unit, and the determined mixing A background replacement device comprising a background replacement unit that replaces the background of a captured image with another background using a ratio. An image storage unit for adding the determination information indicating the determination method selected by the determination method selection unit and the mixture ratio determined by the determination method to the captured image captured under the predetermined imaging condition and storing With
The background replacement unit also executes a background replacement process for the captured image stored by the image storage unit, and is added to the captured image when the background replacement process is performed again. If a determination method of a determination time equal to or less than the determination time of the mixture ratio in the determination method indicated by the determination information is selected by the determination method selection unit, the background replacement is performed using the mixture ratio added to the photographed image. When the determination method selection unit selects a determination time that is longer than the determination time of the mixture ratio in the determination method indicated by the determination information added to the captured image, the selected method is selected. 2. The background replacement apparatus according to claim 1, wherein the mixing ratio is determined by the determination method and the background replacement processing is executed using the determined mixing ratio.   One of the plurality of determination methods is to determine the true subject color and the mixing ratio on the assumption that the color in the captured image is a mixed color obtained by mixing the true subject color and the background color at the mixing ratio. The background replacement device according to claim 1, wherein the background replacement device is a determination method for determination.   2. The determination method according to claim 1, wherein one of the plurality of determination methods is a determination method that determines the mixture ratio using a reduced image obtained by reducing a captured image acquired by the image acquisition unit. The background replacement device described. Based on the captured image acquired by the image acquisition unit, an area distinguishing unit that distinguishes a subject region, a background region, and a boundary region in the captured image,
One of the plurality of determination methods includes a mixing ratio representing a pure background color, a mixing ratio representing a pure subject color, and a mixing ratio representing a pure subject color with respect to the background region, the subject region, and the boundary region, respectively. 2. The background replacement device according to claim 1, wherein the mixing ratio is determined by assigning a constant mixing ratio. Embedded in a computer, on the computer,
An image acquisition unit for acquiring a plurality of photographed images obtained by photographing a common subject under each of a plurality of photographing conditions;
A determination method selection unit that determines a mixing ratio between the subject color and the background color, and selects any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected by the determination method selection unit for a captured image captured under a predetermined capturing condition among a plurality of captured images acquired by the image acquisition unit, and the determined mixing A background replacement program that constructs a background replacement unit that replaces the background of a captured image with another background using a ratio. Embedded in a computer, on the computer,
An image acquisition unit for acquiring a plurality of photographed images obtained by photographing a common subject under each of a plurality of photographing conditions;
A determination method selection unit that determines a mixing ratio between the subject color and the background color, and selects any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected by the determination method selection unit for a captured image captured under a predetermined capturing condition among a plurality of captured images acquired by the image acquisition unit, and the determined mixing A background replacement program storage medium storing a background replacement program for constructing a background replacement unit that replaces a background of a captured image with another background using a ratio. An image acquisition process for acquiring a plurality of captured images obtained by shooting a common subject under each of a plurality of shooting conditions;
A determination method selection process for determining a mixing ratio between the subject color and the background color and selecting any one of a plurality of determination methods having different determination times;
A mixture ratio is determined by a determination method selected in the determination method selection process with respect to a captured image captured under a predetermined imaging condition among a plurality of captured images acquired in the image acquisition process, and the determined mixing A background replacement method comprising a background replacement step of replacing a background of a captured image with another background using a ratio.

Images