JP4943400B2 - Background replacement device and photographing system - Google Patents

Description

  The present invention captures a subject to acquire a captured image, replaces the background in the captured image with another background, and creates a background replacement image, and is suitable for creating this background replacement image. It relates to a photographing system.

  Conventionally, in a photo studio or the like, photography using a background curtain is often performed. However, in such shooting, a large backdrop is used to reliably create a background, and a large shooting space is required. Also, the replacement of the background curtain takes time and results in an increase in shooting cost.

  Therefore, there is a demand for the development of a small and simple photographing system that does not require a large background screen and a wide photographing space as in the past so that photographing can be performed more easily.

  In such a small photographing system, instead of the background curtain as described above, photographing is performed in front of some temporary background, and the background in the photographed image is set by the customer with respect to the photographed image obtained by photographing. Background replacement processing is performed in which the background is replaced with a desired background.

  As an example of the technology related to such background replacement processing, for example, a background panel painted in blue or the like is placed behind the subject to perform shooting, and in the captured image obtained by the shooting, the color of the background panel A process called chroma key processing is known in which the same color part and the different color part are distinguished from each other as the background and the subject in the photographed image, and the part distinguished from the background in the photographed image is replaced with a desired background. Yes.

In this chroma key process, if a shadow of a subject appears on the background panel in the captured image, this shadow may interfere with the distinction between the subject and the background. Therefore, when shooting a photographic image that is subject to chroma key processing, there is a technology that uses a background panel that emits light as a background to cancel the shadow of the subject on the background panel with the light emitted by the background panel. (For example, see Patent Document 1)
In the conventional background replacement process including the chroma key process described above, a photographed image taken under the photographing light condition that is focused on clearly showing the subject is used, and the subject and background in the photographed image are used. However, in many cases, the captured image itself obtained under such a photographing light condition is an image that is not suitable for distinguishing between the subject and the background.
JP 2000-224410 A (page 3-10, FIG. 3)

  Therefore, in order to distinguish the subject in the photographed image from the background, the same subject was photographed under a photographing light condition different from the photographing light condition obtained when the photographed image in which the subject was clearly captured was obtained. It is conceivable to use photographed images, and it is desirable that the difference in the photographing light conditions is extreme.

  For example, if a silhouette of a subject is created by illuminating light from the back side of the subject, it is easy to distinguish between the subject and the background. However, in order to reliably create such a silhouette with various subjects, it is necessary to consider surrounding the subject with a dark curtain so that light does not appear on the subject, which goes against the trend of being small and simple. .

  Therefore, there is a demand for a technique that can more reliably distinguish a subject and a background with a photographed image under simpler photographing conditions so as to contribute to downsizing and simplification of the photographing system.

  In view of the above circumstances, the present invention provides a background replacement device that can reliably distinguish a subject and a background even in a captured image under simple capturing conditions, and a capturing system that can be easily reduced in size and simplified. The purpose is to provide.

The background replacement device of the present invention that achieves the above object is as follows.
A captured image acquisition unit that acquires a captured image obtained by capturing an image of a subject with an imaging device in a format that represents the image with a set of pixel values;
A background estimation unit that estimates a pixel value of the background including a background hidden behind the subject on the assumption that the background of the subject has a predetermined uniformity in the captured image;
A background subtraction unit that subtracts the pixel value estimated by the background estimation unit from each pixel value of the captured image;
Based on the result of subtraction by the background subtracting unit, a region in the captured image obtained by photographing the same subject as the subject is excluded from the subject region in which the subject is reflected and the subject region. An area discriminator for distinguishing from the background area;
And a background replacement unit that replaces the background in the captured image with another background based on the region discrimination by the region discrimination unit.

  In the conventional technique, it is assumed that the background and the subject have extremely different colors and brightness. For this reason, if there is a color or brightness that is confusing with the background in a part of the subject, the accuracy of the discrimination decreases. However, in the background replacement device of the present invention, the estimation is made including the back behind the subject, and the subject and the background are distinguished by the difference between the estimated pixel value and the subject pixel value. If there is a slight difference between the estimated pixel value hidden behind the subject and the subject, even if a simple shooting condition is used that causes a somewhat confusing part of the subject or part of the background, the subject area And the background area. Therefore, according to the background replacement device of the present invention, the background replacement can be performed with high accuracy, which can contribute to the realization of a small and simple photographing system. The term “predetermined uniformity” as used herein is a concept including uniformity as well as uniform change. Further, the “pixel value” herein may be brightness, color, or other so-called feature amount as long as it is information given as a numerical value for each pixel.

  Here, the region discriminating unit compares the subtracted pixel value obtained by subtracting the background pixel value from each pixel value of the captured image by the background subtracting unit and a predetermined threshold value, and compares the subject region and the background. It is preferable to distinguish the region.

  In this way, it is possible to easily distinguish the areas.

The background replacement device of the present invention analyzes each line extending in a predetermined direction in the captured image acquired by the captured image acquisition unit, and confirms whether or not the subject exists on the line. With
The background estimation unit estimates the pixel value of the background using the pixel value of the line where the absence of the subject is confirmed by the subject confirmation unit for the line where the presence of the subject is confirmed by the subject confirmation unit. It is also a preferable aspect that it is a thing.

  In this way, the background can be estimated with higher accuracy.

  In addition, it is preferable that the background estimation unit estimates a pixel value of the background of the subject based on the background lines on both sides sandwiching the subject.

  In this way, the background can be estimated with higher accuracy.

  Here, the background estimation unit estimates a pixel value representing the brightness of the background on the assumption that the brightness of the background has a predetermined uniformity in the captured image. Is preferred.

  This makes it easier to create the difference between the subject and the background because brightness is easier to create than color.

The photographing system of the present invention that achieves the above object provides:
An imaging device that captures a subject and obtains a captured image in a format that represents an image with a set of pixel values;
An illumination device that illuminates the subject with light having a predetermined uniformity along the surface from a surface on the opposite side of the photographing device across the subject; and photographing obtained by the photographing device A captured image acquisition unit for acquiring images;
A background estimation unit that estimates a pixel value of the background including the background hidden behind the subject on the assumption that the background of the subject has the uniformity in the captured image;
A background subtraction unit that subtracts the pixel value estimated by the background estimation unit from each pixel value of the captured image;
An area discriminating unit that discriminates an area in the captured image into a subject area in which the subject is captured and a background area excluding the subject area based on a subtraction result by the background subtraction unit;
A background replacement device comprising: a background replacement unit that replaces a background in a captured image obtained by photographing the same subject as the subject with the photographing device based on the region distinction by the region distinguishing unit. ;
It is provided with.

  According to the imaging system of the present invention, it is possible to provide an imaging system that can be easily reduced in size and simplified.

Here, the illuminating device changes uniformly in a predetermined first direction along the surface, and the subject is irradiated with uniform light in a predetermined second direction along the surface that intersects the first direction. Illuminate
The background replacement device includes a subject confirmation unit that analyzes each line extending in the first direction in the captured image acquired by the captured image acquisition unit and confirms whether or not a subject exists on the line. And
The background estimation unit of the background replacement device uses the pixel value of the background for the line for which the presence of the subject has been confirmed by the subject confirmation unit, and the pixel value of the line for which the absence of the subject has been confirmed by the subject confirmation unit. It is preferable to use it for estimation.

  In this way, it is possible to provide an imaging system that is further miniaturized and simplified, in which background replacement with higher accuracy is performed.

Further, the illumination device illuminates the subject with light having a predetermined uniformity in brightness along the surface,
The background estimation unit may estimate a pixel value representing the brightness of the background on the assumption that the brightness of the background has the uniformity in the captured image.

  According to the present invention, it is possible to provide a background replacement device that can reliably distinguish a subject and a background even in a photographed image under simple photographing conditions, and a photographing system that can be easily reduced in size and simplified. Can do.

  Embodiments of the present invention will be described below.

  FIG. 1 is a diagram illustrating an imaging system.

  The photographing system 1 shown in FIG. 1 distinguishes between a photographing set 10 for photographing a subject and obtaining a photographed image, and the subject and background in the photographed image, and replaces the background in the photographed image with a desired background. A personal computer 20 that creates a background replacement image, a printer 30 that outputs the background replacement image created by the personal computer 20, and a server 40 in which the replacement background image is stocked. The photographing system 1 is an embodiment of the photographing system of the present invention, and the personal computer 20 is a first embodiment of the background replacement device of the present invention.

  The photographing set 10 includes a digital camera 11, a first flash light emitting device 12, and a photographing stand 13, and a second flash light emitting device 130 is provided inside the photographing stand 13.

  The digital camera 11 has a continuous shooting function. When the photographer presses the shutter button once, the digital camera 11 automatically photographs the subject P twice in a short time. A photographed image obtained by each photographing is temporarily stored in a memory in the digital camera 11, but in the photographing system 1, the digital camera 11 is connected to the personal computer 20, and two photographings are performed. At the point of time, the photographed image obtained by the two photographings is immediately sent to the personal computer 20. Further, the digital camera 11 can individually transmit an instruction signal for instructing flash emission to the first flash light emitting device 12 and the second flash light emitting device at the time of shooting. In the first shooting, the instruction signal is transmitted only to the first flash light emitting device 12, and in the second shooting, the instruction signal is transmitted only to the second flash light emitting device 130. .

  The imaging stand 13 includes a milky white plastic plate 131 on which a subject P is placed, a casing 132 that holds the plastic plate 131, and a second flash light emitting device 130 that is built in the casing 132. I have. On the plastic plate 131, another bowl, which is the subject P, is arranged on the back side.

  Here, in the photographing set 10 of the photographing system 1, as described above, the first photographing is performed in a state where only the first flash light emitting device 12 emits a flash, and the second photographing is performed in the second flash light emitting device. Only 130 is performed with a flashing light. As a result, in the first photographing, a photographed image in which the subject P is clearly captured is obtained by photographing in a direct light state where the subject P is illuminated from the digital camera 11 side. Further, in the second shooting performed continuously after the first shooting, the flash light from the second flash light emitting device 130 is emitted from the back side of the curved milky white plastic plate 131 as shown in FIG. Thus, light having uniform brightness is emitted from the plastic plate 131 in the depth direction and the vertical direction in FIG. 1, but the relationship between the orientation of each portion on the curved plastic plate 131 and the orientation of the digital camera 11. Thus, on the captured image, the subject is biased toward the shadow side, whereas the background is uniform in the left-right direction and uniformly darker in the vertical direction downward.

  In the personal computer 20, a background replacement image is created based on the above-described two captured images transmitted from the digital camera 11. The creation of the background replacement image will be described later.

  FIG. 2 is an external perspective view of the personal computer shown in FIG.

  The personal computer 20 shown in FIG. 2 has a loading port for a flexible disk (hereinafter referred to as FD) and a CD-ROM due to its external configuration, and has an access function to the FD and CD-ROM loaded in the loading port. Main body device 210, image display device 220 that displays an image on display screen 220a according to an instruction from main body device 210, keyboard 230 that inputs various information according to key operations to main body device 210, display screen 220a By designating an arbitrary position above, a mouse 240 for inputting an instruction corresponding to, for example, an icon or the like displayed at that position, and a small storage medium used for storing a photographed image by a digital camera or the like are loaded And a media drive 250 for accessing the small storage medium.

  The printer 30 prints an image sent from the personal computer 20, and the imaging system 1 plays a role of printing a background replacement image created by the personal computer 20.

  The server 40 shown in FIG. 1 stores a plurality of background candidates that become replacement backgrounds in the background replacement executed by the personal computer 20. This background candidate is presented by being displayed on the display screen 220a. In addition to the background stored in the server 40, the photographing system 1 also displays a background provided via some input storage medium such as the CD-ROM or the small storage medium. Presented through 220a.

  Next, the flow of work executed in the photographing system 1 shown in FIG. 1 will be described. In the following description, the elements shown in FIG. 1 are referred to by using the reference numerals in FIG.

  FIG. 3 is a diagram showing a flow of work executed in the imaging system of FIG.

  First, when the photographer presses the shutter button of the digital camera 11 after adjusting the focus or exposure of the subject P placed on the plastic plate 131, the subject P is photographed twice consecutively. First, in the first shooting, only the first flash light emitting device 12 is flashed in accordance with an instruction signal transmitted from the digital camera 11, and the subject P is shot (step S1). Subsequently, the second shooting is performed by flashing only the second flash light emitting device 130 (step S2).

  The two captured images obtained by each of these two shootings are temporarily stored in the memory in the digital camera 11 and then transmitted to the personal computer 20. Further, the background desired by the customer is read from the server 40 or some input storage medium by an operation on the personal computer 20 (step S3). Subsequently, a background replacement process, which will be described in detail later, is executed to create a background replacement image based on these two captured images (step S4). The image data representing the background replacement image created by the background replacement processing is first transferred to the image display device 220, and further to at least one of the printer 30 and the output storage medium desired by the customer. Transferred (step S5). Subsequently, the background replacement image is displayed on the display screen 220a (step S6).

  Further, when the image data is transferred to the printer 30, the background replacement image is printed based on the image data in the printer 30 (step S7).

  Hereinafter, the details of the personal computer 20 that executes these processes and the background replacement process in the personal computer 20 will be described.

  First, the internal configuration of the personal computer 20 will be described.

  FIG. 4 is a hardware configuration diagram of the personal computer shown in FIG.

  As shown in FIG. 4, inside the main unit 210, a CPU 211 that executes various programs, a hard disk device 213 storing various programs and data, and a program stored in the hard disk device 213 are read out by the CPU 211. 1 is loaded with a main memory 212, an FD 520 that accesses the loaded FD 520, a CD-ROM drive 215 that accesses a CD-ROM 510, the server 40 and the digital camera 11 in FIG. And an input interface 216 that receives a replacement background, a captured image, and the like from these devices, and an output interface 217 that is connected to the printer 30 of FIG. 1 and outputs the background replacement image and the like. These various elements, and the image display device 220, the keyboard 230, the mouse 240, and the media drive 250 for accessing the small storage medium 530, which are also shown in FIG.

  Here, the CD-ROM 510 stores a background replacement program for operating the personal computer 20 as an example of the background replacement device according to the present invention. When the CD-ROM 510 storing the background replacement program is loaded into the CD-ROM drive 215, the background replacement program stored in the CD-ROM 510 is uploaded to the personal computer 20 and written to the hard disk device 213. It is. Thereby, the personal computer 20 operates as a background replacement device.

  Next, the background replacement program will be described.

  FIG. 5 is a conceptual diagram of a CD-ROM.

  A CD-ROM 510 shown in FIG. 5 stores a background replacement program for operating the personal computer 20 shown in FIG. 1 or the like as an example of the background replacement device according to the present invention.

  A background replacement program 600 is stored in the CD-ROM 510 shown in FIG. 5, and this background replacement program 600 includes an image acquisition unit 610, a replacement processing unit 620, an image data transfer unit 630, and an image display unit 640. ing.

  Details of each part of the background replacement program 600 will be described together with the operation of each part of an example of the background replacement apparatus according to the present invention. In the following description, the elements shown in FIG. 1, FIG. 4, and FIG.

  FIG. 6 is a functional block diagram showing functions when the personal computer operates as an example of the background replacement device according to the present invention. When the background replacement program shown in FIG. 5 is installed in the personal computer shown in FIG. 1 or the like, the personal computer operates as an example of the background replacement device according to the present invention.

  6 also shows the photographing set 10 including the digital camera 11, the first flash light emitting device 12, and the second flash light emitting device 130, the printer 30, and the server 40, which are also shown in FIG.

  In the background replacement device 700 illustrated in FIG. 6, as described above, two captured images captured by the digital camera 11 of the capturing set 10 are acquired, and the second flash light emitting device 130 among these two captured images is acquired. Based on a photographed image obtained by flashing only the first flash light emitting device 12 (hereinafter, this photographed image is referred to as a backlight image), a photographed image obtained by flashing only the first flash light emitting device 12 (hereinafter, this photographed image). An image is referred to as a normal image.) A background replacement process is performed in which a subject area and a background area are distinguished and the background area is replaced with a desired background. The background replacement apparatus 700 includes an image acquisition unit 710, A background replacement processing unit 720, an image data transfer unit 730, and an image display unit 740 are provided.

  The image acquisition unit 710 first receives RGB values representing two captured images that are captured by the digital camera 11 and transmitted from the digital camera 11. The image acquisition unit 710 reads a background desired by the customer from the server 40 or from an input storage medium 540 such as a CD-ROM or a small storage medium. In this image acquisition unit 710, the CPU 211 of the personal computer 20 substantially controls the input interface 216, the FD drive 214, the CD-ROM drive 215, and the media drive 250 in accordance with the image acquisition unit 610 of the background replacement program 600. It is composed by doing.

  The background replacement processing unit 720 creates a background replacement image based on the two captured images (backlight image and normal image) acquired by the image acquisition unit 710. The background replacement processing unit 720 is substantially configured by the CPU 211 of the personal computer 20 operating according to the replacement processing unit 630 of the background replacement program 600.

  In the image data transfer unit 730, image data representing the background replacement image created by the background replacement processing unit 720 is sent to at least one of the printer 30 and the output storage medium 550 desired by the customer, and the background replacement device 700. The image is transferred to the image display unit 740. In the image data transfer unit 730, the CPU 211 of the personal computer 20 is substantially controlled by the output interface 217, the FD drive 214, the CD-ROM drive 215, and the media drive 250 in accordance with the image data transfer unit 630 of the background replacement program 600. It is configured by controlling.

  In the image display unit 740, the background replacement image represented by the image data transferred from the image data transfer unit 730 is displayed on the display screen 220a. The image display unit 740 is substantially configured such that the CPU 211 of the personal computer 20 It is configured by controlling the image display device 220 according to the image display unit 640 of the replacement program 600.

  When image data representing a background replacement image is output from the image data transfer unit 730 to the printer 30, the printer 30 prints a background replacement image based on the image data, and the image data transfer unit When image data representing a background replacement image is output from 730 to any output storage medium 550, the image data is written to the output storage medium 550. Then, both or one of the print paper on which the background replacement image is printed and the output storage medium 550 on which the image data representing the background replacement image is written is provided according to the customer's request.

  Next, details of the background replacement processing unit 720 will be described.

  FIG. 7 is a diagram showing details of the background replacement processing unit shown by one block in FIG.

  The background replacement processing unit 720 illustrated in FIG. 7 includes a background estimation unit 721, a subtraction unit 722, an adjustment unit 725, a binarization unit 723, and a background replacement creation unit 724. Hereinafter, among the RGB value groups of the two captured images acquired by the image acquisition unit 710, an RGB value group representing a captured image (normal image) captured using only the first flash light emitting device 12 is used. The RGB value group representing the captured image (backlight image) captured using only the second flash light emitting device 130 is referred to as a first RGB value group. The background RGB values desired by the customer are referred to as a background RGB value group. The third RGB value group shown in FIG. 7 will be described later.

  Here, FIG. 8 is a diagram conceptually showing an image generated by each unit shown in FIG.

  FIG. 8 (a) shows an image (backlight image) represented by the second RGB value group, which includes a background having a uniform gradation from the top to the bottom and an upper portion of the background. A subject close to the shadow side is shown. FIG. 8B shows a background image represented by the third RGB value group created by estimating the background behind the subject based on the second RGB value group in the background estimation unit 721 shown in FIG. Has been. FIG. 8C shows an image represented by the RGB value group as a result of subtracting each value of the third RGB value group from the second RGB value group in the subtracting unit 722. A state in which the subject P slightly appears on the black background is shown. FIG. 8D shows the processed RGB value group that has been subjected to binarization processing by the binarization unit 723 after the RGB value group resulting from the subtraction has been adjusted by the adjustment unit 725. An image is shown.

  Hereinafter, the operation of each unit illustrated in FIG. 7 will be described in more detail with reference to FIGS.

  First, the second RGB value group representing the captured image shown in FIG. 8A is input to the background estimation unit 721 shown in FIG. 7, and the second RGB value group shown in FIG. The RGB values of the area behind the subject P are estimated from the backlit captured image.

  FIG. 9 is a diagram conceptually showing a method of estimating the RGB value of the region behind the subject. The image shown in FIG. 9 is omitted from the illustration of brightness and darkness, but the description will be made assuming that the image is the same as FIG.

  In the background estimation unit 721, first, the luminance in each pixel from the pixel located at the upper end of the first column on the left side of the image shown in FIG. The luminance of the pixels from which the luminance is read out after this is read out by extrapolation, and the estimated luminance (hereinafter, this estimated luminance is referred to as estimated luminance) and the actually read luminance (hereinafter, this read out luminance) The luminance is referred to as “read luminance”.) Is determined whether there is a deviation of a predetermined value or more, and the luminance reading, luminance estimation, and determination are performed from the left end to the right end of the photographed image shown in FIG. Do it. As a result, a column composed of pixels in which the difference between the read luminance and the estimated luminance is all less than a predetermined value is recognized as a column that passes only through the background area. On the other hand, a column in which a pixel in which the difference between the read luminance and the estimated luminance is greater than or equal to a predetermined value is recognized as a column that passes through the subject region, and the luminance of the pixel belonging to the subject region on the column is The read luminance of the corresponding pixel in the column passing through the background region immediately before the column passing through the subject region is replaced. In this way, the luminance of the area behind the subject P is determined by estimation. In the estimation of the luminance of the area behind the subject P, the luminance reading starts from the upper end of the photographed image because the background of the photographed image photographed in the photographing set 10 is bright upward and downward as described above. For example, even when the background edge is shown as a black frame in the background area, pixels whose brightness is equal to or higher than the predetermined threshold are excluded while excluding dark portions below the predetermined threshold. This is because it can be the pixel at the beginning of the. The background estimation unit 721 corresponds to a combination of an example of the background estimation unit according to the present invention and an example of a subject confirmation unit.

  FIG. 10 is a diagram illustrating the relationship between the vertical position on the captured image and the read luminance.

  FIG. 10A shows a column belonging to the background region of this image when the horizontal axis indicates the vertical position on the backlight image shown in FIG. 9 and the vertical axis indicates the luminance (0 to 255). The relationship between the vertical position and the readout brightness is shown. Here, the readout brightness on the “bright” side close to 255 at the top of the column gently decreases toward the “dark” side near 0 toward the bottom of the column. It shows how it changes. Further, the relationship between the vertical position and the readout luminance in the representative columns A, B, C, E, F, and G among the columns passing through the background area shown in FIG. 10 (a) shows a state in which the plots are overlaid at substantially the same position. This means that the background of the captured image shown in FIG. 9 is uniform in the left-right direction.

  FIG. 10B shows the relationship between the vertical position in the column D and the readout luminance obtained for the column D typically extracted from the column passing through the subject area. For this column D, the pixels before position a belong to the background area, but the pixels between position a and position b belong to the subject area. Is greater than or equal to a predetermined value. For this reason, the read luminance within this range is replaced with the read luminance in the corresponding range of column C, which is the closest column among columns passing only the background region. In FIG. 10B, the range in which the read luminance is replaced is indicated by a dotted line. As described above, the background estimation unit 721 creates the third RGB value group that is the RGB value group of the entire background after estimating the luminance of the pixels behind the subject P. In the subtraction unit 722 shown in FIG. 7, the third RGB value group is subtracted from the second RGB value group.

  In the conventional background replacement technique, it is a requirement for the background replacement processing to be performed with high accuracy that the background and the subject have extremely different colors and brightness. Therefore, if there is no obvious difference in the color or brightness at the boundary between the subject and the background, the accuracy of distinguishing between the background and the subject is lowered. However, the background replacement processing unit 720 subtracts the third RGB value group representing the background image using the estimation behind the subject P from the second RGB value group obtained by photographing the subject with the backlight. The subject area and the background area are distinguished based on the obtained difference. That is, according to the background replacement processing unit 720, even if there is no clear difference in the color or brightness of the boundary between the subject and the background, the RGB value group of the subject region and the background region hidden behind the subject region are estimated. The subject area and the background area can be distinguished if there is a slight difference from the RGB value group.

  In the adjustment unit 725 shown in FIG. 7, the operator can adjust the RGB value group after the third RGB value group is subtracted from the second RGB value group.

  Here, FIG. 11 is a diagram showing a screen display of the image display device of the computer.

  In the computer 20 which is the first embodiment of the background replacement device of the present invention, the operator removes noise components by reflection of light on the RGB value group after the third RGB value group is subtracted from the second RGB value group. Can be manually performed via the adjustment unit 725, and FIG. 11A shows a screen display when this manual adjustment is performed.

  FIG. 11A shows a screen 220a on which an image represented by the RGB value group after the third RGB value group is subtracted from the second RGB value group is displayed. The screen 220a includes an input field 221a. , An “execute” button 222a and an “OK” button 223a are also displayed.

  In the input field 221a shown in FIG. 11A, a threshold value for noise component removal is input. For example, when the value “10” is input to the input field 221a and the “execute” button 222a is operated, first, an isolated pixel group having a different RGB value from that of the surrounding area is detected. In the pixel group in which the number of pixels constituting the group is equal to or less than the value specified in the input field 221a ('10' in the example shown here), the pixel group is assumed to be a noise component, and the RGB value of the pixel group is the surrounding area. It is replaced with the same RGB value as the RGB value of this pixel. As described above, in the computer 20, the operator can change the number of pixels serving as a criterion for removing noise components from the RGB value group after the third RGB value group is subtracted from the second RGB value group. The “OK” button 223a shown in FIG. 11A is a button operated when sending the RGB value group from which the noise component is removed to the binarization unit 723 (see FIG. 7).

  Thereafter, on the screen 220a, as shown in FIG. 11B, RGB values obtained by performing binarization processing in the binarization unit 723 on the RGB value group from which the noise component is removed by the adjustment unit 725 are displayed. An image represented by the group is displayed.

  The binarization unit 723 reads the luminance from the RGB value group sent from the adjustment unit 725 and from which the noise component has been removed. The luminance less than the threshold A is set to “0”, and the luminance greater than the threshold A is obtained. Has a luminance of '255'. As this threshold value, an intermediate value between the maximum luminance and the minimum luminance in the distribution of the appearance frequency of luminance read from the RGB value group from which the noise component has been removed is set by default.

  Here, in the computer 20, the operator can manually adjust the binarization processing for the RGB value group after the noise component removal processing is performed by the adjustment unit 725 via the adjustment unit 725. FIG. 11B shows a screen display when this manual adjustment is performed.

  A 'histogram' button 224a shown in FIG. 11B is a button operated when the luminance of the RGB value group sent from the adjustment unit 725 is represented by the appearance frequency distribution.

  FIG. 11C shows a screen display when the 'histogram' button 224a is operated. Here, the distribution of the appearance frequency of the luminance read from the RGB value group sent from the adjustment unit 725 is shown, and is the maximum luminance in the luminance distribution of the RGB value group from which the noise component has been removed. A straight line S in the figure shows that “A” which is an intermediate value between “2A” and the value “0” which is the minimum luminance is the current threshold value. The operator can change the threshold value by moving left and right using the mouse 240 while pinching the straight line S. The “execute” button 222a shown in FIG. 11B is operated when the binarization process is performed based on a threshold value newly set by the operator. Also, an “OK” button 223 a shown in FIG. 11B is a button operated when sending the binarized RGB value group to the background replacement image creating unit 724. A “decision” button 225a shown in FIG. 11C is a button operated to move the straight line S using the mouse 240 and set it as a threshold value. The 'return' button 226a is a button operated when returning from the screen display shown in FIG. 11C to the screen display shown in FIG. 11B.

  Regarding the luminance of the RGB value group binarized by the binarization unit 723, the luminance below the threshold is set to “0” and the luminance above the threshold is set to “255” based on the set threshold. In the image, they are represented by 'solid black' and 'white', respectively.

  In the background replacement image creating unit 724 shown in FIG. 7, the RGB value of the subject area whose luminance is set to “255” by the binarizing unit 723 is replaced with the corresponding RGB value of the first RGB value group, and binarized. The RGB value of the background area whose luminance is set to “0” by the unit 723 is replaced with the corresponding RGB value of the background RGB value group desired by the customer. As a result, a background replacement image is created in which a subject photographed beautifully with direct light and a background desired by the customer are combined.

  The RGB value group representing the background replacement image created in this way is transmitted to the image data transfer unit 730 shown in FIG. 6 and transferred to each output device and output storage medium 550 (see FIG. 6).

  As described above, in the computer 20 of this embodiment, light having a predetermined uniformity along the plastic plate 131 from the plastic plate 131 located on the opposite side of the digital camera 11 with the subject P interposed therebetween. The background behind the subject P is estimated based on the backlight image obtained by photographing the subject P illuminated with the light to obtain the RGB value group of the entire background, and the RGB value group of the entire background from the RGB value group representing the backlight image By subtracting, the subject area is detected with high accuracy and background replacement is performed. Therefore, according to the computer 20 of the present embodiment, it is possible to contribute to the realization of a small and simple photographing system 1.

  Next, the personal computer which is the second embodiment of the background replacement device of the present invention, which can be replaced with the personal computer 20 which is the first embodiment of the background replacement device of the present invention provided in the photographing system 1. The computer 21 will be described.

  Since the difference between the personal computer 21 of the second embodiment and the personal computer 20 of the first embodiment is in the background estimation method behind the subject, the background estimation in the personal computer 21 of the second embodiment will be described below. Only the method will be described.

  FIG. 12 is a diagram showing a backlight image that is a target of background estimation in the personal computer that is the second embodiment of the background replacement device of the present invention.

  FIG. 12 shows a backlight image in which one subject is arranged at the center of the angle of view.

  In the personal computer 21 according to the second embodiment, from the input second RGB value group representing the backlight image shown in FIG. 12, first, the pixel located at the upper end of the first column on the left side of the image shown in FIG. Sequentially read the luminance in each pixel from the pixel located at the lower end to approximate the curve based on the read luminance for one column read, and the average value of the difference between the approximate curve and the read luminance is less than a predetermined value The luminance reading, curve approximation, and determination are performed from the leftmost column to the rightmost column of the photographed image shown in FIG. As a result of this determination, if the average value is less than the predetermined value, the column is recognized as a column that passes only the background region, and if the average value is equal to or greater than the predetermined value, the column is recognized as a column that passes through the subject region. . The luminance of the pixel belonging to the subject area on the column passing through the subject area is replaced with the readout luminance of the corresponding pixel in the column passing only the background area immediately before the column passing through the subject area. In the personal computer 21 of the second embodiment, the luminance of the area behind the subject P is estimated in this way using the approximate curve.

  Next, the personal computer which is the third embodiment of the background replacement device of the present invention, which can be replaced with the personal computer 20 which is the first embodiment of the background replacement device of the present invention provided in the photographing system 1. The computer 22 will be described.

  Since the difference between the personal computer 22 of the third embodiment and the personal computer 20 of the first embodiment is also the background estimation method behind the subject, the subject in the personal computer 22 of the third embodiment will be described below. Only the background estimation method behind will be described.

  In the personal computer 22 according to the third embodiment, first, the luminance readout, curve approximation, and determination performed by the personal computer 21 according to the second embodiment are performed from the left end to the right end and from the right end to the left end. In both cases, the process is repeated until a column in which the average value of the differences is equal to or greater than a predetermined value is detected.

  In the above-described determination performed from the left end to the right end, a column in which the average value of the differences is equal to or greater than a predetermined value (hereinafter, the detected column is referred to as column L1) is detected, and the above-described determination performed from the right end to the left end. If a column in which the average value of the differences is equal to or greater than a predetermined value is detected in the determination (hereinafter, the detected column is referred to as column R1), column 1 and column including these column L1 and column R1 are included. Each column positioned between R1 and R1 is also recognized as a column that passes through the subject area. The luminance of the pixel on each column that passes through the subject area is the column that passes only the background area and is the column closest to the subject area, the column adjacent to the left of the column L1 (hereinafter, this column is referred to as L0). .) And a new luminance (hereinafter, this new luminance is referred to as an estimated luminance) that is weighted and averaged according to the distance ratio from the column on the right side of the column R1 (hereinafter, the column is referred to as R0). .). For example, the estimated luminance of the predetermined pixel x on the column X that belongs to the subject region and whose ratio of the distance from the column L0 to the distance from the column R0 is m: n is the predetermined pixel x on the column L0. Based on the luminance Lx of the corresponding pixel and the luminance Rx of the pixel corresponding to the predetermined pixel x on the column R0, it is calculated by the equation (m * Rx + n * Lx) / (m + n). The reason why the weighted average according to the ratio of distances is performed is to reflect the luminance unevenness of the background area in the left-right direction also in the area where the luminance is estimated. According to the personal computer 22 of the third embodiment, background replacement can be performed with high accuracy when there is one subject as shown in FIG. In the above description, the case where the determination of whether or not the column passes through the subject area is performed based on the average value of the difference from the approximate curve. The determination may be performed based on the maximum value of the difference from the approximate curve. In the third embodiment, the case where the above-described R1 and L1 are detected by the approximate curve described in the second embodiment is taken as an example. However, the detection of R1 and L1 is the same as that described in the first embodiment. Detection by insertion may be employed.

  Next, a personal computer 23 which is a fourth embodiment of the background replacement device of the present invention will be described. The personal computer 23 can also be replaced with the personal computer 20 that is provided in the photographing system 1 and is the first embodiment of the background replacement device of the present invention.

  Since the difference between the personal computer 23 of the fourth embodiment and the personal computer 20 of the first embodiment is also the background estimation method behind the subject, the subject in the personal computer 23 of the fourth embodiment will be described below. Only the background estimation method behind will be described.

  In the personal computer 23 according to the fourth embodiment, the luminance reading, curve approximation, and determination performed by the personal computer 21 according to the second embodiment are performed in addition to the upper end in addition from the left end to the right end and from the right end to the left end. This is performed for both from the lower end to the lower end and from the lower end to the upper end until a row in which the average value of the differences is equal to or greater than a predetermined value is detected.

  In the personal computer 23 according to the fourth embodiment, when the above determination is performed from the upper end to the lower end, the upper row of the row in which the average value of the differences is not less than a predetermined value (hereinafter, this row is referred to as row U0). .), And the distance from a column below the column in which the average value of the differences is equal to or greater than a predetermined value when the above determination is performed from the lower end to the upper end (hereinafter, this column is referred to as column D0). The average of the luminance of each pixel belonging to the subject area, which is weighted averaged based on the above, and the luminance of each pixel belonging to the subject area, which is weighted averaged based on the distance from the above-described L0 and R0, is used as the above-described estimated luminance. . This background estimation in the personal computer 23 according to the fourth embodiment is effective when it is unclear whether the brightness of the background is uniform in the left-right direction or the up-down direction. In the third embodiment, the case where the above-described columns R1, L1, U1, and D1 are detected by the approximate curve described in the second embodiment is taken as an example, but this has been described in the first embodiment. Detection by extrapolation may be employed.

  In the above embodiment, as an example of the predetermined uniformity of the background referred to in the present invention, an example is given in which the brightness is uniform in the horizontal direction and uniform in the vertical direction. As long as the background can be estimated, the predetermined uniformity of the background may be one in which the brightness is uniform in the vertical direction and uniform in the horizontal direction, and is uniform in the vertical and horizontal directions. It may be.

  In the above embodiment, an example of detecting the luminance of a pixel is given as an example of the background estimation according to the present invention. However, the background estimation according to the present invention detects not the pixel luminance but the pixel color. May be. Furthermore, it may be estimated two-dimensionally instead of for each line.

It is a figure which shows an imaging | photography system. FIG. 2 is an external perspective view of the personal computer shown in FIG. 1. It is a figure which shows the flow of the work performed with the imaging | photography system of FIG. It is a hardware block diagram of the personal computer shown in FIG. It is a conceptual diagram of CD-ROM. It is a functional block diagram showing a function when a personal computer operates as an example of the background replacement device according to the present invention. It is a figure which shows the detail of the replacement process part shown by one block in FIG. It is a figure which shows the image produced | generated by each part shown by FIG. 7 notionally. It is a figure which shows notionally the method of estimation of the RGB value of the area | region behind a subject. It is a figure which shows the relationship between the up-and-down position on a picked-up image, and a brightness | luminance. It is a figure which shows the screen display of the image display apparatus of a computer. It is a figure which shows the backlight image which is the object of the background estimation in the personal computer which is 3rd Embodiment of the background replacement apparatus of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Shooting system 10 Shooting set 11 Digital camera 12 First flash light emitting device 13 Shooting stand 130 Second flash light emitting device 131 Plastic plate 132 Case 20 Personal computer 600 Background replacement program 610 Image acquisition unit 620 Replacement processing unit 630 Image data transfer unit 640 Image display unit 700 Background replacement device 710 Image acquisition unit 720 Background replacement processing unit 721 Background estimation unit 722 Subtraction unit 723 Binarization unit 724 Background replacement image creation unit 725 Adjustment unit 730 Image data transfer unit 740 Image display unit

Claims (8)

A captured image acquisition unit that acquires a captured image obtained by capturing an image of a subject with an imaging device in a format that represents the image with a set of pixel values;
A background estimation unit that estimates a pixel value of the background including a background hidden behind the subject on the assumption that the background of the subject has a predetermined uniformity in the captured image;
A background subtraction unit that subtracts the pixel value estimated by the background estimation unit from each pixel value of the captured image;
Based on the subtraction result by the background subtraction unit, the area in the captured image obtained by photographing the same subject as the subject by the photographing apparatus is excluded from the subject area in which the subject is reflected and the subject area. An area discriminator for distinguishing from the background area;
A background replacement device, comprising: a background replacement unit that replaces a background in the captured image with another background based on the region discrimination by the region discrimination unit.   The region distinguishing unit compares the subject region and the background region by comparing a subtracted pixel value obtained by subtracting the background pixel value from each pixel value of the photographed image by the background subtracting unit and a predetermined threshold value. The background replacement device according to claim 1, wherein the background replacement device is distinguished. Analyzing each line extending in a predetermined direction in the captured image acquired by the captured image acquisition unit, and a subject confirmation unit that confirms whether or not a subject exists on the line,
The background estimation unit uses the pixel value of the background for the subject line for which the presence of the subject has been confirmed by the subject confirmation unit, and the pixel value of the background line for which the absence of the subject has been confirmed by the subject confirmation unit. The background replacement device according to claim 1, wherein the background replacement device is an estimation device.   4. The background replacement device according to claim 3, wherein the background estimation unit estimates a pixel value of a background of the subject based on the background lines on both sides sandwiching the subject.   The background estimation unit estimates a pixel value representing the brightness of the background on the assumption that the brightness of the background has a predetermined uniformity in the captured image. The background replacement device according to any one of claims 1 to 4. An imaging device that captures a subject and obtains a captured image in a form in which the image is represented by a set of pixel values;
An illumination device that illuminates the subject with light having a predetermined uniformity along the surface from a surface located on the opposite side of the photographing device across the subject; and photographing obtained by the photographing device A captured image acquisition unit for acquiring images;
A background estimation unit that estimates the pixel value of the background including the back behind the subject on the assumption that the background of the subject has the uniformity in the captured image;
A background subtraction unit that subtracts the pixel value estimated by the background estimation unit from each pixel value of the captured image;
An area discriminating unit for discriminating an area in the captured image into a subject area in which the subject is reflected and a background area excluding the subject area, based on a subtraction result by the background subtraction unit;
A background replacement device comprising a background replacement unit that replaces a background in a captured image obtained by capturing the same subject as the subject with the imaging device based on the region distinction by the region distinction unit. ;
An imaging system characterized by comprising: The illumination device uniformly changes in a predetermined first direction along the surface, and illuminates the subject with uniform light in a predetermined second direction along the surface and intersecting the first direction. And
The background replacement device includes a subject confirmation unit that analyzes each line extending in the first direction in the captured image acquired by the captured image acquisition unit and confirms whether or not a subject exists on the line. And
The background estimation unit of the background replacement device uses the pixel value of the background for the line where the presence of the subject is confirmed by the subject confirmation unit, and the pixel value of the line where the absence of the subject is confirmed by the subject confirmation unit. The photographing system according to claim 6, wherein the photographing system is used for estimation. The illumination device illuminates the subject with light having a predetermined uniformity of brightness along the surface;
The background estimation unit estimates a pixel value representing the brightness of the background on the assumption that the brightness of the background has the uniformity in the captured image. The imaging system according to claim 6 or 7.

Images