JP2012252648A - Image processing device, image processing method, program, imaging device, and television receiver - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of extracting only a specific subject without the need for preparation in advance even when there are a plurality of subjects having the same depth in an image.SOLUTION: A television receiver 1 acquires an original image having distance information related to a distance in the image depth direction on each of a plurality of positions on the image, and an image to be replaced with a part of the original image and synthesizes an extracted image of a part of the acquired original image and the image to be replaced. The television receiver 1 comprises: a threshold value setting part that sets a threshold value for the information on the basis of each position on the original image, and an extracted image generation part that extracts a part of the original image on the basis of the set threshold value and the distance information and outputs the part as an extracted image.

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, an imaging apparatus, and a television receiver that extract a part of an original image and combine the extracted image with another image.

When a still image or a moving image is acquired by an imaging apparatus, an image obtained by capturing a landscape or an image acquired as a commemorative image does not normally require processing for extracting only the subject and replacing the background and other processing.
On the other hand, instead of using the acquired image as it is, there is a desire to extract a specific subject and replace an image of other parts such as a background image. For example, in a videophone or a video conference, it is only necessary that both faces are displayed, and a background image is basically unnecessary. In particular, when the background area is not neatly arranged, it is desirable to remove the background image.
Also, in a machine that photographs an image of itself and decorates it with letters and patterns as a sticker, such as a print sticker machine, a curtain is attached to the machine itself to block the background. This is to prevent the background from getting in the way, and there are many requests to separate the background from the person.
From such a background, a method of extracting a specific subject from an image and separating it from the background has been proposed.

  For example, a subject such as a person is photographed with one type of color (blue) as a background, and only a portion of a color different from the background color is extracted, so that only the background can be extracted in advance. This is a technique for obtaining an image of only a subject by obtaining a difference between an image obtained by capturing an image of the background and the subject and an image of only the background.

  There is also a method that employs a device that acquires the distance from the camera to the object to be imaged and uses the obtained depth value from the acquired distance. The depth value is a value indicating the distance in the depth direction of each of a plurality of image parts constituting the image. The depth value is calculated using, for example, the distance between the subject and the imaging device used for imaging. Methods for measuring the distance between the imaging device and each subject include, for example, a TOF (Time Of Flight) method, a stereo method, and the like.

The TOF method measures the distance by irradiating invisible light such as infrared rays from a light source such as an LED (Light Emitting Diode), and measuring the time of flight when the light hits the subject and returns. It is. By measuring the measurement for each finely divided area, it is possible to measure not only one point but also various parts of the subject.
In addition, the measurement of the time of flight is a method of measuring the time from the pulse irradiation of the laser light, the time from when the pulse is emitted until the reflected light returns, the phase when the irradiation infrared light is modulated and irradiated, There is a method of calculating from the phase difference with the reflected light.

  In the stereo method, the same region is imaged by two imaging devices arranged substantially in parallel, the parallax of corresponding pixels in the two obtained images is obtained, and the distance is calculated based on the parallax. As another measurement method, there is a method of measuring a distance from the intensity of irradiated infrared rays and the intensity of reflected light.

  In the stereo method, obtaining corresponding pixels in two images is called stereo matching. For example, the following processing is performed. Pixel matching is performed by scanning a pixel of one captured image in the horizontal direction on the other captured image. Pixel matching is performed in units of blocks centered on the pixel of interest, and a SAD (Sum of Absolute Difference) that calculates the sum of absolute value differences of pixels in the block is calculated to determine a block that minimizes the SAD value. Thus, the pixel on the other captured image corresponding to the target pixel of the one captured image is obtained. In addition to SAD calculation methods, there are calculation methods such as SSD (Sum of Squared Intensity Difference), graph cut, and DP (Dynamic Programming) matching. When the corresponding pixel is obtained, the parallax value of the pixel can be calculated.

  The parallax value of all pixels is calculated by performing for all the pixels of one captured image, and the distance from the imaging device to the subject is calculated by performing triangulation using the positional relationship and the parallax value of the two imaging devices. it can. In addition, the parallax value can be calculated even when the two imaging units are arranged in the vertical direction instead of the horizontal direction. In this case, scanning of the captured image may be performed in the vertical direction instead of the horizontal direction.

  28 and 29 are explanatory diagrams illustrating an example of image extraction according to the conventional technique. Assume that the subject F33 is extracted from an image made up of subjects F31, F32, F33, and F34 as shown in FIG. FIG. 29 is a diagram showing the positional relationship between the subjects F31 to F34, and as can be seen from FIG. 29, F33 is positioned closest to the front.

FIG. 30 is an explanatory diagram conceptually showing distance information between the imaging device and the subject. This is a conceptual representation of what is called a depth map. Normally, the depth value associated with each pixel is represented using a 256-tone black-and-white image. It is displayed using hatching.
In the example of FIG. 30, the depth values are indicated by numbers in ascending order of 1, 2,. As is clear from the correspondence between hatching and depth values, the positional relationship shown in FIG. 29 in which the subject F33 is positioned closest to the front and the subject F34 is positioned deepest is represented.

  For example, when it is desired to extract only the subject F33 from the image shown in FIG. 28, a portion having a depth value smaller than the threshold (front portion) is set with the depth represented by the dotted line B0 shown in FIG. By extracting the subject to be extracted and the portion having the depth value larger than the threshold (the back portion) as the background, the subject F33 and the other subjects F31, F32, F34, and the background can be separated. .

  Patent Document 1 discloses a distance corresponding to a captured image by obtaining a captured image by capturing a captured image in the imaging range and simultaneously measuring the distance to the captured image in a plurality of points within the captured range. There has been disclosed a method of obtaining distance information expressing a distribution, separating a person and a background in an imaging range based on the distance information, and extracting only a person image from the captured image.

JP 2001-167276 A

  However, the chroma key technique and the method of obtaining a difference from the background image require a preparatory preparation or a limited shooting environment. The method according to Patent Document 1 can separate and extract a subject existing in front of a certain depth and a background existing in the depth by looking at distance information (depth value). It is impossible to separate the subject existing in the background and separate it from the background. When a plurality of persons are present at the same depth from the camera, even if one person is to be extracted, another person existing at the same depth is also extracted. Also, even if another subject that is not related to the image has the same value as the depth of another subject that is not related to the depth of the person to be extracted, There was a problem of being extracted.

  The present invention has been made in view of the above circumstances, and it is possible to extract only a specific subject even if there are a plurality of subjects having the same depth in the image without any prior preparation. An object of the present invention is to provide an image processing apparatus, an imaging apparatus or a television receiver, an image processing method in the image processing apparatus, and a program for causing a computer to operate as the image processing apparatus.

  The image processing apparatus according to the present invention acquires an original image having distance information related to a distance in the depth direction of the image at each of a plurality of positions on the image, and a replacement image that replaces a part of the original image. In an image processing apparatus that combines an extracted image obtained by extracting a part of an image and the replacement image, a threshold setting unit that sets a threshold for the distance information based on each position on the original image, And an extracted image generating unit that generates the extracted image based on the distance information.

  The image processing apparatus according to the present invention is configured to replace an image portion that has not been extracted by the extraction unit with the replacement image.

  The image processing apparatus according to the present invention further includes a region dividing unit that divides the original image into a plurality of regions, the threshold setting unit sets a threshold for each divided region, and the extracted image generation unit An extracted image is generated for each of the areas.

  The image processing apparatus according to the present invention further includes a region dividing unit that divides the original image into a plurality of regions, the threshold setting unit sets a threshold for each divided region, and the extracted image generation unit An extracted image is generated for each of the areas, a plurality of replacement images are acquired, and different replacement images are used for the divided areas.

  The image processing method according to the present invention acquires an original image having distance information related to the distance in the depth direction of the image at each of a plurality of positions on the image, and a replacement image that replaces a part of the original image. In the image processing method for synthesizing an extracted image obtained by extracting a part of an image and the replacement image, a step of setting a threshold for the distance information based on each position on the original image, the set threshold and the distance Generating the extracted image based on the information.

  A program according to the present invention causes a computer to acquire and acquire an original image having distance information related to the distance in the depth direction of the image at each of a plurality of positions on the image and a replacement image that replaces a part of the original image. In the program for synthesizing the extracted image obtained by extracting a part of the original image and the replacement image, the computer sets a threshold for the distance information based on each position on the original image; And generating the extracted image based on the threshold value and the distance information.

  An imaging apparatus according to the present invention includes an imaging unit that images a subject and a replacement image storage unit that stores a replacement image, and an extracted image obtained by extracting a part of the captured image and a part of the captured image In the imaging device for synthesizing the replacement image that is replaced with the distance measurement unit that measures the distance to the subject at a plurality of positions, and interpolating the distance values of the plurality of positions measured by the distance measurement unit, the imaging A distance information interpolation unit that calculates distance information related to the distance in the depth direction of the image at each of a plurality of positions on the captured image, and a threshold setting unit that sets a threshold for the distance information based on each of the positions on the captured image And an extracted image generation unit that generates the extracted image based on the set threshold and the distance information.

  An imaging apparatus according to the present invention includes a plurality of imaging units that image a subject and a replacement image storage unit that stores a replacement image, an extracted image obtained by extracting a part of the captured image, and a captured image In the imaging apparatus for synthesizing a replacement image that is replaced with a part, a plurality of images captured by each of the plurality of imaging units are used to relate to the distance in the depth direction of each of the plurality of positions on the captured image. A distance information calculation unit that calculates distance information; a threshold setting unit that sets a threshold for the distance information based on each position on the captured image; and the extracted image based on the set threshold and the distance information. And an extracted image generation unit for generation.

  A television receiver according to the present invention includes an image processing apparatus according to any one of the above, a tuner unit that receives a television broadcast, and a display that displays an image related to the television broadcast received by the tuner unit. And the display unit displays an image acquired, extracted, generated or synthesized by the image processing apparatus.

  A television receiver according to the present invention includes an imaging device according to any one of the above, a tuner unit that receives a television broadcast, and a display unit that displays an image related to the television broadcast received by the tuner unit. And the display unit displays an image captured, extracted, generated or synthesized by the imaging device.

  In the present invention, a threshold for distance information is set based on each position on the original image, and an extracted image is generated based on the set threshold and the distance information. Therefore, there are a plurality of subjects having the same depth value. However, since the threshold value at the position on the image where the specific subject exists is different from the threshold value at the position of the image where the other subject exists, an image of only the specific subject may be extracted. It becomes possible.

  In the present invention, the image portion that has not been extracted is configured as a replacement image. Therefore, it is possible to generate a new image using the background image of the original image as a replacement image.

  In the present invention, the original image is divided into a plurality of regions, a threshold is set for each divided region, and an extracted image is generated for each divided region. When it is desired to extract a plurality of subjects, it is possible to easily extract a plurality of subjects by dividing the region for each subject to be extracted.

  In the present invention, the original image is divided into a plurality of regions, a threshold is set for each divided region, an extracted image is generated for each divided region, a plurality of replacement images are acquired, and each divided region is Then, image composition is performed using different replacement images. Therefore, it is possible to generate an image in which different background images for each subject are combined.

  In the present invention, a threshold setting unit that sets a threshold for distance information based on each position on the captured image, and extracts a part of the captured image based on the set threshold and the distance information, An extracted image generation unit that generates an extracted image. Since the threshold is set based on each position on the captured image, even if there are multiple subjects with the same depth value, the threshold at the position of the image where the specific subject exists and the position of the image where other subjects exist Since a value different from the threshold value at is set, it is possible to extract an image of only a specific subject.

  In the present invention, a threshold setting unit that sets a threshold for distance information based on each position on the captured image, and extracts a part of the captured image based on the set threshold and the distance information, An extracted image generation unit that outputs the extracted image. Since the threshold value is set based on the position of each captured image, even if there are multiple subjects with the same depth value, the threshold value at the image location where a specific subject exists and the image location where other subjects exist Since a value different from the threshold value is set, an image of only a specific subject can be extracted.

  In the present invention, since the television receiver includes the image processing device according to any one of the above or the imaging device according to any one of the above, the original image or the imaging unit received from the external device Even if there are multiple subjects with the same depth value in the original image acquired in step 1, the threshold value at the position of the image where the specific subject exists is different from the threshold value at the position of the image where the other subject exists. Therefore, it is possible to extract an image of only a specific subject.

  According to the present invention, even if there are a plurality of subjects located at the same depth in the original image or the captured image, only the image of the specific subject can be extracted from the original image or the captured image.

It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows an example of an original image. It is explanatory drawing which shows the positional relationship of the to-be-photographed object in an original image. It is explanatory drawing which shows the depth map corresponding to an original image. It is explanatory drawing which shows notionally the data structure of a depth map. It is explanatory drawing which shows an extraction range notionally It is explanatory drawing which shows the setting of a threshold value notionally. It is explanatory drawing which shows an example of an extracted image. It is explanatory drawing which shows an example of the synthesized image displayed. It is explanatory drawing in the case of recognizing a subject by face recognition. It is a flowchart which shows the flow of the image processing method which an image processing apparatus implements. It is a flowchart which shows the flow of the threshold value setting process which an image processing apparatus implements. It is explanatory drawing which shows notionally the setting of the threshold value which concerns on the modification 1. FIG. It is explanatory drawing which shows notionally the setting of the threshold value which concerns on the modification 2. FIG. It is a block diagram which shows the structural example of the image processing apparatus which concerns on Embodiment 2 of this invention. It is explanatory drawing which shows the example of the area | region division of the image in Embodiment 3 of this invention. 14 is a flowchart illustrating a flow of processing performed by the image processing apparatus according to the third embodiment of the present invention. It is a flowchart which shows the flow of the process performed with respect to each area | region divided in Embodiment 3 of this invention. It is explanatory drawing which shows the example of a threshold value setting in Embodiment 3 of this invention. It is explanatory drawing which shows the example of an output image in Embodiment 3 of this invention. It is explanatory drawing which shows the example of the user operation screen in the modification 3. It is explanatory drawing which shows the example of the user operation screen in the modification 3. It is explanatory drawing which shows the example of the user operation screen in the modification 3. It is a block diagram which shows the structural example of the imaging device which concerns on Embodiment 4 of this invention. It is a block diagram which shows the structural example of the imaging device which concerns on Embodiment 5 of this invention. It is a block diagram which shows the structural example of the television receiver which concerns on Embodiment 6 of this invention. It is a block diagram which shows the structural example of the television receiver which concerns on Embodiment 7 of this invention. It is explanatory drawing which shows the example of the image extraction by a prior art. It is explanatory drawing which shows the example of the image extraction by a prior art. It is explanatory drawing which shows notionally the distance information of an imaging device and a to-be-photographed object.

(Embodiment 1)
Embodiment 1 of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a hardware configuration of the image processing apparatus according to Embodiment 1 of the present invention. The image processing apparatus 1 according to Embodiment 1 includes a control unit 11, an external interface unit 12, a display interface unit 13, and an operation interface unit 14.

The control unit 11 includes a CPU (Central Processing Unit) 11a that performs calculations, a ROM (Read Only Memory) 11b that stores a control program indicating a control procedure of calculations performed by the CPU 11a, and temporary information associated with calculations performed by the CPU 11a. A RAM (Random Access Memory) 11c is stored.
The control unit 11 functions as a threshold setting unit and an extracted image generation unit of the present invention by the CPU 11a reading out and executing a control program stored in the ROM 11b in the RAM 11c.

  The external interface unit 12 includes a USB (Universal Serial Bus) connector, an HDMI (High-Definition Multimedia Interface) connector, or an IEEE 1394 connector, and has, for example, an imaging apparatus having a distance measuring function including a digital camera and a laser range finder. 2 are connected. The control unit 11 receives an original image and distance information corresponding to the image from the imaging device 2 via the external interface unit 12. The received original image and distance information corresponding to the image are stored in the RAM 11c by the control unit 11.

The display interface unit 13 is connected to the display device 3 such as a television receiver, and causes the display device 3 to display a user operation screen, an original image to be processed, an image after image processing, and the like.
The operation interface unit 14 is connected to an operation device 4 such as a keyboard and a mouse, and receives an operation input from a user via the operation device 4.

Next, an original image having distance information handled in the embodiment of the present invention will be described. In the following description, it is assumed that the original image is a bitmap image and the distance information is a depth map.
FIG. 2 is an explanatory diagram illustrating an example of an original image. As shown in FIG. 2, the original image has four subjects F1, F2, F3, and F4. Subjects F1 to F3 are people, and subject F4 is a building. FIG. 3 is an explanatory diagram showing the positional relationship of the subject in the original image. As shown in FIG. 3, the positions of the subjects are in a longitudinal relationship, and the front side is the subject F2 and the subject F3, then the subject F1, and then the subject F4. It is assumed that the subject F2 and the subject F3 are arranged side by side and the front and rear positions are the same.

  FIG. 4 is an explanatory diagram showing a depth map corresponding to the original image. As is clear from the depth values corresponding to each of the plurality of hatchings, the positional relationship between the front and rear of the subject is described.

FIG. 5 is an explanatory diagram conceptually showing the data structure of the depth map. A of FIG. 5 shows the depth map value of each part of the original image shown in FIG. 4, that is, the depth value of the image, and assumes the values of D1, D2, D3, and D4. D1 is a value corresponding to a subject behind the subject F4, D2 is a value corresponding to the subject F4, D3 is a value corresponding to the subject F1, and D4 is a value corresponding to the subjects F2 and F3.
The upper part of B in FIG. 5 is a part extracted from A in FIG. The lower part of FIG. 5B further shows the data structure of the depth map for the top line of the extracted portion, and shows the depth value for each pixel in order from left to right. Since the value shown on the left corresponds to the leftmost pixel, it takes the value of D3, and as it goes to the right, the value on the right pixel corresponding to it is shown. When going to the right, D3 changes to D1, D1 changes to D2, and the rightmost pixel takes the value of D2.

An operation of the image processing apparatus 1 according to the first embodiment will be described.
The control unit 11 (threshold setting unit) receives the original image and the depth map of the original image from the imaging device 2 via the external interface unit 12, and stores them in the RAM 11c. The control unit 11 sets a threshold value for extracting a target subject. In the present invention, the threshold value is not a single value, but is set differently depending on the position of the original image in the horizontal direction. In the following description, a three-dimensional coordinate system is used. The XY coordinates are the coordinate system of the original image, and the upper left of the image is the origin, the right direction from the origin is the X axis direction, and the lower direction from the origin is the Y axis direction. The remaining axis is the axis in the depth direction, and the larger the value, the deeper the position.

FIG. 6 is an explanatory diagram conceptually showing the extraction range. In the following description, only the subject F2 that is a person is extracted. As shown in FIG. 6, the subject F2 has an X coordinate value in the range of X3 to X4 in the original image (X3 <X4). A method for obtaining X3 and X4 will be described later.
FIG. 7 is an explanatory diagram conceptually showing threshold setting. Similar to FIG. 3, the positional relationship of the subject in the original image is shown, and information necessary for explanation is added.

The control unit 11 (threshold setting unit) acquires depth values for all the pixels whose X coordinate values take values from X3 to X4 from the depth map. Next, an average value d0 of the acquired values is obtained. Further, d1 and d2 are obtained by the following equations (1) and (2).
d1 = d0−c1 (c1> 0) (1)
d2 = d0 + c2 (c2> 0) (2)
Furthermore, X1 and X2 are obtained from the following equations (3) and (4) from X3 and X4.
X1 = X3-c3 (c3 ≧ 0) (3)
X2 = X4 + c4 (c4 ≧ 0) (4)
Therefore, the magnitude relationship from d0 to d2 is d1 <d0 <d2, and the magnitude relationship from X1 to X4 is X1 ≦ X3 <X4 ≦ X2.

Using the values obtained as described above, the threshold values are set as the following equations (5) and (6).
Threshold = d1 (when 0 ≦ X <X1 or X ≧ X2) (5)
Threshold = d2 (when X1 ≦ X <X2) (6)
The setting of the threshold depends only on the X coordinate, and is set as described above regardless of the value of the Y coordinate.
The threshold value set in this way is illustrated as B1 in FIG. The control unit 11 (threshold setting unit) stores such threshold setting information in the RAM 11c.

The control unit 11 (extracted image generation unit) reads the original image, threshold setting information, and depth map from the RAM 11c.
The control unit 11 generates a binarized image based on the threshold setting information and the depth map. That is, for each pixel, the depth value is compared with a set threshold value. If the depth value value is less than or equal to the threshold value, 1 is set, and if the depth value value is greater than the threshold value, 0 is set. By taking the logical product of the binarized image generated in this way and the original image, an image of only the target subject (subject F2 in this embodiment) is extracted. That is, the original image and the binarized image are compared for each pixel. When the pixel value of the binarized image is 1, the pixel value of the original image is adopted, and when the pixel value of the binarized image is 0, The pixel value of the original image is set to a value corresponding to black or white. Thereby, the portion of the subject F2 is output as it is, but the portion other than the subject F2 is output in a solid state (black or white). The control unit 11 (extracted image generation unit) stores the extracted image in the RAM 11c. FIG. 8 is an explanatory diagram illustrating an example of an extracted image. As described above, only the subject F2 is extracted.

  The control unit 11 reads out an extracted image obtained by extracting only the target subject from the RAM 11c. Further, the replacement image to be combined with the extracted image is read from a storage device (not shown). The replacement image is stored in advance in the storage device, and is a background image such as a landscape photograph, a world heritage or a tourist attraction.

  The control unit 11 combines the extracted image and the replacement image. In other words, the control unit 11 inserts the replacement image into the solid portion of the extracted image. The control unit 11 displays the synthesized image on the display device 3 via the display interface unit 13. FIG. 9 is an explanatory diagram illustrating an example of the displayed composite image. As shown in FIG. 9, the extracted subject F2 can obtain an image as if standing in a different landscape from the original image (FIG. 2).

Next, a method for obtaining coordinate values X3 and X4 indicating the range where the subject to be extracted in the original image is located using a known face recognition technique will be described. The face feature amount of the subject F2 to be extracted is stored in advance in a storage unit (not shown) of the image processing apparatus 1. The control unit 11 specifies the face of the subject F2 from the plurality of subjects F1 to F4 of the original image using the face feature amount of the subject F2, and obtains the face position of the subject F2 in the original image.
FIG. 10 is an explanatory diagram when a subject is recognized by face recognition. The control unit 11 recognizes that the face of the subject F2 is in a square C having four points (X5, Y5), (X6, Y5), (X6, Y6), and (X5, Y6) as vertices ( X5 <X6, Y5 <Y6). In this case, X3 and X4 are obtained by the following equations (7) and (8).
X3 = X5-m1 (m1> 0) (7)
X4 = X6 + m2 (m2> 0) (8)
The control unit 11 stores the obtained values of X3 and X4 in the RAM 11c. The stored values of X3 and X4 are used for the process of setting the threshold as described above.

  Next, an image processing method performed by the image processing apparatus 1 will be described. FIG. 11 is a flowchart showing a flow of an image processing method performed by the image processing apparatus 1. The control unit 11 acquires an original image from the imaging device 2 via the external interface unit 12, and stores it in the RAM 11c (S11). The control unit 11 acquires a depth map from the imaging device 2 via the external interface unit 12, and stores it in the RAM 11c (S12).

  The control unit 11 sets a threshold value (S13). FIG. 12 is a flowchart illustrating a flow of threshold setting processing performed by the image processing apparatus 1. The control unit 11 obtains values indicating a range in the original image of the subject F2 to be extracted by face recognition, that is, values of X3 and X4 (S21). The controller 11 uses the obtained X3 and X4 to extract the depth value of the pixel whose X coordinate value is X3 or more and X4 or less from the depth map stored in the RAM 11c (S22). The control unit 11 obtains an average value d0 of the extracted depth values (S23). The control unit 11 obtains an upper limit value (d2) and a lower limit value (d1) of the threshold from the obtained d0 (S24).

  The control unit 11 obtains X coordinate values and X1 and X2 values for changing the threshold from X3 and X4 obtained in S21 (S25). The control unit 11 stores the obtained X1 and X2 and d1 and d2 obtained in S24 as threshold setting information in the RAM 11c (S26).

  Returning to FIG. 11, the control unit 11 generates a binarized image for extracting the image of the subject F2 from the original image based on the depth map and threshold setting information stored in the RAM 11c, and generates the generated binarized image. And the original image stored in the RAM 11c are subjected to a logical product operation to extract an image of the subject F2 (S14). The control unit 11 stores the extracted image in the RAM 11c (S15).

  The control unit 11 acquires a replacement image from a storage device (not shown) (S16). The control unit 11 combines the extracted image and the replacement image, displays the combined image on the display device 3 via the display interface 13 (S17), and ends the process.

  Note that c1 and c2 used when setting the threshold are values that should be determined in consideration of the thickness of the human body in the depth direction of the original image, and may be manually set in advance or according to the original image. That is, depending on the values of X3 and X4 or by known image recognition, the person who is the subject stands (standing up front or standing diagonally) and posture (standing upright or bent). Or the like, and the controller 11 may set an appropriate value accordingly.

  Similarly, c3 and c4 are values to be determined in consideration of processing errors, and may be manually set in advance, or the controller 11 sets appropriate values according to the values of X3 and X4. It's also good.

  The replacement image is stored in the storage device in advance, but may be read from a recording medium such as a CD-ROM or DVD via an external storage device, or via a communication network such as the Internet. It may be obtained.

  M1 and m2 used when recognizing the position of the subject are values in consideration of the width of the human body, and may be manually set in advance, or according to the original image, that is, X5, X6, Y5, Y6. The controller 11 may determine an appropriate value in accordance with the value of the face or the face orientation during face recognition and the face orientation.

  In order to recognize the position of the target subject, the face recognition technique is used here. However, the present invention is not limited to this, and it can recognize where the subject F2 to be extracted is located in the original image. For example, other technical methods may be used.

  Since the subject F2 to be extracted and the other subjects F1 and F3 are arranged side by side in the original image, the threshold value setting depends only on the X coordinate, but it may depend only on the Y coordinate. , And may change depending on both coordinate values of the Y coordinate. Depending on the Y coordinate only, a case where an image of a specific person is extracted from an image obtained by photographing a shoulder wheel or a totem pole of a gymnastic exercise can be considered. When changing by both coordinate values of the X coordinate and the Y coordinate, it is conceivable that only a specific part (upper body of a specific person) of a specific person is extracted.

  The subject to be extracted is F2, but the subject is not limited to this, and other subjects (F1, F3) may be extracted. In this case, a rule for determining a subject to be extracted (for example, in the case of side-by-side, a central person is always extracted) or a process for a user to specify a subject to be extracted from a plurality of subjects is required. is there.

  Although the binarized image is used to extract the image of the subject F2, the extraction can be performed using the original image, the depth map, and the threshold setting information without using the binarized image. The following processing is performed for each pixel. Extract pixels from the original image. A depth value corresponding to the pixel is extracted from the depth map. The extracted depth value is compared with the threshold value, and if the depth value is equal to or smaller than the threshold value, the pixel extracted from the original image is set as the pixel of the output image. If the depth value is larger than the threshold value, a pixel given a pixel value representing white is set as a pixel of the output image. By performing this process on all the pixels, only the pixels constituting the subject F2 to be extracted are output, and the pixels at other positions are all white, and the image of the subject F2 can be extracted.

  It is also possible to combine the image extraction of the subject F2 and the replacement image by similar processing. That is, extraction can be performed using the original image, the depth map, the threshold setting information, and the replacement image. The following processing is performed for each pixel. Extract pixels from the original image. A depth value corresponding to the pixel is extracted from the depth map. The extracted depth value is compared with the threshold value, and if the depth value is equal to or smaller than the threshold value, the pixel extracted from the original image is set as the pixel of the output image. If the depth value is larger than the threshold value, the pixel of the replacement image corresponding to the extracted pixel is extracted, and the pixel is set as the pixel of the output image. By performing this process on all the pixels, only the pixels constituting the subject F2 to be extracted are output, and the pixels at other positions become pixels extracted from the replacement image, and the extraction of the subject F2 and the replacement image are performed. Can be synthesized.

  In the first embodiment, the description has been made on the assumption that the image is a still image. However, the present invention is not limited to this and may be a moving image. Since a still image is composed of one frame image and a moving image is composed of images of a plurality of frames in time series order, in the case of a moving image, the same processing as a still image is performed for each frame of video that is continuous in time series order. Shall be performed. The image may be compressed by a predetermined encoding method, for example, JPEG (Joint Photographic Experts Group), MPEG-2 (Moving Picture Expert Group phase 2) or the like, or may be uncompressed. good. In the case of a configuration that handles encoded images, the image is decoded in advance into an image in RGB format or YUV format, for example, according to a predetermined encoding method, and the image obtained by decoding is input to the image processing apparatus 1 And The same applies to the following embodiments and modifications. The image is not limited to a still image, and may be a moving image.

(Modification 1)
In the first embodiment described above, the threshold value is set like a rectangular wave as shown in FIG. 7, but is not limited to this. FIG. 13 is an explanatory diagram conceptually showing the setting of the threshold value according to the modified example 1, and the threshold value is set to be a curve. Since the difference from the first embodiment is the threshold value setting method, the difference will be mainly described below.

In the first modification, in order to curve the threshold setting, a plurality of points through which the curve should pass are obtained, and an interpolation curve that passes through all the obtained points is obtained.
As shown in FIG. 13, the points through which the curve passes are five points P1, P2, P3, P4, and P5, and the coordinate values are P1 (X21, d1), P2 (X25, d0), and P3 (X20), respectively. , D2), P4 (X26, d0), and P5 (X22, d1).
The method for obtaining the X coordinate values X3 and X4 indicating the range in which the subject F2 to be extracted is located, the lower limit value d1 of the depth value, and the upper limit value d2 is the same as the method described above, and will be omitted.

Next, X20, X21, and X22 are obtained by the following equations (9), (10), and (11).
X20 = (X3 + X4) / 2 (c23> 0) (9)
X21 = X3-c23 (c24> 0) (10)
X22 = X4 + c24 (11)
The relationship among X3, X4, X22, and X21 is X21 <X3 <X4 <X22.
X25 is a value between X21 and X3 (X21 <X25 <X3), and X26 is a value between X4 and X22 as X26 (X4 <X26 <X22).

  When the coordinate values of P1 to P5 are obtained, an interpolation curve passing through all points is obtained, and this is set as a threshold value. The interpolation curve is obtained using a known interpolation method such as Lagrangian interpolation or spline interpolation. The threshold depends only on the value of the X coordinate. When the X coordinate is the same, the same threshold is taken regardless of the value of the Y coordinate. B2 shown in FIG. 13 is an example of a threshold curve. The image of the subject F2 can be extracted according to the obtained threshold curve. Other processes are the same as those in the first embodiment, and are omitted.

  Also in the first modification, an image as if the extracted subject F2 is standing in a landscape different from the original image can be obtained, as in the first embodiment.

Note that the values of c23 and c24 described above may be manually set in advance similarly to c3 and c4 in the first embodiment, or may be determined by the control unit 11 according to the original image.
Further, as in the first embodiment described above, the threshold value setting depends only on the X coordinate. However, it may depend only on the Y coordinate, and may change depending on both the X coordinate and Y coordinate values. good.

(Modification 2)
In the first embodiment described above, the distance information input to the threshold setting unit is such that the distance between the value and the actual value is proportional as in the depth map, but is not limited to this, and the value and the distance are not limited thereto. A disparity map in which is inversely proportional may be used as the distance information. The disparity map is obtained by obtaining a parallax value for each pixel of the entire image. The parallax value is a value indicating how much the corresponding pixel is shifted in the stereo image when the left image and the right image are compared. The parallax value takes a larger value as the imaging device 2 and the subject are closer, and becomes a smaller value as the imaging device 2 and the subject are farther from each other.
Therefore, in the above-described first embodiment, when the disparity map is used as distance information instead of the depth map, only the magnitude relationship of the values is different and the other parts are the same. The matter is mainly explained.

FIG. 14 is an explanatory diagram conceptually illustrating threshold setting according to the second modification. The control unit 11 (threshold setting unit) obtains the parallax values for all the pixels whose X coordinate values take values from X3 to X4 from the disparity map (X3 <X4). Next, an average value d10 of the acquired values is obtained. Next, d11 and d12 are obtained by the following equations (12) and (13).
d11 = d10 + c11 (c11> 0) (12)
d12 = d10−c12 (c12> 0) (13)
The relationship between the values of d10, d11, and d12 is d11>d10> d12.
Further, the method of obtaining X1 from X3 and X4 from X2 is the same as that in the first embodiment, and is omitted (X1 <X3 <X4 <X2).

Using the values obtained as described above, the threshold values are set as the following equations (14) and (15).
Threshold = d11 (when 0 ≦ X <X1 or X ≧ X2) (14)
Threshold = d12 (when X1 ≦ X <X2) (15)
The setting of the threshold depends only on the X coordinate, and is set as described above regardless of the value of the Y coordinate. The threshold value set in this way is shown in the figure as B11 in FIG.

  The control unit 11 stores such threshold setting information in the RAM 11c. The subsequent processing in the control unit 11 is the same as that in the first embodiment described above, but since the magnitude relationship between values is reversed between the depth map and the disparity map, processing for generating a binarized image is performed. Different. That is, the disparity value is compared with a set threshold value for each pixel, and 1 is set if the disparity value is greater than or equal to the threshold value, and 0 is set if the distance information value is less than the threshold value.

  In the second modification, a disparity map is used as distance information. Since the disparity map can be used as distance information without conversion, it is possible to perform a process of setting a threshold in the same time as when using a depth map.

  Note that c11 and c12 are values that should be determined in consideration of the thickness of the human body, similarly to c1 and c2, and c13 and c14 are values that should be determined in consideration of processing errors, similarly to c3 and c4. Any value of c11 to c14 may be determined in advance by a person, or the controller 11 may set an appropriate value according to the original image.

  For those skilled in the art, it is possible for a person skilled in the art to use a disparity map as distance information in place of the depth map by making the same changes as described in Modification 2. It is obvious.

(Embodiment 2)
In the first embodiment, the control program is stored in the ROM 11b in advance. However, the present invention is not limited to this, and a computer program as a control program may be read from the outside.
FIG. 15 is a block diagram showing a configuration example of an image processing apparatus according to Embodiment 2 of the present invention. The image processing apparatus 5 according to the second embodiment of the present invention is realized by executing the computer program 6a according to the present invention.

  The image processing device 5 is further provided with an external storage device 15 and a communication unit 16 in the image processing device 1 shown in the first embodiment. The same components as those in the first embodiment are denoted by the same reference numerals and description thereof is omitted.

  The external storage device 15 reads the computer program 6a from the recording medium 6 on which the computer program 6a according to the embodiment is recorded, for example, a CD-ROM. The communication unit 16 acquires the computer program 6a according to the embodiment of the invention through a communication network N such as the Internet. The computer program 6 a may be acquired from the recording medium 6 through the external storage device 15 or may be acquired from the communication network N through the communication unit 16. Moreover, as what is acquired from the recording medium 6 or the communication network N for every program module, it is good also as acquiring the computer program 6a using both the recording medium 6 and the communication network N.

  The processing procedure of the control unit 11 is as shown in FIG. 11, and the processing procedure of steps S11 to S17 is executed. Since the processing procedure is the same as the processing content in the image processing apparatus 1 according to the first embodiment, detailed description thereof is omitted.

  The image processing apparatus 5 and the computer program 6a according to the second embodiment function as the image processing apparatus according to the present embodiment, and can execute the image processing method according to the present embodiment. The same effects as those of the first embodiment of the invention can be obtained.

(Embodiment 3)
In the first embodiment, the image is extracted with the same threshold setting throughout the original image. However, in the third embodiment, the original image is divided into a plurality of regions, and the image is extracted for each region. This is different from the first and second embodiments. The hardware configuration is the same as in the first embodiment. Below, this difference is mainly demonstrated.

The control unit 11 (region dividing unit) divides the original image into a plurality of regions, and stores coordinate information indicating each region in the RAM 11c.
The control unit 11 (threshold setting unit, extracted image generation unit) divides the original image into a plurality based on the coordinate information indicating each region stored in the RAM 11c, and performs processing for each divided region image. The control unit 11 sets a threshold value for extracting a specific subject for each divided area, and stores the threshold information in the RAM 11c. The control unit 11 extracts a subject for each area based on the coordinate information of each area and the threshold information of each area stored in the RAM 11c, and stores the extracted image in the RAM 11c. The control unit 11 combines the replacement image for each area and the extracted image for each area based on the coordinate information indicating each area stored in the RAM 11c and the extracted image for each area. After the synthesis for all the regions is completed, the control unit 11 stitches the combined images of all the regions as one image, and displays an image having the same size as the original image via the display interface unit 13. Output to.

  FIG. 16 is an explanatory diagram showing an example of image region division in the third embodiment of the present invention. In the image composed of the subjects F11, F12, F13, and F14, the area is divided into two areas (areas A1 and A2).

  FIG. 17 is a flowchart showing the flow of processing performed by the image processing apparatus 1 according to Embodiment 3 of the present invention. The control unit 11 acquires the original image from the imaging device 2 and stores it in the RAM 11c (S31). The control part 11 acquires distance information from the imaging device 2, and memorize | stores it in RAM11c (S32). The control unit 11 determines whether or not the user has designated image segmentation (S33). When area division is designated (YES in S33), the coordinates of each area are stored in the RAM 11c, and processing for each area is performed (S34). The process for each area will be described later. Next, coordinate information and a composite image for each area stored in the RAM 11c are acquired, all the composite images are connected as one image, and combined into an image having the same size as the original image (S35) and displayed. It outputs to the display apparatus 3 via the interface part 13, and complete | finishes a process. If area division has not been performed (NO in S33), the process from S36 to S40 is performed and the process ends. Since the processes from S36 to S40 are the same as the processes from S13 to S17 (FIG. 11) in the first embodiment, the description thereof is omitted.

  FIG. 18 is a flowchart showing the flow of processing performed on each area divided in the third embodiment. The control unit 11 acquires coordinate information of each area from the RAM 11c (S41). From the acquired coordinate information of each area, the control unit 11 extracts the coordinate information of the unprocessed area and sets it as the coordinate information of the processing image area (S42). The control unit 11 reads distance information corresponding to the processing image area from the RAM 11c based on the coordinate information (S43). The control unit 11 performs threshold setting based on the image of the processing image area portion of the original image and the read distance information (S44). The threshold setting is the same as the processing shown in FIG. That is, a threshold setting process is performed on the image in the processing image area portion of the original image. Based on the coordinate information and threshold setting of the processing image area, the control unit 11 extracts a subject from the processing image area portion of the original image and stores it in the RAM 11c (S45). The control unit 11 acquires a replacement image corresponding to the processing image area from an external device or a storage device (not shown) (S46). The control unit 11 combines the extracted image read from the RAM 11c and the acquired replacement image (S47). The control unit 11 associates the coordinate information of the processing image area and the composite image and stores them in the RAM 11c (S48). The control unit 11 determines whether or not processing has been completed for all regions (S49). If the process has not been completed for all the regions (NO in S49), the control unit 11 returns to S42. If the processing has been completed for all the regions (YES in S49), the control unit 11 terminates the processing for each region and proceeds to S35 in FIG.

  FIG. 19 is an explanatory diagram showing an example of threshold setting in the third embodiment of the present invention. FIG. 19 shows the context of the subject shown in FIG. 16 and the set threshold value. The threshold curve B31 is used for the area A1 of the original image shown in FIG. 16, and the subject curve F32 is used for the area A2 and the subject F12 is used for the area A2 by using the threshold curve B32 for the area A2. Can be extracted respectively. As a method for obtaining the threshold curves B31 and B32, for example, in the case of region division, a person who has a large area in the image in each region, and a person in front and a person in front of that person are selected. A rule of extraction is set in advance. Using a known technique such as pattern matching, a person is recognized in the original image, and a threshold curve is generated in each region (A1, A2) so that an image of a person corresponding to the rules described above can be extracted. Thus, threshold curves B31 and B32 can be obtained.

  FIG. 20 is an explanatory diagram illustrating an example of an output image in the third embodiment. In FIG. 20, after extracting the subject F12 and the subject F13 in each of the two areas A1 and A2 obtained by dividing the original image, a replacement image is synthesized in each area, and a composite image of the areas A1 and A2 is obtained. Are combined to create an image having the same size as the original image. In the third embodiment, it is possible to extract a subject by setting a plurality of regions with respect to the original image, and to replace an arbitrary image at an arbitrary position such as between the background and the subject or in front of the subject. That is, by using the distance information, a pseudo three-dimensional space can be created on a two-dimensional plane, and a new image expression can be realized.

  In the above description, the original image is divided into two parts in the upper and lower directions. However, the present invention is not limited to this, and the original image may be divided into left and right, and the number of areas may be set to three or more. In addition, for the designation of the area, the number and arrangement of the areas to be divided in advance may be stored in the image processing apparatus 1 and divided based on the area. 1 may be set. Alternatively, the user may specify with the operation device 4 such as a mouse.

(Modification 3)
In the above-described third embodiment, the image processing apparatus 1 performs region division and threshold setting for each region based on a predetermined rule. However, the user may perform similar division and setting. In this modification, the user uses the operation device 4 to set the area to be divided and the threshold value for each area.
FIG. 21 to FIG. 23 are explanatory diagrams showing examples of the user operation screen 20 in the third modification. The user operation screen 20 is displayed on the display device 3 via the display interface unit 13 of the image processing apparatus 1. As shown in FIG. 21, the user operation screen 20 includes a processed image display unit 21, a pointer 22, a parallax value display column 23, threshold input columns 24a and 24b, an original image name display column 25a, a disparity map name display column 25b, and a replacement. An image a display field 25c, a replacement image b display field 25d, and selection buttons 26a, 26b, 26c, and 26d are configured.

  The processed image display unit 21 displays an original image to be processed. In the present modification, an original image composed of subjects F21, F22, and F23 and a pointer 22 are displayed on the processed image display unit 21. The original image name display column 25a is a column in which the name of the image to be processed is displayed. When the selection button 26a is selected by the operation device 4, a selection window (not shown) is opened, and an image to be processed is displayed from the displayed image list. By selecting, the name of the image to be processed is displayed in the original image name display field 25a. If the name of the image to be processed is known, the name of the image can be directly input in the original image name display field 25a. The disparity map name display column 25b is a column in which the name of the disparity map is displayed. Similar to the original image name, a disparity map corresponding to the original image is selected in a selection window (not shown) displayed by selecting the selection button 26b using the operation device 4, and a disparity map name display field 25b displays a disparity map. By directly entering the parity map name, the disparity map used for processing is specified. The replacement image a display field 25c and the replacement image b display field 25d are fields in which replacement image names are displayed, respectively, and selection buttons 26c and 26d correspond to the respective fields. The input method is the same as that of the original image and disparity map.

  The parallax value display column 23 is a column for displaying the parallax value of the pixel designated by the user. When an arbitrary position of the image is designated by the pointer 22 and selected using the operation device 4 such as a mouse, the parallax value at the position indicated by the pointer 22 is displayed in the parallax value display field 23. The threshold value input field 24a and the threshold value input field 24b are fields for inputting a parallax value threshold value used when extracting a subject image.

  Next, a user operation will be described. First, the user designates an area. In the example shown in FIG. 21, the area A3 is set. At this time, it is possible not to set the area, in which case it is equivalent to selecting the entire image as one area. The area designation may use a square or a rounded figure, or may be determined freehand using the operation device 4 such as a mouse.

  When the subject F22 to be extracted is selected by the pointer 22, the parallax value 127 at the selected position is displayed in the parallax value display field 23. Based on this parallax value, the user determines a threshold in the setting area A3 and inputs the threshold in the threshold input field 24a. For example, a value smaller than the parallax value of the subject F22, that is, the parallax value behind the subject F22 is set as the threshold value. Here, since the parallax value of the subject F22 is 127, the threshold value is set to 100 (FIG. 23). Further, a threshold value of an area other than the setting area A3 is also determined, and the threshold value is input to the threshold value input field 24b. For example, a value smaller than the parallax value of the subject F21, that is, the parallax value behind the subject F21 is set as the threshold value. Here, as shown in FIG. 22, since the parallax value of the subject F21 is 64, the threshold is set to 50 (FIG. 23). In addition, when the area is not set, if the threshold is input to the threshold input field 24a, the threshold is set with the entire image as one area.

  By setting the threshold values as described above, the subjects F21 and F22 can be extracted as shown in FIG. A subject F22 is extracted in the setting area A3, a part of the body of the subject F21 is extracted in an area other than the setting area A3, and an image excluding the background subject F23 is displayed on the processed image display unit 21.

  Subsequently, the image specified in the replacement image a display field 25c is inserted at a position corresponding to the threshold distance input in the threshold value input field 24a of the setting area A3. That is, in the setting area A3, the image specified in the replacement image a display field 25c becomes the background image of the subject F22. In the area other than the setting area A3, the image specified in the replacement image b display field 25d is inserted at a position corresponding to the threshold distance input in the threshold value input field 24b. That is, in the area other than the setting area A3, the image specified in the replacement image b display field 25d becomes the background image of the subject F21. As shown in FIG. 23, the processed image display unit 21 displays a composite image in which the background image is replaced. The created image may be stored in a storage device (not shown) and displayed on the display device 3.

  Thus, for example, by using a GUI (Graphical User Interface), the user can freely perform image processing using the held image and distance information (parallax information), and display the image on an arbitrary display device. Is possible. In the present embodiment, as an example, the method of dividing the image into two areas, that is, the area within the selected area and the other area, and setting the threshold value and performing the image replacement has been described. It is also possible to change the image. In addition, one value is set for one threshold value in one area (surface), but the range to be extracted is further limited by changing the threshold value in one area as shown in the threshold curve B2 in FIG. Is also possible.

(Embodiment 4)
FIG. 24 is a block diagram illustrating a configuration example of an imaging apparatus according to Embodiment 4 of the present invention. The imaging device 7 according to the fourth embodiment is an example of the imaging device according to the present invention. In the image processing device 1 according to the first embodiment, the imaging unit 17, the distance measuring unit 18, and the replacement image storage unit are further provided. 19 is provided. The same components as those in the image processing apparatus 1 in the first embodiment described above are denoted by the same reference numerals and description thereof is omitted.

  The image capturing unit 17 captures an object such as a person or a background and outputs an image. The image capturing unit 17 captures an image such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) using the received light as an electrical signal. An optical system such as an element and a lens for condensing light from the subject on the image sensor is provided. The imaging unit 17 outputs the captured image to the control unit 11 (threshold setting unit, extracted image generation unit). The distance measuring unit 18 measures the distance between the imaging device 7 and the subject. The distance measuring unit 18 measures the distance to the subject using a distance measuring method such as TOF (Time of Flight), and the measurement result is transmitted to the control unit 11 (distance information interpolation). Part). The control unit 11 performs an interpolation process on the plurality of distance information acquired from the distance measuring unit 18, and calculates distance information related to the distance in the depth direction at each of the plurality of positions of the original image.

  The replacement image storage unit 19 stores a replacement image to be combined with the subject image extracted from the original image. The replacement image is output to the control unit 11. The other operations of the imaging device 7 are the same as the operations of the image processing device 1 according to the first embodiment described above, and are therefore omitted.

  As described above, since the imaging device 7 according to the present embodiment includes the imaging unit 17 and the distance measuring unit 18, the user can handle the original image and the original image without preparing the imaging device separately. Distance information can be acquired. Further, since the user can use the replacement image stored in the replacement image storage unit 19, image processing can be performed without separately preparing a replacement image.

(Embodiment 5)
FIG. 25 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 5 of the present invention. The imaging device 8 according to the present embodiment is an example of the imaging device according to the present invention. In the image processing device 1 according to the first embodiment, a pair of imaging units 17a and 17b and a replacement image storage unit 19 are further provided. Is provided. The same components as those of the image processing apparatus 1 according to the first embodiment described above are denoted by the same reference numerals, and the description of the components and the operation of each unit is omitted.

  The imaging units 17a and 17b are arranged in stereo. The stereo arrangement refers to an arrangement in which the two imaging units 17a and 17b are arranged side by side so that their optical axes are substantially parallel. In the present embodiment, two imaging units 17a and 17b having the same configuration are used as an example. However, it is possible to capture the same area by two imaging units 17a and 17b and take correspondence between pixels. For example, imaging units having different configurations such as resolution and angle of view may be used. The imaging units 17a and 17b each capture a subject such as a person or a background and output an image. The received light is converted into an electrical signal and used as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). And an optical system such as a lens for condensing light from the subject on the image sensor. The imaging unit 17 a outputs the captured image to the control unit 11 (distance calculation unit, threshold setting unit, extracted image generation unit), and the imaging unit 17 b outputs the captured image to the control unit 11.

  The control unit 11 calculates the distance between the imaging device 8 and the subject from the two images received from the imaging units 17a and 17b, and generates a depth map based on the calculated distance. Subsequently, the control unit 11 corrects the depth map as necessary, and stores the corrected depth map in the RAM 11c. The replacement image storage unit 19 stores a replacement image to be combined with the subject image extracted from the original image, and outputs the replacement image to the control unit 11. Other operations are the same as those in the first embodiment described above, and are therefore omitted.

  Here, the two imaging units 17a and 17b are arranged in the left-right direction. However, when the parallax value can be calculated even if arranged in the up-down direction, the captured image is scanned horizontally. Instead of the direction, the vertical direction may be used. Moreover, although the example which calculated distance from the parallax value was shown, it is also possible to use a parallax value as distance information, without calculating a distance. In this case, the control unit 11 generates a disparity map instead of a depth map. Further, the operation of the imaging device 8 is the same as the operation shown in the second modification of the first embodiment.

  As described above, since the imaging device 8 in the present embodiment includes the imaging units 17a and 17b, the original image and the original image can be obtained even if the user does not separately prepare the imaging device, as in the fourth embodiment. Distance information corresponding to the image can be acquired. Further, since the user can use the replacement image stored in the replacement image storage unit 19, image processing can be performed without preparing a replacement image separately.

(Embodiment 6)
In the above-described embodiment, the image processing apparatuses 1 and 5 have been described as the image processing apparatus alone. However, the image processing apparatuses 1 and 5 are replaced with other devices such as a television receiver, a cellular phone, or a personal computer (PC). It is also possible to incorporate it into an information processing apparatus such as
FIG. 26 is a block diagram showing a configuration example of a television receiver according to Embodiment 6 of the present invention. The television receiver 9 according to Embodiment 6 includes a control unit 11, an external interface unit 12, a tuner unit 91, a signal processing unit 92, an audio output unit 93, a display unit 94, and an operation unit 95. The television receiver 9 incorporates the image processing apparatus 1 according to the first embodiment. In addition, the same code | symbol is attached | subjected about the structure similar to the image processing apparatus 1. FIG.

The control unit 11 reads out the control program stored in advance in the ROM 11b to the RAM 11c and executes it by the CPU 11a, thereby controlling the operation of each hardware unit included in the television receiver 9 and controlling the entire apparatus according to the image processing of the present invention. To function as a device.
Since the function of the external interface unit 12 is the same as that of the external interface unit 12 included in the image processing apparatus 1, the description thereof is omitted.

  The tuner unit 91 is a digital tuner for receiving a digital broadcast signal, and is connected to the antenna AN. The tuner unit 91 detects the radio wave received by the antenna AN according to the broadcast channel selected by the user via the operation unit 95, for example, acquires the broadcast signal from the obtained broadcast wave, and acquires the acquired broadcast signal. It is sent to the signal processing unit 92.

  The signal processing unit 92 separates the broadcast signal acquired by the tuner unit 91 into a video signal (RGB signal (R: red, G: green, B: blue)) and an audio signal. The signal processing unit 92 performs a predetermined decoding / decompression process on the separated audio signal, and sends the obtained audio signal to the audio output unit 93. The audio output unit 93 amplifies the audio signal sent from the signal processing unit 92 and outputs audio based on the audio signal through a speaker (not shown).

  The signal processing unit 92 performs a predetermined decoding / decompression process on the separated video signal. The signal processing unit 92 converts the video signal (Y signal and C signal) obtained by the decoding / decompression process into an RGB video signal (RGB signal), and then, for each color component of the video signal, the input gradation ( (Luminance) is converted into output gradation (output luminance) corresponding to each input gradation, and the obtained video signal (output gradation) is sent to the display unit 94.

The display unit 94 is obtained by integrating the display interface unit 13 and the display device 3 included in the image processing apparatus 1. When the television receiver 9 functions as a normal television receiver, the display unit 94 displays the video signal transmitted from the signal processing unit 92 on a liquid crystal module or the like provided in the display unit 94 according to a predetermined timing.
When the television receiver 9 functions as an image processing device, the display unit 94 displays an original image, an extracted image, a composite image, or an operation screen.

The operation unit 95 is obtained by integrating the operation interface unit 14 included in the image processing apparatus 1 and the operation device 4. A function switching operation of whether the television receiver 9 functions as a normal television receiver or an image processing apparatus is accepted.
When the television receiver 9 functions as a normal television receiver, the operation unit 95 accepts operations such as selection of a broadcast channel and volume control.
When the television receiver 9 functions as an image processing device, selection of an original image, selection of a depth map, designation of divided areas, and the like are performed.
When designating a divided area, a cross button (an operation button including an up button, a down button, a right button, a left button, a determination button, and the like) provided in the operation unit 95 is used.

Since the operation when the television receiver 9 functions as an image processing apparatus is the same as that of the image processing apparatus 1 according to Embodiment 1 described above, the description thereof is omitted.
In the same manner as in this embodiment, the image processing apparatuses according to Modification 1, Modification 2, Embodiment 2, Embodiment 3, and Modification 3 described above can be incorporated into a television receiver.
The same effects as those of the image processing apparatus according to the first embodiment, the first modification, the second modification, the second embodiment, the third embodiment, and the third modification are obtained.

(Embodiment 7)
As in the above-described sixth embodiment, the imaging devices 7 and 8 can be incorporated in an information processing device such as a television receiver, a mobile phone, or a personal computer (PC) instead of the imaging device alone. Since the difference from the sixth embodiment is only the difference between whether the apparatus to be incorporated is an image processing apparatus or an imaging apparatus, the difference will be mainly described below.
FIG. 27 is a block diagram showing a configuration example of a television receiver according to Embodiment 7 of the present invention. The television receiver 10 according to the seventh embodiment includes a control unit 11, an imaging unit 17, a distance measuring unit 18, a replacement image storage unit 19, a tuner unit 91, a signal processing unit 92, an audio output unit 93, and a display unit 94. And an operation unit 95. The television receiver 10 incorporates the imaging device 7 according to the fourth embodiment. In addition, the same code | symbol is attached | subjected about the structure similar to the imaging device 7 or the television receiver 9, respectively.

The control unit 11 has a function similar to that of the control unit 11 included in the imaging device 7 and performs control for causing the television receiver 10 to function as a normal television receiver.
The functions of the imaging unit 17, the distance measuring unit 18, and the replacement image storage unit 19 are the same as those of the imaging unit 17, the distance measuring unit 18, and the replacement image storage unit 19 included in the imaging device 7, respectively. Omitted.
Similarly, functions of the tuner unit 91, the signal processing unit 92, the audio output unit 93, and the display unit 94 are the same as those of the television receiver 9 according to the sixth embodiment, and thus the description thereof is omitted.

The operation unit 95 has a function similar to that of the above-described television receiver 9 and also has a function of receiving a command from the user to the imaging unit such as releasing a shutter and transmitting the command to the control unit 11.
Since the operation when the television receiver 10 functions as an imaging processing apparatus is the same as that of the imaging apparatus 7 according to Embodiment 4 described above, the description thereof is omitted.
Similarly to the present embodiment, the above-described imaging device 8 according to the fifth embodiment can be incorporated into a television receiver.
And the same effect as the imaging device concerning Embodiment 4 and 5 is produced.

  It should be understood that the above-described embodiment is illustrative in all respects and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the meanings described above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1, 5 Image processing device 7, 8 Imaging device 11 Control unit 11a CPU
11b ROM
11c RAM
DESCRIPTION OF SYMBOLS 12 External interface part 13 Display interface part 14 Operation interface part 15 External storage device 16 Communication part 17, 17a, 17b Imaging part 18 Distance measuring part 19 Replacement image memory | storage part 2 Imaging device 3 Display apparatus 4 Operation apparatus 6 Recording medium 6a Computer Program N Communication network 20 User operation screen 21 Processed image display section 22 Pointer 23 Parallax value display field 24a, 24b Threshold value input field 25a Original image name display field 25b Disparity map name display field 25c Replacement image a display field 25d Replacement image b display Field 26a, 26b, 26c, 26d Selection button 9, 10 Television receiver

Claims (10)

An extracted image obtained by acquiring an original image having distance information related to the distance in the depth direction of the image at each of a plurality of positions on the image, and a replacement image replacing a part of the original image, and extracting a part of the acquired original image And an image processing apparatus for synthesizing the replacement image,
A threshold setting unit that sets a threshold for the distance information based on each position on the original image;
An image processing apparatus comprising: an extracted image generation unit configured to generate the extracted image based on a set threshold and the distance information. The image processing apparatus according to claim 1, wherein an image portion that has not been extracted from the original image is replaced with the replacement image. An area dividing unit for dividing the original image into a plurality of areas;
The threshold setting unit sets a threshold for each divided area,
The image processing apparatus according to claim 1, wherein the extracted image generation unit generates an extracted image for each divided region. An area dividing unit for dividing the original image into a plurality of areas;
The threshold setting unit sets a threshold for each divided area,
The extracted image generation unit generates an extracted image for each divided region,
The image processing apparatus according to claim 2, wherein a plurality of replacement images are acquired, and a different replacement image is used for each divided region. An extracted image obtained by acquiring an original image having distance information related to the distance in the depth direction of the image at each of a plurality of positions on the image, and a replacement image replacing a part of the original image, and extracting a part of the acquired original image And an image processing method for synthesizing the replacement image,
Setting a threshold for the distance information based on each position on the original image;
An image processing method comprising: generating the extracted image based on a set threshold and the distance information. A computer acquires an original image having distance information related to a distance in the depth direction of the image at each of a plurality of positions on the image, and a replacement image that replaces a part of the original image, and acquires a part of the acquired original image In the program for synthesizing the extracted image and the replacement image,
In the computer,
Setting a threshold for the distance information based on each position on the original image;
A program for executing the step of generating the extracted image based on a set threshold and the distance information. An imaging unit for imaging a subject;
A replacement image storage unit for storing replacement images;
In an imaging device that synthesizes an extracted image obtained by extracting a part of a captured image and a replacement image that replaces a part of the captured image,
A distance measuring unit for measuring the distance to the subject at a plurality of positions;
A distance information interpolation unit that calculates distance information related to the distance in the depth direction of the image at each of a plurality of positions on the captured image by interpolating the distance values of the plurality of positions measured by the distance measuring unit;
A threshold setting unit that sets a threshold for the distance information based on each position on the captured image;
An image pickup apparatus comprising: an extracted image generation unit configured to generate the extracted image based on a set threshold and the distance information. A plurality of imaging units for imaging a subject;
A replacement image storage unit for storing replacement images;
In an imaging device that synthesizes an extracted image obtained by extracting a part of a captured image and a replacement image that replaces a part of the captured image,
A distance information calculation unit that calculates distance information related to the distance in the depth direction of the image at each of a plurality of positions on the captured image using a plurality of images captured by the plurality of imaging units;
A threshold setting unit that sets a threshold for the distance information based on each position on the captured image;
An image pickup apparatus comprising: an extracted image generation unit configured to generate the extracted image based on a set threshold and the distance information. An image processing apparatus according to any one of claims 1 to 4,
A tuner for receiving television broadcasts;
A display unit for displaying an image related to television broadcast received by the tuner unit,
The display unit is configured to display an image acquired, extracted, generated or synthesized by the image processing apparatus. The imaging device according to claim 7 or 8,
A tuner for receiving television broadcasts;
A display unit for displaying an image related to television broadcast received by the tuner unit,
The television receiver, wherein the display unit displays an image captured, extracted, generated or synthesized by the imaging device.

Images