JP2015005925A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to check, when an image in which an obstruction is removed is displayed in live view, the obstruction from the live view.SOLUTION: When a distance to an obstruction is a predetermined value or more, an imaging device displays an image in which the obstruction is removed as image data, and when the distance to the obstruction is smaller than the predetermined value, performs collision prevention processing, such as creating display image data allowing the visual recognition of the obstruction.

Description

  The present invention relates to an image processing apparatus and method for displaying an image obtained by removing a specific object from a captured image.

  Currently, various attempts have been made to remove specific obstacles included in video captured by a handheld camera by image processing. For example, a method of removing an obstacle from an image in which an object such as a building having a planar structure is shielded by an obstacle such as a utility pole in the foreground than the object is known (see Patent Document 1). reference). In Patent Document 1, each frame in a video is converted by projective transformation so that the position of a typical planar structure included in the video matches. Then, foreground objects in the video are removed by using a spatio-temporal filter.

  As another method, there is also known a method of removing an obstacle using a difference in parallax between a foreground object and a background object between images having different viewpoints (see Patent Document 2). In Patent Document 2, the foreground object region in the image data of the viewpoint A is obtained by utilizing the fact that the parallax between the foreground object region of the image data of the viewpoint A and the foreground object region of the image data of the viewpoint B is larger than the background object region. Recognize Then, the foreground object area in the image data of the viewpoint A is corrected using the image data of the viewpoint B, thereby removing the foreground object (obstacle) from the image data of the viewpoint A. Further, Patent Document 2 describes an image display method on a display screen at the time of imaging in order to obtain captured images with different viewpoints suitable for removing an obstacle. That is, a plurality of captured images captured from different viewpoints are superimposed and displayed on the live view. Thereby, the photographer can confirm at the time of imaging whether sufficient parallax is obtained.

JP 2011-118495 A JP 2010-41586 A

  In an imaging field, for example, a scene is assumed in which an object that is a background object is imaged through a fence that is a foreground object. However, neither the technique disclosed in Patent Document 1 nor the technique disclosed in Patent Document 2 is a technique that actually considers the usage mode at the imaging site.

  For example, when an image from which the foreground object is removed is displayed as in Patent Document 1, the foreground object cannot be confirmed on the display screen. For this reason, when the image obtained as a result of removing the foreground object is displayed in live view, the distance to the foreground object is not known and there is a possibility of colliding with the foreground object. On the other hand, when the live view display is performed by superimposing the captured images as in Patent Document 2, the foreground object is displayed in a translucent manner, so the distance of the foreground object can be known, but the result of removing the foreground object is displayed in the live view. Cannot confirm by display. As described above, there has been no method for appropriately presenting the user with confirmation of the foreground object on the display screen and confirmation of the result of removing the foreground object on the display screen at the imaging site.

  An imaging apparatus according to the present invention generates image data obtained by imaging a background object partially shielded by a foreground object and acquiring image data, and removing the foreground object from the image data. Display image data is generated from at least one of the image data and the foreground object removal image data based on the distance, a removal unit, a distance acquisition unit that acquires a distance from the imaging unit to the foreground object, and And display means for displaying.

  In the image processing apparatus that generates an image obtained by removing a specific obstacle from a captured image, the present invention presents an image from which the obstacle has been removed during imaging to the photographer and notifies the imager not to collide with the shielding object. it can.

It is a figure which shows the external appearance of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the internal structure of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the image process part which concerns on Embodiment 1 of this invention. It is a flowchart which shows the flow of the process which concerns on Embodiment 1 of this invention. It is a figure which shows arrangement | positioning of the imaging device which concerns on Embodiment 1 of this invention, a shield, and a to-be-photographed object. It is a flowchart showing the flow of the obstruction removal process which concerns on Embodiment 1 of this invention. It is a figure showing the example of the coordinate transformation image data which concerns on Embodiment 1 of this invention. It is a figure showing the example of the obstruction removal image data concerning Embodiment 1 of this invention. It is a flowchart showing the flow of the display image generation process which concerns on Embodiment 1 of this invention. The figure which shows the relationship between the distance which concerns on Embodiment 1 of this invention, and the transmittance | permeability. It is a figure which shows the example of the live view display which concerns on Embodiment 1 of this invention. It is a figure which shows the example of the imaging device which has several imaging means concerning Embodiment 1 of this invention. It is a flowchart which shows the flow of the display image generation process which concerns on Embodiment 2 of this invention. It is a figure which shows the example of the live view display which concerns on Embodiment 2 of this invention. It is an example which shows the example of the imaging device which concerns on Embodiment 3 of this invention. It is a flowchart which shows the flow of the control method of the optical part which concerns on Embodiment 3 of this invention. It is a figure which shows the example of the optical system control process which concerns on Embodiment 3 of this invention. It is a flowchart which shows the flow of the optical system control process which concerns on Embodiment 4 of this invention.

[Embodiment 1]
<Appearance of imaging device>
1A and 1B are views showing an external appearance of an imaging apparatus according to the present embodiment. FIG. 1A shows the external appearance of the front surface and FIG. The imaging apparatus 101 includes an optical unit 102, an imaging button 103, a strobe 104, a distance image acquisition unit 105, a display unit 106, and operation buttons 107.

  The optical unit 102 is a lens barrel including a zoom lens, a focus lens, a shake correction lens, a diaphragm, and a shutter, and collects light information of a subject. The imaging button 103 is a button for the user to instruct the imaging apparatus 101 to start imaging. The strobe 104 can emit light at the start of imaging in accordance with a user instruction. The distance image acquisition unit 105 acquires distance image data of the subject according to the imaging instruction.

  The display unit 106 uses a liquid crystal display or the like, and displays image data processed by the imaging apparatus 101 and other various data. In this embodiment, since no optical finder is provided, a framing operation (confirmation of focus and composition) is performed using the display unit 106. Since the imaging is performed while confirming the live view image on the display unit 106, it can be said that the display unit 106 functions as an electronic viewfinder while performing the framing and focusing operations. The display unit 106 displays a camera setting menu in addition to performing live view display for displaying the imaging range in real time.

  The operation button 107 is a button for the user to instruct the imaging apparatus 101 about imaging condition parameters, imaging modes, and the like. In the present embodiment, the shielding removal mode is included as one of the imaging modes. It is assumed that the user can use the operation button 107 or the imaging button 103 to instruct to start the shielding removal mode, end the shielding removal mode, and output the shielding removal image data. Note that the display unit 106 may have a touch screen function. In that case, a user instruction using the touch screen can be handled as an input of the operation button 107.

<Internal configuration of imaging device>
FIG. 2 is a block diagram illustrating an internal configuration of the imaging apparatus 101 according to the present embodiment.

  The CPU 202 is involved in all the processes of each component, and sequentially reads and interprets instructions stored in a ROM (Read Only Memory) 203 and a RAM (Rondom Access Memory) 204, and executes processes according to the results. The system bus 212 is a bus for transmitting and receiving data.

  The control unit 206 controls the start and end of the imaging operation according to a user instruction from the imaging button 103 and the operation button 107. In the present embodiment, the control unit 206 controls SW1 and SW2 according to a user instruction. Specifically, when instructed to start the shield removal mode, SW1 is turned on, when instructed to end the shield removal mode, SW1 is turned off, and when the output of the shield removal image is instructed. Turns on SW2. Actually, SW1 and SW2 are bit signals in the RAM 204, and the state where the bit is 0 is turned OFF and the state where the bit is 1 is turned ON, and the control unit 206 switches the state of these bits.

  The optical system control unit 205 performs control instructed to the optical unit 102 by the CPU 202 such as focusing, opening a shutter, and adjusting an aperture.

  The color imaging element unit 201 is an element that converts light information collected by the optical unit 102 into a current value, and acquires color information by combining with a color filter or the like.

  The distance image acquisition unit 105 includes an infrared light emitting unit that emits infrared light and a light receiving unit that receives infrared light reflected by the subject, and a time until the emitted infrared light is reflected by the subject and received. The distance from the imaging device to the subject is calculated based on the above and acquired as distance image data.

  The A / D conversion unit 208 converts the amount of light of the subject detected by the optical unit 102 into a digital signal value to obtain RAW image data. In this embodiment, it is assumed that distance image data and RAW image data captured at the same time can be acquired.

  An image processing unit 209 performs development processing on the above RAW image data to generate color image data. Further, the image processing unit 209 performs various image processes including a shielding object removal process (foreground object removal process) based on the color image data and the distance image data, thereby obtaining shielding object removal image data (foreground object removal image data). Generate. The internal structure of the image processing unit 209 will be described in detail later.

  In addition, the character generation unit 207 generates characters and graphics. The characters and graphics generated here are displayed on the display unit 106 so as to be superimposed on the obstruction removal image data generated by the image processing unit 209.

  The encoder unit 210 converts various image data including color image data processed by the image processing unit 209 and shielding object removal image data generated by the shielding object removal process into a file format such as Jpeg.

  The media I / F 211 is an interface for transmitting / receiving image data to / from a PC / media 213 (for example, a hard disk, a memory card, a CF card, an SD card, etc.).

<Internal configuration of image processing unit>
FIG. 3 is a block diagram showing an internal configuration of the image processing unit 209 in the present embodiment.

  The development processing unit 301 performs white balance processing, demosaic processing, noise reduction processing, color conversion processing, edge enhancement processing, gamma processing, and the like on the RAW image data acquired from the A / D conversion unit 208, and converts the color image data. Generate. The generated color image data is stored in a storage device such as the RAM 204 and the PC / media 213.

  The shielding object removal processing unit 302 removes the shielding object included in the subject based on the plurality of color image data generated by the development processing unit 301, and generates shielding object removal image data. The generated shield removal image data can be output to and stored in a storage device such as the RAM 204 or the PC / media 213 or output to the display unit 106 for display.

  The display image generation unit 303 generates display image data based on the distance image data, color image data, and shield removal image data acquired from the distance image acquisition unit 105. The generated display image data is output to the display unit 106.

<Processing flow of image processing unit>
FIG. 4 is a flowchart showing an operation procedure of the image processing unit 209 in the image processing apparatus of this embodiment. First, shielding object removal image data is generated based on a plurality of color image data captured at different imaging positions. Then, display image data is generated from the color image data and the shield removal image data based on the distance image data, and is output to the display unit 106 to perform live view display. At this time, if the imaging device is away from the shielding object, display image data is generated from the image from which the shielding object is removed, and if the imaging device approaches the shielding object, display image data is generated so that the shielding object can be visually recognized. To do. Details of the processing will be described below.

  In step S <b> 401, the development processing unit 301 generates color image data based on the RAW image data acquired from the A / D conversion unit 208. Here, the color image data captured at time t is assumed to be I (t). The generated color image data is stored in a storage device such as the RAM 204 and the PC / media 213. Note that the storage device also stores N pieces of color image data I (t−1) to I (t−N) generated before I (t).

  In step S <b> 402, the display image generation unit 303 acquires distance image data from the distance image acquisition unit 105. Here, the distance image data captured at time t is L (t). FIG. 5 shows an example of color image data and distance image data. In this embodiment, the distance from the imaging device to the subject is d1, the distance from the imaging device to the shielding object is d2, and each pixel of the distance image data is a pixel value as a distance from the imaging device to each object such as the subject or the shielding object. Assume that a value is stored.

  In step S403, the shielding object removal processing unit 302 performs shielding object removal processing based on the plurality of color image data, and generates shielding object removal image data. Here, the state of SW1 is ON. Details of the shielding removal process will be described later.

  In step S404, it is determined whether or not the shielding object removal processing unit 302 outputs shielding object removal image data based on the state of SW2. If SW2 is OFF, the process proceeds to step S405. If SW2 is ON, SW2 is turned OFF and then the process proceeds to step S407. SW2 is turned on when the user instructs the output of the shield removal image.

  In step S405, the display image generation unit 303 generates display image data from the color image data and the shield removal image data based on the distance image data. Details of the display image generation processing will be described later.

  In step S406, the display image generation unit 303 outputs the display image data to the display unit 106, thereby displaying the display image data in a live view.

  In step S407, the shielding object removal processing unit 302 stores the shielding object removal image data in a storage device such as the PC / media 213. Further, the shield removal image data is output to the display unit 106 and displayed.

  In step S408, the shielding object removal processing unit 302 determines whether or not to continue the processing based on the state of SW1. If SW1 is ON, the process proceeds to step S401, where color image data I (t + 1) at the next time is acquired and the process is continued. If SW1 is OFF, the process ends.

<Shield removal process>
Here, the shielding removal process performed in step S403 will be described. In the shielding object removal process, the shielding object included in the color image data is removed based on the plurality of color image data to generate shielding object removal image data. Hereinafter, the details of the shielding removal process will be described with reference to the flowchart shown in FIG.

  First, in step S601, the obstruction removal processing unit 302 uses the image data I (t) as a reference image, and N−1 pieces of past image data I (t−1) to I (t−N) are coordinate systems of the reference image. To coordinate-converted image data I ′ (t−1) to I ′ (t−N).

  Specifically, first, feature points useful for associating each image are extracted. A feature point is a point characteristic of information (luminance or RGB value) around the point, and is extracted using SIFT, SURF, or the like. Next, the reference points are associated with feature points of other images. The peripheral information of the feature points is compared with the peripheral information of the feature points of other images, and the matching information is associated. Finally, a matrix for converting each image into a reference image is calculated from the associated feature points, and each image is converted into the coordinate system of the reference image. In the present embodiment, feature points are associated by RANSAC (RANdom Sample Consensus), and a transformation matrix is calculated. The coordinate conversion image data is generated by converting the multi-viewpoint image data into the coordinate system of the reference image using the obtained conversion matrix.

  Next, in step S602, the shielding object removal processing unit 302 generates shielding object removal image data obtained by removing the shielding object from the reference image data and the coordinate conversion image data. A shielding object (foreground object) such as a net shown in FIG. 5 partially shields the subject, and occupies a smaller area in the color image data than the subject (background object). For this reason, when a pixel at the same position is referred to in the base image data and the coordinate conversion image data, the number of images in which the subject is reflected in the pixel tends to be larger than the number of images in which the shielding object is reflected. An example is shown in FIG. FIG. 7 shows a total of five examples of standard image data I (t) and coordinate-converted image data I ′ (t−1) to I ′ (t-4) (reference image data). The pixels indicated by the marks indicate the same position on each image data, and the shielding object is shown in the reference image data, and the subject is shown in the four coordinate conversion image data. In this way, when a pixel at a specific position is referred to on each image data, there is more image data showing a subject than image data showing a shielding object. Therefore, for example, a vector median filter is used to remove the shielding object.

  The vector median filter is an extension of the one-dimensional median filter to multiple dimensions, and is a filter that selects a vector that minimizes the sum of distances from other vectors. The expression for the vector median filter is shown below.

Here, m is a vector median, v is an input vector representing the pixel value of each pixel, and N is the number of input vectors. As the pixel value, an RGB value or a Lab value is handled as a three-dimensional vector v. A vector median filter is applied to the pixel group at each coordinate in the reference image coordinate system. Thereby, the pixel value m of the subject having a large number of images is selected, and the shielding object is removed from the image.

  FIG. 8 is an example of the shield removal image data, and it can be seen that the shield has been removed. With the above processing, the shielding object included in the color image data can be removed based on the plurality of color image data, and the shielding object removal image data H (t) can be generated.

<Display image generation processing>
Here, the display image generation process performed in step S405 will be described. In the display image generation process, display image data S (t) is generated based on the color image data I (t), the distance image data L (t), and the shield removal image data H (t). Hereinafter, the details of the display image generation processing will be described with reference to the flowchart shown in FIG.

  In step S901, the display image generation unit 303 estimates the distance value from the viewpoint position of the color image data to the shielding object based on the distance image data L (t). Here, the viewpoint position of the color image data is regarded as the position of the imaging device, and the distance value is estimated. First, a histogram is generated based on the distance value stored in each pixel in the distance image data L (t). Next, bins in which the number of included pixels is equal to or greater than a threshold are extracted from the bins in the histogram. Then, the bin with the smallest distance value is selected from the extracted bins. The median value of the distance values corresponding to the selected bin is set as the distance value d to the shielding object.

  In step S902, the display image data is added by weighting the color image data I (t) and the obstruction removal image data H (t) based on the distance value d to the obstruction acquired by the display image generation unit 303 in step S901. S (t) is generated. At this time, the weight α (the transmittance of the shielding object) for the shielding object removal image H (t) is set to be smaller as d is smaller. In the present embodiment, as shown in FIG. 10A, α is set so as to increase monotonously with respect to d using predetermined threshold values D1 and D2 (predetermined values) (D1 <D2). . Α is a parameter that sets the maximum value of α. Specifically, it is represented by the following formula.

Then, the display image data S (t) is calculated by weighted addition of the color image data I (t) and the obstruction removal image data H (t) according to the following equation using the weight α according to the following equation.

Through the above processing, the display image data S (t) can be generated so that the shielding object becomes easier to visually recognize as d becomes smaller.

  An example in which the display image data S (t) is displayed on the display unit 106 is shown in FIG. Here, the maximum value A of α is set to 1. Since α = 0 when d ≦ D1, a shielding object is displayed as shown in FIG. In the case of D1 <d <D2, 0 <α <1, so that the shielding object is displayed as shown in FIG. 11B. In the case of D2 ≦ d (greater than or equal to a predetermined value), α = 1, so that the shielding object is not displayed as shown in FIG.

  The display image generation method is not limited to this. For example, the method for setting the weight α may be as shown in FIG. Specifically, it is represented by the following formula.

In this case, the change in the display image data can be increased when d is around D1 (predetermined value). Here, the maximum value A of α is also set to 1. The manner in which the display image data S (t) generated in this way is displayed on the display unit 106 is shown in FIGS. 11 (a) and 11 (c). That is, in the case of d <D1, α = 0, so that a shielding object is displayed as shown in FIG. Since α = 1 in the case of D1 ≦ d, the shielding object is not displayed as shown in FIG.

  The distance acquisition process is not limited to the process described above. For example, the imaging device may be a stereo camera as shown in FIG. 12, and distance image data may be acquired from the acquired color image data having parallax using triangulation. In FIG. 12, two color image data having parallax are acquired using two optical units 1201 and 1202. However, the imaging apparatus includes three or more optical units and acquires three or more color image data having parallax. You may do it. Or you may use the proximity sensor which can detect that the distance from an imaging device to a shield became below a specific value.

  In addition, the process which estimates the distance value from the viewpoint position of color image data to a shield is not restricted to the process mentioned above. For example, a distance value at which the distance from the viewpoint position is considered to be the minimum may be estimated from the distance image data L (t) and used as the distance value to the shielding object. Specifically, the minimum value of the distance value stored in each pixel in the distance image data L (t) may be used as the distance value to the shielding object.

  The shielding removal process is not limited to the process described above. For example, the area of the shielding object in the image may be designated, and the designated area may be filled using the pixel values of other areas.

  As described above, the shielding object removal result can be confirmed in the live view, and when the imaging apparatus approaches the shielding object, the shielding object can be visually recognized in the live view.

[Embodiment 2]
In the first embodiment, the display image is controlled so as to display the shielding object when the imaging apparatus approaches the shielding object. In the present embodiment, a warning is given to the user when the imaging apparatus approaches the shield.

  This embodiment is different from the first embodiment in the display image generation process. The flow of the display image generation process in this embodiment is shown in FIG.

  In step S1301, the display image generation unit 303 acquires the distance value d to the shielding object as in step S901 of the first embodiment.

  In step S1302, the display image generation unit 303 compares the distance value d with a threshold value D1 (predetermined value). If the distance value d is less than or equal to the threshold value D1, the process proceeds to step S1303. If the distance value d is greater than the threshold value D1, the process proceeds to step S1304.

  In step S1303, the display image generation unit 303 generates an image in which the distance value d is superimposed on the shield removal image data as a display image. First, the character generation unit 207 generates a graphic corresponding to the distance value d. And display image data is produced | generated by superimposing the produced | generated graphic on obstruction removal image data, and a process is complete | finished.

  In step 1304, the display image generation unit 303 converts the obstruction removal image data into a display image, and the process ends.

  An example in which the display image data S (t) in this embodiment is displayed on the display unit 106 is shown in FIG. As shown in FIG. 14A, when the distance d from the imaging device to the shielding object is equal to or less than D1, the distance d to the shielding object is displayed on the display unit 106. Here, an example in which D1 = 25 cm and d = 20 cm is shown. When the distance d becomes larger than D1, the distance value d is not displayed as shown in FIG.

  In the present embodiment, the warning is given to the user by displaying the distance value d on the display unit 106, but the warning method is not limited to this. For example, the warning screen set in advance in step S1303 may be displayed. Alternatively, a warning by voice or sound may be performed. In step S1303, a preset voice or warning sound may be reproduced. In the present embodiment, the distance value d is displayed on the display unit 106 only when the distance value d is equal to or less than the threshold value D1, but the distance value d may be always displayed. In this case, only when the distance value d is equal to or less than the threshold value D1, a warning is given by highlighting the distance value d. Specifically, the display of the distance value d may be enlarged or the color may be changed. Alternatively, a warning may be given by voice or sound instead of emphasizing the distance value d.

  Note that this embodiment can also be implemented in combination with the first embodiment.

  As described above, the shield removal result can be confirmed in the live view, and the user can be warned when the imaging apparatus approaches the shield.

[Embodiment 3]
In the first and second embodiments, information is presented to the user when the imaging apparatus approaches the shield. In this embodiment, the optical unit 102 is protected.

  First, the configuration of the imaging apparatus in the present embodiment will be described. In this embodiment, the optical unit 102 is a collapsible imaging optical system, and is movable in the feeding direction and the feeding direction. For this reason, in accordance with a user instruction such as turning on the power, the optical unit 102 can be extended so as to be ready for imaging as shown in FIG. Alternatively, the optical unit 102 can be protected by retracting the optical unit 102 as shown in FIG. 15B in accordance with a user instruction such as power OFF. Hereinafter, the state as shown in FIG. 15A is referred to as the extended state, and the state as shown in FIG. 15B is referred to as the extended state. The optical system control unit 205 controls the operation of moving the optical unit 102 in the retraction direction and the operation of moving in the retraction direction according to the imaging parameters including the focal length and the focus position determined by the user instruction. Further, the optical system control unit 205 records the lens state including the extended state and the extended state in a RAM (Rondom Access Memory) 204 or the like.

  FIG. 16 is a flowchart showing the operation procedure of the image processing apparatus of this embodiment. In the present embodiment, when the distance value d from the imaging device to the shielding object becomes equal to or less than a predetermined distance value D1 (less than a predetermined value), control is performed so that the optical unit 102 is retracted. Thereby, the optical unit 102 is protected. Details of the processing will be described below.

  In step S1601, the optical system control unit 205 performs control to feed out the optical unit 102 in response to a user operation such as power ON.

  In step S1602, the image processing unit 209 acquires the distance value d to the shielding object based on the distance image data acquired from the distance image acquisition unit 105 as in step S901 of the first embodiment.

  In step S1603, the image processing unit 209 compares the distance value d with a predetermined distance value D1. If the distance value d is equal to or less than D1, the process proceeds to step S1604. If the distance value d is greater than D1, the process proceeds to step S1607.

  In step S1604, the optical system control unit 205 determines the state of the optical unit 102. If the optical unit 102 is not in the retracted state, the process proceeds to step S1605. If the optical unit 102 is in the retracted state, the process proceeds to step S1611.

  In step S1605, the optical system control unit 205 stores imaging parameters such as a focal length and a focus position related to the state of the optical unit 102 as a reference state in a storage device such as the RAM 204.

  In step S1606, the optical system control unit 205 moves the optical unit 102 forward, and the process proceeds to step S1611.

  In step S1607, the optical system control unit 205 determines the state of the optical unit 102. If the optical unit 102 is in the retracted state, the process proceeds to step S1608. If the optical unit 102 is not in the retracted state, the process proceeds to step S1610.

  In step S <b> 1608, the optical system control unit 205 reads out imaging parameters such as a focal length and a focus position related to the state of the optical unit 102 from a storage device such as the RAM 204.

  In step S1609, the optical system control unit 205 extends the optical unit 102 so as to reproduce the reference state indicated by the imaging parameter read in step S1608.

  In step S1610, imaging processing according to the imaging mode set by the user instruction is performed. Here, the imaging process described in the second embodiment is performed.

  In step S1611, the image processing unit 209 determines whether to continue the processing based on the state of SW1. If SW1 is ON, the process proceeds to step S1602, and if SW1 is OFF, the process ends.

  An example in which the optical unit 102 is protected by the above processing is shown in FIG. In the case of d ≦ D1, the optical unit 102 is brought into a retracted state as shown in FIG. Although the image cannot be taken, the optical unit 102 can be protected. When D1 <d, as shown in FIG. 17B, the optical unit 102 is in the extended state, and imaging is possible.

  Note that this embodiment can be implemented in combination with the first and second embodiments.

  As described above, the optical unit 102 can be protected when the imaging apparatus approaches the shield.

[Embodiment 4]
In the third embodiment, after the optical unit 102 is brought into the retracted state, it is determined based on the distance value d that the optical unit 102 is brought into the retracted state. In this case, it cannot be applied to a configuration in which the distance value d is calculated using color image data. Therefore, in this embodiment, after the optical unit 102 is brought into the retracted state, the optical unit 102 is fed out after a predetermined time has elapsed.

  FIG. 18 is a flowchart showing the operation procedure of the image processing apparatus of this embodiment. Steps S1801, S1802, and S1803 are the same processes as steps S1601, S1602, and S1603 of the third embodiment, respectively. If it is determined in step S1803 that d is equal to or smaller than D1 (predetermined value), the process proceeds to step S1804. If d is greater than D1, the process proceeds to step S1809. Note that steps S1804 and S1805 are the same processes as steps S1605 and S1606 of the third embodiment, respectively.

  In step S1806, the process waits for a predetermined time. After a predetermined time has elapsed, the process proceeds to step S1807.

  Thereafter, step S1807, step S1808, step S1809, and step S1810 are the same processes as step S1608, step S1609, step S1610, and step S1611 of the third embodiment, respectively.

  Note that this embodiment can be implemented in combination with the first and second embodiments.

  As described above, the optical unit 102 can be changed from the retracted state to the extended state even for the configuration in which the distance value d is calculated using the color image data.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (14)

Imaging means for capturing image data by capturing a background object partially shielded by the foreground object;
Removing means for removing the foreground object from the image data and generating foreground object removal image data;
Distance acquisition means for acquiring a distance from the imaging means to the foreground object;
An image processing apparatus comprising: display means for generating and displaying display image data from at least one of the image data and the foreground object removal image data based on the distance.   The display means displays the foreground object removal image data as the display image data when the distance is greater than or equal to a predetermined value, and the image data and the foreground object removal image data when the distance is smaller than a predetermined value. The image processing apparatus according to claim 1, wherein the display image data is generated and displayed so that the foreground object passes through the display.   The image processing apparatus according to claim 2, wherein the image processing apparatus is set such that the transmittance of the foreground object decreases as the distance decreases.   The display means displays the foreground object removal image data as the display image data when the distance is greater than or equal to a predetermined value, and displays the image data including the foreground object when the distance is smaller than a predetermined value. The image processing apparatus according to claim 1, wherein the image processing apparatus displays the display image data.   The image processing apparatus according to claim 1, wherein the display unit generates and displays the display image data by superimposing information on the distance on the foreground object removal image data.   The image processing apparatus according to claim 1, further comprising a warning unit that warns a user when the distance is equal to or less than a predetermined value.   The image processing apparatus according to claim 1, further comprising a control unit that moves an imaging optical system of the imaging unit in a retraction direction when the distance is equal to or less than a predetermined value.   The image according to claim 7, wherein the control unit moves the imaging optical system in a feeding direction when the distance is larger than a predetermined value in a state where the imaging optical system is retracted. Processing equipment.   The said control means moves the said imaging optical system to a paying-out direction, when it determines with predetermined | prescribed time having passed in the state where the said imaging optical system was retracted. Image processing apparatus.   The image processing apparatus according to claim 8, wherein the control unit stores a reference state determined by a user instruction, and reproduces the reference state when the imaging optical system is moved in a feeding direction. . The imaging means continuously performs imaging,
The image processing apparatus according to claim 1, wherein the display unit performs live view display.   The removing unit generates reference image data by performing coordinate conversion of past image data having a different viewpoint acquired before the image data by the imaging unit so as to have the same viewpoint as the image data, and generating the reference image data. The image processing apparatus according to claim 11, wherein the foreground object removal image data is generated based on the image data and the image data. An imaging step of capturing image data by capturing a background object partially shielded by the foreground object;
Removing the foreground object from the image data and generating foreground object removal image data;
A distance acquisition step of acquiring a distance from the imaging means to the foreground object;
An image processing method comprising: a display step of generating display image data from at least one of the image data and the foreground object removal image data based on the acquired distance.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 12.