JP2004328657A - Image input device, image input method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image input method, and an image input device using it, capable of obtaining the depth information on a substance in an image or a picture comprising the depth information easily and moreover at a low cost. <P>SOLUTION: This image input device is equipped with an imaging means for taking an image of an object of imaging by a set exposure time interval, and a light emitting means for radiating light to the object of imaging. When the direction for imaging is inputted, the exposure time is set to a first time interval, light is radiated to the object of imaging with the light emitting means to take a first image of the object with the imaging means. The exposure time is set to the first time, and a second picture of the object of imaging is taken without radiating light to the object. The difference between the first image and the second image is found, and a third picture which is a picture representing the intensity distribution of reflected light by the object of imaging of light radiated to the object by the light emitting means, and comprises depth information on the object, is obtained. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a digital still camera configured using a solid-state imaging device such as a MOS imaging device or a CCD imaging device.
[0002]
[Prior art]
In recent years, image input devices using solid-state imaging devices such as CCD image sensors have been widely used. Recently, it has become possible to easily transfer an image to a personal computer (hereinafter, referred to as a PC) or the like, and to print the image on a widely used color printer.
[0003]
On the other hand, imaging of a three-dimensional image has been attempted for a long time. In the old days, stereoscopic viewing could be performed by capturing parallax images with a two-eye silver halide camera and viewing them with a dedicated observation device. In recent years, it has been required to input an image with a depth value instead of two images with parallax in order to not only see a three-dimensional image but also variously process and reuse it. There are many. As a result, it is possible to reproduce a three-dimensional image of the imaged object viewed from another viewpoint or to combine the three-dimensional image with a three-dimensional CG (computer graphics).
[0004]
Here, when referring to the depth value, it is an exact distance value from a reference point or a reference plane, an attribute divided into several layers (for example, near, medium, far, etc.), or acquired. The distance values may have a certain degree of error due to the characteristics of the system, but here, they are broadly referred to as depth values.
[0005]
Also, there may be one depth value per pixel of the corresponding image, or it may not. Further, the distribution of the depth values of the measured object or the scene will be referred to as a depth image.
[0006]
A device for acquiring a depth image already exists. A typical example is a laser range finder. In this method, an object to be measured is mainly irradiated with slit light by a laser, the reflected light is imaged from different positions, and the distance of each part is obtained from the distortion of the shape of the slit light by using a trigonometric method ( For example, see Patent Document 1).
[0007]
There is also a method in which pattern light such as a striped pattern is irradiated instead of slit light, the reflected light is imaged, and the depth is obtained from a change in the pattern light (for example, see Patent Document 2).
[0008]
When the laser range finder or the space code method is used, there are many products for specific applications, and there is a problem that the apparatus becomes large-scale and the cost increases. In addition, since it is generally necessary to acquire and process a plurality of images, it takes time to capture images. Therefore, what can be acquired is limited to a stationary object.
[0009]
In some cases, the distance is measured by irradiating the object with irradiation light and measuring the time until the reflected light returns (for example, see Non-Patent Document 1). However, this type is also mainly used for dedicated applications such as broadcasting stations, and is very large and expensive.
[0010]
A stereo camera method is an approach for obtaining depth information from a difference between images of a twin-lens camera in the same manner as a human determines the depth of an object. The conventional stereo camera method had problems such as strict camera calibration.However, recently, an object was placed on the floor where reference points were placed, and multiple images including the reference points were recorded. There is also a method of obtaining both a plurality of images and the position where each image was captured by obtaining the three-dimensional shape by using the correlation between the images (for example, see Patent Document 3).
[0011]
In the stereo camera method, generally, it is necessary to strictly adjust the positions of a plurality of cameras, which leads to difficulty in operation and an increase in cost. In addition, the stereo camera method calculates a distance by finding a corresponding point (corresponding area) in which the same part is reflected among a plurality of images. There is a tendency that the accuracy of the search significantly deteriorates depending on the quality and the imaging conditions. In actual use, it is necessary to cover the background with a single color material and make the lighting uniform so that the images from multiple viewpoints do not differ too much, so that it is difficult to place corresponding point search errors. It is. In the type of technique using a reference chart or the like for specifying the camera position, there is no problem of calibrating the camera position because a plurality of cameras are not used, but otherwise, the same problem occurs. Further, in this case, since a plurality of images are captured by the same camera for a long time, it is difficult to capture an object such as a person that is difficult to be stationary. Also, it is difficult to image an object that is too large for the chart.
[0012]
There is also an approach of acquiring the depth of an image from a plurality of images obtained by irradiating a target object with a plurality of known illuminations.
[0013]
However, in general, in a technology of a type that irradiates light, when an image is captured in a bright place, depth acquisition cannot often be performed correctly due to the influence of external light. For this reason, there is also inconvenience that it is necessary to prepare a dedicated shooting place.
[0014]
As described above, the conventional technology for acquiring an image including depth information is mainly an expensive and large-scale device mainly for a specific application, and there is no easy imaging. That is, with the existing technology, it is impossible to acquire a depth image as if many digital camera users are now easily taking and using digital photographs.
[0015]
In recent years, three-dimensional image display apparatuses have also been remarkably evolved and cost-reduced, and are now mounted on displays of mobile phones. However, since there is no environment where a depth image can be easily obtained as described above, the content to be displayed is either downloaded from a specialized company and displayed, or the depth is estimated by image processing with a certain target assumed. You will use the tools you create. The depth image created by such a tool is only an estimation and differs from the actual situation. If the object is not the assumed object, the optimal estimation is often not performed.
[0016]
[Patent Document 1]
Patent No. 3317093 (Fig. 1)
[0017]
[Patent Document 2]
Patent No. 3269484 (Fig. 3)
[0018]
[Patent Document 3]
JP-A-11-96374 (FIG. 10)
[0019]
[Non-patent document 1]
3DV Systems' homepage, [online], [searched March 25, 2003], Internet <URL: http: // www. 3dvsystems. com / >>
[0020]
[Problems to be solved by the invention]
As described above, the conventional techniques for acquiring an image including depth information are mainly expensive and large-scale apparatuses mainly for specific applications, and easily and at low cost to acquire depth information of an object in an image. There was a problem that it was not possible.
[0021]
In view of the above problems, the present invention provides an image input method and an image input device using the same, which can easily and at low cost acquire depth information of an object in an image or an image including depth information. The purpose is to do.
[0022]
In addition, by simply performing the same operation as an imaging operation of a general digital camera or the like, it is possible to acquire a normal image (natural image) and an image including depth information or depth information of an object in the image. It is an object of the present invention to provide a method and an image input device using the same.
[0023]
[Means for Solving the Problems]
(1) The present invention provides an image input device having an image pickup device for picking up an image of an image pickup target with a set exposure time, and a light emitting unit for irradiating the image pickup target with light. Is input, the exposure time is set to a first time, the light emitting unit irradiates the imaging target with light, the imaging unit captures a first image of the imaging target, and A time is set to the first time, a second image of the imaging target is captured by the imaging unit without irradiating the imaging target with light by the light emitting unit, and the first image and the second image are captured. A third image including depth information of the imaging target, which is an image representing an intensity distribution of light reflected by the imaging target of light emitted to the imaging target by the light emitting unit, by calculating a difference between the two images. It is characterized by obtaining.
[0024]
Further, after the second image is captured, the exposure time is set to a second time different from the first time (for example, longer than the first time), and the imaging object is set by the imaging unit. Is captured.
[0025]
According to the present invention, it is possible to easily and at low cost acquire depth information of an object in an image or an image including depth information. A first image with an emission time of a first time and light emission by the light emitting means, a second image with an exposure time of the first time and no light emission by the light emission means, and Three images such as a fourth image having an exposure time of a second time are captured, and a third image including depth information can be obtained from the first image and the second image. Further, with a single imaging instruction, it is possible to obtain a fourth image (natural image) that is a normal image (an image captured with a normal appropriate exposure time) together with the third image including the depth information. it can.
[0026]
Note that the second image may be captured after capturing the first image, or the first image may be captured after capturing the second image. Whichever image is taken first has no effect on the above-described effects.
[0027]
(2) The present invention relates to an image input apparatus having an image pickup unit for picking up an image of an image pickup target with a set exposure time, and a light emitting unit for irradiating the image pickup target with light. When an instruction is input, the exposure time is set to a first time, the light emitting unit irradiates the imaging target with light, and the imaging unit captures a first image of the imaging target, An exposure time is set to a second time different from the first time (for example, longer than the first time), and the light-emitting unit does not irradiate the image-capturing object with light. A second image of the second image, and from each pixel value of the second image, an image component imaged during the first time of the second time is extracted. A third image, which is an extracted image component, is obtained, and the first image and the third image are obtained. Obtaining a difference from an image to obtain a fourth image including an intensity distribution of reflected light of the light irradiated on the imaging target by the light emitting unit and reflected by the imaging target and including depth information of the imaging target. It is characterized by the following.
[0028]
The third image is obtained by multiplying each pixel value of the second image by a ratio of the first time to the second time.
[0029]
The third image is obtained by dividing each pixel value of a plurality of pixels constituting the second image by a pixel value of the first image corresponding to each of the plurality of pixels. From among the values corresponding to each of the pixels, a value corresponding to the ratio of the first time to the second time is selected, and each pixel value of the second image is multiplied by the selected value. , The third image is obtained.
[0030]
According to the present invention, it is possible to easily and at low cost acquire depth information of an object in an image or an image including depth information. In addition, a first image with an exposure time of a first time and light emission by a light emitting unit and a second image with an exposure time of a second time and no light emission by a light emitting unit in one imaging instruction. , And a fourth image including depth information can be obtained from the first image and the second image. Further, with a single imaging instruction, it is possible to obtain a second image (natural image) that is a normal image (an image captured with a normal appropriate exposure time) together with the fourth image including the depth information. it can.
[0031]
Note that the second image may be captured after capturing the first image, or the first image may be captured after capturing the second image. Whichever image is taken first has no effect on the above-described effects.
[0032]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0033]
In the present embodiment, an object is to realize an image input device capable of acquiring a depth image at low cost. Therefore, it is characterized in that it can be realized with a configuration as close as possible to a conventional digital camera. Therefore, before describing this embodiment, the configuration and operation of a conventional digital camera will be described first.
[0034]
FIG. 16 shows a schematic configuration of a conventional digital camera. The subject forms an image on the image sensor 104 through the lens 101. An aperture 102 and a shutter 103 are provided between the lens 101 and the image sensor 104. To be precise, the aperture 102 and the shutter 103 may be inside a lens system composed of a plurality of lenses, but are simplified in FIG.
[0035]
The aperture 102 adjusts the amount of light that enters the image sensor 104, and the shutter 103 opens at the time of imaging to allow light to reach the image sensor 104. The image sensor is typically a CCD (Charge Coupled Device) image sensor and a CMOS image sensor. Here, the description will be made mainly using a CCD image sensor. The image sensor 104 two-dimensionally arranges a plurality of photodiodes (PD) corresponding to each pixel that converts light into electric charge, and accumulates electric charge according to the intensity of light received by each photodiode. It comprises a photoelectric conversion unit 104a for converting an optical image into a charge image, and a transfer unit 104b for transferring charges (charge images) accumulated in each photodiode to a signal processing circuit. The charge (charge image) accumulated in each photodiode of the photoelectric conversion unit 104a is temporarily transferred to the transfer unit 104b. The transfer unit 104b outputs the charge image to the signal processing circuit 105 as a one-dimensional electric signal (image signal).
[0036]
Since the image sensor 104 itself can control the charge accumulation time of the PD, it operates without a mechanical shutter unlike a film camera. However, since the image sensor 104 causes various problems such as noise when light strikes the sensor even when no image is captured, a digital still camera often uses a mechanical shutter together.
[0037]
The output (image signal) from the image sensor 104 is stored in the memory 106 via the signal processing circuit 105. The signal processing circuit 105 performs color image processing for generating a color image, noise reduction, contrast adjustment, gamma conversion, and the like from the signals of the R, G, and B pixels on the image sensor 104.
[0038]
In the case of a CCD image sensor, since the output is generally an analog voltage output, an amplifier or an A / D converter is often provided before the above signal processing. However, it is the structure containing these. In a chip set of a CCD image sensor, it is often realized by one LSI including this amplifier, A / D converter, color processing, and the like.
[0039]
The strobe 112 emits light to the subject. It is used as an auxiliary light source when shooting in a dark environment or when the subject is darker than the surroundings due to backlight. The control unit 113 controls each of the above components.
[0040]
The actual imaging operation timing will be described with reference to FIG. Here, the operation when flash photography is performed will be described. 17A shows the operation of the shutter 103, FIG. 17B shows the charge accumulation operation of the photoelectric conversion unit 104a in the image sensor 104, and FIG. 17C shows the operation of the image sensor 104 (of the transfer unit 104b). FIG. 17D shows an output operation, and FIG. 17D shows a flash emission operation. In each case, the horizontal axis is the time axis, and represents each operation timing with the passage of time (from left to right in FIG. 17).
[0041]
First, each PD (photoelectric conversion unit 104a) of the image sensor 104 is reset all at once by a user's photographing instruction (generally by pressing a shutter button, not shown) (step S201). Specifically, by discharging the charges stored in each PD until then, new charge accumulation starts from this point. After the reset of the photoelectric conversion unit 104a, the shutter opens (step S202). In the operation of the shutter 103 in FIG. 17A, “1” indicates that the shutter is open, and “0” indicates that the shutter is closed. Strictly speaking, since the mechanical shutter has a response delay and an operation time, the shutter is not fully opened instantaneously and rises smoothly from “0” to “1”. However, in FIG. In this case, it is shown that the signal rises instantaneously from “0” to “1”.
[0042]
After the shutter 103 opens, the flash 112 emits light (step S203). Note that the light emission operation of the strobe 112 in FIG. 17D indicates “1” when emitting light, and “0” indicates when not emitting light.
[0043]
After a certain time tx from the opening of the shutter 103, the shutter 103 closes (step S204), and the electric charge of each PD of the photoelectric conversion unit 104a is transferred to the transfer unit 104b of the image sensor 104 (step S205). The charges output to the transfer unit 104b are sequentially output from the transfer unit 104b to the signal processing circuit 105 (Step S206).
[0044]
The timing of resetting the photoelectric conversion unit 104a or transferring electric charges from the photoelectric conversion unit 104a to the transfer unit 104b and the timing of opening and closing the shutter 103 are not necessarily as illustrated. For example, the photoelectric conversion after the shutter 103 is opened may be performed. The unit 104a may be reset, or the charge may be transferred to the transfer unit 104b before the shutter 103 is closed.
[0045]
In the operation of the photoelectric conversion unit 104a, a period from reset (discharge of electric charge) to transfer of electric charge to the transfer unit 104b is a time period in which the photoelectrically converted electric charge is accumulated.
[0046]
Actually, no light enters if the shutter 103 is closed even when the CCD image sensor 104 can store electric charges. Therefore, the time during which the CCD image sensor 104 can store charges and the shutter is open is the so-called exposure time. In FIG. 17, the exposure time is tx. Hereinafter, this state (exposure time period) will be referred to as an “imaging state”. Further, when shooting without using a strobe, it is only necessary that the strobe is not emitted by the above operation. As to how to set the exposure time tx, a value determined by manual setting or automatic exposure control is used.
[0047]
Another example of the imaging operation of the conventional digital camera shown in FIG. 16 will be described with reference to a flowchart shown in FIG. The flowchart starts according to a photographing request from the user. The shutter 103 is opened (step S211), and the charge of each PD of the photoelectric conversion unit 104a is discharged to start charge accumulation (step S212). Immediately after the start, the flash 112 emits light (step S213). After that, it waits tx after the start of imaging (step S214), transfers the charge of each PD of the photoelectric conversion unit 104a to the transfer unit 104b (step S215), and closes the shutter 103 (step S217). The output of the image signal from the CCD image sensor 104 is started (step S218).
[0048]
Although not shown in the flowchart of FIG. 18, the image signal output from the image sensor 104 is input to a signal processing circuit 105, subjected to predetermined signal processing by the signal processing circuit 105, and Images are stored.
[0049]
Next, a first embodiment of the present invention will be described with reference to the drawings.
[0050]
Here, the depth information is, for example, a distance value from a certain reference point or a reference plane, an attribute for dividing into several layers (for example, near, medium, far, etc.), or imaging of an image. The distance value may have a certain error due to the characteristics of the system. That is, information on the three-dimensional shape of the imaging target in the image, the positional relationship between the plurality of imaging targets in the image (absolute or relative to a certain reference point), level, etc. Is information representing
[0051]
The reflected light image is an image representing the intensity distribution of light reflected on the imaging target by light emitted to the imaging target by a light emitting unit such as a strobe. Since the numerical values such as the distance value and the level as the depth information can be calculated based on each pixel value of the reflected light image, the reflected light image can be said to be an image including the depth information.
[0052]
Strictly speaking, the depth image is an image whose pixel value is the depth information and can be obtained from the reflected light image. However, here, the reflected light image is regarded as a kind of the depth image because its pixel value includes the depth information.
[0053]
<First embodiment>
FIG. 1 shows a schematic configuration example of the image input device according to the first embodiment. In FIG. 1, the same parts as those in FIG. 16 are denoted by the same reference numerals. This configuration is very similar to the configuration of the general digital still camera shown in FIG. The difference from the configuration of the digital camera shown in FIG. 16 is that FIG. 1 has an arithmetic unit 111 which is not included in the conventional digital camera shown in FIG. 16, and further includes a control unit 113 shown in FIG. 1, the control unit 114 is connected. Further, three images (first image 107 to third image 109) are captured by one shutter operation (depression of the shutter button) by the user, and these are stored in the memory 106. Then, as described later, a depth image is generated from the first image 107 and the second image 108, and is also stored in the memory 106.
[0054]
First, each component of the image input device of FIG. 1 will be described.
[0055]
The subject forms an image on the image sensor 104 through the lens 101. An aperture 102 and a shutter 103 are provided between the lens 101 and the image sensor 104. To be precise, the aperture 102 and the shutter 103 may be inside a lens system including a plurality of lenses, but are simplified in FIG. In some cases, an integrated lens is used as a stop and a shutter.
[0056]
The aperture 102 adjusts the amount of light that enters the image sensor 104, and the shutter 103 opens at the time of imaging to allow light to reach the image sensor 104. The image sensor is typically, for example, a CCD (charge-coupled device) image sensor, a CMOS image sensor, or the like. Here, the description will be made mainly using a CCD image sensor. The image sensor 104 two-dimensionally arranges a plurality of photodiodes (PD) corresponding to each pixel that converts light into electric charge, and accumulates electric charge according to the intensity of light received by each photodiode. It comprises a photoelectric conversion unit 104a for converting an optical image into a charge image, and a transfer unit 104b for transferring charges (charge images) accumulated in each photodiode to a signal processing circuit. The charge (charge image) accumulated in each photodiode of the photoelectric conversion unit 104a is temporarily transferred to the transfer unit 104b. The transfer unit 104b outputs the charge image to the signal processing circuit 105 as a one-dimensional electric signal (image signal).
[0057]
The output of the image sensor 104 is stored in the memory 106 via the signal processing circuit 105. The signal processing circuit 105 performs color image processing for generating a color image, noise reduction, contrast adjustment, gamma conversion, and the like from the signals of the R, G, and B pixels on the image sensor 104. In the case of a CCD image sensor, since the output is generally an analog voltage output, an amplifier or an A / D converter is often provided before the above signal processing. However, it is the structure containing these.
[0058]
The first image 107, the second image 108, the third image 109, and the depth image 110 in the memory 106 each represent image data. Here, three images from the image sensor 104 are first stored in the memory 106, and then a calculation unit 111 described later generates a depth image 110 from the first image 107 and the second image 108, and stores the depth image 110 in the memory 106. accumulate.
[0059]
The calculation unit 111 calculates and generates a depth image 110 in which each pixel represents depth information of a subject. The strobe 112 emits light to the subject. Usually, it is used as an auxiliary light source, but in the present embodiment, it also plays a different role. The control unit 114 controls the above components.
[0060]
Specifically, the image sensor 104 is, for example, a CCD imaging device having a configuration as shown in FIG. Here, for simplicity of description, a case of a total of 16 pixels of 4 × 4 pixels is shown. The sixteen photodiodes PD are arranged in four rows and four columns, vertical transfer CCDs are arranged between the photodiodes in each column, and one horizontal transfer CCD is adjacent to the vertical transfer CCD in the last row. Book is provided.
[0061]
When an optical image of a subject is formed on this surface by the lens 101, photoelectric conversion is performed in each photodiode PD, and charges are accumulated according to the intensity of light and time. Thus, a charge image is completed. In order to take out this charge image as a one-dimensional electric signal to the outside, the signal is sequentially transferred vertically and horizontally, and the signal is taken out from one signal output terminal. A transfer unit such as a vertical transfer CCD and a horizontal transfer CCD plays this role.
[0062]
First, charges accumulated in the photodiodes PD of 16 pixels are simultaneously transferred to a part of the vertical transfer CCD by a field shift pulse. Next, the four rows of vertical transfer CCDs start transferring charges in parallel in the vertical direction. The vertically transferred signal charges are successively sent to the horizontal transfer CCD. Each time a signal of four pixels for one row is input, the horizontal CCD quickly transfers the signal in the horizontal direction and outputs the signal through an output circuit provided at the left end. In this way, a signal for each line is obtained from the output terminal.
[0063]
When the image sensor 104 is a CCD image pickup device having a configuration as shown in FIG. 2, the arrangement of the photodiodes PD in FIG. 2 in four rows and four columns corresponds to the photoelectric conversion unit 104a in FIG. The transfer CCD and the horizontal transfer CCD correspond to the transfer unit 104b in FIG.
[0064]
Note that the exposure time is a time during which the image sensor 104 is in a state capable of accumulating charges and the shutter is open. The exposure time is set by the control unit 114 when capturing the first to third images. The exposure time zone is a “state where the subject is being imaged”.
[0065]
Next, the operation of the image input apparatus of FIG. 1 according to the first embodiment will be described with reference to the timing chart shown in FIG. 3 and the flowcharts shown in FIGS.
[0066]
3A illustrates the operation of the shutter 103, FIG. 3B illustrates the charge accumulation operation of the photoelectric conversion unit 104a in the image sensor 104, and FIG. 3C illustrates the operation of the image sensor 104 (of the transfer unit 104b). FIG. 3D shows an output operation, and FIG. In each case, the horizontal axis is a time axis, which represents each operation timing with the passage of time (from left to right in FIG. 3).
[0067]
In the operation of the shutter 103 in FIG. 3A, “1” indicates that the shutter is open, and “0” indicates that the shutter is closed. Strictly speaking, since the mechanical shutter has a response delay and an operation time, the shutter is not fully opened instantaneously and rises smoothly from "0" to "1". However, in FIG. In this case, it is shown that the signal rises instantaneously from “0” to “1”. 3D, the light emitting operation of the strobe 112 is "1" when it is emitting light, and when it is "0" it is when it is not emitting light.
[0068]
4 and 5 are flowcharts for explaining the processing operation of the image input device of FIG. 1. This operation is stored in, for example, storage means such as a ROM (Read Only Memory) of the control unit 114. It is realized by controlling each component of FIG. 1 according to a program.
[0069]
Note that the same steps in FIG. 3 and FIGS. 4 and 5 are denoted by the same reference numerals.
[0070]
First, an operation is started by the control unit 114 in response to a shooting instruction (for example, pressing down a shutter button) by a user. First, after the shutter is opened (Step S1 in FIG. 4), each photodiode PD of the photoelectric conversion unit 104a is reset all at once to start imaging. (Step S2). Immediately after the start of imaging, the strobe 112 emits light (step S3). After the imaging is performed for a time t1 set by the control unit 114 as the exposure time (step S4), each photodiode of the photoelectric conversion unit 104a is exposed during this exposure time. Is transferred to the transfer unit 104b (step S5), and at the same time, the transfer unit 104b starts an output operation (step S6). Further, the shutter 103 is also closed (step S7).
[0071]
In step S6, the transfer unit 104b outputs the charge corresponding to each pixel sent from the photoelectric conversion unit 104a to the signal processing circuit 105 as a first image signal (first signal). In the signal processing circuit 105, the input first signal is subjected to color image processing for generating a color image, noise reduction, contrast adjustment, gamma conversion, and the like. The image is stored in the memory 106 as one image.
[0072]
After the output of the signal of one image (first image) is completed in the transfer unit 104b (step S8), the shutter is opened again (step S9), and the photodiodes PD of the photoelectric conversion unit 104a are simultaneously opened. It resets and starts the second imaging (step S10). The exposure time set by the control unit 114 in the second imaging is the same as the exposure time in the first imaging (t1), except that the strobe 112 does not emit light.
[0073]
After the set exposure time, that is, after the elapse of the time t1 (step S11), the charge accumulated in each photodiode of the photoelectric conversion unit 104a is transferred to the transfer unit 104b (step S12), and at the same time, the transfer unit 104b performs an output operation. Is started (step S13). Further, the shutter 103 is also closed (step S14).
[0074]
In step S12, similarly to step S6, the transfer unit 104b converts the electric charge corresponding to each pixel sent from the photoelectric conversion unit 104a into an image signal (second signal) of a second image, and the signal processing circuit 105 Output to In the signal processing circuit 105, the input second signal is subjected to color image processing for generating a color image, noise reduction, contrast adjustment, gamma conversion, and the like. The second image is stored in the memory 106.
[0075]
After the output of the signal of the second image (second image) is completed in the transfer unit 104b (step S15), the shutter is opened again (step S16 in FIG. 5), and each photodiode of the photoelectric conversion unit 104a is opened. The PDs are reset all at once, and the third imaging is started (step S17). The exposure time set by the control unit 114 in the third imaging is set to a time t2 different from the first and second exposures.
[0076]
After the set exposure time, that is, after the elapse of time t2 (step S18), the charge accumulated in each photodiode of the photoelectric conversion unit 104a is transferred to the transfer unit 104b (step S19), and at the same time, the transfer unit 104b performs an output operation. Is started (step S20). Further, the shutter 103 is also closed (Step S21).
[0077]
In step S20, the transfer unit 104b converts the charge corresponding to each pixel transmitted from the photoelectric conversion unit 104a into an image signal (third signal) of a third image as in steps S6 and S12. Output to the processing circuit 105. In the signal processing circuit 105, the input third signal is subjected to color image processing for generating a color image, noise reduction, contrast adjustment, gamma conversion, and the like. 3 is stored in the memory 106.
[0078]
Here, the first to third images will be described.
[0079]
The first image of the first sheet is an image that is exposed for a very short time (during time t1) while emitting the flash 112. Since the strobe 112 emits light for an extremely short time, such as 1/1000 to 1 / 10,000 seconds, the exposure time t1 is set to about the same time as the strobe 112 emission time. Since the light of the strobe 112 is intense, if the subject is at a distance of about 1 to 3 m, it is possible to take a picture sufficiently even if the time t1 is short. Since the time t1 is short, in a relatively dark environment, the influence of light other than the strobe light is small, but in a bright environment, a background or the like is captured by background light (sunlight or indoor illumination light).
[0080]
The second image of the second image is an image obtained at the same exposure time t1 as the first image without emitting the flash 112. Here, since the external light conditions when the first image and the second image are captured are almost the same, the external light component disappears when the second image is subtracted from the first image. That is, the difference image between the first image and the second image is an image in the same state as when the image is captured using only the light of the strobe 112 in a completely dark state. This difference image can also be rephrased as an image that captures only the light reflected by the subject from the light of the strobe 112. This difference image is called a reflected light image.
[0081]
The reflected light image is considered to include information on the depth of the subject (imaging target).
[0082]
The reflected light from the object greatly decreases as the distance from the image input device to the object increases. When the surface of the object disperses light uniformly, the amount of light received per pixel of the reflected light image decreases in inverse proportion to the square of the distance to the object.
[0083]
Each pixel value of the reflected light image represents the amount of reflected light corresponding to that pixel. The reflected light is affected by the properties of the object (specular reflection, confusion, absorption, etc.), the orientation of the object, the distance of the object, etc., but the entire object is an object that disperses the light uniformly In this case, the amount of reflected light has a close relationship with the distance to the object. For example, since a human hand or the like has such a property, the reflected light image obtained when the hand is extended in front of the image input device may include a distance to the hand, a tilt of the hand (partially different distances), and the like. Is represented as a three-dimensional image.
[0084]
When shooting an object having the property of diffusing light, the intensity of reflected light attenuates in inverse proportion to the square of the distance of the object. For example, a person is at a position of 1 to 2 m, and 10 m or more behind it When a background object exists in a distant place, sufficient reflected light returns from a person, but reflected light hardly returns from the background. Therefore, using a reflected light image makes it possible to separate a person from a background. If the intensity of the reflected light is inversely proportional to the square of the distance, the three-dimensional shape of the object can be restored.
[0085]
A pixel value Q of a reflected light image of a certain object that is imaged is a distance d between a portion of the object corresponding to the pixel and the image input device, a color or reflectance of the surface of the object, gloss, material , And k is a coefficient having a role of correcting the intensity of the reflected light of the elementary value according to any of the inclination of the surface and the like.
Q = k / d ² … (1)
It can be expressed as. By solving the equation (1) for d, it is possible to obtain the distance d as depth information corresponding to the unevenness of the object surface and the positional relationship between a plurality of objects.
[0086]
An image in which each pixel value represents the depth information is called a depth image.
[0087]
Due to differences in the intensity of reflected light depending on the color, reflectance, inclination, gloss, material, etc. of the surface of the object, in the reflected light image, it is actually captured as if it is near but far away, The reverse is also possible. Therefore, the reflected light image is not strictly the depth image, but an appropriate value of k based on the color, reflectance, inclination, gloss, material, etc. of the surface of the object shown in the reflected light image. If is selected, each pixel can be converted into an image representing the depth information from the above equation (1). That is, since it can be said that each pixel of the reflected light image contains or represents depth information, the reflected light image is treated here as a type of depth image.
[0088]
The third third image is captured with an exposure time of time t2 different from time t1. The time t2 is an exposure time set so as to appropriately photograph a subject, as in the case of imaging using a conventional digital still camera.
[0089]
As a result, both a reflected light image (difference image) having depth information and a natural image (third third image) are obtained.
[0090]
Returning to the description of FIG. 5, when the output of the signal of the third image (third image) is completed by the transfer unit 104b (step S22), the first image 107 and the second image are stored in the memory 106. An image 108 and a third image 109 are stored. After that, the calculation unit 111 reads the first image 107 and the second image 108 (step S23), and calculates a difference between the two to obtain a reflected light image (difference image). The reflected light image may be used as it is as the depth image 110, or an image obtained by converting each elementary value of the reflected light image into the depth information may be used as the depth image 110 (step S24). The obtained depth image 110 is stored again in the memory 106 (step S25).
[0091]
The arithmetic unit 111 converts the elementary value of the reflected light image or a plurality of pixels having similar pixel values into depth information using the above equation (1), and converts the surface reflectance, gloss state, surface The depth image 110 may be generated in consideration of the influence of the inclination of the image. When the object surface is inclined in the reflected light image, the intensity of the reflected light increases as the surface of the object surface is inclined toward the image input device side. The inclination state can be determined from the distribution of pixel values.
[0092]
In addition, for example, by comparing the pixel values of adjacent pixels of a reflected light image or a depth image obtained by converting each elementary value into depth information as described above, a subject surrounded by pixels whose pixel values are equal to or greater than a certain value α. A portion may be extracted, and the rest may be regarded as a background portion, and an image of only the subject portion may be used as the depth image 101.
[0093]
<Second embodiment>
Next, a second embodiment will be described. The configuration of the image input device according to the second embodiment is the same as that of FIG. The difference from the first embodiment is the control method of the control unit 114.
[0094]
The operation of the image input device of FIG. 1 according to the second embodiment will be described with reference to the timing chart shown in FIG. 6 and the flowcharts shown in FIGS. Here, the description of the parts common to the first embodiment will be omitted, and different parts will be described. 6A and FIG. 6B, FIG. 6A shows the operation of the shutter 103, FIG. 6B shows the charge accumulation operation of the photoelectric conversion unit 104a in the image sensor 104, and FIG. The output operation of the sensor 104 (of the transfer unit 104b) and the light emission operation of the strobe are shown in FIG.
[0095]
FIGS. 8 and 9 are flowcharts for explaining a processing operation according to the second embodiment of the image input apparatus of FIG. 1. This operation is performed, for example, by a storage unit such as a ROM of the control unit 114. Is realized by controlling each component of FIG. 1 in accordance with the program stored in.
[0096]
The same steps in FIG. 6 and FIGS. 8 to 9 are denoted by the same reference numerals.
[0097]
First, the operation starts in accordance with a photographing instruction (for example, pressing down a shutter button) by a user. After the shutter is opened (step S31 in FIG. 8), the photodiodes PD of the photoelectric conversion unit 104a are reset all at once to start imaging (step S32). Immediately after the start of imaging, the strobe 112 emits light (step S33). After the imaging is performed for the time t1 set by the control unit 114 as the exposure time (step S34), each photodiode of the photoelectric conversion unit 104a is exposed during this exposure time period. Is transferred to the transfer unit 104b (step S35), and at the same time, the transfer unit 104b starts the output operation (step S36). In step S36, the transfer unit 104b outputs the electric charge corresponding to each pixel transmitted from the photoelectric conversion unit 104a to the signal processing circuit 105 as an image signal (first signal) of a first image.
[0098]
Up to this point, the operation is the same as in the first embodiment. In the first embodiment, the shutter 103 is closed thereafter, but is not closed here. After the charge is transferred to the transfer unit 104b in step S35, each photodiode of the photoelectric conversion unit 104a is in the same state as the reset state, and the charge is empty. Therefore, immediately thereafter, the photoelectric conversion unit 104a starts the charge accumulation operation in step S36. That is, imaging of the second image is started. Then, for a further time t1, after the charge is accumulated in the photoelectric conversion unit 104a (step S37), the shutter 103 is closed (step S38).
[0099]
In step S35, after the photoelectric conversion unit 104a transfers the electric charge accumulated in each photodiode to the transfer unit 104b, the accumulation operation is started again, and the photoelectric conversion is performed for a time t1 until the shutter 103 is closed in step S38. The electric charge accumulated in the portion 104b is the second image, and in this case, the strobe 112 does not emit light. After the shutter 103 is closed, the photoelectric conversion unit 104a itself is in a state capable of accumulating electric charge. However, since the shutter 103 is closed and no light enters, the electric charge does not increase.
[0100]
When the output of the first signal corresponding to the first image is completed (step S39), the charge corresponding to the second image and currently stored in each photodiode of the photoelectric conversion unit 104b is transferred to the transfer unit 104b. (Step S40), and starts outputting a second signal corresponding to the second image (step S41).
[0101]
When the output of the second signal ends (step S42), the shutter 103 is opened again (step S43), the photodiodes of the photoelectric conversion unit 104a are reset, and the third imaging is started (step S44). The exposure time set by the control unit 114 in the third imaging is set to time t2. After a lapse of time t2 (step S45), the charge accumulated in each photodiode of the photoelectric conversion unit 104a is transferred to the transfer unit 104b (step S46), and at the same time, the transfer unit 104b starts an output operation (step S47). Further, the shutter 103 is also closed (step S48). In step S47, as in steps S36 and S41, the transfer unit 104b converts the electric charge corresponding to each pixel transmitted from the photoelectric conversion unit 104a into a signal (third signal) of a third image. Output to the processing circuit 105.
[0102]
When the output of the signal of the third image (third image) is completed by the transfer unit 104b (step S49), the first image 107, the second image 108, and the third image are stored in the memory 106. 109 are stored. After that, the calculation unit 111 reads the first image 107 and the second image 108 (step S50), and calculates a difference between the two to obtain a reflected light image (difference image).
[0103]
As in the first embodiment, the calculation unit 111 uses the reflected light image as it is as the depth image 110, or converts each elementary value of the reflected light image into the depth information as the depth image 110 (step S51). Is stored again in the memory 106 (step S52).
[0104]
An advantage of the second embodiment is that the times at which the first image 107 and the second image 108 are captured are close to each other. When trying to obtain a natural image corresponding to a depth image from a plurality of images as in the present embodiment, it is ideal that an object as a subject is stationary. However, actually, when photographing a person or the like, it is not always stationary. If you want to take natural photos, you're going to capture the people who are working. In addition, even if the object is stationary, if a camera shake occurs during shooting, it is the same as moving the object.
[0105]
It is assumed that the object has moved slightly when obtaining the reflected light image from the difference between the first and second images. Then, a correct reflected light image cannot be obtained when the difference is obtained. This will be described with reference to FIGS. 10 and 11, (a) is a graph showing a first image and a luminance value of each pixel on a horizontal line in the center of the first image, and (b) is a second image. And a graph showing the luminance value of each pixel on the horizontal line at the center of the second image. FIG. 3C shows a difference image (reflected light image) between the first image and the second image, It is a graph showing the luminance value of each pixel on the horizontal line at the center of the difference image.
[0106]
A circular object is nearby and the rest is a distant background. The case where the circular style is imaged by the image input device in FIG. 1 in this state will be described as an example.
[0107]
As shown in each of FIGS. 10 and 11B, in the second image, the nearby circular object is not so much exposed to external light and is dark, but the background is well exposed to external light. And bright.
[0108]
First, a description will be given with reference to FIG. In the first image of FIG. 10A, since the strobe light shines on a nearby object, it becomes brighter than the second image of FIG. 10B. On the other hand, since the background is far, the light of the strobe 112 does not reach and the brightness does not change. This is also evident from the graphs shown in FIGS. 7A and 7B showing the luminance of the horizontal line at the center of the height of each image. The higher the brightness value is, the brighter the image becomes, and the lower the brightness value is, the darker the image becomes.
[0109]
Here, a difference image is obtained by subtracting the second image from the first image. Since the background has the same brightness, the subtraction results in zero (black). Since the first image is brighter in the object portion, the reflected light component of the strobe appears as a subtraction result. As a result, a difference image as shown in FIG.
[0110]
Next, look at FIG. This is because the position of the circular object is shifted between the first image in FIG. 7A and the second image in FIG. Then, the difference obtained by subtracting the second image from the first image is a result obtained by subtracting the luminance of the object part from the luminance of the minus part and the luminance of the background, as shown in the graph of FIG. This is because, in the difference image shown on the left side, the right end of the object becomes bright as shown at the lower left even if the minus remains black. This is due to the fact that the displacement of the object has resulted in the calculation of the luminance of the parts that do not originally correspond.
[0111]
In order to prevent this from occurring, it is necessary that the object (or the image input device) does not move as much as possible between the first image and the second image. Unless it is unavoidable that the object moves to some extent, the times at which the first image and the second image are captured should be as close as possible.
[0112]
In the second embodiment, since the charge stored in the photoelectric conversion unit 104a corresponding to the first image is transferred to the transfer unit 104b, the operation of storing the charge corresponding to the second image is started immediately. The time period for acquiring the image and the time period for acquiring the second image are very close to those in the first embodiment. Therefore, the embodiment is superior to the first embodiment in that a reflected light image can be obtained correctly without being affected by the movement of the object or the camera shake of the photographer.
[0113]
A variation of the second embodiment will be described with reference to FIG. The same parts as those in FIG. 6 are denoted by the same reference numerals, and only different parts will be described. That is, the process is the same up to the point where the output of the second signal corresponding to the second image is started (steps S31 to S41).
[0114]
In FIG. 7, immediately after the output of the second signal is started, the shutter 103 is opened again immediately (step S61), the photodiodes PD of the photoelectric conversion unit 104a are reset all at once, and the imaging of the third image is started. (Step S62). After the image is captured for the time t2 set by the control unit 114 (step S63), if the output of the second signal has already been completed in the transfer unit 104b, the image is immediately stored in each photodiode of the photoelectric conversion unit 104a. The transferred charge is transferred to the transfer unit 104b, and the transfer unit 104b starts outputting a third signal corresponding to the third image (Step S65).
[0115]
In step S63, if the transfer unit 104b has not yet finished outputting the second signal at the time t2, the charge stored in each photodiode of the photoelectric conversion unit 104a is waited for the completion. The data is transferred to the transfer unit 104b, and the transfer unit 104b starts outputting a third signal corresponding to the third image.
[0116]
An advantage of the operation shown in FIG. 7 is that the times at which the first to third images are taken are closer to each other than the operation of FIG. In the operation of FIG. 6, the motion of the object between the imaging of the first and second images is so small that it can be ignored, but there is still a time difference between the first and second images and the third image. large. This indicates that a shift occurs between the depth image created from the first and second images and the third image (natural image). That is, when an attempt is made to cut out only the object in front using the depth image as a clue, the cut-out image may be shifted.
[0117]
The operation shown in FIG. 7 has the effect of reducing the shift by slightly bringing the imaging timing of the third image closer to that of the first and second images.
[0118]
<Third embodiment>
Next, a description will be given of a third embodiment, which aims at bringing the imaging timing between the depth image and the natural image as close as possible. In the first and second embodiments, three images are captured. In the third embodiment, a depth image 110 and a natural image (third image 109) are similarly obtained from two images (first image 107 and second image 108).
[0119]
FIG. 12 shows an example of the configuration of an image input device according to the third embodiment. The same parts as those in FIG. 1 are denoted by the same reference numerals, and only different parts will be described. That is, in FIG. 12, the images acquired by the image sensor 104 are only the first image 107 and the second image 108. The third image (natural image) 109 and the depth image 110 are generated by the calculation unit 111 from the first and second images.
[0120]
Next, the operation of the image input apparatus of FIG. 12 will be described with reference to timing charts shown in FIGS. 13 and 14 and a flowchart shown in FIG.
[0121]
Here, the description of the parts common to the first and second embodiments will be omitted, and different parts will be described. FIGS. 13 and 14 show the operation of the shutter 103 in FIG. 13A and the charge accumulation operation of the photoelectric conversion unit 104a in the image sensor 104 in FIG. (C) shows the output operation of the image sensor 104 (of the transfer unit 104b), and (d) shows the strobe light emission operation.
[0122]
FIG. 15 is a flowchart for explaining the processing operation of the image input device of FIG. 12 according to the third embodiment. This operation is stored in a storage unit such as a ROM of the control unit 114, for example. It is realized by controlling each component in FIG. 12 according to the program.
[0123]
13 and FIG. 14, and the same steps in FIG. 15 are denoted by the same reference numerals.
[0124]
First, the operation starts in accordance with a photographing instruction (for example, pressing down a shutter button) by a user. After the shutter is opened (step S71 in FIG. 15), the photodiodes PD of the photoelectric conversion unit 104a are reset all at once to start imaging (step S72). Immediately after the start of imaging, the strobe 112 emits light (step S73). After the imaging is performed for a time t1 set by the control unit 114 as the exposure time (step S74), each photodiode of the photoelectric conversion unit 104a is exposed during this exposure time. Is transferred to the transfer unit 104b (step S75), and at the same time, the transfer unit 104b starts the output operation (step S76). The shutter 103 is not closed, and the second imaging is started continuously (step S76). Up to this point, the operation is the same as in the example of the second embodiment.
[0125]
In the second embodiment, the exposure time when capturing the second image is the same time t1 as the exposure time when capturing the first image. However, here, the exposure time for the time t2 different from the time t1 is set. The second image is captured during the exposure time.
[0126]
In other words, as shown in FIG. 13, if the transfer unit 104b has already finished outputting the first signal corresponding to the first image after the exposure time t2 set by the control unit 114 has elapsed (step S77). (Step S79) Immediately, the charge accumulated in each photodiode of the photoelectric conversion unit 104a is transferred to the transfer unit 104b (Step S80), and the transfer unit 104b outputs the output of the second signal corresponding to the second image. The process starts (step S81). Then, the shutter 103 is closed (Step S78).
[0127]
On the other hand, as shown in FIG. 14, after the exposure time t2 set by the control unit 114 has elapsed (step S77), when the transfer unit 104b has not yet output the first signal corresponding to the first image, (Step S79) Once the shutter 103 is closed, the imaging of the second image is ended (Step S78), and the transfer unit 104b waits for the end of the first signal output. Since the shutter is closed while waiting, no charge is accumulated in the photoelectric conversion unit 104a. When the output of the first signal is completed by the transfer unit 104b (Step S79), the charge of the photoelectric conversion unit 104 is transferred to the transfer unit 104b (Step S80), and the transfer unit 104b transmits the second signal corresponding to the second image. (Step S81).
[0128]
When the output of the signal of the second image (second image) is completed by the transfer unit 104b, the first image 107 and the second image 108 are stored in the memory 106. After that, the calculation unit 111 reads the first image 107 and the second image 108 (Step S82), and generates a depth image 110 and a third image (natural image) 109 from these.
[0129]
Next, a description will be given of an operation of the arithmetic unit 111 for generating the depth image 110 and the third image 109 from the first image 107 and the second image 108.
[0130]
First, the first image is captured at the exposure time t1 with the flash unit 103 emitting light, and the second image is captured at the exposure time t2 without the flash unit 103 emitting light. In order to obtain a reflected light image, as described in the first and second embodiments, an image is captured at the same exposure time, one image is an image captured by flash emission, and the other image is not flashed. Are required. The difference between the two images was obtained as a reflected light image. However, in the third embodiment, although there is the first image 107 captured at the exposure time t1 by emitting the flash, there is no image captured at the exposure time t1 without emitting the flash. However, from the second image 108, an image captured at the exposure time t1 can be obtained by calculation without firing the strobe.
[0131]
That is, since the exposure time of the second image is the time t2, the brightness value of each pixel of the second image is multiplied by t1 / t2 to obtain an image captured at the exposure time t1 without emitting a strobe light. And almost the same. Therefore, if each pixel of the reflected light image is H (x, y), each pixel of the first image 107 is G1 (x, y), and each pixel of the second image is G2 (x, y), Each pixel value of the reflected light image can be obtained from Expression (2).
[0132]
H (x, y) = G1 (x, y)-(t1 / t2) G2 (x, y) (2)
The third image (natural image) 109 is the second image 108 itself in this case.
[0133]
The natural image may generally be the second image itself. However, if the shooting environment is too dark, an image using the strobe 103 as an auxiliary light source may be desirable as a natural image. In that case, the image obtained by adding the first image 107 and the second image 108 may be a natural image. The added image is basically an image obtained by emitting light from the strobe 103 and capturing the image for the exposure time (t1 + t2). Therefore, in this case, the exposure time t2 'of the second image is set so that the sum of the exposure time t1 of the first image and the exposure time t2' becomes t2 as the exposure time t2 'of the second image. The exposure time t2 'may be set.
[0134]
Each prime value of the third image (natural image) 109 is N (x, y), each prime value of the first image is G1 (x, y), and each pixel value of the second image is G2 (x, y). y), the third image can be calculated from the following equation (3) or (4).
[0135]
When the shooting environment is bright:
N (x, y) = G2 (x, y) (3)
If the shooting environment is dark:
N (x, y) = G1 (x, y) + G2 (x, y) (4)
In the first and second embodiments, if the shooting environment is too dark, a natural image may be created by adding the first image and the third image.
[0136]
<Fourth embodiment>
In the first to third embodiments, a general CCD image sensor is used to capture a plurality of images by controlling the imaging timing. However, as described above, the image is captured up to twice. If so, it can be done at a very close time, but it is difficult for the third time or more. This is because the signal output speed in the transfer unit 104b is low.
[0137]
However, for example, one or more charge storage units are further provided corresponding to each pixel of the photoelectric conversion unit 104b (a plurality of charge storage units are provided in total). If a charge image can be accumulated, three imagings can be performed at very close times.
[0138]
The transfer unit 104b of the image sensor 104 is generally formed of a CCD (Charge Imaging Element). The transfer unit 104b separately stores two or more charge images and transfers them at different timings. Even with such a configuration, three imagings can be performed at very close times.
[0139]
However, providing the image sensor 104 with such a configuration means that a dedicated solid-state imaging device having such a configuration is newly developed, which causes an increase in cost. In order to obtain a very large effect by slightly changing the control method of the digital still camera which is very widely used at present, the method described in the first to third embodiments is also used. Is desirable.
[0140]
<Control of aperture, shutter speed, flash control, etc.>
Here, an example of how to control the aperture, shutter speed, strobe light control, and the like will be described. Here, the third embodiment will be described as an example. In a conventional digital still camera, an aperture, a shutter speed, and the like are often set by an automatic exposure function. Of course, either the aperture or the shutter speed can be fixed and the other can be automatically set, or both can be set manually. Now, it is assumed that the aperture f and the shutter speed t0 for capturing an image of a subject with an appropriate exposure have been determined.
[0141]
When capturing the first image 107, the purpose is to obtain a reflected light image, so that an exposure time longer than the light emission time of the strobe 103 is not required. The time t1 is selected as a value slightly larger than the maximum emission time of the strobe light 103. In principle, this can be a fixed value. For the aperture, f is used as it is.
[0142]
When capturing the first image 107, the strobe 103 is caused to emit light, but it is necessary to set an appropriate amount of light emission. Here, it is possible to perform dimming by a method called TTL dimming, in which light is temporarily emitted in advance, the amount of reflection by the subject is captured by a sensor, and the amount of light emitted from the strobe is adjusted.
[0143]
Since the second image itself is also used as a natural image, the exposure time is set to t2 = t0. The aperture remains at f. When the exposure is finely adjusted again and the shutter is closed, t2 is different from t0, but at that time, the time t2 is measured. If the value is not known, the calculation for obtaining the reflected light image cannot be performed.
[0144]
From these, the opening time of the mechanical shutter is controlled so as to be opened for the time t1 + t2 after resetting the photoelectric conversion unit 104b after opening. Actually, it takes time to close the mechanical shutter. Therefore, t2 is selected so as to have an appropriate value in consideration of the time. In addition, when the time t2 is very short and a shutter in which the curtain runs like a focal plane shutter is used, the brightness gradually changes in the second image due to the time difference between the start and end of the curtain closing. Shading may occur. In such a case, it is necessary to perform an operation for correcting shading based on the closing speed of the shutter.
[0145]
In addition, there is a shutter system such as a lens shutter that gives a shutter effect to all pixels at the same timing. Even in this case, a value corresponding to t2 is calculated by taking into consideration the time when the shutter is actually opened and closed. Equation (2) should correctly guide the reflected light image. Conversely, the timing to start closing the shutter in FIG. 14 is determined in consideration of the time delay of opening and closing the shutter.
[0146]
The values of the time t1 and the time t2 are required to obtain the reflected light image, but in practice, when t2 is finely adjusted, it may be difficult to measure accurately. However, the reflected light image can be calculated even if the time t2 is not known. Assuming that the background is located somewhere far from the subject so that the flash does not reach, the ratio between time t1 and time t2 can be determined from the first image 107 and the second image 108.
[0147]
A value obtained by dividing the luminance value of each pixel of the second image 108 by the luminance value of the corresponding pixel of the first image 107 is calculated for all the pixels of the first image 107. Then, assuming that the value of the minimum value is a, it is equal to t1 / t2. Therefore, the reflected light image is obtained by the following equation (5). Here, each elementary value of the reflected light image is H (x, y), each elementary value of the first image 107 is G1 (x, y), and each elementary value of the second image is G2 (x, y). And
[0148]
H (x, y) = G1 (x, y) −a × G2 (x, y) (5)
Actually, due to the influence of noise and the like, t1 / 12 often does not exactly become the minimum value a. It is necessary to suppress the influence of noise by taking a value slightly larger than the minimum value or using a histogram together.
[0149]
The exposure time t1 for capturing the first image 107 may be an extremely short time, which is substantially equal to the light emission time of the strobe light 103. When a strobe is used as a light source, the light emission time of the strobe is generally very short, about one thousandth to one thousandth of a second. Even under such a bright environment, the influence of external light can be minimized. Conversely, imaging in a very short time also means an exposure time in which the brightness of an image can be suppressed to a sufficiently low level when an image is captured without emitting light from the light emitting unit (for example, the strobe light 103).
[0150]
In some cases, such as when the environment for obtaining the depth image is dark or when the imaging object and the image input device are stationary, the exposure time does not always need to be 1 / 1000th to 1 / 1,000th of a second. In such a case, a sufficient effect can be obtained even with an exposure time longer than the time t1.
[0151]
In the first to fourth embodiments, the flash unit 103 is mainly used as the light emitting unit. However, other light emitting units may be used. For example, a device such as an LED flash may be used. The type of light is not particularly limited, and the strobe is mainly white, but may be colored light or invisible light such as near infrared light.
[0152]
In each of the above embodiments, taking an example of a still image as an image to be taken is an operation of taking one still image, and actually three or two images (first to third images) are taken. Has been described above, and an image including depth information and a natural image are acquired. However, not limited to this case, the image input method and apparatus according to each embodiment can be applied to a moving image. In this case, when capturing one frame of a moving image, an operation of acquiring three or two images is performed as described in each of the above embodiments, and this operation is repeated for each frame of the moving image. Just fine.
[0153]
As described above, according to the first to third embodiments, the depth image or the reflected light image including the depth information and the natural light corresponding to these images can be obtained simply by devising the conventional digital camera control method. Images can be easily obtained.
[0154]
In addition, it is possible to provide a technology capable of acquiring a depth image with the same ease as a general digital camera user enjoying a digital photograph. Specifically, it can be realized as an optional function of the digital still camera. Thereby, depth information or an image including depth information can be acquired as a part of the action of the digital camera user.
[0155]
The image input device according to the above embodiment is almost the same as the components of a conventional digital still camera, and can be realized by devising a control method thereof. Therefore, it can be expected to make a great contribution to popularizing the apparatus for acquiring the depth image to the general public.
[0156]
According to the first embodiment, it is possible to acquire a depth image or a reflected light image including depth information and a natural image corresponding to these images by devising the simplest control method with respect to a conventional digital camera. can do.
[0157]
According to the second embodiment, even when a relatively moving subject is photographed or camera shake occurs, a reflected light image or a depth image including depth information can be obtained correctly without being affected by the movement. effective.
[0158]
According to the third embodiment, there is an effect of preventing displacement of an object in a reflected light image or depth image including depth information and an object in a natural image from occurring due to movement or camera shake of a subject.
[0159]
The method of the present invention described in the embodiment of the present invention as shown in the flowcharts of FIGS. 4 and 5, FIGS. 8 and 9 and FIG. Disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and storage media such as semiconductor memories can also be distributed.
[0160]
Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying constituent elements in an implementation stage without departing from the scope of the invention. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Further, components of different embodiments may be appropriately combined.
[0161]
【The invention's effect】
As described above, according to the present invention, depth information of an object in an image or an image including depth information can be easily acquired at low cost. Further, a normal image (natural image) and an image including depth information or an object including the depth information of an object in the image can be acquired only by performing the same operation as an imaging operation of a general digital camera or the like.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration example of an image input device according to first and second embodiments of the present invention.
FIG. 2 is a diagram showing a specific configuration example of an image sensor.
FIG. 3 is a diagram for explaining operation timings of main parts of the image input device according to the first embodiment.
FIG. 4 is a flowchart for explaining the operation of the image input device according to the first embodiment.
FIG. 5 is a flowchart for explaining the operation of the image input apparatus according to the first embodiment.
FIG. 6 is a view for explaining the operation timing of the main part of the image input device according to the second embodiment of the present invention.
FIG. 7 is a diagram for explaining another operation timing of a main part of the image input apparatus according to the second embodiment.
FIG. 8 is a flowchart for explaining the operation of the image input device according to the second embodiment.
FIG. 9 is a flowchart for explaining the operation of the image input device according to the second embodiment.
FIG. 10 is a diagram for explaining a difference calculation for obtaining a reflected light image.
FIG. 11 is a view for explaining a reflected light image obtained when there is a gap between two images used for obtaining the reflected light image.
FIG. 12 is a diagram showing a configuration example of an image input device according to a third embodiment of the present invention.
FIG. 13 is a view for explaining the operation timing of the main part of the image input device according to the third embodiment.
FIG. 14 is a diagram for explaining the operation timing of the main part of the image input device according to the third embodiment.
FIG. 15 is a flowchart for explaining the operation of the image input device according to the third embodiment.
FIG. 16 is a diagram illustrating a configuration example of a conventional digital still camera.
FIG. 17 is a diagram for explaining the operation timing of the main part of the conventional digital still camera.
FIG. 18 is a view for explaining the operation of a conventional digital still camera.
[Explanation of symbols]
Reference numeral 101: lens, 102: aperture, 103: shutter, 104: image sensor, 104a: photoelectric conversion unit, 104b: transfer unit, 105: signal processing circuit, 106: memory, 107: first image, 108: second Image 109, third image, 110 depth image, 111 arithmetic unit, 112 strobe, 114 control unit.

Claims (22)

Imaging means for capturing an image of an imaging target with a set exposure time,
Light emitting means for irradiating the imaging target with light,
When an imaging instruction is input, the exposure time is set to a first time, the light emitting unit irradiates light to the imaging target, and the imaging unit captures a first image of the imaging target. 1 means;
After capturing the first image, the exposure time is set to the first time, and the light emitting unit does not irradiate the imaging target with light. Second means for imaging the
A third means for obtaining a difference between the first image and the second image to obtain a third image representing an intensity distribution of light reflected on the object by the light emitted from the light emitting unit and illuminating the object; When,
An image input device comprising: A fourth unit configured to set the exposure time to a second time different from the first time after capturing the second image, and to capture the fourth image of the imaging target by the imaging unit; The image input device according to claim 1, further comprising: The image input device according to claim 1, further comprising a calculating unit configured to calculate depth information of the imaging target from the third image. Imaging means for capturing an image of an imaging target with a set exposure time,
Light emitting means for irradiating the imaging target with light,
When an imaging instruction is input, the exposure time is set to a first time, the light emitting unit irradiates light to the imaging target, and the imaging unit captures a first image of the imaging target. 1 means;
After capturing the first image, the exposure time is set to a second time different from the first time, and the light emitting unit does not irradiate the imaging target with light. Second means for capturing a second image of the imaging target;
An image component captured during the first time of the second time is extracted from each pixel value of the second image, and each pixel value is the extracted image component. A third means for obtaining an image of
Calculating a difference between the first image and the third image to obtain a fourth image representing an intensity distribution of reflected light of the light emitted to the imaging target by the light emitting unit and reflected by the imaging target; Means,
An image input device comprising: 5. The method according to claim 4, wherein the third unit obtains the third image by multiplying each pixel value of the second image by a ratio of the first time to the second time. The image input device according to the above. The third means is configured to divide the pixel values of a plurality of pixels constituting the second image by pixel values of the first image corresponding to each of the plurality of pixels. From among the values corresponding to each of the pixels, a value corresponding to the ratio of the first time to the second time is selected, and each pixel value of the second image is multiplied by the selected value. 5. The image input device according to claim 4, wherein the third image is obtained. The image input device according to claim 4, further comprising a calculating unit configured to calculate depth information of the imaging target from the fourth image. 5. The image input device according to claim 4, further comprising a fifth unit for adding the first image and the second image to obtain a fifth image. The imaging means,
A plurality of photoelectric conversion elements corresponding to each of a plurality of pixels constituting the image,
Output means for outputting the charge accumulated during the exposure time in the plurality of photoelectric conversion elements as an image signal,
The first means transfers the electric charge corresponding to the first image accumulated in the plurality of photoelectric conversion elements during the first time to the output means, and then transfers the first electric charge to the output means. The image input device according to claim 1, wherein output of a first image signal corresponding to the image is started, and imaging of the second image by the second unit is started. 2. The method according to claim 1, wherein when the timing of starting or ending one of the exposures is performed by the operation of the mechanical shutter, control or calculation is performed in consideration of a time delay required for the operation of the mechanical shutter. 4. The image input device according to 4. An image input method in an image input device, comprising: an imaging unit that captures an image of an imaging target with a set exposure time; and a light emitting unit that irradiates light to the imaging target.
When an imaging instruction is input,
A first step of setting the exposure time to a first time, irradiating the imaging target with light by the light emitting unit, and capturing a first image of the imaging target by the imaging unit;
A second step of setting the exposure time to the first time, and irradiating the imaging target with light with the light emitting unit, and capturing a second image of the imaging target with the imaging unit;
A third step of calculating a difference between the first image and the second image to obtain a third image representing an intensity distribution of light reflected by the imaging target of light emitted to the imaging target by the light emitting unit; When,
An image input method comprising: After capturing the first image and the second image, the exposure time is set to a second time different from the first time, and a fourth image of the imaging target is captured by the imaging unit. The image input method according to claim 11, further comprising a fourth step of imaging. The image input method according to claim 11, further comprising a calculating step of calculating depth information of the imaging target from the third image. An image input method in an image input device, comprising: an imaging unit that captures an image of an imaging target with a set exposure time; and a light emitting unit that irradiates light to the imaging target.
When an imaging instruction is input,
A first step of setting the exposure time to a first time, irradiating the imaging target with light by the light emitting unit, and capturing a first image of the imaging target by the imaging unit;
The exposure time is set to a second time different from the first time, and the light-emitting unit does not irradiate the imaging target with light, and the imaging unit captures a second image of the imaging target. The second step;
An image component captured during the first time of the second time is extracted from each pixel value of the second image, and each pixel value is the extracted image component. A third step for obtaining an image of
Calculating a difference between the first image and the third image to obtain a fourth image representing an intensity distribution of reflected light of the light irradiated on the imaging target by the light emitting unit and reflected by the imaging target; Steps and
An image input method comprising: 15. The method according to claim 14, wherein the third step calculates the third image by multiplying each pixel value of the second image by a ratio of the first time to the second time. Image input method described. The third step includes the step of dividing each pixel value of a plurality of pixels constituting the second image by a pixel value of the first image corresponding to each of the plurality of pixels. From among the values corresponding to each of the pixels, a value corresponding to the ratio of the first time to the second time is selected, and each pixel value of the second image is multiplied by the selected value. 15. The image input method according to claim 14, wherein the third image is obtained. The image input method according to claim 14, further comprising a calculating step of calculating depth information of the imaging target from the fourth image. The image input method according to claim 14, further comprising a fifth step of obtaining a fifth image by adding the first image and the second image. An imaging unit that captures an image of an imaging target at a set exposure time, and a light emitting unit that irradiates the imaging target with light, wherein the imaging unit includes a plurality of pixels that form the image. An image input method in an image input device having a plurality of corresponding photoelectric conversion elements and an output unit that outputs, as an image signal, an electric charge accumulated during the exposure time in the plurality of photoelectric conversion elements,
When an imaging instruction is input,
A first step of setting the exposure time to a first time, irradiating the imaging target with light by the light emitting unit, and starting imaging of a first image of the imaging target by the imaging unit;
When the first time has elapsed since the imaging of the first image was started in the first step, the first image accumulated by the plurality of photoelectric conversion elements during the first time is added to the first image. A second step of transferring a corresponding charge to the output means and starting to output a first image signal corresponding to the first image at the output means;
A third step of setting the exposure time to the first time, and starting imaging of a second image of the imaging target by the imaging unit without irradiating the imaging target with light by the light emitting unit; ,
In the third step, when the first time elapses after the imaging of the second image is started, after the output of the first image signal is completed by the output means, the first time In the meantime, the electric charge corresponding to the second image accumulated by the plurality of photoelectric conversion elements is transferred to the output unit, and the output unit starts outputting the second image signal corresponding to the second image. The fourth step;
When the output of the second image is completed by the output unit, a difference between the first image and the second image is obtained, and the light emitted from the light emitting unit to the object is reflected by the object. A fifth step of obtaining a third image representing the light intensity distribution;
An image input method comprising: An imaging unit that captures an image of an imaging target at a set exposure time, and a light emitting unit that irradiates the imaging target with light, wherein the imaging unit includes a plurality of pixels that form the image. An image input method in an image input device having a plurality of corresponding photoelectric conversion elements and an output unit that outputs, as an image signal, an electric charge accumulated during the exposure time in the plurality of photoelectric conversion elements,
When an imaging instruction is input,
A first step of setting the exposure time to a first time, irradiating the imaging target with light by the light emitting unit, and starting imaging of a first image of the imaging target by the imaging unit;
When the first time has elapsed since the imaging of the first image was started in the first step, the first image accumulated by the plurality of photoelectric conversion elements during the first time is added to the first image. A second step of transferring a corresponding charge to the output means and starting to output a first image signal corresponding to the first image at the output means;
The exposure time is set to a second time different from the first time, and the imaging means captures a second image of the imaging target without irradiating the imaging target with light by the light emitting means. A third step to start;
In the third step, when the second time has elapsed from the start of the imaging of the second image, after the output of the first image signal has been completed by the output means, the second time has elapsed. In the meantime, the electric charge corresponding to the second image accumulated by the plurality of photoelectric conversion elements is transferred to the output unit, and the output unit starts outputting the second image signal corresponding to the second image. The fourth step;
When the output of the second image is completed by the output unit, an image component captured during the first time of the second time is extracted from each pixel value of the second image. A fifth step of obtaining a third image in which each pixel value is the extracted image component;
Calculating a difference between the first image and the third image to obtain a fourth image representing an intensity distribution of reflected light of the light irradiated on the imaging target by the light emitting unit by the imaging target; Steps and
An image input method comprising: An imaging unit that captures an image of an imaging target with a set exposure time, and a program for inputting an image using a light emitting unit for irradiating the imaging target with light,
When the imaging instruction is input, the computer
A first step of setting the exposure time to a first time, irradiating the imaging target with light by the light emitting unit, and capturing a first image of the imaging target by the imaging unit;
A second step of setting the exposure time to the first time, and irradiating the imaging target with light with the light emitting unit, and capturing a second image of the imaging target with the imaging unit;
A third step of calculating a difference between the first image and the second image to obtain a third image representing an intensity distribution of light reflected by the imaging target of light emitted to the imaging target by the light emitting unit; When,
A fourth step of setting the exposure time to a second time different from the first time, and capturing a fourth image of the imaging target by the imaging unit;
A program that executes An imaging unit that captures an image of an imaging target with a set exposure time, and a program for inputting an image using a light emitting unit for irradiating the imaging target with light,
When the imaging instruction is input, the computer
A first step of setting the exposure time to a first time, irradiating the imaging target with light by the light emitting unit, and capturing a first image of the imaging target by the imaging unit;
The exposure time is set to a second time different from the first time, and the light-emitting unit does not irradiate the imaging target with light, and the imaging unit captures a second image of the imaging target. The second step;
An image component captured during the first time of the second time is extracted from each pixel value of the second image, and each pixel value is the extracted image component. A third step for obtaining an image of
Calculating a difference between the first image and the third image to obtain a fourth image representing an intensity distribution of reflected light of the light irradiated on the imaging target by the light emitting unit and reflected by the imaging target; Steps and
A program that executes

Images