JP2001141430A - Image pickup device and image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup device and an image processing device capable of easily and accurately acquiring information on the depth of a subject. SOLUTION: This image pickup device and image processing device comprise a projecting part 14 to project a shape pattern to a subject by projecting light, an image pickup part 12 to converge the light from the subject to which the shape pattern is projected and to pick up the image of the subject at a location optically different from the divergence center of light which propagates the shape pattern projected by the projecting part 14, a shape pattern detecting part 304 to detect the size of the image of the shape pattern projected to a specific area in the subject of which the image is picked up by the image pickup part 12, and a depth computing part 308 to compute the depth value of the specific area in the subject on the basis of the size detected by the shape pattern detecting part 304.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image pickup apparatus, an image processing apparatus, an image processing method, and a recording medium for acquiring information on the depth of a subject. In particular, the present invention relates to an image capturing apparatus, an image processing apparatus, an image processing method, and a recording medium that acquire information on the depth of a subject by capturing a projection pattern.

[0002]

2. Description of the Related Art In order to obtain information on the distance to an object and information on the position of the object, three-dimensional image measurement in which a pattern light such as a slit or a striped pattern is projected on the object, and the pattern projected on the object is photographed and analyzed. Are known. Typical measurement methods include slit light projection (also known as light sectioning) and coded pattern light projection, which are detailed in Seiji Iguchi and Kosuke Sato, "Three-dimensional image measurement" (Shokodo).

Also, Kahoku et al., "Axi-Vision three-dimensional imaging device"
Camera Development ”(3D Image Conference 99, 199
9 years), ultra-high-speed intensity modulation is applied to the projection light, and a subject illuminated with the intensity-modulated light is photographed with a camera equipped with a high-speed shutter function. Is disclosed.

[0004]

The conventional measuring method based on the light projection method measures a distance to a region of a subject on which a projection pattern is projected based on the principle of triangulation. Therefore, in order to obtain a high resolution of the distance measurement, it is necessary to arrange the projection optical system and the photographing optical system sufficiently in principle, and there has been a problem that the size of the measuring apparatus cannot be avoided. Also, since the optical axes of the projection optical system and the imaging optical system are far apart, when viewed from the imaging optical system, there is a place where the projected pattern is hidden behind the subject and cannot be observed, and distance information can be obtained. There was also the problem that there was no "blind spot".

Further, the above-mentioned method using intensity modulation requires that light modulation and optical shutter operation be performed at a very high speed, and the measuring device becomes large and expensive, and cannot be easily measured. is there.

In order to solve the above-mentioned problems, the present invention provides an image pickup apparatus, an image processing apparatus, an image processing method, and a recording medium that can easily and efficiently acquire information on the depth of a subject. The purpose is to:
This object is achieved by a combination of features described in the independent claims. The dependent claims define further advantageous embodiments of the present invention.

[0007]

According to a first aspect of the present invention, there is provided an image pickup apparatus for acquiring information relating to the depth of a subject. An imaging unit that converges light on a subject on which a shape pattern is projected at an optically different position from a projection unit that projects the shape pattern and a divergence center of light that transmits the shape pattern projected by the projection unit, and captures an image of the subject. And a unit.

[0008] A shape pattern detecting section for detecting the size of the image of the shape pattern projected on the specific area of the subject imaged by the imaging section, and a specific area of the subject based on the size detected by the shape pattern detecting section. The apparatus may further include a depth calculator that calculates a depth value.

The optical axis when the projection unit projects light onto the subject and the optical axis when the imaging unit captures the subject may be substantially the same. The imaging unit and the projection unit may perform imaging of a subject and projection of light onto the subject, respectively, using optical elements having the same optical axis.

[0010] The shape pattern is a plurality of figure patterns arranged two-dimensionally. The projecting unit projects the plurality of figure patterns on the subject simultaneously, and the photographing unit captures the subject on which the plurality of figure patterns are projected. Then, the shape pattern detection unit detects the size of each of the plurality of figure pattern images, and the depth calculation unit based on the size of each of the plurality of figure pattern images detected by the shape pattern detection unit.
The depth value of each region of the subject on which a plurality of graphic patterns are projected may be calculated. Shape patterns include points, lines, circles,
Or it may be any of polygons.

According to a second aspect of the present invention, there is provided an image pickup apparatus for acquiring information on the depth of a subject,
A projection unit that projects light having two different optical divergence centers in two directions perpendicular to the optical axis to project a shape pattern on a subject, and optically communicates with one of the two divergence centers At the convergence position at substantially the same position, the light on the subject on which the shape pattern is projected is converged,
An imaging unit for imaging a subject.

A shape pattern detection unit for detecting the size of an image of a shape pattern projected on a specific region of a subject to be imaged by an imaging unit, with respect to a divergence direction of light at a divergence center optically different from the converging position. And a depth calculation unit that calculates a depth value of a specific region of the subject based on the size of the divergence direction of the image of the shape pattern detected by the shape pattern detection unit.

[0013] The optical axis when the projection unit projects light onto the subject and the optical axis when the imaging unit captures the subject may be substantially the same. The projection unit may project light having two different optical divergence centers in two directions using an optical element having power only in one direction. A drive unit for rotating an optical element having lens power only in one direction around the optical axis may be further provided.

The shape pattern detecting section detects the size of the shape pattern image in the direction of the power of the optical element before and after the driving section rotates the optical element by 90 degrees around the optical axis, and the depth calculating section Alternatively, the shape pattern detection unit may calculate the depth value of the specific region of the subject using the size of the image of the shape pattern detected before and after the rotation of the optical element.

According to a third aspect of the present invention, there is provided an image pickup apparatus for acquiring information on the depth of a subject,
A projection unit that projects light having two different optical divergence centers in two directions perpendicular to the optical axis to project a shape pattern on a subject, and a position that is optically different from any of the two divergence centers And an image capturing unit that captures an image of the subject by converging light on the subject on which the shape pattern is projected at the converging position.

A shape pattern detector for detecting two sizes of an image of a shape pattern projected on a specific area of a subject to be imaged by the imaging unit in two directions of light at two divergence centers; and a shape pattern detector. And a depth calculation unit that calculates a depth value of the specific region of the subject in consideration of two sizes in two directions of the image of the shape pattern detected by.

The optical axis when the projection unit projects light onto the subject and the optical axis when the imaging unit images the subject may be substantially the same. The projection unit may project light having two different optical divergence centers in two directions using optical elements having independent powers in two directions perpendicular to the optical axis.

According to a third aspect of the present invention, there is provided an image processing method for acquiring information relating to the depth of a subject,
By projecting light, the shape pattern is projected on the subject, and at a position optically different from the divergence center of the light transmitting the shape pattern, the light on the subject on which the shape pattern is projected is converged to capture an image of the subject. Then, the size of the image of the shape pattern projected on the specific region of the subject is detected, and the depth value of the specific region of the subject is calculated based on the size.

According to a fourth aspect of the present invention, there is provided an image processing method for acquiring information relating to the depth of a subject,
By projecting light having two different optical divergence centers in two directions perpendicular to the optical axis, a shape pattern is projected on the subject, and optically substantially at the same position as one of the two divergence centers At the convergence position, the light of the object on which the shape pattern is projected is converged to image the object, and the image pickup unit images the divergence direction of light at the divergence center at a position optically different from the convergence position Detecting the size of the image of the shape pattern projected on the specific area of the object, and calculating the depth value of the specific area of the subject based on the divergence direction of the image of the shape pattern detected by the shape pattern detection unit. It is characterized by.

According to a fifth aspect of the present invention, there is provided an image processing method for acquiring information relating to the depth of a subject,
Projects light having two different optical divergence centers in two directions perpendicular to the optical axis to project a shape pattern on a subject, and converges at optically different positions from any of the two divergence centers At the position, converge the light at the object on which the shape pattern is projected, image the object, and in two directions of light at the two divergence centers,
The image pickup unit detects two sizes of the shape pattern image projected on the specific area of the subject, and detects two sizes of the shape pattern image detected by the shape pattern detection unit in two directions.
The depth value of the specific region of the subject is calculated in consideration of the three sizes.

According to a sixth aspect of the present invention, there is provided an image processing apparatus for acquiring information relating to the depth of a subject,
An input unit for inputting an image of the subject on which the shape pattern is projected, a shape pattern detection unit for detecting the size of the shape pattern image projected on a specific area of the subject, and a size detected by the shape pattern detection unit An image processing apparatus comprising: a depth calculation unit that calculates a depth value of a specific area of a subject based on the depth calculation unit.

According to a seventh aspect of the present invention, there is provided a recording medium storing a computer program for acquiring information on the depth of a subject, the program comprising an input for inputting an image of the subject on which a shape pattern is projected. A module, a shape pattern detection module for detecting the size of the image of the shape pattern projected on the specific region of the subject, and calculating a depth value of the specific region of the subject based on the size detected by the shape pattern detection module. A depth calculation module.

The above summary of the present invention does not list all of the necessary features of the present invention, and sub-combinations of these features may also be included in the present invention.

[0024]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described through embodiments of the present invention. However, the following embodiments do not limit the invention according to the claims and are described in the embodiments. Not all combinations of the features described above are essential to the solution of the invention.

(Embodiment 1) A first embodiment of the present invention will be described. FIG. 1 is a configuration diagram of a digital camera 10 as an example of an image capturing device. The digital camera 10 includes a digital still camera, a digital video camera capable of capturing a still image, and the like. Digital camera 10
Mainly includes a projection imaging unit 20, a control unit 40, a processing unit 60, a display unit 100, and an operation unit 110.

The projection imaging unit 20 includes a mechanical member and an electric member related to projection, photographing, and imaging. The projection imaging unit 20 is an imaging unit 1 that forms an image of a subject and shoots it.
2 and a projection unit 14 for projecting light onto a subject. The imaging unit 12 is a photographing lens 2 that captures an image and performs processing.
2, includes an optical path dividing element 23, an aperture 24, a shutter 26, an optical LPF (low-pass filter) 28, a CCD 30 as an example of a solid-state image sensor, and an image signal processor 32.
The projection unit 14 includes a projection lens 34, a coded pattern 3
6, and a light source 38.

The photographing lens 22 includes a focus lens, a zoom lens, and the like. Light passing through the taking lens 22 passes through the optical path splitting element 23, is filtered by the optical LPF 28, and is received by the CCD 30. With this configuration, a subject image is formed on the light receiving surface of the CCD 30.
Charges are accumulated in each sensor element (not shown) of the CCD 30 according to the amount of light of the formed subject image (hereinafter, the charges are referred to as “accumulated charges”). The accumulated charge is read out to a shift register (not shown) by a read gate pulse, and is sequentially read out as a voltage signal by a register transfer pulse.

Since the digital camera 10 generally has an electronic shutter function, a mechanical shutter such as the shutter 26 is not essential. In order to realize the electronic shutter function, the CCD 30 is provided with a shutter drain via a shutter gate. When the shutter gate is driven, accumulated charges are swept out to the shutter drain. By controlling the shutter gate, the time for accumulating the electric charges in each sensor element, that is, the shutter speed can be controlled.

A voltage signal output from the CCD 30, that is, an analog signal is color-separated into R, G, and B components by an image signal processing unit 32, and the white balance is adjusted first.
Subsequently, the imaging signal processing unit 32 performs gamma correction, sequentially A / D converts the R, G, and B signals at necessary timing, and obtains digital image data (hereinafter, simply referred to as “digital image data”) obtained as a result. ) Is output to the processing unit 60.

The projection unit 14 projects the pattern light onto a subject. The coded pattern 36 is illuminated by the light source 38, and the light transmitting the pattern is projected onto the subject by the projection lens 34 and the taking lens 22. Here, the optical path dividing element 23 reflects the light of the light source 38 passing through the projection lens 34 and makes the light enter the photographing lens 22. Optical path splitting element 23
May be, for example, a half mirror. Projection unit 14
May include a condenser lens (not shown) that collects light from the light source 38.

The projection imaging unit 20 further has a finder and a strobe (not shown). An LCD (not shown) may be provided inside the finder. In this case, various information from the main CPU 62 and the like described later can be displayed in the finder. The strobe works by emitting light when energy stored in a capacitor is supplied to a discharge tube.

The CCD 30 of this embodiment is an example of a solid-state image sensor. A solid-state imaging device is a semiconductorized and integrated imaging device, and is a two-dimensionally arranged pixel group having a function of photoelectric conversion and charge accumulation on a semiconductor substrate in terms of structure. The solid-state imaging device receives light formed by the imaging lens 22 and accumulates electric charges by a photoelectric conversion action. The stored charge images are scanned in a certain order and read out as electric signals.

The solid-state image sensor basically includes a semiconductor element including a light receiving element for receiving light incident from the outside and performing photoelectric conversion, a package accommodating the semiconductor element,
In order to enable light to enter the light receiving element, a transparent protective member disposed at a position facing the semiconductor element of the package, and a light shielding property higher than the transparent protective member on the outer surface or inside of the transparent protective member. It is preferable that the light-shielding member is formed from a light-shielding member having the light-shielding member. Thereby, the quality of the captured image can be improved. Further, the transparent protective portion may have a function of a microlens to improve the resolution of an image to be formed. A color filter may be provided between the light receiving element section and the transparent protection section, or on or in the transparent protection section, so that a color image can be captured.

The CCD 30 of this embodiment is a charge-coupled device (CCD) one-dimensional image sensor (CCD) having a sufficiently high resolution so that the size of the image of the projection pattern projected on the subject by the pattern light can be detected with high accuracy. It is desirable to be a linear sensor) or a two-dimensional image sensor (area sensor) image sensor. C as a solid-state image sensor
In addition to CD, MOS image sensor, CdS-Se contact type image sensor, a-Si (amorphous silicon)
Either a contact image sensor or a bipolar contact image sensor may be used.

The control unit 40 includes a zoom drive unit 42,
It has a focus drive unit 44, an aperture drive unit 46, a shutter drive unit 48, a control system CPU 50 that controls them, a distance measurement sensor 52, and a photometry sensor 54. Zoom drive unit 4
The drive units such as 2 each have a drive unit such as a stepping motor. When a release switch 114 described later is pressed, the distance measurement sensor 52 measures the distance to the subject,
The photometric sensor 54 measures the brightness of the subject. The measured distance data (hereinafter simply referred to as “distance measurement data”) and the subject luminance data (hereinafter simply referred to as “photometry data”) are sent to the control system CPU 50. The control system CPU 50 is configured to perform the following operations based on shooting information such as a zoom magnification designated by the user.
The zoom drive unit 42 and the focus drive unit 44 are controlled to adjust the zoom magnification and focus of the taking lens 22.

The control system CPU 50 controls the R of one image frame.
An aperture value and a shutter speed are determined based on an integrated value of a digital signal of GB, that is, AE information. According to the determined values, the aperture driving unit 46 and the shutter driving unit 48 adjust the aperture amount and open and close the shutter 26, respectively.

The control system CPU 50 also controls the flash emission based on the photometric data, and at the same time adjusts the aperture of the aperture 26. When the user instructs to capture video, C
The CD 30 starts charge storage, and after a lapse of a shutter time calculated from the photometric data, the stored charge is transferred to the imaging signal processing unit 3.
2 is output. The control system CPU 50 also controls the projection unit 14 of the projection imaging unit 20 to project the pattern light onto the subject, and controls the imaging unit 12 of the projection imaging unit 20 to capture the image of the subject on which the pattern light is projected. .

The processing unit 60 includes the digital camera 10
The main C that controls the whole, especially the processing unit 60 itself
PU 62 and memory control unit 6 controlled by PU 62
4, a YC processing unit 70, an optional device control unit 74, a compression / decompression processing unit 78, and a communication I / F unit 80. Main C
The PU 62 is connected to the control CPU 5 by serial communication or the like.
Necessary information is exchanged with 0. Main CPU6
The second operation clock is provided from a clock generator 88. The clock generator 88 provides clocks of different frequencies to the control system CPU 50 and the display unit 100, respectively.

The main CPU 62 is provided with a character generator 84 and a timer 86. The timer 86 is backed up by a battery and always counts the date and time.
From this count value, information about the shooting date and time and other time information are given to the main CPU 62. The character generation unit 84 generates character information such as a shooting date and time and a title, and the character information is appropriately combined with the captured image.

The memory control unit 64 includes a nonvolatile memory 66
And the main memory 68. Non-volatile memory 66
Is constituted by an EEPROM (electrically erasable and programmable ROM) or a flash memory, and stores data to be retained even when the power of the digital camera 10 is off, such as setting information by a user and adjustment values at the time of shipment. ing. The non-volatile memory 66 may have a main CP
A boot program or a system program of U62 may be stored. On the other hand, the main memory 68 generally has a D
It is composed of a relatively inexpensive and large-capacity memory such as a RAM. The main memory 68 has a function as a frame memory for storing data output from the imaging unit 20, a function as a system memory for loading various programs, and a function as a work area. The non-volatile memory 66 and the main memory 68
Data is exchanged with each unit inside and outside the unit 0 via the main bus 82.

The YC processing section 70 performs YC conversion on the digital image data, and outputs a luminance signal Y and a color difference (chroma) signal B-
Generate Y, RY. The luminance signal and the color difference signal are temporarily stored in the main memory 68 by the memory control unit 64.
The compression / decompression processing unit 78 sequentially reads out the luminance signal and the color difference signal from the main memory 68 and compresses them. The data thus compressed (hereinafter simply referred to as “compressed data”) is written to a memory card, which is a type of the optional device 76, via the optional device control section 74.

The processing unit 60 further includes an encoder 72
With. The encoder 72 receives a luminance signal and a color difference signal, converts them into a video signal (NTSC or PAL signal), and outputs the video signal from a video output terminal 90. When a video signal is generated from data recorded in the optional device 76, the data is first supplied to the compression / decompression processing unit 78 via the optional device control unit 74. Subsequently, the data subjected to the necessary expansion processing by the compression / expansion processing unit 78 is converted into a video signal by the encoder 72.

The optional device control section 74 generates necessary signals between the main bus 82 and the optional device 76, performs logical conversion, or converts voltage according to the signal specifications recognized by the optional device 76 and the bus specifications of the main bus 82. And so on. The digital camera 10 may include, for example, a PCM as an optional device 76 in addition to the above-described memory card.
A standard CIA-compliant standard I / O card may be supported. In that case, the optional device control unit 74
It may be constituted by an IA bus control LSI or the like.

The communication I / F unit 80 is provided for the digital camera 10
Communication specifications supported by, for example, USB, RS-23
It performs control such as protocol conversion according to specifications such as 2C, Ethernet, Bluetooth, and IrDA. The communication I / F unit 80 includes a driver IC as necessary, and communicates with an external device including a network via a connector 92. In addition to such standard specifications, data may be exchanged with an external device such as a printer, a karaoke machine, a game machine, or the like by a unique I / F.

The display unit 100 includes an LCD monitor 10
2 and an LCD panel 104. These are the LCD driver monitor driver 106, panel driver 10
8 respectively. LCD monitor 102
Is provided on the back of the camera with a size of, for example, about 2 inches, and displays a current shooting and playback mode, a zoom ratio for shooting and playback, a remaining battery level, a date and time, a screen for mode setting, a subject image, and the like. . The LCD panel 104 is, for example, a small black-and-white LCD provided on the upper surface of the camera and has an image quality (FINE).
/ NORMAL / BASIC etc.), information such as strobe light emission / non-light emission, standard number of recordable images, number of pixels, battery capacity, etc. are simply displayed.

The operation unit 110 includes mechanisms and electrical members necessary for the user to set or instruct the operation and the mode of the digital camera 10. The power switch 112 determines whether the power of the digital camera 10 is turned on or off. The release switch 114 has a two-stage pressing structure of half pressing and full pressing. As an example, AF and AE are locked by a half-press, captured images are captured by a full-press, and are recorded in the main memory 68, the optional device 76, etc. after necessary signal processing, data compression, and the like. The operation unit 110 may receive a setting using a rotary mode dial, a cross key, or the like in addition to these switches, and these are collectively referred to as a function setting unit 116 in FIG. Examples of operations or functions that can be specified by the operation unit 110 include “file format”, “special effects”, “printing”, “determine / save”, and “display switching”. The zoom switch 118 determines a zoom magnification.

The main operation of the above configuration is as follows. First, the power switch 11 of the digital camera 10
2 is turned on, and power is supplied to each part of the camera. The main CPU 62 determines whether the digital camera 10 is in the shooting mode or the playback mode by reading the state of the function setting unit 116.

When the camera is in the shooting mode, the main C
The PU 62 monitors the half-pressed state of the release switch 114. When the half-pressed state is detected, the main CPU 62
Obtains photometry data and distance measurement data from the photometry sensor 54 and the distance measurement sensor 52, respectively. The control unit 40 operates based on the obtained data, and the focus, the aperture, and the like of the photographing lens 22 are adjusted. When the adjustment is completed, L
Characters such as "standby" are displayed on the CD monitor 102 to inform the user of the fact, and then the release switch 114 is monitored for a full press. Release switch 11
When the button 4 is fully pressed, the shutter 26 is closed after a predetermined shutter time, and the accumulated charges of the CCD 30 are swept out to the image signal processing unit 32. The digital image data generated as a result of the processing by the imaging signal processing unit 32 is output to the main bus 82. The digital image data is temporarily stored in the main memory 68, thereafter processed by the YC processing unit 70 and the compression / decompression processing unit 78, and recorded in the optional device 76 via the optional device control unit 74. The recorded image is frozen for a while on the LCD monitor 1
02, the user can know the captured image. Thus, a series of photographing operations is completed.

On the other hand, when the digital camera 10 is in the reproduction mode, the main CPU 62 reads out the last photographed image from the main memory 68 via the memory control unit 64,
This is displayed on the LCD monitor 102 of the display unit 100. In this state, when the user instructs “forward” or “reverse” on the function setting unit 116, images taken before and after the currently displayed image are read and displayed on the LCD monitor 102.

In this embodiment, the projection imaging unit 2
0 projects the pattern light on the subject and captures an image of the subject on which the pattern light is projected. FIG. 2 is a diagram illustrating the projection and imaging of pattern light by the projection imaging unit 20. In the drawing, only the main parts of the imaging unit 12 and the projection unit 14 of the projection imaging unit 20 are shown.

The projection unit 14 projects the pattern light onto a subject. The coded pattern 36 is a plurality of two-dimensionally arranged graphic patterns, and as an example, is a two-dimensionally arranged circular figure pattern as shown in FIG. Light emitted from the light source 38 onto the coded pattern 36 (hereinafter simply referred to as “projection pattern light”) passes through the projection lens 34, is reflected by the optical path splitting element 23, passes through the photographing lens 22, and is projected on the subject. With concave as the projection lens 34, projected pattern of light is converged to the diverging center P ₀ by the projection lens 34 and the imaging lens 22, it is emitted is projected onto the object from a divergent center P _0. With a convex lens as the projection lens 34, the divergence center P ₀ of the projected pattern of light is made in front of the position from the photographing lens 22 on the optical path, since the effect of the present invention is the same, a concave lens as the projection lens 34 in the following The case of using this will be described.

If the subject is located at a distance Z ₁ from the position O of the photographing lens 22 by the projection pattern light, the pattern P ₁ is projected on the subject, and if the subject is located at a distance Z ₂ , the pattern P ₁ is projected on the subject. pattern _{P 2} is projected.

Next, the image pickup section 12 picks up an image of the subject on which the pattern is projected by the projection pattern light. The taking lens 22 converges the light on the subject on which the pattern is projected at the center O of the taking lens 22,
3 to form an image on the CCD 30. Shooting lens 2
The captured image on the CCD 30 of the projection pattern P ₁ on the subject located at a distance Z ₁ from the position O of position 2 is I ₁ , and the projection pattern on the subject located at a distance Z ₂ from the position O of the imaging lens 22. capturing images in CCD30 of P ₂ is _{I 2.}

[0054] and converging position O of the taking lens 22, due to a position different from the divergence center P ₀ of the projected pattern of light, projected by the light emitted from the same divergence center P ₀ pattern P
_1, even in _{P 2,} the captured image _I 1, _{I 2} are different.
In the present invention, the captured images I ₁ and I ₂ of the projection patterns P ₁ and P ₂ are provided.
By utilizing the fact that a difference occurs in the depth Z ₁ of the object,
To measure the Z _2. If the projection lens 34 of the subject is a flat glass and the CCD 30 and the coded pattern 36 are at optically equivalent positions, the divergence center P _{0 of the} projection pattern light
For but to match the convergence position O of the taking lens 22, pickup image _I 1 of the projection pattern _P 1, _{P 2} by the projection pattern light,
I ₂ becomes the same, the effect of the present invention are lost.

Next, an image of the subject on which the projection pattern is projected is taken.
Describes how to calculate the depth of a subject using an image.
You. Projection pattern P₁, P₂Image I of₁, I₂Figure 4
As shown in the figure, the sizes of the circular graphic patterns are different.
Image I₁The size of the circular figure at is S₁Is
To the photographed image I₂The size of the circular figure at is S
₂It is. Projection pattern P from taking lens 22₁, P₂
Distance Z to₁, Z₂The size S of the graphic pattern
₁, S₂Is different, the larger the distance, the more
The size of the button increases.

FIG. 5 is a diagram for explaining the relationship between the distance of the projection pattern and the size of the photographed image of the projection pattern. The distance from the position O of the taking lens 22 to the divergence center P ₀ of the projection pattern light is Z ₀ , and the distance from the position O of the taking lens 22 to the CCD 3 is
The distance to 0 is set as d. Let the radius of the taking lens 22 be S ₀
And then, an angle projected pattern of light is emitted from a divergent center P ₀ and alpha. Also, let T ₁ and T ₂ be the sizes of the projection patterns P ₁ and P ₂ on the subject. The sizes of the captured images of the graphic patterns of the projection patterns P ₁ and P ₂ on the CCD 30 are S ₁ and S ₂ as described above. Let β be the angle when the graphic pattern of the projection pattern P1 is viewed from the taking lens 22. Focusing on the projection pattern P1 using the projection pattern light,

T0 / Z0 = T1 / (Z1-Z0) = tanα (1)

When focusing on the photographed image of the projection pattern P1, the following holds: T1 / Z1 = S1 / d = tanβ (2) When the unknown T1 is deleted from (1) and (2),

(Z1−Z0) · T0 / Z0 = S1 · Z1 / d (3) is obtained. Solving (3) for Z1,

Z1 = T0 / (T0 / Z0−S1 / d) (4) Therefore, the size S of the image of the projection pattern P1
When 1 is detected, the distance Z1 from the taking lens 22 to the projection pattern P1 can be obtained from Expression (4).

In the above embodiment, the photographing lens 22 and the projection lens 34 used in the photographing unit 12 and the projecting unit 14 of the projection imaging unit 20 are optical lenses as an example of an optical element. The optical lens may be an optical lens system combining a plurality of optical lenses. The optical lens may be a Fresnel lens which is a thin lens having a saw-toothed surface and an oblique component added to have the optical properties of a thick lens. The optical lens is
It may also be a gradient index lens whose surface is generally flat and whose index of refraction inside varies from place to place but is not constant. In addition to the optical lens, a concave mirror or a convex mirror may be used as the optical element instead of the lens. Further, the optical element may be a hologram optical element.

The processing unit 60 of this embodiment acquires depth information of a subject based on the image of the subject captured by the imaging unit 12 of the projection imaging unit 20. FIG. 6 is a functional block diagram of the processing unit 60. The processing unit 60 includes an image storage unit 302 and a shape pattern detection unit 30
4, a depth calculation unit 308, and a recording unit 310.

The image storage unit 302 stores an image of a subject captured by the image capturing unit 12. The image of the subject captured by the imaging unit 12 includes the image of the figure pattern projected by the projection unit 14 of the projection imaging unit 20. The shape pattern detection unit 304 extracts an image of the graphic pattern projected on a specific region of the subject from the image of the subject stored in the image storage unit 302, and detects the size of the image of the graphic pattern.
The shape of the projected graphic pattern is coded pattern 3
6, the image processing technology
An image of a graphic pattern can be easily extracted.

The depth calculation unit 308 calculates the depth value of the specific area of the subject from the equation (4) using the size of the figure pattern detected by the shape pattern detection unit 304. The shape pattern detection unit 304 and the depth calculation unit 308 calculate the depth value of the subject for a partial region or the entire region of the subject on which the graphic pattern is projected.

The depth calculator 308 inputs the calculated depth information of the subject to the control unit 40, and
0 may control the focus drive, the aperture drive 46, and the shutter drive 48 based on the depth information of the subject to adjust the focus, aperture, and shutter speed.

The recording unit 310 causes the option device 76 to record the depth information of the subject calculated by the depth calculation unit 308 and the image of the subject stored in the image storage unit 302.

The shape pattern detection unit 304 extracts the image of the projection pattern by the image processing technique and detects the size of the image of the projection pattern. It is preferred that FIG. 7 is a slit pattern which is another example of the coding pattern 36. The slit pattern is the simplest as a figure and is easy to extract by image processing. What is necessary is just to detect the height of the slit pattern in the vertical direction as the size.

FIG. 8 shows a polygon pattern which is another example of the coding pattern 36. In the case of a polygon, the size is calculated by extracting edges by image processing. In the case of the slit pattern, it is necessary to detect the end point of the line, and an error is likely to occur. On the other hand, in the case of a polygon pattern, an edge can be found by relying on the shape, so that an error can be reduced in size detection. FIG. 9 is a cross-character pattern which is another example of the coding pattern 36. In the case of a cross pattern, the shape and center of the cross are extracted by image processing, and the size of the cross is detected.

The projection pattern does not necessarily have to be a figure, but may be a set of points. FIG. 10 is a dot pattern, which is another example of the coding pattern 36. The point patterns are uniformly arranged in a grid pattern, and four points of a square grid may be extracted by image processing, and the size may be detected from the length of one side of the square. However, point detection is often difficult in image processing. Therefore, by making the color of the point a characteristic color, it is possible to easily extract the point pattern projected by the image processing.

In order to easily extract the image of the projection pattern from the image of the subject, it is preferable to use a figure that is not included in the subject as the figure of the projection pattern. For example, if the subject is a stripe pattern or a polka dot pattern, and a projection pattern such as a slit pattern or a circular pattern is projected on the subject, it becomes difficult to extract the image of the projection pattern from the image of the subject by image processing. Or more. Therefore, it is preferable to artificially create and project a graphic pattern that is not included in the pattern of the subject.
The same can be said for the color of the projection pattern. It is preferable to project the projection pattern in a color that is not included in the color of the subject in order to facilitate the extraction of the image of the projection pattern in the image processing.

The shape pattern detector 3 of the processing unit 60
1 and the functions of the depth calculation unit 308 are shown in FIG.
Of the main CPU 62 and a program stored or loaded in the main memory 68 or the nonvolatile memory 66. When the main CPU 62 has a built-in memory, necessary programs may be stored in the memory and various functions may be realized as firmware. Image data to be stored in the image storage unit 302 of the processing unit 60 can be stored in the main memory 68 or the nonvolatile memory 66. The image data may be compressed by the compression / decompression processing unit 78. The function of the recording unit 310 of the processing unit 60 can be realized by the optional device control unit 74, for example. Also,
The operation unit 110 that receives the user's instruction may instruct the processing unit 60 to specify a specific area of the subject image, and the depth calculation unit 308 may calculate the depth value for the specific area specified by the user. There is considerable flexibility in designing the digital camera 10 to implement the above functions of the processing unit 60.

FIG. 11 is a flowchart of the image capturing method according to the present embodiment. Projection unit 1 of projection imaging unit 20
4 projects light transmitting the shape pattern onto the subject (S1).
00), the imaging unit 12 of the projection imaging unit 20 converges the light on the subject on which the shape pattern has been projected at a position optically different from the divergence center of the projection pattern light projected by the projection unit 14, and (S10)
2). A specific area of the image of the subject is selected (S10
4). The selection of the specific area may be performed by appropriately dividing the area of the image of the subject and sequentially selecting the automatically divided areas, or selecting the area of the image of the subject specified by the user using the operation unit 110. May be. The shape pattern detection unit 304 extracts an image of the shape pattern projected on the specific region of the subject by an image processing technique, and detects the size of the image of the extracted shape pattern (S106). The depth calculation unit 308 calculates the depth value of the specific area using the size of the image of the detected shape pattern (S108).

The accuracy and processing cost of extracting the projection pattern image and detecting the size of the projection pattern image by the shape pattern detection unit 304 depend on the resolution of the CCD 30 of the imaging unit 12 and the main CPU 6 of the processing unit 60 that performs image processing.
2. System functions such as the performance dependent on hardware performance, such as the processing performance of the nonvolatile memory 66, the main memory 68, the main bus 82, and the like, and the algorithm performance of the image processing method for detecting the figure pattern image and its size. Depends on configuration. Therefore, the image processing method and the calculation method from extracting the projection pattern image to calculating the depth value may vary considerably depending on the hardware configuration and software configuration of the digital camera 10. In any case, according to the image capturing apparatus of the present embodiment, the divergence center of the projection pattern light for projecting the pattern and the convergence position of the light for imaging the projected pattern are made different from each other, so that the image is projected on the subject. It has an essential feature that it is possible to efficiently calculate the depth value of the subject by utilizing the difference in size according to the depth value of the subject in the pattern image that has been obtained. The method can have many variations.

As described above, according to the image pickup apparatus of the present embodiment, a shape pattern is projected on a subject, and the depth distribution of the subject can be calculated from the image of the subject on which the shape pattern is projected. According to the present embodiment, a shape pattern is projected on the entire subject, an image of the projected image is captured, and a simple image processing technique is used from the captured image.
The depth distribution of the entire subject can be calculated. Therefore, the depth distribution of the subject can be calculated easily and efficiently as compared with the conventional method of sequentially scanning the area of the subject with projection light such as slit light.

According to the present embodiment, the optical axes of the projection optical system and the photographing optical system can be made coaxial, and both the projection and photographing can be performed by sharing a part of the optical lens. Can be miniaturized and manufactured at low cost. Further, since the optical axes of the projection optical system and the photographing optical system are coaxial, depth information of the entire area of the subject on which the projection pattern is projected can be calculated, and there is no occurrence of a blind spot where the depth cannot be calculated.

(Embodiment 2) A second embodiment of the present invention will be described. The image capturing apparatus according to the present embodiment differs from the image capturing apparatus according to the first embodiment only in part of the configuration of the projection unit 14 of the projection imaging unit 20. Therefore, the same components will be described. Are omitted, and only different components will be described.

FIG. 12 is a diagram for explaining the projection and imaging of pattern light by the projection imaging unit 20. In the present embodiment, a cylindrical lens 330 is used as a projection lens of the projection unit 14. Cylindrical lens 330
Is a concave cylindrical lens having a lens power only in the vertical direction and having a focal line in the horizontal direction. Light source 38
Irradiates the coded pattern 36 (hereinafter simply referred to as “projected pattern light”) with a cylindrical lens 330.
, Is reflected by the optical path splitting element 23, passes through the taking lens 22, and is projected onto the subject.

[0078] by the cylindrical lens 330, projected pattern of light is vertically converging diverging center P _0.
On the other hand, since the cylindrical lens 330 does not diffuse light in the horizontal direction, the projection pattern light converges on the center O of the photographing lens 22 in the horizontal direction. Cylindrical lens 3
When a convex cylindrical lens is used as 30,
Although the vertical convergence position (the center of divergence) is located on the optical path before the taking lens 22, the effect of the present invention is the same. Therefore, a case where a concave cylindrical lens is used as the cylindrical lens 330 will be described below. I do.

When the subject is located at a distance Z ₁ from the position O of the photographing lens 22 by the projection pattern light, the pattern P ₁ is projected on the subject, and when the subject is located at a distance Z ₂ , the pattern P ₁ is projected on the subject. pattern _{P 2} is projected. Due to the effect of the cylindrical lens 330, the projection pattern P
1, P2 is enlarged at a larger magnification in the vertical direction than in the horizontal direction.

Next, the image pickup section 12 picks up an image of the subject on which the pattern is projected by the projection pattern light. The taking lens 22 converges the light on the subject on which the pattern is projected at the center O of the taking lens 22,
3 to form an image on the CCD 30. Shooting lens 2
The captured image on the CCD 30 of the projection pattern P ₁ on the subject located at a distance Z ₁ from the position O of position 2 is I ₁ , and the projection pattern on the subject located at a distance Z ₂ from the position O of the imaging lens 22. capturing images in CCD30 of P ₂ is _{I 2.}

FIG. 13 shows the projection pattern P₁, P₂Shooting
Statue I₁, I₂FIG. Receiving lens 22
Convergence position O and vertical divergence center P of projection pattern light₀
Is located at a different position from the vertical
P₁, P₂Image I of₁, I₂Is different. Cylindri
Due to the vertical diffusion effect of the Cull lens 330,
The difference in size occurs in the₁Circular figure in
The size of the shape is S₁, Whereas the photographed image I₂Smell
The size of a circular figure is S₁Greater S₂It is. one
The center of the horizontal divergence of the projected pattern light is the taking lens
22 is located at the same position as the convergence position O,
Is the projection pattern P₁, P₂Image I of ₁, I₂Difference
Does not occur. Therefore, the photographed image I₁Circular figure in
The horizontal dimension L of the shape₁And image I₂Circular in
The horizontal size of the figure is L₂Are the same.

As described above, due to the effect of the cylindrical lens 330, the photographed image of the projection pattern is obtained by expanding the graphic pattern of the coded pattern 36 only in the vertical direction according to the depth of the subject. From the vertical size of the image of the projection pattern, the depth value of the area of the subject on which the projection pattern is projected can be obtained as in the first embodiment.

As the distance to the subject increases, the image of the projection pattern expands greatly in the vertical direction, so that the density of measurement points at which depth information can be calculated in the vertical direction becomes coarse. On the other hand, since the captured image of the projection pattern does not expand in the horizontal direction, the density of measurement points at which depth information can be calculated does not become coarse in the horizontal direction even if the distance to the subject increases. Therefore, it is preferable to increase the arrangement density of the graphic patterns of the coded pattern 36 in the horizontal direction. Further, since the projected pattern image is expanded only in the vertical direction, the figure pattern may have any shape whose size can be easily detected only in the vertical direction, and the figure pattern can be simplified. Further, since the size can be detected by focusing only on the vertical direction, image processing is facilitated.

Although the accuracy of calculating the depth value in the vertical direction increases due to the effect of the cylindrical lens 330, the density of the measurement points of the depth value becomes coarse. Is high, and depth distribution information of the subject having a low measurement point density in the vertical direction is obtained.
The projection unit 14 may include a driving unit 340 that rotates the cylindrical lens 330 in the optical axis direction. Drive unit 340
By rotating the cylindrical lens 330 by 90 degrees in the optical axis direction, the measurement point density in the vertical direction is high,
In the horizontal direction, depth distribution information of a subject having a low measurement point density can be obtained.

By rotating the cylindrical lens 330 in this manner, the measurement point density is high in any direction,
In a direction perpendicular to that direction, depth distribution information with a low measurement point density can be obtained. Therefore, by rotating the cylindrical lens 330 by 90 degrees to synthesize two pieces of depth distribution information obtained before and after the rotation, it is possible to obtain depth distribution information having a high measurement point density in both horizontal and vertical directions. it can. Further, the driving unit 340 includes the coding pattern 3 together with the cylindrical lens 330.
6 may be rotated.

FIG. 14 is a flowchart of the image capturing method according to the present embodiment. The cylindrical lens 330 is
A case in which power is provided in the vertical direction will be described. The projection unit 14 of the projection imaging unit 20 has two horizontal and vertical directions.
The light transmitting the shape pattern having two divergence centers is projected onto the subject (S200), and the imaging unit 12 of the projection imaging unit 20 is optically identical to the horizontal divergence center of the projection pattern light projected by the projection unit 14. At the position, the light on the subject on which the shape pattern is projected is converged to capture an image of the subject (S202). A specific area of the image of the subject is selected (S204). For the selection of the specific area, the area of the image of the subject may be appropriately divided, and the automatically divided areas may be sequentially selected.
May be used to select a region of the image of the designated subject. The shape pattern detection unit 304 extracts the image of the shape pattern projected on the specific area of the subject by using the image processing technique (S206), and detects the size of the extracted image of the shape pattern in the vertical direction (S208). Depth calculator 30
8, using the size of the image of the detected shape pattern,
The depth value of the specific area is calculated (S210).

When the cylindrical lens 330 has a lens power in the horizontal direction, the imaging unit 12
At a position optically the same as the vertical divergence center of the projection pattern light projected by 4, the light on the subject on which the shape pattern is projected is converged to capture an image of the subject. The shape pattern detection unit 304 detects the horizontal size of the shape pattern. Other processes are the same.

FIG. 15 is a flowchart of another image pickup method according to the present embodiment. The projection unit 14 has a driving unit 340, and can rotate the cylindrical lens 330 by 90 degrees in the optical axis direction to project the pattern light. The projection unit 14 of the projection imaging unit 20 has two horizontal and vertical directions.
The light transmitting the shape pattern having two divergence centers is projected onto the subject (S300), and the imaging unit 12 of the projection imaging unit 20 is optically identical to the horizontal divergence center of the projection pattern light projected by the projection unit 14. At the position, the light on the subject on which the shape pattern is projected is converged to capture an image of the subject (S302). Shape pattern detector 3
In step S304, an image of the shape pattern projected on the subject is extracted by an image processing technique (S304), and the size of the extracted image of the shape pattern in the vertical direction is detected (S306).
The depth calculation unit 308 calculates a first depth distribution of the entire subject on which the shape pattern is projected, using the size of the image of the detected shape pattern (S308). Projection unit 1
4 projects light transmitting a shape pattern having two divergence centers in the horizontal and vertical directions onto the subject (S310), and the imaging unit 12 of the projection imaging unit 20 causes the projection pattern light projected by the projection unit 14 to be vertical. At the optically same position as the divergence center in the direction, the light on the subject on which the shape pattern is projected is converged to capture an image of the subject (S31).
2). The shape pattern detection unit 304 extracts an image of the shape pattern projected on the subject by an image processing technique (S
314), the horizontal size of the image of the extracted shape pattern is detected (S316). The depth calculation unit 308 calculates a second depth distribution of the entire subject on which the shape pattern is projected, using the size of the image of the detected shape pattern (S318). The depth distribution of the subject is obtained by combining the first depth distribution and the second depth distribution (S32
0).

The first depth distribution has a high measurement point density in the horizontal direction and a low measurement point density in the vertical direction. Conversely, the second depth distribution has a high density of measurement points in the vertical direction and a low density of measurement points in the horizontal direction. Therefore, by combining the first depth distribution and the second depth distribution,
A depth distribution of a subject having a high measurement point density in both the horizontal and vertical directions can be obtained.

Further, instead of the cylindrical lens 330, an anamorphic optical lens system having different lens powers in the horizontal and vertical directions may be used.
The anamorphic optical system may be configured by combining a cylindrical lens having power in the vertical direction and a cylindrical lens having power in the horizontal direction.
6, an anamorphic lens having different lens powers in the vertical and horizontal directions as shown in FIGS. 17 and 18 may be used. FIG. 16 is a perspective view of an anamorphic lens having a convex front surface and a concave rear surface. FIG. 17 is a perspective view of an anamorphic lens whose front surface is weakly convex and whose back surface is strongly convex. FIG. 18 is a perspective view of an anamorphic lens having a slightly concave front surface and a strongly concave rear surface.

Each of the anamorphic lenses in FIGS. 16 to 18 has different lens power in the vertical and horizontal directions. Therefore, the projection pattern light projected by the projection unit 14 using the anamorphic optical system has a divergence center in the vertical direction and the horizontal direction at a position optically different from the center O of the photographing lens 22. Therefore, the projection pattern image captured by the imaging unit 12 is expanded at different magnifications in both the vertical and horizontal directions according to the depth of the subject, so that either the vertical or horizontal size of the projection pattern image is used. However, the depth value can be calculated.

FIG. 19 is a flowchart of another image pickup method according to the present embodiment. An anamorphic lens is used as a projection lens of the projection unit 14 of the projection imaging unit 20. The projection unit 14 projects light transmitting a shape pattern having two divergence centers in the horizontal and vertical directions onto the subject (S
400), the imaging unit 12 of the projection imaging unit 20 emits light from the subject on which the shape pattern is projected at a position optically different from any of the horizontal and vertical divergence centers of the projection pattern light projected by the projection unit 14. The convergence is performed, and an image of the subject is captured (S402). A specific area of the subject image is selected (S404). Shape pattern detector 3
In step S406, the image of the shape pattern projected on the specific area of the subject is extracted by an image processing technique (S406), and the horizontal and vertical sizes of the extracted image of the shape pattern are detected (S408). The depth calculation unit 308 calculates the depth value of the specific area based on the horizontal and vertical sizes of the image of the detected shape pattern (S210).
The average of the horizontal and vertical sizes may be calculated and used for calculating the depth value, or the one with better detection accuracy for the horizontal and vertical sizes may be used for calculating the depth value.

As described above, when the anamorphic lens having the lens power in the horizontal and vertical directions is used, the size is detected in both the horizontal and vertical directions, and is used for calculating the depth value. Accuracy can be improved.

In the above embodiment, the cylindrical lens and the anamorphic lens are optical lenses as an example of the optical element. The cylindrical lens and anamorphic lens do not need to be cylindrical lenses, but have a sawtooth-shaped surface with a diagonal component to have the optical properties of a cylindrical lens and anamorphic lens. The lens may be a Fresnel lens.

As described above, according to the image pickup apparatus of the present embodiment, the shape pattern is projected on the subject by the projection light having two divergence centers in the horizontal and vertical directions, and the shape of the subject on which the shape pattern is projected is obtained. From the image, the depth distribution of the subject can be calculated. Since the projection pattern can be expanded only in the horizontal or vertical direction, image processing for size detection is facilitated. In addition, the projection pattern can be expanded at different magnifications in both the horizontal and vertical directions, and the size can be detected in both the horizontal and vertical directions to improve the accuracy of calculating the depth value.

(Embodiment 3) A third embodiment of the present invention will be described. FIG. 20 is a configuration diagram of a lab system 200 that develops and edits a photographic image as an example of an image processing apparatus. The lab system 200 of the present embodiment includes an input unit 210, a processing unit 220, a recording unit 240, and an output unit 2
50.

The input section 210 inputs image data of a subject. As the image data, an image of the subject on which the shape pattern is projected is input. When inputting a digital image of a subject photographed by a digital camera or the like, the input unit 210
For this, a reading device for reading image data from a removable recording medium such as a semiconductor memory card is used. When image data is read from a floppy disk, MO, CD-ROM, or the like, a floppy drive, MO drive, CD
A drive or the like may be used.

The processing section 220 stores the image of the subject input by the input section 210 and calculates depth information of the subject. The processing unit 220 outputs the calculated depth information to the recording unit 240 together with the image of the subject. The processing unit 220 may process the image of the subject based on the calculated depth information and output the processed image to the recording unit 240 and the output unit 250.

The recording section 240 records the depth information or image data output from the processing section 220 on a removable recording medium. Writable CD-RO as recording medium
Optical recording media such as M and DVD, magneto-optical recording media such as MO, and magnetic recording media such as floppy disks are used. As the recording unit 240, a CD-R drive, a DVD drive, an MO drive, a floppy drive, or the like is used. The recording unit 240 may record depth information or image data in a semiconductor memory such as a flash memory or a memory card.

The output section 250 outputs the processed image data of the subject output by the processing section 220 as an image.
For example, when displaying an image on a screen, a monitor that displays the image is used as the output unit 250. For example, when printing an image, a printer such as a digital printer or a laser printer is used as the output unit 250.

FIG. 21 is a functional block diagram of the processing section 220. The processing unit 220 includes an image storage unit 302, a shape pattern detection unit 304, a depth calculation unit 308, and an image processing unit 312.

The image storage unit 302 stores the image data of the subject input by the input unit 210 in a semiconductor memory such as a RAM or a magnetic recording medium such as a hard disk. The shape pattern detection unit 304 extracts an image of the shape pattern projected on a specific area of the subject stored in the image storage unit 302, and detects the size of the image of the shape pattern. The depth calculation unit 308 calculates the depth value of the specific region of the subject using the size detected by the shape pattern detection unit 304.

The processing in which the shape pattern detecting unit 304 and the depth calculating unit 308 calculate the depth information of the subject for a partial or entire region of the subject captured in the subject image is described in the first embodiment. Therefore, the description is omitted.

The image processing section 312 includes a depth calculating section 308
The image of the subject is processed based on the depth information of the subject calculated by. The image processing unit 312 stores the depth information of the subject, the image of the subject, or the processed image in the storage unit 24.
0 and output to the output unit 250.

The image processing section 312 may extract a specific part of the subject, for example, an image of the main subject or the background, based on the depth information of the subject.

According to the image processing apparatus of the present embodiment, it is possible to input the image of the subject on which the shape pattern is projected, and calculate the depth information of the subject. Further, image processing can be performed based on the calculated depth information to extract images of the main subject and the background.

(Embodiment 4) A fourth embodiment of the present invention will be described. FIG. 22 is a configuration diagram of the image processing apparatus. The basic configuration and operation of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus according to the second embodiment. The present embodiment differs from the second embodiment in that an electronic computer such as a personal computer or a workstation is used as the processing unit 220 of the image processing apparatus.

The hardware configuration of the processing section 220 according to the present embodiment will be described with reference to FIG. CPU 230
Operate based on programs stored in the ROM 232 and the RAM 234. Data is input by a user via an input device 231 such as a keyboard and a mouse. The hard disk 233 stores data such as images and the CPU 2
30 is stored. CD-ROM
The drive 235 reads data or a program from the CD-ROM 290,
33 and the CPU 230.

The functional configuration of the program executed by the CPU 230 is the same as that of the processing unit 220 of the image processing apparatus according to the third embodiment.
The image storage module, the shape pattern detection module, the depth calculation module,
An image processing module.

The processing that the image storage module, the shape pattern detection module, the depth calculation module, and the image processing module cause the CPU 230 to perform are respectively performed by the processing unit 220 of the image processing apparatus of the third embodiment.
The functions and operations of the parallax image storage unit 302, the shape pattern detection unit 304, the depth calculation unit 308, and the image processing unit 312 are the same, and a description thereof will be omitted. These programs are stored in a recording medium such as the CD-ROM 290 and provided to the user. CD-R as an example of a recording medium
The OM 290 can store some or all functions of the operation of the image processing apparatus described in the present application.

The above program is directly stored in the RA from the recording medium.
It may be read by M234 and executed by the CPU 230. Alternatively, the above program is installed on the hard disk 233 from a recording medium,
And may be executed by the CPU 230.

As a recording medium, in addition to the CD-ROM 290, a hard disk, a memory such as a ROM or a RAM,
An optical recording medium such as a DVD or PD, a magnetic recording medium such as a floppy disk or a mini disk (MD), a magneto-optical recording medium such as an MO, a tape-shaped recording medium, and a nonvolatile semiconductor memory card can be used.

The program described above may be stored on a single recording medium, or may be divided and stored on a plurality of recording media. Further, the program may be compressed and stored in a recording medium. The compressed program may be decompressed, read to another recording medium such as the RAM 234, and executed. Furthermore, the compressed program is a CPU
After being decompressed by 230 and installed on the hard disk 233 or the like, it may be read to another recording medium such as the RAM 234 and executed.

Furthermore, a CD-R as an example of a recording medium
The OM 290 may store the program provided by the host computer via the communication network. The above program stored in the recording medium is
It may be stored in the hard disk of the host computer, transmitted from the host computer to the computer via a communication network, read out to another recording medium such as the RAM 234, and executed.

The recording medium storing the above program is:
It is apparent that the invention is used only for manufacturing the image processing apparatus of the present application, and that the manufacture and sale of such a recording medium as a business constitute infringement of a patent right based on the present application.

As described above, according to the image capturing apparatus and the image processing apparatus of the present invention, it is possible to easily and efficiently calculate the depth information of a subject with high accuracy.

As described above, the present invention has been described using the embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is apparent to those skilled in the art that various changes or improvements can be added to the above embodiment. It is apparent from the description of the appended claims that embodiments with such changes or improvements can be included in the technical scope of the present invention.

[0118]

As is apparent from the above description, according to the present invention, information on the depth of a subject can be obtained easily and efficiently.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a digital camera 10 as an example of an image capturing apparatus.

FIG. 2 is a view for explaining projection and imaging of pattern light by a projection imaging unit 20;

FIG. 3 shows an example of a coding pattern 36, which is a circular figure pattern arranged two-dimensionally.

[4] projection pattern _P 1, pickup image _I 1 of _{P 2, I 2}
FIG.

FIG. 5 is a diagram illustrating the relationship between the distance of a projection pattern and the size of a captured image of the projection pattern.

FIG. 6 is a functional block diagram of the processing unit 60.

FIG. 7 is a slit pattern, which is another example of the coding pattern 36;

FIG. 8 is another example of a coded pattern 36, which is a polygonal pattern.

FIG. 9 shows another example of the coded pattern 36, which is a cross-shaped pattern.

FIG. 10 is a point pattern which is another example of the coding pattern 36.

FIG. 11 is a flowchart of an image capturing method according to the present embodiment.

FIG. 12 is a diagram for describing projection and imaging of pattern light by the projection imaging unit 20.

FIG. 13 shows captured images I ₁ , I of projection patterns P ₁ , P _2
2 is a diagram illustrating a.

FIG. 14 is a flowchart of an image capturing method according to the present embodiment.

FIG. 15 is a flowchart of another image capturing method according to the present embodiment.

FIG. 16 is a perspective view of an anamorphic lens having a convex front surface and a concave rear surface.

FIG. 17 is a perspective view of an anamorphic lens whose front surface is weakly convex and whose back surface is strongly convex.

FIG. 18 is a perspective view of an anamorphic lens having a slightly concave front surface and a strongly concave rear surface.

FIG. 19 is a flowchart of another image capturing method according to the present embodiment.

FIG. 20 is a configuration diagram of a lab system 200 that develops and edits a photographic image, as an example of an image processing apparatus.

FIG. 21 is a functional configuration diagram of a processing unit 220.

FIG. 22 is a configuration diagram of an image processing apparatus.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 10 Digital camera 12 Imaging part 14 Projection part 20 Projection imaging unit 22 Imaging lens 23 Optical path splitting element 30 CCD 34 Projection lens 36 Coding pattern 38 Light source 40 Control unit 60 Processing unit 100 Display unit 110 Operation unit 200 Image processing device 210 Input part 220 processing unit 240 recording unit 250 output unit 290 recording medium 302 image storage unit 304 shape pattern detection unit 308 depth calculation unit 310 recording unit 312 image processing unit 330 cylindrical lens 340 driving unit

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/225 G01B 11/24 E 5C061 13/02 G06F 15/62 415 F Term (Reference) 2F065 AA04 AA06 DD00 DD02 FF00 FF04 GG03 GG08 HH05 HH06 HH07 JJ03 JJ26 LL04 LL06 LL08 LL10 LL26 LL30 QQ03 QQ23 QQ24 QQ26 QQ31 SS06 SS13 UU01 2F112 AD10 BA05 BA09 CA12 DA05 DA06 DA32 DA40 FA07 FA02 AFA0A02 0 AA13 AB12 AB21 AB66 AB68 AC01 AC03 AC32 AC42 AC52 AC69 AC74 AC78 5C061 AB03 AB06 AB08

Claims (21)

[Claims] 1. An image capturing apparatus for acquiring information on a depth of a subject, wherein the projection section projects light to project a shape pattern on the subject, and transmits the shape pattern projected by the projection section. At a position optically different from the divergence center of light, converge light on the subject on which the shape pattern is projected,
An image pickup apparatus, comprising: an image pickup unit for picking up an image of the subject. 2. A shape pattern detection unit for detecting a size of an image of the shape pattern projected on a specific area of the subject imaged by the imaging unit, based on the size detected by the shape pattern detection unit A depth calculation unit configured to calculate a depth value of the specific area of the subject.
An image pickup device according to claim 1. 3. The optical axis when the projection unit projects light onto the subject and the optical axis when the imaging unit captures the subject are substantially the same. The image pickup apparatus according to any one of the preceding claims. 4. The image according to claim 3, wherein the imaging unit and the projection unit use an optical element having the same optical axis to image the subject and project light onto the subject, respectively. Imaging device. 5. The shape pattern is a plurality of figure patterns arranged two-dimensionally, the projection unit simultaneously projects the plurality of figure patterns onto the subject, and the imaging unit The object on which the pattern is projected is imaged, the shape pattern detection unit detects the size of each of the plurality of figure patterns, and the depth calculation unit includes the plurality of shape patterns detected by the shape pattern detection unit. The apparatus according to claim 2, wherein the depth value of each area of the subject on which the plurality of graphic patterns are projected is calculated based on the size of each image of the graphic pattern. 6. The apparatus according to claim 5, wherein the graphic pattern is one of a point, a line, a circle, and a polygon. 7. An image capturing apparatus for acquiring information relating to the depth of a subject, wherein the apparatus projects light having two different optical divergence centers in two directions perpendicular to an optical axis to the subject. A projection unit for projecting a shape pattern; and a light source on the object on which the shape pattern is projected converges at a convergence position optically substantially at the same position as one of the two divergence centers to image the object. An image pickup apparatus, comprising: 8. The size of an image of the shape pattern projected on a specific region of the subject imaged by the imaging unit, with respect to a divergence direction of light at the divergence center located at a position optically different from the convergence position. And a depth calculation unit that calculates a depth value of the specific area of the subject based on the size of the divergent azimuth of the image of the shape pattern detected by the shape pattern detection unit. The image capturing apparatus according to claim 7, further comprising: 9. The optical axis when the projection unit projects light onto the subject and the optical axis when the imaging unit captures the subject are substantially the same. The image pickup apparatus according to any one of the preceding claims. 10. The projector according to claim 1, wherein the projection unit projects light having the two different optical divergence centers in the two directions using an optical element having power in only one direction. 10. The image pickup device according to 9. 11. The image pickup apparatus according to claim 10, further comprising a driving unit that rotates the optical element having power only in one direction around an optical axis. 12. The shape pattern detection unit may determine the size of an image of the shape pattern in the direction of power of the optical element before and after the driving unit rotates the optical element by 90 degrees around an optical axis. The depth calculation unit detects the depth value of the specific area of the subject using the size of the shape pattern image detected before and after the rotation of the optical element by the shape pattern detection unit. The image pickup device according to claim 11, wherein: 13. An image capturing apparatus for acquiring information relating to the depth of a subject, wherein the apparatus projects light having two different optical divergence centers in two directions perpendicular to an optical axis to the subject. A projection unit that projects a shape pattern; and an imaging unit that converges light on the subject on which the shape pattern is projected at a convergence position that is optically different from any of the two divergence centers to capture the subject. An image pickup apparatus comprising: 14. A shape pattern detection for detecting two sizes of the shape pattern image projected on a specific area of the subject imaged by the imaging unit in the two directions of light at the two divergence centers. And a depth calculation unit that calculates a depth value of the specific region of the subject in consideration of the two sizes in the two directions of the image of the shape pattern detected by the shape pattern detection unit. 2. The device according to claim 1, wherein
4. The image pickup device according to 3. 15. The optical axis when the projection unit projects light onto the subject and the optical axis when the imaging unit captures an image of the subject are substantially the same.
An image pickup device according to claim 1. 16. The projection unit according to claim 2, wherein the projection unit is perpendicular to an optical axis.
Using an optical element having independent power in directions,
16. The image pickup apparatus according to claim 15, wherein light having the two different optical divergence centers is projected according to three directions. 17. An image processing method for acquiring information relating to the depth of a subject, wherein the method projects light to project a shape pattern onto the subject, and the divergence center of light transmitting the shape pattern is optically determined. At different positions, converges light on the subject on which the shape pattern is projected, captures an image of the subject, detects the size of the shape pattern image projected on a specific region of the subject, An image processing method for calculating a depth value of the specific region of the object based on the depth value. 18. An image processing method for acquiring information relating to the depth of a subject, comprising projecting light having two different optical divergence centers in two directions perpendicular to an optical axis to the subject. Projecting a shape pattern, at a convergence position optically substantially at the same position as one of the two divergence centers, converging light on the subject on which the shape pattern is projected, imaging the subject, For the divergence direction of light at the divergence center at a position optically different from the convergence position, the size of an image of the shape pattern projected on a specific region of the subject imaged by the imaging unit is detected. Calculating a depth value of the specific region of the subject based on the size of the divergence direction of the image of the shape pattern detected by the pattern detection unit; Image processing method for the. 19. An image processing method for acquiring information relating to the depth of a subject, comprising projecting light having two different optical divergence centers in two directions perpendicular to an optical axis to the subject. Projecting a shape pattern, converging light on the subject on which the shape pattern is projected at a convergence position that is optically different from any of the two divergence centers, imaging the subject, In the two directions of light at the center, the image capturing unit detects two sizes of the image of the shape pattern projected on a specific region of the subject, and the shape pattern detected by the shape pattern detecting unit Calculating a depth value of the specific area of the subject in consideration of the two sizes of the image in the two directions. Management method. 20. An image processing apparatus for acquiring information on a depth of a subject, comprising: an input unit for inputting an image of the subject on which a shape pattern is projected; and an input unit for inputting an image of the shape pattern projected on a specific region of the subject. A shape pattern detection unit that detects a size of an image; and a depth calculation unit that calculates a depth value of the specific region of the subject based on the size detected by the shape pattern detection unit. Image processing apparatus. 21. A recording medium storing a computer program for acquiring information on the depth of a subject, the program comprising: an input module for inputting an image of the subject on which a shape pattern is projected; A shape pattern detection module for detecting a size of an image of the shape pattern projected on a specific region; and calculating a depth value of the specific region of the subject based on the size detected by the shape pattern detection module. A recording medium comprising a depth calculation module.