JP2001141422A - Image pickup device and image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup device and an image processor capable of obtaining information relating to the depth of an object over a wide visual field. SOLUTION: This device is provided with an image pickup part 20 for picking up the parallactic image of the object obtained in the case of viewing the object from two different view point positions, a size ratio detection part 306 for detecting the ratio of the size of the image of the specified area of the object in the parallactic image, a position deviation detection part 304 for detecting the deviation of the position of the image of the specified area of the object in the parallactic image and a depth calculation part 308 for calculating the depth value of the specified area of the object based on the position deviation and the size ratio.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image pickup apparatus, an image processing apparatus, an image processing method, and a recording medium for acquiring information on the depth of a subject. In particular, the present invention relates to an image capturing apparatus, an image processing apparatus, an image processing method, and a recording medium that acquire information on the depth of a subject based on a parallax image.

[0002]

2. Description of the Related Art In order to obtain positional information of a subject, two cameras are arranged side by side to imitate the function of binocular stereopsis of a human, and a parallax image is obtained when the subject is viewed from two different viewpoints. There is an old technique of stereo photography that measures the depth value of an image. Based on the difference in the viewpoint positions, it is detected that the image of the subject is displaced on the parallax image, and the distance from the camera to the subject is determined based on the principle of triangulation based on the displacement of the image and the focal length of the camera lens. The method of measuring is adopted.

[0003]

However, even if two cameras are arranged, a blind spot area of parallax is generated in the moving direction of the viewpoint due to the limitation of the viewing angle of the lens, and the depth information of the subject over a wide field of view can be obtained with high accuracy. Had a problem that it was not possible to get it.

[0004] In order to solve the above-mentioned problems, the present invention provides an image pickup apparatus, an image processing apparatus, and an image processing method capable of acquiring information on the depth of a subject over a wide field of view.
It is an object to provide an image processing method and a recording medium. This object is achieved by a combination of features described in the independent claims. The dependent claims define further advantageous embodiments of the present invention.

[0005]

According to a first aspect of the present invention, there is provided an image pickup apparatus for acquiring information relating to the depth of a subject. An imaging unit that captures a parallax image of a subject obtained when the subject is viewed, a size ratio detection unit that detects a size ratio of an image of a specific region of the subject in the parallax image, and A depth calculation unit for calculating a depth value of the specific area.

[0006] The image processing apparatus further includes a position shift detecting unit for detecting a position shift of the image of the specific region of the subject in the parallax image, and the depth calculating unit calculates a depth value of the specific region of the subject based on the position shift and the size ratio. It may be calculated.

[0007] The depth calculation section provides, for each specific region of the subject,
The depth value of the specific region of the subject may be calculated by changing the ratio in which the displacement and the size ratio are considered. The depth calculation unit calculates the depth value of the specific region of the subject by changing the ratio of considering the position shift and the size ratio depending on the azimuth when the specific region of the subject is viewed from the midpoint of the two viewpoint positions. Is also good. As the specific region of the subject approaches the direction of the straight line connecting the two viewpoint positions, the ratio of considering the size ratio is increased, and the specific region of the subject passes through the midpoint of the two viewpoint positions and the two viewpoint positions are shifted. The depth value of the specific region of the subject may be calculated by increasing the rate of considering the positional deviation as approaching a plane perpendicular to the connecting straight line.

The imaging section has an optical lens having a wide viewing angle, and further includes a driving section for moving the optical lens. The imaging section uses the position at which the driving section has moved the optical lens as a viewpoint position to determine the parallax of the subject. An image may be taken. The imaging unit may include two optical lenses having different viewing points and wide viewing angles, and may capture a parallax image of the subject using the two optical lenses. The optical lens may be a fisheye lens, and the depth calculation unit may calculate a depth value for an omnidirectional area of the subject imaged by the imaging unit with the fisheye lens. The imaging unit may include a solid-state imaging device, and image a parallax image of the subject on the solid-state imaging device using an optical lens.

According to a second aspect of the present invention, there is provided an image processing apparatus for acquiring information relating to the depth of a subject,
An input unit for inputting a plurality of parallax images of the subject obtained when the subject is viewed from a plurality of different viewpoint positions, and a size ratio detection for detecting a ratio of an image size of a specific region of the subject in the plurality of parallax images And a depth calculator for calculating a depth value of a specific area of the subject based on the size ratio.

[0010] The image processing apparatus may further include an image conversion unit that converts the image by performing coordinate conversion of the image of the subject input by the input unit based on the depth value of the specific region of the subject calculated by the depth calculation unit. When the image of the subject input by the input unit is an omnidirectional image obtained by capturing the subject with a fisheye lens, the image conversion unit may convert the omnidirectional image into a perspective projection image by coordinate conversion. The image conversion unit may generate an orthographic image of the subject by performing coordinate conversion.

According to a third aspect of the present invention, there is provided an image processing method for acquiring information on the depth of a subject,
A plurality of parallax images of a subject obtained when the subject is viewed from a plurality of different viewpoint positions are input, and a position shift and a size ratio of an image of a specific region of the subject in the plurality of parallax images are detected. The depth value of the specific region of the subject is calculated by changing the ratio in which the displacement and the size ratio are considered for each specific region.

According to a fourth aspect of the present invention, there is provided a recording medium storing a computer program for acquiring information on the depth of a subject, the program being provided when the subject is viewed from a plurality of different viewpoint positions. An input module for inputting a plurality of parallax images of a subject to be input, a position shift detection module for detecting a position shift of an image of a specific region of the subject in the plurality of parallax images, and an image of a specific region of the subject in the plurality of parallax images. A size ratio detection module that detects a size ratio, and a specific region of a subject,
A depth calculation module that calculates a depth value of a specific area of the subject by changing a ratio in which a position shift and a size ratio are considered.

The above summary of the present invention does not list all of the necessary features of the present invention, and sub-combinations of these features may also constitute the present invention.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described through embodiments of the present invention. However, the following embodiments do not limit the invention according to the claims and are described in the embodiments. Not all combinations of the features described above are essential to the solution of the invention.

(Embodiment 1) A first embodiment of the present invention will be described. FIG. 1 is a configuration diagram of a digital camera 10 as an example of an image capturing device. The digital camera 10 includes a digital still camera, a digital video camera capable of capturing a still image, and the like. Digital camera 10
Mainly includes an imaging unit 20, an imaging control unit 40, a processing unit 60, a display unit 100, and an operation unit 110.

The imaging unit 20 includes a mechanism member and an electric member related to photographing and image formation. Imaging unit 20
First, a photographing lens 22 for capturing and processing images,
An aperture 24, a shutter 26, an optical LPF (low-pass filter) 28, a CCD 30 which is an example of a solid-state image sensor, and an image signal processor 32 are included. The taking lens 22 includes a focus lens, a zoom lens, and the like. With this configuration, a subject image is formed on the light receiving surface of the CCD 30. Charges are accumulated in each sensor element (not shown) of the CCD 30 according to the amount of light of the formed subject image (hereinafter, the charges are referred to as “accumulated charges”). The accumulated charge is read out to a shift register (not shown) by a read gate pulse, and is sequentially read out as a voltage signal by a register transfer pulse.

Since the digital camera 10 generally has an electronic shutter function, a mechanical shutter such as the shutter 26 is not essential. In order to realize the electronic shutter function, the CCD 30 is provided with a shutter drain via a shutter gate. When the shutter gate is driven, accumulated charges are swept out to the shutter drain. By controlling the shutter gate, the time for accumulating the electric charges in each sensor element, that is, the shutter speed can be controlled.

A voltage signal output from the CCD 30, that is, an analog signal is color-separated into R, G, and B components by an image signal processing unit 32, and a white balance is adjusted first.
Subsequently, the imaging signal processing unit 32 performs gamma correction, sequentially A / D converts the R, G, and B signals at necessary timing, and obtains digital image data (hereinafter, simply referred to as “digital image data”) obtained as a result. ) Is output to the processing unit 60.

The image pickup unit 20 further includes a finder 3
4 and a strobe 36. The finder 34 may be equipped with an LCD (not shown). In this case, various information from the main CPU 62 and the like described later can be displayed in the finder 34. The strobe 36 functions by emitting light when energy stored in a capacitor (not shown) is supplied to the discharge tube 36a.

The imaging control unit 40 includes a lens driving unit 4
2. It has a focus drive unit 44, an aperture drive unit 46, a shutter drive unit 48, an imaging system CPU 50 that controls them, a distance measurement sensor 52, and a photometry sensor 54. The driving units such as the lens driving unit 42 each have a driving unit such as a stepping motor. When a release switch 114 described later is pressed, the distance measurement sensor 52 measures the distance to the subject, and the photometry sensor 54 measures the brightness of the subject. The measured distance data (hereinafter simply referred to as “distance measurement data”) and the subject luminance data (hereinafter simply referred to as “photometry data”) are sent to the imaging system CPU 50. Imaging system CPU 50
Controls the lens drive unit 42 and the focus drive unit 44 to adjust the zoom magnification and focus of the photographing lens 22 based on the photographing information such as the zoom magnification designated by the user. Further, the imaging system CPU 50 controls the lens driving unit 42 to move the position of the imaging lens 22 in order to capture a parallax image.

The image pickup system CPU 50 controls the R of one image frame.
An aperture value and a shutter speed are determined based on an integrated value of a digital signal of GB, that is, AE information. According to the determined values, the aperture driving unit 46 and the shutter driving unit 48 adjust the aperture amount and open and close the shutter 26, respectively.

The imaging system CPU 50 controls the emission of the strobe light 36 based on the photometric data, and at the same time adjusts the aperture of the aperture 26. When the user instructs the capture of an image, the CCD 30 starts to accumulate electric charges, and the accumulated electric charges are output to the imaging signal processing unit 32 after a lapse of a shutter time calculated from the photometric data.

The processing unit 60 includes the digital camera 10
The main C that controls the whole, especially the processing unit 60 itself
PU 62 and memory control unit 6 controlled by PU 62
4, a YC processing unit 70, an optional device control unit 74, a compression / decompression processing unit 78, and a communication I / F unit 80. Main C
The PU 62 is connected to the imaging CPU 5 by serial communication or the like.
Necessary information is exchanged with 0. Main CPU6
The second operation clock is provided from a clock generator 88. The clock generator 88 also provides clocks having different frequencies to the imaging system CPU 50 and the display unit 100, respectively.

The main CPU 62 is provided with a character generator 84 and a timer 86. The timer 86 is backed up by a battery and always counts the date and time.
From this count value, information about the shooting date and time and other time information are given to the main CPU 62. The character generation unit 84 generates character information such as a shooting date and time and a title, and the character information is appropriately combined with the captured image.

The memory control unit 64 includes a nonvolatile memory 66
And the main memory 68. Non-volatile memory 66
Is constituted by an EEPROM (electrically erasable and programmable ROM) or a flash memory, and stores data to be retained even when the power of the digital camera 10 is off, such as setting information by a user and adjustment values at the time of shipment. ing. The non-volatile memory 66 may have a main CP
A boot program or a system program of U62 may be stored. On the other hand, the main memory 68 generally has a D
It is composed of a relatively inexpensive and large-capacity memory such as a RAM. The main memory 68 has a function as a frame memory for storing data output from the imaging unit 20, a function as a system memory for loading various programs, and a function as a work area. The non-volatile memory 66 and the main memory 68
Data is exchanged with each unit inside and outside the unit 0 via the main bus 82.

The YC processing unit 70 performs YC conversion on the digital image data, and outputs a luminance signal Y and a color difference (chroma) signal B-
Generate Y, RY. The luminance signal and the color difference signal are temporarily stored in the main memory 68 by the memory control unit 64.
The compression / decompression processing unit 78 sequentially reads out the luminance signal and the color difference signal from the main memory 68 and compresses them. The data thus compressed (hereinafter simply referred to as “compressed data”) is written to a memory card, which is a type of the optional device 76, via the optional device control section 74.

The processing unit 60 further includes an encoder 72
With. The encoder 72 receives a luminance signal and a color difference signal, converts them into a video signal (NTSC or PAL signal), and outputs the video signal from a video output terminal 90. When a video signal is generated from data recorded in the optional device 76, the data is first supplied to the compression / decompression processing unit 78 via the optional device control unit 74. Subsequently, the data subjected to the necessary expansion processing by the compression / expansion processing unit 78 is converted into a video signal by the encoder 72.

The optional device control unit 74 generates a necessary signal between the main bus 82 and the optional device 76, performs logical conversion, or converts voltage according to the signal specifications recognized by the optional device 76 and the bus specifications of the main bus 82. And so on. The digital camera 10 may include, for example, a PCM as an optional device 76 in addition to the above-described memory card.
A standard CIA-compliant standard I / O card may be supported. In that case, the optional device control unit 74
It may be constituted by an IA bus control LSI or the like.

The communication I / F unit 80 is a digital camera 10
Communication specifications supported by, for example, USB, RS-23
It performs control such as protocol conversion according to specifications such as 2C, Ethernet, Bluetooth, and IrDA. The communication I / F unit 80 includes a driver IC as necessary, and communicates with an external device including a network via a connector 92. In addition to such standard specifications, data may be exchanged with an external device such as a printer, a karaoke machine, a game machine, or the like by a unique I / F.

The display unit 100 includes an LCD monitor 10
2 and an LCD panel 104. These are the LCD driver monitor driver 106, panel driver 10
8 respectively. LCD monitor 102
Is provided on the back of the camera with a size of, for example, about 2 inches, and displays a current shooting and playback mode, a zoom ratio for shooting and playback, a remaining battery level, a date and time, a screen for mode setting, a subject image, and the like. . The LCD panel 104 is, for example, a small black-and-white LCD provided on the upper surface of the camera and has an image quality (FINE).
/ NORMAL / BASIC etc.), information such as strobe light emission / non-light emission, standard number of recordable images, number of pixels, battery capacity, etc. are simply displayed.

The operation unit 110 includes mechanisms and electrical members necessary for the user to set or instruct the operation and mode of the digital camera 10. The power switch 112 determines whether the power of the digital camera 10 is turned on or off. The release switch 114 has a two-stage pressing structure of half pressing and full pressing. As an example, AF and AE are locked by a half-press, captured images are captured by a full-press, and are recorded in the main memory 68, the optional device 76, etc. after necessary signal processing, data compression, and the like. The operation unit 110 may receive a setting using a rotary mode dial, a cross key, or the like in addition to these switches, and these are collectively referred to as a function setting unit 116 in FIG. Examples of operations or functions that can be specified by the operation unit 110 include “file format”, “special effects”, “printing”, “determine / save”, and “display switching”. The zoom switch 118 determines a zoom magnification.

The main operation of the above configuration is as follows. First, the power switch 11 of the digital camera 10
2 is turned on, and power is supplied to each part of the camera. The main CPU 62 determines whether the digital camera 10 is in the shooting mode or the playback mode by reading the state of the function setting unit 116.

When the camera is in the shooting mode, the main C
The PU 62 monitors the half-pressed state of the release switch 114. When the half-pressed state is detected, the main CPU 62
Obtains photometry data and distance measurement data from the photometry sensor 54 and the distance measurement sensor 52, respectively. The imaging control unit 40 operates based on the obtained data, and the focus, aperture, and the like of the photographing lens 22 are adjusted. Once the adjustment is complete,
Characters such as "standby" are displayed on the LCD monitor 102 to inform the user of the fact, and then the state of the release switch 114 being fully pressed is monitored. Release switch 1
When the shutter 14 is fully pressed, the shutter 26 is closed after a predetermined shutter time, and the accumulated charges in the CCD 30 are swept out to the imaging signal processing unit 32. The digital image data generated as a result of the processing by the imaging signal processing unit 32 is output to the main bus 82. The digital image data is temporarily stored in the main memory 68, and then processed by the YC processing unit 70 and the compression / decompression processing unit 78.
Is recorded in the option device 76 via the. The recorded image is displayed on the LCD monitor 102 for a while in a frozen state, so that the user can know the captured image. Thus, a series of photographing operations is completed.

On the other hand, when the digital camera 10 is in the reproduction mode, the main CPU 62 reads out the last photographed image from the main memory 68 via the memory control unit 64,
This is displayed on the LCD monitor 102 of the display unit 100. In this state, when the user instructs “forward” or “reverse” on the function setting unit 116, images taken before and after the currently displayed image are read and displayed on the LCD monitor 102.

In this embodiment, the imaging unit 20
Captures an image of a subject, particularly a parallax image of the subject.
FIG. 2 is an example of the configuration of the imaging lens 22 of the imaging unit 20 of the digital camera 10 and the CCD 30 that is an example of the solid-state imaging device. In the figure, other components included in the imaging unit 20 are omitted, and only the configuration of the imaging lens 22 and the CCD 30 of the imaging unit 20 is shown. In this example, two photographing lenses 22 and two CCDs 30 are provided, and an image of a subject formed by each photographing lens 22 is a CCD3.
0 is received. There is no need to provide two CCDs 30,
One common CCD 30 that can receive the image of the subject formed by the two photographing lenses 22 may be provided.

The photographing lens 22 is preferably an optical lens having a wide viewing angle, and may be a wide-angle lens or a fish-eye lens. The fisheye lens is designed to have a viewing angle of 180 degrees, and is most preferable for capturing an object with a wide field of view.

The CCD 30 of this embodiment is an example of a solid-state image sensor. A solid-state imaging device is a semiconductorized and integrated imaging device, and is a two-dimensionally arranged pixel group having a function of photoelectric conversion and charge accumulation on a semiconductor substrate in terms of structure. The solid-state imaging device receives light formed by the imaging lens 22 and accumulates electric charges by a photoelectric conversion action. The stored charge images are scanned in a certain order and read out as electric signals.

The solid-state image sensor basically includes a semiconductor element including a light receiving element for receiving light incident from the outside and performing photoelectric conversion, a package accommodating the semiconductor element,
In order to enable light to enter the light receiving element, a transparent protective member disposed at a position facing the semiconductor element of the package, and a light shielding property higher than the transparent protective member on the outer surface or inside of the transparent protective member. It is preferable that the light-shielding member is formed from a light-shielding member having the light-shielding member. Thereby, the quality of the captured image can be improved. Further, the transparent protective portion may have a function of a microlens to improve the resolution of an image to be formed. A color filter may be provided between the light receiving element section and the transparent protection section, or on or in the transparent protection section, so that a color image can be captured.

The CCD 30 of this embodiment is a charge-coupled device (CCD) having a sufficiently high resolution, a one-dimensional image sensor (linear sensor) or a two-dimensional image sensor (area sensor), so that parallax in a parallax image can be accurately detected.
An image sensor is desirable. MOS image sensor, CdS-S other than CCD as solid-state image sensor
Any of an e-contact image sensor, an a-Si (amorphous silicon) contact image sensor, or a bipolar contact image sensor may be used.

FIG. 3 shows the photographing lens 2 of the image pickup unit 20.
2 is another example of the configuration of the CCD 30. In this example,
One photographing lens 22 is provided, and the lens driving unit 42 moves the photographing lens 22 to capture a parallax image when the subject is viewed from a different viewpoint on the CCD 30.

FIG. 4 shows the photographing lens 2 of the image pickup unit 20.
2 is another example of the configuration of the CCD 30. Shooting lens 2
2 is provided for main photographing and parallax photographing. A photographing lens 22 for main photographing captures an image of a subject on the CCD 30, and a photographing lens 22 for parallax photographing captures a parallax image of the subject on the CCD 30. I do. Shooting lens 22 for actual shooting
May be a standard optical lens, a wide-angle lens with a wide viewing angle, or a fish-eye lens. The position of the imaging lens 22 for parallax imaging is moved by the lens driving unit 42 in order to capture a parallax image. The CCD 30 for allowing the photographing lens 22 for main photographing to receive an image and the CCD 30 for allowing the photographing lens 22 for parallax photographing to receive an image have a C
The resolution and sensitivity of the CD may be different.

The processing unit 60 of this embodiment acquires depth information of a subject based on a parallax image of the subject captured by the imaging unit 20. FIG. 5 shows the processing unit 6.
0 is a functional block diagram. The processing unit 60 includes a parallax image storage unit 302, a displacement detection unit 304, a size ratio detection unit 306, a depth calculation unit 308, and a recording unit 31.
0.

The parallax image storage unit 302 includes the imaging unit 2
0 stores a plurality of parallax images of the captured subject. The displacement detection unit 304 detects an amount by which the position of the image of the specific area of the subject shifts due to the parallax in the plurality of parallax images stored in the parallax image storage unit 302. Size ratio detector 3
Reference numeral 06 denotes a ratio in the parallax image stored in the parallax image storage unit 302, at which the size of the image of the specific region of the subject changes due to the parallax.

The depth calculator 308 is provided with the position shift detector 3
The depth value of the specific region of the subject is calculated using the position shift detected by the position detection unit 04 and the size ratio detected by the size ratio detection unit 306.

The position shift detector 304 and the size ratio detector 3
06, and the depth calculation unit 308 calculates the depth value of the subject with respect to a partial region or the entire region of the subject captured in the parallax image.

The depth calculation unit 308 inputs the calculated depth information of the subject to the imaging control unit 40, and the imaging control unit 40 executes the focus driving unit 44, the aperture driving unit 46, and the shutter driving unit based on the depth information of the subject. 4
8 may be controlled to adjust the focus, aperture, and shutter speed.

The recording section 310 stores the depth information of the subject calculated by the depth calculation section 308 and the parallax image storage section 302.
Is recorded in the optional device 76.

The displacement detector 30 of the processing unit 60
4. As an example, the functions of the size ratio detection unit 306 and the depth calculation unit 308 can be realized by cooperation of the main CPU 62 of FIG. 1 with a program stored or loaded in the main memory 68 or the nonvolatile memory 66. it can. When the main CPU 62 has a built-in memory, necessary programs may be stored in the memory and various functions may be realized as firmware. The parallax image data to be stored in the parallax image storage unit 302 of the processing unit 60 can be stored in the main memory 68 or the nonvolatile memory 66. The parallax image data is supplied to the compression / decompression processing unit 78.
May be compressed. The function of the recording unit 310 of the processing unit 60 is, for example, an optional device control unit 74.
It can be realized by. Further, the operation unit 110 that receives the user's instruction instructs the processing unit 60 to specify the specific area of the image of the subject, and the depth calculation unit 308
May calculate a depth value for a specific area specified by the user. There is considerable flexibility in designing the digital camera 10 to implement the above functions of the processing unit 60.

Next, lens characteristics of a fisheye lens as an example of an optical lens having a wide viewing angle, which is the most desirable form of the taking lens 22, will be described. FIG. 6 shows a fisheye lens 32.
0 is the incident angle θ of the object point to be imaged, and the omnidirectional image 32
FIG. 4 is a diagram for explaining the relationship between the position of an image of a subject point and the position of the subject in FIG. The center of the fisheye lens 320 is located at the origin of the xy plane. The azimuth of the point of the subject is represented by an incident angle θ formed by the light incident on the fisheye lens 320 from the point of the subject and the optical axis of the fisheye lens 320. The incident angle θ is a half angle of view of a point of the subject. The center of the omnidirectional image 322 is at the position of the origin on the XY plane. The position of the object point on the omnidirectional image 322 is
It is represented by a distance r from the origin of the XY plane. This distance r is also called an image height. Assuming that the focal length of the fisheye lens is f, there is a relationship of r = f · θ between the position r of the point of the subject and the incident angle θ.

As described above, the fisheye lens has a viewing angle of 18
It is 0 degrees, and an omnidirectional image of the subject can be formed on the imaging surface. In particular, a fisheye lens in which a relational expression indicating a lens characteristic satisfying a position r of a point of a subject and an incident angle θ satisfies r = f · θ is called an “fθ lens” and is widely used. Instead of the fθ lens, a fish-eye lens called “fsin θ lens” in which a relational expression indicating lens characteristics is r = f · sin θ may be used as the taking lens 22. In general, the position r monotonically increases with respect to the incident angle θ, and a point in the omnidirectional region of the subject may be a fisheye lens that is imaged at a finite position r, and the characteristics of the fθ lens and fsin θ lens are not necessarily required. The fisheye lens need not be shown. The fisheye lens is designed to have a viewing angle of 180 degrees, and is most preferable for capturing an object with a wide field of view. However, the taking lens 22 is not necessarily a fisheye lens as long as it is an optical lens having a sufficiently wide viewing angle and capable of imaging an object over a wide field of view, and a wide-angle lens having a wide viewing angle may be used.

FIG. 7 is an explanatory diagram of parallax when an object is viewed using two fisheye lenses. The fisheye lenses 324 and 326 are placed so that the positions of the points L and R are the viewpoint positions. Point L and point R are separated by a distance 2d. Lines 328 and 330 are fisheye lens 324, respectively.
And 326 are optical axes. The midpoint between points L and R is point O. The depth value of the subject is defined by the distance from the point O. The depth value of the point A of the subject is the length Z of the line segment OA. The angle formed by the line segment OA and the perpendicular 329 drawn from the point O is defined as θ. Angle θ is point A when point A is viewed from point O
(Half angle of view).

The line segment LA is the optical axis 328 of the fisheye lens 324.
, The incident angle of the point A to the fisheye lens 324 is θ1, and the line segment RA is the optical axis 330 of the fisheye lens 326.
, That is, the angle of incidence of the point A on the fisheye lens 326 is θ2. The incident angle θ1 is the azimuth of point A when viewing point A from viewpoint L, and the incident angle θ2 is the azimuth of point A when viewing point A from viewpoint R. Since the viewpoints are different, a difference θ1−θ2 occurs in the azimuth of the point A. This is called a parallax angle. Line segment L
Assuming that the angle between A and the line segment RA is θA, θA = θ1−θ
For convenience, θA is defined as point A, and point A is defined as a different viewpoint L,
It may be considered as a parallax angle when viewed from R.

When the fisheye lenses 322 and 324 are fθ lenses, the relationship of r = f · θ is established between the image height r and the incident angle θ, so that the image height when the point A is viewed from the viewpoints L and R is obtained. Between the difference between r _L and r _{R and} the difference between the incident angles θ1 and θ2,
Next proportional relationship

The following holds: r _L −r _R = f · (θ1−θ2)

Therefore, the fisheye lenses 324 and 326
In the parallax image captured by the above, the image height difference r _L −r _R of the point A
Is detected, if the focal length f of the fisheye lens is known, the difference θ1−θ2 between the incident angles can be calculated, and the parallax angle θA can be calculated.

Next, the parallax angle θA, the depth value Z, and the azimuth angle θ
A relational expression that satisfies is derived. Angle LAO is θL, angle RA
Let O be θR. Focusing on triangular LAH and triangular RAH,

Tan (θ + θL) = (Z · sin θ +
d) / (Z · cos θ) tan (θ−θR) = (Z · sin θ−d) / (Z · c
os θ) holds.

Therefore, the parallax angle θA can be written as θA = θL + θR = tan ⁻¹ (Z · sin θ + d) / (Z · cos θ) −tan ⁻¹ (Z · sin θ−d) / (Z · cos θ) (1) .

FIG. 8 is a graph of equation (1) for the parallax angle θA. The value of the parallax angle θA was graphed by changing the value of the azimuth angle θ while fixing the inter-viewpoint distance 2d and the depth value Z to certain values. When the azimuth angle θ = 0, the parallax angle θA is the maximum. When the azimuth θ = π / 2, the parallax angle θA becomes 0.

As is clear from the graph of FIG. 8, the parallax angle becomes very small for an object located in a region close to the horizontal direction of the array of fisheye lenses, so that the displacement of the image position on the parallax image is small. Extremely small. FIG. 9 is a diagram illustrating the difference in the magnitude of the parallax angle of the subject. For points of the subject in an area where the azimuth angle θ is close to 0, such as points A and B, the parallax angles θA and θB are sufficiently large, and it is easy to detect the displacement of the image position on the parallax image. is there. Since the parallax angle of the point B is larger than that of the point A, the positional deviation is larger.
This means that the closer the subject is from the point O, the greater the displacement of the image on the parallax image. For points of the subject such as points C and D in which the azimuth angle θ is close to π / 2, the parallax angles θC and θD are small, and it is difficult to detect the displacement of the image position on the parallax image. Become. The parallax angle of the point D is smaller than that of the point C, and the position of the image on the parallax image is smaller for a subject farther from the point O.

As described above, in the horizontal direction of the arrangement of the fisheye lenses, there is an area where the positional deviation is small in the parallax image. This area is called a “blind spot” area. In the blind spot area, it is difficult to obtain the depth value of the subject due to the positional shift. Therefore, the depth value is obtained using another method. In the horizontal direction of the fisheye lens, the fact that the size of a partial area where a subject is present changes in a parallax image is used.

FIG. 10 is a diagram for explaining a difference in size when the subject area is viewed from different viewpoints. The specific area 332 of the subject is at the position shown in the figure, and a certain point on the specific area 332 is C. The azimuth angle of the point C viewed from the point O is defined as θ. The fisheye lens 324 is located on the imaging surface 334 in the specific area 33.
Let _{SL be} the size of the image formed by imaging No. 2. Fisheye lens 32
6 the size of the image obtained by imaging the particular region 332 to the imaging surface 336 and _{S R.} Apparent angles when the specific region 332 is viewed from the viewpoint L and the viewpoint R are θ _L and θ _R , respectively.
Further, the distance from point O to the point C is Z, the point L, respectively the distance R to the point C _Z L, and _{Z R.}

A specific area 332 is created at different viewpoint positions L and R.
When viewed, the apparent angle θ_L, Θ _R, The size of the statue
S_L, S_RMakes a difference. From the characteristics of fθ lens

S_L= F · θ_L, S_R= F · θ_R  Holds. Therefore

S _L / S _R = θ _L / θ _R (2), and the image size ratio S _L / S _R is equal to the apparent angle ratio θ _L / θ _R.

On the other hand, there is an inverse relationship between the distance to the subject and the apparent angle, and the closer the distance, the larger the apparent angle. Therefore, the distance ratio Z _L / Z
_R is equal to the reciprocal of the apparent angle ratio θ _L / θ _R. Ie

Z _L / Z _R = θ _R / θ _L (3) Therefore, from (2) and (3),

S _L / S _R = Z _R / Z _L (4), and the image size ratio S _L / S _R is the distance ratio Z _L
/ Z _{R is} the reciprocal of _R. Thus, the distance ratio can be calculated by detecting the image size ratio in the parallax image.

Next, a relational expression that satisfies the distance ratio Z _L / Z _R , the subject depth value Z, and the viewpoint distance 2d is derived.
FIG. 11 is a diagram illustrating a difference in distance to a point of a subject when viewed from different viewpoints. Point C is one point in the specific area 332 of the subject in FIG. Description will be made using the same symbols as in FIG. Paying attention to the triangle CRH and the triangle CLH,

Z _R ² = (Z · cos θ) ² + (Z · sin θ−d) ² Z _L ² = (Z · cos θ) ² + (Z · sin θ + d) ² holds. From this, the distance ratio Z _L / Z _R is

(Z _L / Z _R ) ² = {(Z · cos θ) ² + (Z · sin θ + d) ² } / {(Z · cos θ) ² + (Z · sin θ−d) ² } (5) . Therefore, when the image size ratio is detected by (4) and (5), the azimuth angle θ of the subject and the inter-viewpoint distance 2
From d, the depth value of the subject can be calculated.

FIG. 12 is a graph of the image size ratio S _L / S _R. The size ratio S _L determined by equations (4) and (5)
/ _S in the formula of _R, d is 50 mm, Z
Is 100, 200, 400, or 800 millimeters, the value of the azimuth angle θ is changed and the magnitude ratio S
And the change in the _L / _{S R} in the graph. As is clear from the figure, when the azimuth angle θ = 0, the size ratio when viewed with the left and right fisheye lenses is 1, and the size of the image in the parallax image is the same. When the azimuth angle θ = π / 2, the size ratio becomes smaller than 1 and a difference occurs in the size of the image in the parallax image.

As described above, in the horizontal direction of the arrangement of the fisheye lenses, the size of the image in the parallax image is greatly different.
The depth value can be easily calculated from the size ratio. On the other hand, in a region where the azimuth angle θ is close to 0, there is almost no difference in the size of the image in the parallax image, so that it is difficult to obtain the depth value from the size ratio.

As described above, regarding the displacement of the image position and the ratio of the image size, in the region where the azimuth angle θ is close to 0, it is easy to detect the positional deviation, but it is difficult to detect the size ratio. It is. On the other hand, in the region where the azimuth angle θ is close to π / 2,
Although it is easy to detect the size ratio, it is difficult to detect the displacement.

Next, the relationship between the displacement and the size ratio will be examined. FIG. 13 is a diagram illustrating the relationship between the displacement and the size ratio on a celestial sphere. The celestial sphere is at infinity in the field of view, and only the upper hemisphere is shown in the figure. The viewpoint L of the fisheye lenses 324 and 326,
A plane passing through the midpoint O of R and perpendicular to the straight line LR is a semicircle 350
And The points where the straight line connecting the viewpoints L and R intersect with the celestial sphere are points 352 and 354.

In the celestial sphere, the azimuth θ when the point of the subject is viewed from the midpoint O is the angle formed by a straight line connecting the point of the subject and the midpoint O with the semicircle 350. In a region where the azimuth angle θ is 0 (zero), the positional deviation appears most strongly, and the size ratio completely disappears. This area is a semicircle 350.

On the other hand, the azimuth θ when viewed from the midpoint O is π
/ 2, that is, a region at a right angle, the size ratio appears most strongly, and the positional displacement completely disappears. This area is the point 35
2 and a point on the line segment connecting the point 354 and the viewpoint L.

The values of the displacement and the size ratio are determined by the viewpoints L and R.
, Ie, symmetric about the x-axis. Half circle 3
Considering the cross section when the celestial sphere is cut on a plane parallel to 50,
This cross section is a semicircle, and points on the circumference of the semicircle have the same positional deviation and size ratio values. Therefore, the values of the displacement and the size ratio can be discussed in a section obtained by cutting the celestial sphere at an arbitrary plane passing through the x-axis. As an example of a plane passing through the x-axis, an xy plane, which is a ground plane of a celestial sphere, is considered.

FIG. 14 shows the relationship between the displacement and the size ratio by x.
It is the figure demonstrated in the y-plane. Consider a cone formed by rotating the triangle OAH around the x-axis, assuming that the point A of the subject is at the azimuth θ. Thus, when a certain azimuth angle θ is determined, the vertex of the cone is set to the midpoint O, and the cone angle is π /
A cone that is 2-θ is defined.

When the azimuth θ is close to π / 2, the point in the cone is in a region near the x-axis. For the points within this cone, the size ratio appears strongly, and the misregistration almost disappears. On the other hand, when the azimuth angle θ is close to 0, the cone angle is π / 2, and the cone includes almost all points except points near the y-axis. For points outside the cone near the y-axis,
Displacement appears strongly, and the size ratio almost disappears.

As described above, the positional deviation and the magnitude of the size ratio are in a mutually complementary relationship depending on the orientation of the subject. Therefore, depending on the azimuth of the subject, the depth value is calculated using any of the displacement and the size ratio, or the depth value obtained from the displacement and the depth value obtained from the size ratio are combined with appropriate weighting. , The depth value can be calculated.

For example, as the azimuth angle θ of the region of the subject is closer to 0, the ratio of considering the positional shift is increased, and as the azimuth angle is closer to π / 2, the ratio of considering the size ratio is increased. do. Also, a predetermined value of the azimuth θM,
For θN, the depth value may be determined by the size ratio for a region having an azimuth angle θ larger than θM, and for the region having an azimuth angle θ smaller than θN, the depth value may be determined by displacement. For a region having an azimuth angle θ smaller than θM and larger than θN, a depth value may be obtained by appropriately determining a ratio in which the displacement and the size ratio are considered.

As another method, when calculating the depth value of a specific area of a subject shown in a parallax image, both the positional deviation and the size ratio of the specific area are detected, and the higher detection accuracy is determined. It may be used for depth value calculation. In particular, when a subject having an overlap is photographed, it may change depending on not only the azimuth but also the depth distribution of the subject, which of the positional deviation and the size ratio has higher detection accuracy. Therefore, instead of determining the ratio of the consideration of the positional deviation and the size ratio only for the azimuth of the subject, for each region of the subject in the parallax image,
It is more preferable to determine which of the displacement and the size ratio is used, or to determine a ratio in which the displacement and the size ratio are considered.

Equation (1) for the parallax angle and the size ratio S _L / S
_When the expressions (4) and (5) of _R are analyzed in detail, the region of the subject where the ratio of the detection accuracy of the displacement and the size ratio is constant is:
It can be seen that it is not determined only by the azimuth of the subject, but also depends on the depth value of the subject. Therefore, more precisely, even if the direction of the subject is the same, if the distance to the subject is different, it is necessary to change the ratio of consideration of the positional deviation and the size ratio.
For this reason, when calculating the depth value of the specific area of the subject captured in the parallax image, the position shift and the size ratio of the specific area are detected, and the one with higher detection accuracy of the position shift and the size ratio is prioritized. To calculate a temporary depth value of the specific area. After that, it is more preferable to determine the ratio of consideration of the positional deviation and the size ratio based on the azimuth of the specific area and the provisional depth value, and to obtain the depth value again.

As described above, the calculation method for obtaining the depth value of the subject by using the positional deviation and the size ratio may be modified in various ways. In any case, the positional deviation and the size ratio are combined to obtain the subject. There is an essential feature in obtaining the depth value for each area.

FIG. 15 is a flowchart of the processing for calculating the depth value of the subject. The parallax image captured by the imaging unit 20 is input (S100), and a specific region of the subject shown in the parallax image is selected (S102). The specific region may be selected by appropriately dividing the region of the parallax image and sequentially selecting the automatically divided regions, or by selecting the region of the subject specified by the user using the operation unit 110. Good. The position shift detection unit 304 detects a position shift of the image of the specific region of the subject (S104). Size ratio detector 3
06 detects the size ratio of the image of the specific area of the subject (S106). The depth calculation unit 308 calculates the depth value of the specific area using the positional deviation and the size ratio (S10).
8).

FIG. 16 is a flowchart of the depth value calculation processing S108. Based on the azimuth angle θ of the specific area of the subject viewed from the midpoint between the viewpoints, a ratio γ that considers the positional deviation and the size ratio is determined (S200). As the azimuth angle θ is closer to 0, the ratio of considering the positional deviation is higher, and as the azimuth angle θ is closer to π / 2, the ratio of considering the size ratio is higher.
Based on the displacement, the depth value Z1 of the specific area is calculated using Expression (1) (S202). On the basis of the size ratio, the depth value Z2 of the specific area is calculated using Expressions (4) and (5).
Is calculated (S204). The depth values Z1 and Z2 are combined based on the ratio γ in which the position shift and the size ratio are considered to calculate the depth value Z of the specific area. As a method of calculating the depth value Z by combining the depth values Z1 and Z2, for example, a weighted average of Z1 and Z2 is calculated at a ratio γ to be Z.

Several modifications are conceivable for the depth value calculation processing S108. FIG. 17 shows a depth value calculation process S1.
It is a flowchart of the modification of 08. By comparing the positional deviation of the specific region of the subject with the detection accuracy of the size ratio, a ratio γ in which the positional deviation and the size ratio are considered is determined (S201).
Subsequent processes in S202, S204, and S206 are shown in FIG.
Same as 6. In this modification, the ratio γ is determined not to determine the ratio γ based on the azimuth angle θ of the specific region of the subject, but to increase the degree of consideration of the one with higher detection accuracy of the positional deviation and the size ratio of the specific region. .

FIG. 18 is a flowchart of another modification of the depth value calculation processing S108. The positional deviation of the specific region of the subject and the detection accuracy of the size ratio are compared, and the provisional depth value Z3 of the specific region is calculated from the value with the higher accuracy (S210). A ratio γ in which the positional deviation and the size ratio are considered based on the azimuth angle θ of the specific area and the temporary depth value Z3.
Is determined (S212). S202 and S20 thereafter
4. The processing in S206 is the same as that in FIG. This modification is different in that the ratio γ is determined from the azimuth angle θ and the temporary depth value Z3.

FIG. 19 is a flowchart of another modification of the depth value calculation processing S108. If the azimuth θ of the specific area of the subject is larger than the predetermined value θM (S220),
The depth value Z of the specific area is calculated based on the size ratio (S
222) If not, proceed to S224. If the azimuth angle θ of the specific area is smaller than θN (S224), the depth value Z of the specific area is calculated based on the displacement (S22).
6) If not, proceed to S228. In S228, the depth value Z of the specific area is calculated by determining the ratio in which the positional deviation and the size ratio are considered. The process of S228 is performed as shown in FIG.
Any of the processes 6, 17, and 18 can be used.

The accuracy and processing cost of detecting the displacement and the size ratio are determined by the resolution of the CCD 30 of the imaging unit 20 and the main CPU 62 of the processing unit 60 for performing image processing, the non-volatile memory 66, the main memory 68, the main bus 82, and the like. There are aspects that depend on the hardware performance, such as the processing performance, and aspects that depend on the functional configuration of the system, such as the algorithm performance of the image processing method for detecting the position and size. For this reason, the rate of consideration of the positional deviation and the size ratio cannot be determined simply by the rate of the detection accuracy theoretically derived from Equations (1), (4), and (5).

Further, the ease of detection differs between the positional deviation and the size ratio in the parallax image. In the case of misalignment, a specific area having a certain size of the subject is found in a plurality of parallax images, and a certain point in the specific area, for example, a center or a center of gravity, or a characteristic point, is detected between the parallax images. Detect the positional deviation at. On the other hand, in the case of the size ratio, it is necessary to find a specific region having a certain size of the subject among a plurality of parallax images and detect a change in the size of the specific region. The cost in image processing may be higher than in the case of detection. Also, in the calculation of the depth value after the positional deviation and the size ratio have been detected, the calculation using the size ratio generally has a higher calculation cost. Therefore, it is preferable to determine not only the detection accuracy but also the processing cost when determining the ratio of consideration of the position shift and the size ratio.

As described above, the calculation method for obtaining the depth value of the subject using the displacement and the size ratio includes the azimuth angle of the subject area, the estimated value of the subject depth, and the detection of the displacement and the size ratio. Many variations are possible based on accuracy, processing cost, hardware performance, system functional configuration, and the like. In any case, according to the image capturing apparatus of the present embodiment, when the accuracy of any one of the positional deviation and the size ratio is low, or when any one of the processing costs is high, it is complemented by the other. In addition, depth information of an omnidirectional area of a subject can be efficiently calculated with high accuracy.

As described above, according to the image pickup apparatus of the present embodiment, a parallax image of a subject is photographed using two lenses having a wide viewing angle, and the ratio between the displacement of the subject image and the size of the subject is calculated. Is detected, and the depth value of the subject can be obtained using the positional deviation and the size ratio. Since the size ratio can be used in an area where the detection accuracy of the position ratio is reduced, and the position deviation can be used in an area where the detection accuracy of the size ratio is reduced, the depth information of the subject can be calculated with high accuracy over a wide field of view. .

The image pickup device of the present embodiment can also be used as a monitoring camera. Conventional surveillance cameras had to be driven to obtain omnidirectional images. When the image capturing apparatus of the present embodiment is used as a surveillance camera, it is possible to photograph an omnidirectional subject and calculate depth information without driving the camera. Since the main subject such as a person can be easily extracted from the obtained depth information, it can be installed in a bank or a retail store and used for crime prevention.

(Embodiment 2) A second embodiment of the present invention will be described. FIG. 20 is a configuration diagram of a lab system 200 that develops and edits a photographic image as an example of an image processing apparatus. The lab system 200 of the present embodiment includes an input unit 210, a processing unit 220, a recording unit 240, and an output unit 2
50.

The input section 210 inputs image data of a subject. As image data, a parallax image when the subject is viewed from different viewpoints is input. When inputting a digital image of an object photographed by a digital camera or the like, the input unit 2
A reading device for reading image data from a detachable recording medium such as a semiconductor memory card is used for 10. Also, floppy disk, MO, CD-ROM
When reading image data from a device such as a floppy drive, MO drive, or CD,
A drive or the like may be used.

The processing section 220 stores the parallax image input by the input section 210 and calculates depth information of the subject. The processing unit 220 outputs the calculated depth information to the recording unit 240 together with the parallax image. The processing unit 220 processes the image of the subject based on the calculated depth information,
40 and the output unit 250.

The recording section 240 records the depth information or image data output from the processing section 220 on a removable recording medium. Writable CD-RO as recording medium
Optical recording media such as M and DVD, magneto-optical recording media such as MO, and magnetic recording media such as floppy disks are used. As the recording unit 240, a CD-R drive, a DVD drive, an MO drive, a floppy drive, or the like is used. The recording unit 240 may record depth information or image data in a semiconductor memory such as a flash memory or a memory card.

The output section 250 outputs the processed image data of the subject output by the processing section 220 as an image.
For example, when displaying an image on a screen, a monitor that displays the image is used as the output unit 250. For example, when printing an image, a printer such as a digital printer or a laser printer is used as the output unit 250.

FIG. 21 is a functional block diagram of the processing section 220. The processing unit 220 includes a parallax image storage unit 302, a displacement detection unit 304, a size ratio detection unit 306, a depth calculation unit 308, and an image conversion unit 312.

The parallax image storage unit 302 stores data of a plurality of parallax images of the subject input by the input unit 210 in a semiconductor memory such as a RAM or a magnetic recording medium such as a hard disk. The displacement detection unit 304 stores the parallax image storage unit 3
In a plurality of parallax images stored in No. 02, an amount by which the position of the image of the specific region of the subject shifts due to parallax is detected. The size ratio detection unit 306 detects, in the parallax image stored in the parallax image storage unit 302, a ratio at which the size of the image of the specific region of the subject changes due to the parallax. Depth calculator 3
08 calculates the depth value of the specific area of the subject using the position shift detected by the position shift detection unit 304 and the size ratio detected by the size ratio detection unit 306.

[0103] Position shift detector 304, size ratio detector 3
06 and the process in which the depth calculation unit 308 calculates the depth information of the subject for a partial region or the entire region of the subject captured in the parallax image is the same as in the first embodiment. Is omitted.

The image conversion unit 312 includes a depth calculation unit 308
The image of the subject is processed based on the depth information of the subject calculated by. The image conversion unit 312 outputs the depth information of the subject, the parallax image, or the processed image to the storage unit 240 and the output unit 250.

The image conversion unit 312 may transform the omnidirectional image of the subject photographed by the fisheye lens into a perspective projection image based on the depth information of the subject. FIG. 22 is a diagram illustrating conversion from an omnidirectional image to a perspective projection image. Since the depth value has been calculated for a point or area on the omnidirectional image 356 of the subject captured by the fisheye lens, the point or area can be mapped on the perspective projection image 358 by performing coordinate transformation. . Perspective projection image 3
Reference numeral 58 denotes an image obtained by photographing a subject with a normal standard lens.

The image conversion unit 312 generates an orthographic projection image such as a front view, a side view, and a top view of the subject from the omnidirectional image of the subject taken by the fisheye lens based on the depth information of the subject. Is also good. FIG. 23 is a schematic diagram of a floor plan of a room taken by a fisheye lens. The fisheye lens shoots the room in all directions from near the ceiling. FIG. 24 is a plan view of a room obtained by performing coordinate conversion on an omnidirectional image. By using the omnidirectional depth information of the room, the omnidirectional image can be converted into such a plan view. FIG. 25 is a side view of a room obtained by performing coordinate conversion on an omnidirectional image. Thus, the image conversion unit 312
Generates an orthographic image of a front view, a side view, a top view, and the like of a subject from an omnidirectional image of the subject based on omnidirectional depth information of the subject, which can be useful for creating a design drawing and the like. Such image conversion processing is widely used in the construction and city planning scenes.

According to the image processing apparatus of the present embodiment, it is possible to input a parallax image of a subject photographed with a lens having a wide viewing angle and calculate depth information of the subject over a wide field of view. Further, image processing can be performed based on the calculated depth information to create drawing data such as CAD. Image data having omnidirectional depth information can be used for CG (computer graphics) and simulation.

(Embodiment 3) Next, a third embodiment of the present invention will be described. FIG. 26 is a configuration diagram of the image processing apparatus. The basic configuration and operation of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus according to the second embodiment. The present embodiment differs from the second embodiment in that an electronic computer such as a personal computer or a workstation is used as the processing unit 220 of the image processing apparatus.

The hardware configuration of the processing section 220 according to the present embodiment will be described with reference to FIG. CPU 230
Operate based on programs stored in the ROM 232 and the RAM 234. Data is input by a user via an input device 231 such as a keyboard and a mouse. The hard disk 233 stores data such as images and the CPU 2
30 is stored. CD-ROM
The drive 235 reads data or a program from the CD-ROM 290,
33 and the CPU 230.

The functional configuration of the program executed by the CPU 230 is the same as that of the processing unit 220 of the image processing apparatus according to the second embodiment.
Parallax image storage module is the same as the functional configuration of
It has a displacement detection module, a size ratio detection module, a depth calculation module, and an image conversion module.

The processing performed by the parallax image storage module, the displacement detection module, the size ratio detection module, the depth calculation module, and the image conversion module by the CPU 230 is the processing unit of the image processing apparatus of the second embodiment. The functions and operations of the parallax image storage unit 302, the displacement detection unit 304, the size ratio detection unit 306, the depth calculation unit 308, and the image conversion unit 312 in 220 are the same as those in FIG. These programs are
It is stored on a recording medium such as a CD-ROM 290 and provided to the user. CD-ROM 29 as an example of a recording medium
0 can store some or all of the functions of the operation of the image processing apparatus described in the present application.

The above program is directly stored in the RA from the recording medium.
It may be read by M234 and executed by the CPU 230. Alternatively, the above program is installed on the hard disk 233 from a recording medium,
And may be executed by the CPU 230.

As a recording medium, in addition to the CD-ROM 290, a hard disk, a memory such as a ROM or a RAM,
An optical recording medium such as a DVD or PD, a magnetic recording medium such as a floppy disk or a mini disk (MD), a magneto-optical recording medium such as an MO, a tape-shaped recording medium, and a nonvolatile semiconductor memory card can be used.

The above program may be stored on a single recording medium, or may be divided and stored on a plurality of recording media. Further, the program may be compressed and stored in a recording medium. The compressed program may be decompressed, read to another recording medium such as the RAM 234, and executed. Furthermore, the compressed program is a CPU
After being decompressed by 230 and installed on the hard disk 233 or the like, it may be read to another recording medium such as the RAM 234 and executed.

Furthermore, a CD-R as an example of a recording medium
The OM 290 may store the program provided by the host computer via the communication network. The above program stored in the recording medium is
It may be stored in the hard disk of the host computer, transmitted from the host computer to the computer via a communication network, read out to another recording medium such as the RAM 234, and executed.

The recording medium storing the above program is
It is apparent that the invention is used only for manufacturing the image processing apparatus of the present application, and that the manufacture and sale of such a recording medium as a business constitute infringement of a patent right based on the present application.

(Embodiment 4) Next, a fourth embodiment of the present invention will be described. An example of the image capturing device according to the present embodiment includes:
Electronic devices such as a notebook computer with a built-in camera and a portable electronic terminal with a built-in camera. In these cases, the computer section of the notebook computer or portable electronic terminal mainly functions as the processing section 220 shown in FIG. The image capturing apparatus according to the present embodiment is obtained by replacing the processing unit 60 of the image capturing apparatus according to the first embodiment with the hardware configuration of the processing unit 220 illustrated in FIG. The basic configuration and operation of the image pickup apparatus of the present embodiment
This is the same as the image pickup device of the embodiment.

The hardware configuration of the processing unit 220 according to the present embodiment is the same as the hardware configuration of the processing unit 220 according to the third embodiment, and a description thereof will be omitted. The functional configuration of the program executed by the CPU 230 is the same as the functional configuration of the processing unit 60 of the image capturing apparatus according to the first embodiment, and includes a parallax image storage module, a displacement detection module, a size ratio detection module, It has a depth calculation module and a recording module.

The processes that the parallax image storage module, the displacement detection module, the size ratio detection module, the depth calculation module, and the image conversion module cause the CPU 230 to perform are respectively the processing units of the image pickup apparatus of the first embodiment. The functions and operations of the parallax image storage unit 302, the displacement detection unit 304, the size ratio detection unit 306, the depth calculation unit 308, and the recording unit 310 in 220 are the same as those in FIG. These programs are on CD
-Stored in a recording medium such as the ROM 290 and provided to the user. The CD-ROM 290 as an example of the recording medium can store some or all of the functions of the operation of the image capturing apparatus described in the present application.

The recording medium storing the above program is
It is clear that the present invention is used only for manufacturing the image pickup apparatus of the present application, and that such production and sale of a recording medium as a business constitutes infringement of a patent right based on the present application.

Next, experimental results of the image pickup apparatus of the present embodiment will be shown. 27, 28, 29, 30, and 31 show experimental results obtained by measuring the size ratio of a subject using the image pickup device of the present embodiment. In the experiment, a digital camera equipped with a fish-eye lens having a lens characteristic of fθ was used, and by moving the digital camera, subjects photographed from two different left and right viewpoints were sequentially photographed. The subject was instructed not to move during shooting. FIG. 27 is a photograph of a halftone image in which a parallax image of a subject taken from a left viewpoint is displayed on a display. FIG. 28 is a photograph of a halftone image in which a parallax image of a subject taken from a right viewpoint is displayed on a display. The focal length f of the left and right fisheye lenses is 8 mm, and the distance 2d between the viewpoints is about 20 cm. The captured parallax image is an omnidirectional image.

FIG. 29 is a photograph of a halftone image in which an image obtained by enlarging the vicinity of the center of an omnidirectional image of a subject is displayed on a display. In the region near the center, there is almost no difference in size between the left and right parallax images, and the size ratio becomes 1.

FIG. 30 is a photograph of a halftone image in which an image obtained by enlarging the periphery of an omnidirectional image of a subject is displayed on a display. It can be seen that in the peripheral area of the omnidirectional image in the moving direction of the viewpoint, the size of the human image is significantly different between the left and right parallax images.

FIG. 31 is a photograph of a halftone image in which an image obtained by enlarging the vicinity of an omnidirectional image of a subject is displayed on a display. Similarly, the difference in the size of the human image can be detected, but the difference in the size is smaller than that of the human image in FIG. This is because the person shown in FIG. 31 is located farther than the person in FIG. As described above, in the area in the moving direction of the viewpoint, the depth information of the subject can be obtained from the difference in the size ratio.

As described above, according to the present experiment, a favorable result was obtained in which it was possible to detect the size ratio, which was conventionally considered difficult, by comparing the parallax images captured by the CCD. As a result, it was confirmed that depth value measurement in a “blind spot” region, which was difficult in the past, was possible.

As described above, according to the image pickup apparatus and the image processing apparatus of the present invention, the ratio between the positional shift and the size of the image of the subject is detected from the parallax image of the subject, and the positional shift and the size are detected. The depth value of the subject can be obtained using the ratio. Since the size ratio can be used in an area where the detection accuracy of the position ratio is reduced, and the position deviation can be used in an area where the detection accuracy of the size ratio is reduced, the depth information of the subject can be calculated with high accuracy over a wide field of view. .

As described above, the present invention has been described using the embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is apparent to those skilled in the art that various changes or improvements can be added to the above embodiment. It is apparent from the description of the appended claims that embodiments with such changes or improvements can be included in the technical scope of the present invention.

[0128]

As is apparent from the above description, according to the present invention, information on the depth of a subject can be obtained with high accuracy over a wide field of view.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a digital camera 10 as an example of an image capturing apparatus.

FIG. 2 shows a photographing lens 22 and a CC of an imaging unit 20;
It is an example of the configuration of D30.

FIG. 3 shows a photographing lens 22 and a CC of an imaging unit 20;
It is another example of the structure of D30.

FIG. 4 shows a photographing lens 22 and a CC of the image pickup unit 20.
It is another example of the structure of D30.

FIG. 5 is a functional block diagram of a processing unit 60.

6 is a diagram illustrating a relationship between an incident angle θ of a point of a subject formed by the fisheye lens 320 and a position of an image of the point of the subject in the omnidirectional image 322. FIG.

FIG. 7 is an explanatory diagram of parallax when a subject is viewed using two fisheye lenses.

FIG. 8 is a graph of Expression (1) of the parallax angle θA.

FIG. 9 is a diagram illustrating a difference in the magnitude of a parallax angle of a subject.

FIG. 10 is a diagram illustrating a difference in size when a subject area is viewed from different viewpoints.

FIG. 11 is a diagram illustrating a difference in distance to a point of a subject when viewed from different viewpoints.

FIG. 12 is a graph of an image size ratio S _L / S _R.

FIG. 13 is a diagram illustrating the relationship between the positional deviation and the size ratio on a celestial sphere.

FIG. 14 is a diagram illustrating the relationship between the displacement and the size ratio in the xy plane.

FIG. 15 is a flowchart of processing for calculating a depth value of a subject.

FIG. 16 is a flowchart of a depth value calculation process S108.

FIG. 17 is a flowchart of a modified example of the depth value calculation processing S108.

FIG. 18 is a flowchart of another modified example of the depth value calculation processing S108.

FIG. 19 is a flowchart of another modified example of the depth value calculation processing S108.

FIG. 20 is a configuration diagram of a lab system 200 that develops and edits a photographic image, as an example of an image processing apparatus.

FIG. 21 is a functional configuration diagram of a processing unit 220.

FIG. 22 is a diagram illustrating conversion from an omnidirectional image to a perspective projection image.

FIG. 23 is a schematic diagram of a floor plan of a room taken by a fisheye lens.

FIG. 24 is a plan view of a room obtained by performing coordinate transformation on an omnidirectional image.

FIG. 25 is an orthographic projection image of a room obtained by performing coordinate transformation on an omnidirectional image.

FIG. 26 is a configuration diagram of an image processing apparatus.

FIG. 27 is a photograph of a halftone image in which a parallax image of a subject taken from a left viewpoint is displayed on a display.

FIG. 28 is a photograph of a halftone image in which a parallax image of a subject captured from a right viewpoint is displayed on a display.

FIG. 29 is a photograph of a halftone image in which an image obtained by enlarging the vicinity of the center of an omnidirectional image of a subject is displayed on a display.

FIG. 30 is a photograph of a halftone image in which an image obtained by enlarging the periphery of an omnidirectional image of a subject is displayed on a display.

FIG. 31 is a photograph of a halftone image in which an image obtained by enlarging the vicinity of an omnidirectional image of a subject is displayed on a display.

[Explanation of symbols]

Reference Signs List 10 digital camera 20 imaging unit 40 imaging control unit 60 processing unit 100 display unit 110 operation unit 200 image processing device 210 input unit 220 processing unit 240 recording unit 250 output unit 290 recording medium 302 parallax image storage unit 304 displacement detection unit 306 size Depth calculation unit 308 Depth calculation unit 310 Recording unit 312 Image conversion unit 324, 326 Fisheye lens

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2F065 AA06 FF05 FF15 GG08 JJ03 JJ26 KK01 LL06 LL10 LL22 LL30 PP22 QQ03 QQ24 QQ26 QQ33 QQ38 SS02 SS13 UU05 UU07 2F112 AC06 BA03 CA02 CA12 CA20 DA05 FA22 A02A03 AB AC69 5C061 AA29 AB02 AB04 AB06 AB08 AB21 AB24

Claims (19)

[Claims] 1. An image capturing apparatus for acquiring information on the depth of a subject, wherein the capturing unit captures a parallax image of the subject obtained when the subject is viewed from two different viewpoint positions; A size ratio detection unit that detects a size ratio of the image of the specific region of the subject in, and a depth calculation unit that calculates a depth value of the specific region of the subject based on the size ratio. An image pickup apparatus characterized by the above-mentioned. 2. The image processing apparatus according to claim 1, further comprising: a position shift detecting unit configured to detect a position shift of an image of the specific region of the subject in the parallax image, wherein the depth calculating unit determines the position shift based on the position shift and the size ratio. The apparatus according to claim 1, wherein a depth value of the specific area of the subject is calculated. 3. The depth calculation unit calculates the depth value of the specific region of the subject by changing a ratio in which the position shift and the size ratio are considered for each of the specific regions of the subject. The image pickup device according to claim 2, wherein: 4. The depth calculation unit changes a ratio in which the position shift and the size ratio are considered according to an azimuth when the specific region of the subject is viewed from a midpoint of the two viewpoint positions, The apparatus according to claim 3, wherein the depth value of the specific area of the subject is calculated. 5. The depth calculating section increases a ratio of considering the size ratio as the specific area of the subject is closer to a direction of a straight line connecting the two viewpoint positions, and increases the specific area of the subject. However, the closer to the plane perpendicular to the straight line connecting the two viewpoint positions, passing through the midpoint of the two viewpoint positions, the greater the rate of considering the position shift, and the greater the depth value of the specific area of the subject. The image pickup apparatus according to claim 4, wherein is calculated. 6. The imaging unit has an optical lens having a wide viewing angle, and further includes a driving unit that moves the optical lens, wherein the imaging unit determines a position where the driving unit has moved the optical lens. The image capturing apparatus according to claim 1, wherein the parallax image of the subject is captured as the viewpoint position. 7. The apparatus according to claim 1, wherein the imaging unit has two optical lenses having different viewing positions and having a wide viewing angle, and capturing the parallax image of the subject with the two optical lenses. The image pickup apparatus according to any one of the preceding claims. 8. The apparatus according to claim 6, wherein the optical lens is a fisheye lens, and the depth calculation unit calculates the depth value for an omnidirectional area of the subject imaged by the imaging unit with the fisheye lens. Or the image pickup device according to 7. 9. The image pickup apparatus according to claim 6, wherein the image pickup unit has a solid-state image pickup device, and the parallax image of the subject is picked up by the solid-state image pickup device by the optical lens. apparatus. 10. An image processing apparatus for acquiring information on a depth of a subject, comprising: an input unit configured to input a plurality of parallax images of the subject obtained when the subject is viewed from a plurality of different viewpoint positions; A size ratio detection unit that detects a size ratio of the image of the specific region of the subject in a plurality of parallax images; and a depth calculation unit that calculates a depth value of the specific region of the subject based on the size ratio. An image processing apparatus comprising: 11. The image processing apparatus according to claim 1, further comprising: a position shift detecting unit configured to detect a position shift of an image of the specific area of the subject in the plurality of parallax images, wherein the depth calculating unit determines the position shift based on the position shift and the size ratio. And calculating a depth value of the specific area of the subject.
An image processing apparatus according to claim 1. 12. The depth calculation unit calculates the depth value of the specific region of the subject by changing a ratio of considering the position shift and the size ratio for each of the specific regions of the subject. The image processing apparatus according to claim 11, wherein: 13. The depth calculation unit changes a ratio in which the position shift and the size ratio are considered according to an azimuth when the specific region of the subject is viewed from a midpoint between the two viewpoint positions. The image processing apparatus according to claim 12, wherein the depth value of the specific area of the subject is calculated. 14. The method according to claim 1, wherein the specific area of the subject is the 2
As approaching the direction of the straight line connecting the two viewpoint positions, the ratio considering the size ratio is increased, and the specific region of the subject passes through the midpoint of the two viewpoint positions, and the straight line connects the two viewpoint positions. As you approach a plane perpendicular to
The image processing apparatus according to claim 13, wherein the depth value of the specific region of the subject is calculated by increasing a ratio of considering the position shift. 15. An image conversion for converting the image by performing coordinate conversion of an image of the subject input by the input unit based on the depth value of the specific area of the subject calculated by the depth calculation unit. The image processing apparatus according to claim 10, further comprising a unit. 16. When the image of the subject input by the input unit is an omnidirectional image of the subject captured by a fisheye lens, the image conversion unit perspectively projects the omnidirectional image by the coordinate transformation. The image processing apparatus according to claim 15, wherein the image processing apparatus converts the image into an image. 17. The image processing apparatus according to claim 15, wherein the image conversion unit generates an orthographic image of the subject by the coordinate conversion. 18. An image processing method for acquiring information on the depth of a subject, comprising: inputting a plurality of parallax images of the subject obtained when the subject is viewed from a plurality of different viewpoint positions; Detecting the ratio of the position shift and the size of the image of the specific region of the subject in the image, and changing the ratio considering the position shift and the size ratio for each of the specific regions of the subject, An image processing method comprising calculating a depth value of the specific area. 19. A recording medium storing a computer program for acquiring information on the depth of a subject, wherein the program is provided with a plurality of the subjects obtained when the subject is viewed from a plurality of different viewpoint positions. An input module for inputting a parallax image, a position shift detection module for detecting a position shift of an image of a specific region of the subject in the plurality of parallax images, and an image of the specific region of the subject in the plurality of parallax images. A size ratio detection module that detects a size ratio, and for each of the specific regions of the subject, changing the ratio of considering the displacement and the size ratio to change the depth value of the specific region of the subject. A recording medium comprising a depth calculation module for calculating the depth.