JP2018081534A - Image processing device, imaging apparatus, and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of generating stereoscopic image data effectively having depth feeling even when a photographing angle corresponding to the image data is a wide angle.SOLUTION: A digital camera 100 for generating stereoscopic image data corresponding to image data is provided. The digital camera 100 acquires image data, and depth information indicating a depth in a depth direction of a subject corresponding to the image data. The digital camera 100 sets a background plane for the stereoscopic image data in accordance with photographing information corresponding to the image data. The digital camera 100 generates stereoscopic image data by using the background plane that has been set and on the basis of the image data and the depth information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an imaging device, and a control method.

  3D printers, which are devices for modeling solids by laminating recording materials such as resin and gypsum powder, are beginning to become widespread. An apparatus or system has been proposed that uses a 3D printer to output a stereoscopic relief of a captured image, that is, a stereoscopic image, based on the captured image and distance information indicating the depth.

  As a method for outputting a stereoscopic image, in addition to a method using a 3D printer, various methods such as a method of modeling a solid by thermally deforming a thermally expansible sheet, embossing printing, laser engraving, etc. Can be considered. In any of the output methods, there is a restriction on the maximum height difference of the stereoscopic image to be output, and the more recording materials are stacked as the height difference is added, the cost increases. Patent Document 1 discloses a stereoscopic image output apparatus that performs stereoscopic output by emphasizing the positional relationship in the depth direction for each subject so that the boundary lines of a plurality of adjacent subjects are more conspicuous. Patent Document 2 discloses a stereoscopic image output apparatus that outputs a stereoscopic image so as to be within the upper limit value of the thickness.

JP 2008-213320 A Japanese Patent No. 4998317

  A case is assumed where shooting is performed using a lens capable of shooting with a wide angle of view, such as a fish-eye lens or a wide-angle lens with a short focal length. When an image shot using a wide-angle lens such as a fisheye lens is output by a conventional stereoscopic image output device, the shooting range is wide, so a wide range of subjects can be shot in the depth direction from a short distance to a long distance. Many. Therefore, a large number of recording materials are required to three-dimensionally represent all the areas in the image. In addition, in a stereoscopic image output device having an upper limit in thickness, even if a stereoscopic image is output, there are few regions to be stereoscopicized and a desired stereoscopic effect cannot be obtained. An object of the present invention is to provide an image processing apparatus that can generate stereoscopic image data with a sense of depth effectively even when the shooting angle of view corresponding to the image data is a wide angle.

  An image processing apparatus according to an embodiment of the present invention is an image processing apparatus that generates stereoscopic image data corresponding to image data, and indicates the depth in the depth direction of the image data and a subject corresponding to the image data. A first acquisition unit that acquires depth information; a setting unit that sets a background surface of the stereoscopic image data according to a shooting angle of view corresponding to the image data; and the image data using the background surface. And generating means for generating the stereoscopic image data based on the depth information.

  According to the image processing apparatus of the present invention, even when the shooting angle of view corresponding to the image data is a wide angle, it is possible to generate stereoscopic image data with a sense of depth effectively.

It is a figure which shows the structural example of the image processing apparatus of this embodiment. It is a figure explaining a picked-up image and depth information. It is a figure which shows the relationship between an image sensor size, a focal distance, and an imaging | photography angle of view. It is a figure explaining the characteristic of a lens. It is a figure explaining the conversion process from an imaging | photography angle of view to a background surface. It is a figure explaining the production | generation process of stereo image data. It is a flowchart explaining the production | generation process of stereo image data.

FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus according to the present embodiment.
The image processing apparatus of this embodiment is a digital camera 100 as an example of an imaging apparatus. The digital camera 100 is an imaging device having a function of generating stereoscopic image data corresponding to image data acquired by imaging. The digital camera 100 includes an internal bus 102 through an LCD 116. LCD is an abbreviation for Liquid Crystal Display. The internal bus 102 connects each block. The system control unit 101 controls each block connected by the internal bus 102.

  The FLASH memory controller 103 is controlled by the system control unit 101 to read / write nonvolatile data from / to the FLASH memory 104. The system control unit 101 operates based on a program stored in the FLASH memory 104 and stores non-volatile data such as image data obtained by imaging in the FLASH memory 104. The DRAM controller 105 reads and writes data to and from the DRAM 106 based on requests from each block. DRAM is an abbreviation for Dynamic Random Access Memory. The DRAM 106 stores temporary data during operation.

  The operation unit 107 transmits a command to the system control unit 101 in accordance with a user operation. The operation unit 107 has buttons and the like provided on the exterior of the digital camera 100. The imaging unit 108 is an imaging unit that photographs a subject. The imaging unit 108 includes a lens and an imaging element (image sensor) such as a CCD or CMOS element that photoelectrically converts subject light and outputs an image signal. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. The imaging unit 108 passes the image signal obtained by the imaging element to the image signal processing unit 109 under the control of the system control unit 101. Further, the imaging unit 108 measures the amount of defocus, that is, the defocus amount, for each region of two images (viewpoint images) with different viewpoints obtained by the imaging device, under the control of the system control unit 101. Further, the imaging unit 108 acquires shooting information at the time of shooting according to the control of the system control unit 101. That is, the system control unit 101 functions as a second acquisition unit that acquires shooting information. The shooting information includes at least one of a focal length at the time of shooting, a lens characteristic used for shooting, and an image sensor size.

  The image signal processing unit 109 stores image data obtained by photographing in the DRAM 106 based on an instruction from the system control unit 101. The image signal processing unit 109 stores the defocus amount measured by the imaging unit 108 in the DRAM 106 together with the image data at the same time as shooting. Further, the image signal processing unit 109 reduces the image data for display and stores it in a display frame memory provided in the DRAM 106.

  The depth information generation unit 110 generates depth information for each area of the image based on the image data stored in the DRAM 106 and the defocus amount under the control of the system control unit 101, and stores the depth information in the DRAM 106. That is, the system control unit 101 functions as a first acquisition unit that acquires image data and depth information corresponding to the image data. In the present embodiment, the depth information indicates the depth in the depth direction of the subject corresponding to the image data. The depth information includes data representing the distance from the camera as the imaging means to the subject (subject distance) as an absolute value, and data indicating the relative distance relationship (depth of the image) in the image data (parallax amount). Distribution, defocus amount distribution, etc.).

  There are various embodiments for depth information. That is, the information indicated by the data corresponding to the depth of the subject directly represents the subject distance from the imaging device to the subject in the image, or the relative relationship between the distance (subject distance) and the depth of the subject in the image. May be used as long as it represents information. For example, the system control unit 101 performs control to change the in-focus position, and a plurality of photographed captured image data is acquired. Depth information can be acquired from the focus area of each captured image data and the focus position information of the captured image data. In addition, when the imaging device has a pupil-divided pixel configuration, depth information for each pixel can be acquired from the phase difference between a pair of image signals. Specifically, the imaging device converts a pair of light beams that pass through different pupil partial regions of the imaging optical system into optical signals, and converts a plurality of pairs of image data (viewpoint images). Output from the photoelectric converter. The image shift amount of each region is calculated by the correlation calculation between the paired image data, and an image shift map representing the distribution of the image shift amount is calculated. Alternatively, the image shift amount is further converted into a defocus amount, and a defocus map representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image) is generated. When this defocus amount is converted into the subject distance based on the conditions of the imaging optical system and the imaging element, distance map data representing the distribution of the subject distance is obtained. Image shift map data, defocus map data, or distance map data of the subject distance converted from the defocus amount can be acquired.

  In addition, the subject distance from the imaging device to the subject in the image is directly measured using the TOF (Time Of Flight) method that measures the delay time from the light projection to the subject and receiving the reflected light to measure the distance to the subject. May be acquired automatically. In the TOF method, pulse light is projected onto a subject (target object) by a light projecting means, the reflected light is received by the imaging unit 108, and the flight time (delay time) of the pulse light is measured to measure the subject distance ( Measure the distance to the object) and obtain depth information.

  The shooting information acquisition unit 111 acquires shooting information at the time of shooting from the imaging unit 108 and stores it in the DRAM 106. The angle-of-view information generation unit 112 generates a shooting range (hereinafter referred to as “shooting angle of view”) of the recorded shot image based on the shooting information information acquired by the shooting information acquisition unit 111. The background surface conversion unit 113 sets the background surface of the stereoscopic image data based on the shooting angle of view generated by the angle of view information generation unit 112. The stereoscopic image data generation unit 114 acquires image data from the DRAM 106. In addition, the stereoscopic image data generation unit 114 acquires the depth information generated by the depth information generation unit 110. Then, the stereoscopic image data generation unit 114 generates stereoscopic image data based on the image data and the depth information using the background surface under the control of the system control unit 101. By connecting an external 3D printer or the like as the output destination of the stereoscopic image data, the output stereoscopic image data can be materialized.

  The LCD controller 115 reads the content of the display frame memory stored in the DRAM 106 and displays it on the LCD 116. The configuration of the digital camera 100 illustrated in FIG. 1 is an example, and the configuration of the digital camera 100 is not limited to the configuration illustrated in FIG. 1 as long as operations described below can be performed. Absent.

FIG. 2 is a diagram for explaining a captured image and depth information.
In the example shown in FIG. 2, the depth information is a distance map. FIG. 2A is a diagram illustrating a shooting situation when an image is shot. A person 201 is positioned as d1 in the depth direction as a subject. Reference numeral 101 denotes a shooting angle of view of the digital camera 100. z indicates the optical axis direction of the camera. y indicates an axis perpendicular to the ground.

  FIG. 2B shows a photographed image 200 obtained using the digital camera 100 according to the photographing situation shown in FIG. In addition to the person 201, a tree 202 is shown in the background. x indicates the horizontal direction. y represents a vertical axis.

  FIG. 2C shows a distance map 220 for the captured image 200 obtained in the imaging state shown in FIG. For example, the depth information generation unit 110 generates the distance map 220 using the defocus amount for each area of the image data acquired from the imaging unit 108. In the distance map 220, it is shown that a dark color area is located on the near side and a thin area is located on the back side. The lower part of the image close to the camera is the darkest, and then the person 201 is a slightly darker color. In the present embodiment, the digital camera 100 acquires a captured image 200 and a defocus amount for generating the distance map 220 at the time of shooting.

  Next, stereoscopic image data generation processing based on the captured image 200 and the distance map 220 will be described. FIG. 2D shows the relationship between the image capturing situation and the stereoscopic image data. Reference numeral 250 denotes generated stereoscopic image data. As shown in FIG. 2D, the stereoscopic image data generation unit 114 sets a region farther than the person 201 as the background surface 240 of the stereoscopic image data 250 based on the distance map 220, and sets the depth from the background surface 240 to the person 201. wear. The stereoscopic image data generation unit 114 adds depth using the background surface 240 set by the background surface conversion unit 113.

  Hereinafter, with reference to FIG. 3, the calculation process of the shooting angle of view will be described. The angle-of-view information generation unit 112 calculates a shooting angle of view using two relational expressions based on the shooting information acquired by the shooting information acquisition unit 111. The first relational expression is an expression showing the relation between the focal length and the shooting angle of view.

FIG. 3 is a diagram illustrating the relationship among the image sensor size, the focal length, and the shooting angle of view included in the shooting information.
Reference numeral 301 in FIG. 3 denotes an image sensor that constitutes the imaging unit 108, and X denotes a vertical size of the image sensor. Reference numeral 302 denotes a lens constituting the imaging unit 108, and F denotes a focal length indicating a distance from the lens to the image sensor. θ is a shooting angle of view.

From FIG. 3, the shooting angle of view θ, the focal length F, and the image sensor vertical size X can be expressed by the following equations.
tan (θ / 2) = X / (2 × F) Equation 1
In the above formula 1, the relational expression changes depending on the characteristics of the lens described later. Expression 1 shows a relational expression in the case of the flat projection method used for a general lens. From the above relational expression, the shooting angle of view θ is expressed as follows, and the shooting angle of view is calculated from the following expression.
θ = 180 / π × 2 arctan (X / (2 × F)) [degrees] Equation 2
The angle-of-view information generation unit 112 calculates the horizontal shooting angle of view by the same calculation method as the vertical shooting angle of view.

The second relational expression is an expression showing the relationship between the lens characteristics and the shooting angle of view. The characteristics and types of lenses will be described below. FIG. 4 is a diagram illustrating the characteristics of the lens.
In general, the type of lens is defined by the projection method. With reference to FIG. 4, the principle of the projection method, in particular, the principle of the orthographic projection method will be described. The hemispherical surface G is a virtual spherical surface whose diameter is a range of an image formed on the imaging surface by the lens. Here, the center O of the image corresponds to the zenith of the spherical surface G in all projection methods. Then, pay attention to one point P (referred to as an image point) in the image. The distance (referred to as image height) from the center O to the image point P in each projection method image has a specific relationship with the zenith angle (referred to as the incident angle) of the incident light.

The relationship is expressed by a mathematical expression where the image height is rp and the incident angle is φ.
(A) Equidistant projection method (Equidistant Projection)
The image height rp is proportional to the incident angle φ. Many common fisheye lenses.
rp∝φ ... Formula 3
(B) Stereographic Projection (Stereographic Projection)
The image height rp is proportional to the tangent of the half angle of the incident angle φ. It is often used for general lenses.
rp∝tan (φ / 2) Equation 4
(C) Equi-solid angle projection method
The image height rp is proportional to the half sine of the incident angle φ.
rp∝sin (φ / 2) Equation 5
The feature is that the solid angle taken by the subject is proportional to the area of the image, and the area ratio of the subject can be calculated correctly by measuring the area on the screen. This feature is used for surveying the amount of cloud cover and the distribution of forest vegetation.
(D) Orthographic Projection (Orthographic Projection)
The image height rp is proportional to the sine of the incident angle φ.
rp∝sinφ ・ ・ ・ Formula 6

The relationship expressed by Equation 6 is equivalent to the celestial sphere projected directly onto the film surface and the image side. It is characterized by the fact that the area of the surface light source occupying the screen is proportional to the illuminance at the location where the image was taken.
Since the image sensor vertical size X / 2 in Expression 1 indicates the maximum value of the image height rp, when it is replaced with X / 2 = rp, the following expression is obtained.
rp = F · tan (θ / 2) Equation 7
Similar to Equation 4, it can be seen that the incident angle φ, that is, the photographing field angle θ, varies depending on the tangent of the photographing field angle θ, which is the lens characteristic, the focal length F, and the image sensor size rp. For the image height and the incident angle, the above-described relational expression is established according to the lens characteristics. The angle-of-view information generation unit 112 calculates a shooting angle of view based on at least one of the image sensor size, the focal length, and the lens characteristics that are the shooting information acquired by the shooting information acquisition unit 111. .

FIG. 5 is a diagram for explaining the conversion process from the shooting angle of view to the background plane by the background plane conversion unit.
Reference numeral 501 in FIG. 5A denotes a shooting angle of view. Reference numeral 502 denotes a horizontal field of view. Reference numeral 503 denotes a vertical field of view. Reference numeral 504 denotes a background surface converted by the background surface conversion unit 113. When the shooting field angle is larger than the threshold, the background surface conversion unit 113 performs planar spherical conversion from the shooting field angle (angle) as shown in FIG. This sets the background surface to a curved surface. If the shooting angle of view is smaller than the threshold, the background surface is set to a plane.

  Planar spherical transformation will be described with reference to FIGS. FIG. 5B is a diagram illustrating an imaging region provided on a virtual spherical surface. The center O is the position of the digital camera 100 at the time of shooting. S indicates the center of the shooting angle of view 501.

  FIG. 5C is a plan view when the spherical surface shown in FIG. 5B is cut out by a plane on the horizontal axis passing through the center S of the shooting angle of view 501. O in FIG. 5C indicates the center of the sphere. r indicates the radius of the sphere. P indicates coordinates within the shooting angle of view. The triangle OPS and the triangle OP ′S ′ are similar, and a spherical point P ′ (x ′, y ′) can be calculated from the point P (x, y) within the shooting angle of view. The background surface conversion unit 113 moves the point P (x, y) within the range of the shooting angle of view, and calculates the coordinate point P ′ (x ′, y ′) on the spherical surface, thereby making the background surface a spherical surface. Convert.

FIG. 6 is a diagram for explaining a process of generating stereoscopic image data when a curved surface is set on the background surface.
Reference numeral 600 in FIG. 6A indicates image data obtained when a circumferential fisheye lens (equal distance projection type lens) that can be photographed at a photographing field angle of 180 degrees is photographed. A point 601 is the center of the image, and is the center of the shooting direction of the digital camera 100. The inside of the circle 602 is an area in which light that has passed through the circumferential fisheye lens is captured by the image sensor. Since the field angle of view is 180 degrees, an area 90 degrees from the photographing direction of the digital camera is photographed in the area along the circle 602. In addition, since an equidistance projection type lens is used, an area of 60 degrees is photographed on the circle 603. Further, an area of 30 degrees is photographed in the circle 604.

  Reference numeral 610 in FIG. 6B denotes a background surface of stereoscopic image data using the image data 600. Since the shooting angle of view is 180 degrees, the background surface 610 is a hemisphere of 180 degrees. A point 611 in FIG. 6B corresponds to the point 601 in FIG. 6A on the background surface 610. A point 615 corresponds to the point 605 in FIG. 6A on the background surface 610.

  A point 620 is the center of the hemisphere of the background surface 610 and is the shooting position of the digital camera 100. A point 611 corresponds to a point 601 at the center of the image data 600. Therefore, photographing is performed with the direction of the arrow 630 as the center. The stereoscopic image data generation unit 114 adds a depth from each region on the background surface 610 toward the center 620 based on the depth information. Taking the region corresponding to the point 615 as an example, the stereoscopic image data generation unit 114 performs depthing in the direction of the arrow 625. Accordingly, the subject corresponding to the image data protrudes from the background surface according to the depth indicated by the depth information. The stereoscopic image data generation unit 114 generates stereoscopic image data by performing depthing processing on the entire area of the background surface 610.

FIG. 7 is a flowchart for describing the generation processing of stereoscopic image data.
In step S <b> 701, the system control unit 101 stores the captured image and the defocus amount in the DRAM 106. Further, the shooting information acquisition unit 111 acquires shooting information from the imaging unit 108 and stores it in the DRAM 106. Subsequently, in step S702, the depth information generation unit 110 generates depth information corresponding to the captured image. For example, the depth information generation unit 110 generates a distance map using the defocus amount acquired in step S701.

  Next, in step S <b> 703, the view angle information generation unit 112 generates a shooting angle of view corresponding to the shot image based on the shooting information acquired by the shooting information acquisition unit 111. Subsequently, in step S704, the background surface conversion unit 113 determines whether the shooting angle of view generated in step S703 is larger than a threshold value. If the shooting angle of view is larger than the threshold, the process proceeds to step S705. If the shooting angle of view is equal to or smaller than the threshold, the process proceeds to step S706. Subsequently, in step S705, the background surface conversion unit 113 calculates a curved surface based on the generated shooting angle of view, and sets the background of the stereoscopic image data as the calculated curved surface. Then, the process proceeds to step S707. In step S706, the background surface conversion unit 113 sets the background of the stereoscopic image data as a plane. Then, the process proceeds to step S707. In step S707, the stereoscopic image data generation unit 114 generates stereoscopic image data using the background surface set by the background surface conversion unit 113.

  In this embodiment, a 3D printer output method is employed as a method for outputting stereoscopic image data as an entity, but a stereoscopic image may be output by another method employed in the prior art. The present invention is also applicable to an output device such as a 3D printer that inputs an externally captured image and depth information thereof and outputs stereoscopic image data, and a program on a computer.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Digital Camera 101 System Control Unit

Claims (11)

An image processing apparatus that generates stereoscopic image data corresponding to image data,
First acquisition means for acquiring the image data and depth information indicating the depth of the subject corresponding to the image data;
Setting means for setting a background surface of the stereoscopic image data according to a shooting angle of view corresponding to the image data;
An image processing apparatus comprising: generating means for generating the stereoscopic image data based on the image data and the depth information using the background surface. Second acquisition means for acquiring the imaging information;
Calculating means for calculating the shooting angle of view based on the shooting information;
The image processing apparatus according to claim 1, wherein the setting unit sets the background surface according to the calculated shooting angle of view. The image processing apparatus according to claim 2, wherein the second acquisition unit acquires at least one of a size, a focal length, and lens characteristics of the image sensor as the shooting information. The setting means includes
Determining whether the angle of view is greater than a threshold;
The image processing apparatus according to claim 2, wherein the background surface is set to be a curved surface when the shooting angle of view is larger than the threshold value. The image processing apparatus according to claim 4, wherein the setting unit sets the background surface to a plane when the shooting angle of view is equal to or less than the threshold value. The said generation | production means produces | generates the said stereo image data by attaching | subjecting the depth according to the depth which the said depth information shows for every area | region of the said image data from the said background surface. The image processing apparatus according to any one of the above. The image processing apparatus according to claim 6, wherein the generation unit causes a subject corresponding to the image data to protrude from the background surface according to a depth indicated by the depth information. The depth information is obtained by an image shift map based on the amount of parallax of a plurality of viewpoint images to be photographed, a defocus map based on a defocus amount, a distance map indicating a relative distance relationship between subjects in image data, or a TOF method. The image processing apparatus according to claim 1, wherein the image processing apparatus is any one of acquired distance information indicating a distance to a subject. An image pickup apparatus that functions as an image processing apparatus according to claim 1, further comprising: an image pickup device that photoelectrically converts subject light and outputs the image data. A control method of an image processing apparatus for generating stereoscopic image data corresponding to image data,
Obtaining the image data and depth information indicating a depth in a depth direction of a subject corresponding to the image data;
Setting a background surface of the stereoscopic image data according to a shooting angle of view corresponding to the image data;
And a step of generating the stereoscopic image data based on the image data and the depth information using the background surface.   A program that causes a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 7.