JP7263493B2 - Electronic devices and notification methods - Google Patents

Description

Embodiments of the present invention relate to electronic devices and notification methods.

A method using a stereo camera is known as a method for simultaneously obtaining a captured image and depth information. However, since two cameras are required and the distance between the two cameras must be increased in order to ensure the parallax, the size of the apparatus that implements this method is increased. Therefore, a technique for obtaining depth information with a single camera is desired. As one example, a technology has been proposed that obtains depth information based on the difference in shape of blur in an image captured by an image sensor for each of a plurality of color components. An image is captured with a filter consisting of at least two regions that pass different color components to create a per-color component blur in the image. A mechanism for obtaining depth information by using filters having transmission characteristics of different color components is called a color aperture.

JP 2016-102733 A

In order to generate blur for each color component in the image at any portion such as the center and periphery of the image, it is preferable to arrange filters having transmission characteristics of different color components at the lens aperture. However, this requires customizing a camera lens or modifying an off-the-shelf camera lens. Therefore, it was not easy to obtain depth information with a single camera.

SUMMARY OF THE INVENTION An object of the present invention is to provide an electronic device and a notification method that can easily obtain depth information with a single camera.

An electronic device according to an embodiment is an image captured by a camera including a filter having a first region that transmits a first wavelength band and a second region that transmits a second wavelength band, the image transmitted through the first region. input means for inputting a photographed image including a first color component image based on one wavelength band and a second color component image based on a second wavelength band transmitted through the second region; An electronic device comprising: an image; and a processing means for outputting in a state in which an effective area for calculation of depth information based on bias of color information in the second color component image is identified. The processing means acquires or calculates the effective area based on a distance between the aperture of the lens of the camera and the filter, which is at least one shooting parameter among a plurality of shooting parameters.

FIG. 10 is a diagram showing an example of changes in light rays depending on the distance to the subject when a filter is added to the aperture of the lens; It is a figure which shows an example of a filter. It is a figure which shows an example of the transmittance|permeability characteristic of the 1st filter area|region of a filter, and a 2nd filter area|region. It is a figure which shows an example of the change of a light ray when a filter is arrange|positioned in a lens opening. FIG. 10 is a diagram showing an example of changes in light rays when a filter is arranged outside the tip of a lens; It is a figure which shows an example of the external appearance of the electronic device by 1st Embodiment. 1 is a block diagram showing an example of an electrical configuration of an electronic device according to a first embodiment; FIG. It is a figure which shows an example of the structure of an image sensor. FIG. 3 is a diagram showing an example of color filters of an image sensor; It is a figure which shows an example of the relationship between the light ray change by a filter, and the shape of blur. 1 is a block diagram showing an example of a functional configuration of an electronic device according to a first embodiment; FIG. FIG. 4 is a diagram showing an example of a blurring function of a reference image in the first embodiment; FIG. FIG. 4 is a diagram showing an example of a blurring function of a target image in the first embodiment; FIG. FIG. 4 is a diagram showing an example of a blur correction filter in the first embodiment; FIG. It is a figure which shows an example of the effective area determination part in 1st Embodiment. FIG. 4 is a diagram showing an example of determining an effective area in the first embodiment; FIG. 4 is a diagram showing an example of a valid area table in the first embodiment; FIG. It is a figure which shows an example of the guideline which shows an effective area in 1st Embodiment. 7 is a flow chart showing an example of effective area notification in the first embodiment; It is a figure which shows an example of the external appearance of the electronic device by 2nd Embodiment. FIG. 10 is a diagram illustrating an example of a system including electronic devices according to a second embodiment; FIG. 10 is a flow chart showing an example of effective area notification by a system including an electronic device according to the second embodiment;

Embodiments will be described below with reference to the drawings. The disclosure is merely an example, and the invention is not limited by the contents described in the following embodiments. Modifications that can be easily conceived by those skilled in the art are naturally included in the scope of the disclosure. In order to make the explanation clearer, in the drawings, the size, shape, etc. of each part may be changed from the actual embodiment and shown schematically. Corresponding elements in multiple drawings may be denoted by the same reference numerals and detailed descriptions thereof may be omitted.

[Principle of acquisition of depth information]
FIG. 1 shows an example of changes in light rays depending on the distance to an object when a filter 10 having transmission characteristics of different wavelength bands is added to the aperture of a lens 12. FIG. FIG. 2 is a diagram showing an example of the filter 10. As shown in FIG. The filter 10 includes a plurality, for example two, first filter regions 10a and second filter regions 10b. The x direction is the horizontal direction, the y direction is the vertical direction, and the horizontal direction is the horizontal direction when viewed from the object. The center of filter 10 coincides with the optical center 16 of the lens. The first filter region 10a and the second filter region 10b are arranged so as not to overlap each other. In the example shown in FIG. 2, the first filter region 10a and the second filter region 10b each have a semicircular shape in which the circular filter 10 is divided into left and right by a vertical line segment passing through the optical center 16. ing. When the image sensor captures a color component image of three colors of red (R), green (G), and blue (B), the first filter area 10a is a combination of the first two colors among R, G, and B. and the second filter region 10b transmits a wavelength band corresponding to light of a second two-color combination different from the first two-color combination.

FIG. 3 shows an example of transmittance characteristics of the first filter region 10a and the second filter region 10b. For example, the first filter region 10a is a yellow (Y) filter that transmits red and green light wavelength regions, and the second filter region 10b is a cyan (C) filter that transmits green and blue light wavelength regions. consists of The transmittance characteristic 18a of the first filter (yellow filter) region 10a has a high transmittance for light in the wavelength ranges corresponding to the R image and the G image, and hardly transmits light in the wavelength range corresponding to the B image. The transmittance characteristic 18b of the second filter (cyan filter) region 10b has a high transmittance for light in the wavelength ranges corresponding to the B image and the G image, and hardly transmits light in the wavelength range corresponding to the R image.

Therefore, as shown in FIG. 1, light rays passing through the aperture change for each of the R, G, and B color component images. Light in the wavelength region of the G image passes through both the first filter (yellow filter) region 10a and second filter (cyan filter) region 10b, whereas light in the wavelength region of the R image passes through the first filter (yellow filter) region. 10a only, and the light in the B image wavelength region passes only through the second filter (cyan filter) region 10b. Therefore, the shape of the blur of the R image and the B image of the captured image changes according to the distance d to the subject. Also, since the aperture is divided into two, the biased directions of the blurring of the R image and the B image are reversed before and after the in-focus plane df. Therefore, front blur and back blur can be distinguished, and depth information corresponding to the distance d to the subject can be obtained from the shape of the blur.

Instead of arranging the filter 10 at the lens aperture, the lens 12 may be arranged between the filter 10 and the image sensor 14 on the optical path of the light incident on the image sensor 14, or the lens 12 and the image sensor 14 may be arranged. A filter 10 may be placed between 14 . Furthermore, when the lens 12 consists of a plurality of lenses, the filter 10 may be arranged between any of the lenses 12 . Also, instead of the Y filter or C filter, a magenta (M) filter that transmits the wavelength regions of red and blue light may be used. Also, the number of regions of the filter 10 is not limited to two, and may be three or more. The shape of the regions is not limited to the configuration in which the whole is divided by a straight line passing through the center of the circle, and may be a configuration in which a plurality of regions are arranged in a mosaic pattern.

[Unnecessary color shift for depth estimation]
In order to blur the image for each of a plurality of color components in any part of the image such as the center and the periphery, it is preferable to place the filter at the lens aperture. However, this requires customizing a camera lens or modifying an off-the-shelf camera lens. Therefore, it has been considered to arrange a color file outside the lens. FIGS. 4 and 5 show an example of the difference in change in light rays when the filter is placed at the lens aperture and when the filter is placed outside the lens tip (object side). 4 and 5 are views viewed from the plus side (upper side) of the y-axis. As shown in FIGS. 4A and 4B, when the filter 10 is placed at the aperture of the lens 12, light rays from the central and peripheral parts of the object pass through both the first filter area 10a and the second filter area 10b. The light passes evenly and enters the image sensor 14 . However, the filter 10 is fixed to the object side from the tip of the lens 12, for example, by screwing the filter 10 into a filter attachment screw at the tip of the lens, or the filter 10 is fixed to the front of the lens 12 with a clip or the like like a wide-angle lens attachment for a smartphone. In this case, as shown in FIG. 5(a), light rays from the center of the subject equally pass through both the first filter area 10a and the second filter area 10b and are incident on the image sensor 14. FIG. However, as shown in FIG. 5(b), light rays from the periphery of the object pass through either one of the first filter region 10a and the second filter region 10b more and pass through the other less and reach the image sensor. 14. In this way, when the filter is divided into left and right first and second filter areas, the image sensor 14 cannot evenly capture the R image and the B image at both the left and right ends of the subject. The balance with the B image is lost, and undesirable color shifts unnecessary for depth estimation occur.

However, when the filter 10 is placed outside the lens aperture, it is not impossible to measure the depth information at all. Although it is possible to obtain the depth information of the subject in the central area of the screen, it is necessary to accurately obtain the depth information of the subject in the peripheral area. There are only cases where it is not possible. Since the area of the subject in which the image sensor 14 can evenly capture the R image and the B image is determined by various parameters related to the optical characteristics of the camera at the time of image capturing (hereinafter referred to as "capturing parameters"), depth information measurement errors are a practical problem. It is possible to identify an area that is small enough to be nonexistent, that is, an area in which the reliability of depth information is higher than a certain reference reliability. Also, it is not always necessary to obtain depth information for the entire screen, and it may be sufficient to obtain depth information for the part of the subject located in the central area.

In the embodiment, a filter 10 that causes different blurring of a plurality of color components depending on the distance is arranged outside the lens aperture to notify the user of the reliability of the depth information. If the reliability is low, depth information with a high reliability can be obtained by changing the composition of the camera and taking a picture again. This provides an electronic device and notification method that can easily obtain depth information with a single camera without the need to custom order a camera lens or modify a commercially available camera lens.

[Appearance of the first embodiment]
FIG. 6 shows the appearance of the electronic device of the first embodiment. FIG. 6(a) is a perspective view, and FIG. 6(b) is a side view showing a cross-sectional structure. A smart phone 40 can be used as the electronic device of the first embodiment. The smartphone 40 is an example of an electronic device that has a function of capturing an image and processing the captured image. The embodiment is not limited to the smart phone 40, but a mobile information terminal such as a camera, a mobile phone with a camera function, a PDA (Personal Digital Assistant, Personal Data Assistant), or a personal computer with a camera function can be used.

A clip-type attachment 44 is attached to the smartphone 40 so as to cover the front surface of the camera lens 42 of the smartphone 40 . The attachment 44 has a first lens barrel 46 having a diameter larger than that of the camera lens 42 , and a second lens barrel 48 is inserted inside the first lens barrel 46 . A filter 52 is arranged at the tip of the second lens barrel 48 . Similar to the filter 10 shown in FIG. 2, the filter 52 consists of a yellow first filter area 52a and a cyan second filter area 52b. When the user applies a rotational force to the second barrel 48 , the second barrel 48 is rotated with respect to the first barrel 46 . Thereby, the direction of the filter 52 (the direction of the dividing line between the first filter region 52a and the second filter region 52b) can be adjusted. The second barrel 48 remains stationary with respect to the first barrel 46 when the user does not apply a rotational force. A set screw (not shown) or the like may be provided on the first lens barrel 46 to prevent rotation of the second lens barrel 48 after the orientation of the filter 52 is adjusted.

One of the subject conditions for which depth information is required is the direction in which the edge extends (edge direction). The edge direction of a vertical object such as a pillar is horizontal, and the edge direction of a horizontal object such as a desk is vertical. If the region division direction of the filter and the edge direction of the subject are not parallel, depth information can be obtained. In the filter shown in FIG. 2, the two regions are arranged horizontally, so the direction of division is the vertical direction. Therefore, the object for which depth information can be obtained using the filter shown in FIG. 2 is a vertical object that is a horizontal edge. Here, it is assumed that the edge direction of the object for which depth information is to be obtained is the horizontal direction. The attachment 44 is mounted so that the direction of dividing the area of the filter intersects the edge direction of the object so as to be suitable for such an object. Therefore, a marker 46 a for determining the mounting position of the attachment 44 is marked on the surface of the first lens barrel 46 , and a marker 48 a is marked on the surface of the second lens barrel 48 . The attachment 44 is attached to the smartphone 40 so that the marker 46a is aligned with the LED 40a of the smartphone 40, and the second lens barrel 48 is positioned with respect to the first lens barrel 46 so that the marker 48a and the marker 46a are aligned. In this case, depth information of a subject whose edge direction is horizontal is obtained. If the edge direction of the subject is not horizontal, then the second lens barrel 48 is rotated by the user so that the direction of the area dividing line of the filter 52 intersects the edge direction, and the orientation of the filter 52 is adjusted. .

Although the second lens barrel 48 is provided inside the first lens barrel 46 , it may be provided outside the first lens barrel 46 .
Further, when the edge direction of the subject is parallel to the area dividing line of the filter 52, the smartphone 40 itself may be rotated so that the direction of the area dividing line intersects the edge direction. In this case, since the filter 52 does not have to be rotatable, the second lens barrel 48 is unnecessary and the filter 52 may be provided in the first lens barrel 46 .

[Electrical configuration of the first embodiment]
FIG. 7 is a block diagram showing an example of an electrical configuration of smartphone 40. As shown in FIG. The smartphone 40 includes a filter 52, an image sensor 54 that captures a subject image incident through a lens 42, a CPU 56, a RAM 58, a display 60, a communication section 62, a nonvolatile memory 64, a memory card slot 66, and an audio output section 68. Prepare. The image sensor 54 , CPU 56 , RAM 58 , display 60 , communication section 62 , nonvolatile memory 64 , memory card slot 66 and audio output section 68 are interconnected via a bus line 69 .

The CPU 56 is a processor that controls operations of various components within the smartphone 40 . CPU 56 executes various programs loaded into RAM 58 from non-volatile memory 64, which is a storage device. In addition to various application programs, the nonvolatile memory 64 also stores an OS (Operating System) 64a. Examples of various application programs are depth calculation program 64b and coverage area notification program 64c. The non-volatile memory 64 can also store images captured by the image sensor 14 and processing results of the images by the programs 64b and 64c.

The communication unit 62 is an interface device configured to perform wired communication or wireless communication. The communication unit 62 includes a transmitting unit that transmits signals by wire or wirelessly, and a receiving unit that receives signals by wire or wirelessly. For example, the communication unit 62 can connect the smartphone 40 to an external server 94 via the Internet 92 .

The display 60 is, for example, an LCD (Liquid Crystal Display). A display 60 displays a screen image based on a display signal generated by the CPU 56 or the like. For example, the display 60 has a function of displaying captured images and depth information, and a function of notifying the user of the reliability by displaying the reliability itself or information corresponding to the reliability as text, icons, marks, or the like. Display 60 may be a touch screen display. In that case, for example, a touch panel is arranged on the upper surface of the LCD. A touch panel is a capacitive pointing device for inputting on an LCD screen. The touch panel detects a contact position on the screen where the finger contacts, movement of the contact position, and the like.

An audio output unit 68 is a speaker, an earphone jack, or the like, and notifies the user of the reliability itself or information corresponding to the reliability by voice.
Memory cards, which are various portable storage media such as SD (registered trademark) memory cards and SDHC (registered trademark) memory cards, can be inserted into the memory card slot 66 . When a storage medium is inserted into memory card slot 66, data can be written to and read from the storage medium. The data are image data and distance data, for example.

The application program does not have to be built into the smart phone 40 . For example, the depth calculation program 64b may be provided in the server 94 instead of the smart phone 40 . In this case, an image captured by the image sensor 54 may be transmitted to the server 94 by the communication unit 62 , depth information may be obtained by a depth calculation program provided in the server 94 , and the depth information may be returned to the smartphone 40 . Moreover, both the depth calculation program 64b and the effective area notification program 64c may be provided in the server 94 instead of the smart phone 40 . In this case, the image captured by the image sensor 54 is transmitted to the server 94 by the communication unit 62, the depth information is obtained by the depth calculation program provided in the server 94, and the effective area is determined by the effective area notification program provided in the server 94. , depth information, and effective area notification information are returned to the smartphone 40 .

[Image sensor]
The image sensor 54, for example, a CCD-type image sensor comprises photodiodes as light receiving elements arranged in a two-dimensional matrix and a CCD for transferring signal charges generated by photoelectric conversion of incident light by the photodiodes. Become. FIG. 8 shows an example of the cross-sectional structure of the photodiode portion. A large number of n-type semiconductor regions 72 are formed in the surface region of the p-type silicon substrate 70, and pn junctions between the p-type substrate 42 and the n-type semiconductor regions 72 form a large number of photodiodes. Here, two photodiodes constitute one pixel. Each photodiode is therefore also referred to as a sub-pixel. A light shielding film 74 for suppressing crosstalk is formed between each photodiode. A multi-layered wiring layer 76 is formed on the p-type silicon substrate 70, in which transistors and various wirings are provided.

A color filter 80 is formed on the wiring layer 76 . The color filter 80 is composed of a two-dimensional array of a large number of filter elements that transmit red (R), green (G), or blue (B) light, for example, for each pixel. Therefore, each pixel generates only image information of one of R, G, and B color components. The image information of the other two color components not generated by the pixel is obtained by interpolation from the color component image information of surrounding pixels. Moiré and false colors may occur during interpolation when capturing a periodically repeated pattern. In order to prevent this, although not shown, an optical low-pass filter made of quartz or the like that slightly blurs the repetitive pattern may be placed between the lens 42 and the image sensor 54 . Instead of providing an optical low-pass filter, a similar effect may be obtained by signal processing of the image signal.

A microlens array 82 is formed on the color filter 80 . The microlens array 82 consists of a two-dimensional array arrangement of a pair of microlenses 82a and 82b corresponding to one pixel. Although FIG. 8 shows a front-illuminated image sensor 54, it may be a back-illuminated image sensor. Two photodiodes constituting one pixel are configured to receive light passing through different regions of the exit pupil of the lens 42 via a pair of microlenses 82a and 82b, thus realizing so-called pupil division. It is

FIG. 9A is a plan view showing an example of the relationship between photodiodes 84a and 84b and microlenses 82a and 82b that constitute each pixel. As shown in FIG. 9A, light that has passed through the first filter region 10a of the filter 52 is incident on the photodiode 84a as the first sub-pixel. Light passing through the second filter region 10b of the filter 52 is incident on the photodiode 84b as the second sub-pixel.

FIG. 9B shows an example of the color filter 80. FIG. The color filter 80 is, for example, a Bayer arrangement primary color filter. Thus, the image sensor 54 includes R pixels that sense red light, G pixels that sense green light, and B pixels that sense blue light. Note that the color filter 80 may be a complementary color filter. Furthermore, if it is not necessary to capture a color image and only depth information is to be obtained, the image sensor 54 need not be a color sensor, but may be a monochrome sensor, and the color filter 80 may be omitted.

The image sensor 14 is not limited to a CCD type sensor, and a CMOS (Complementary Metal Oxide Semiconductor) type sensor may be used.
[Bokeh shape by filter]
The shape of the light beam change and the blur due to the filter 52 will be described with reference to FIG. 10 . The filter 52 consists of a yellow first filter area 52a and a cyan second filter area 52b. When the object 88 is located further than the focal distance df (d>df), the image picked up by the image sensor 54 is blurred. A blur function (PSF: Point Spread Function) indicating the shape of the blur on the image is different for each of the R image, the G image, and the B image. The blur function 101R for the R image shows the shape of the blur skewed to the left, the blur function 101G for the G image shows the shape of the unbiased blur, and the blur function 101B for the B image shows the shape of the blur skewed to the right. there is

When the object 88 is at the focal distance df (d=df), the image captured by the image sensor 54 is hardly blurred. Blur functions indicating the shape of blur on this image are substantially the same for the R, G and B images. That is, the R-image blur function 102R, the G-image blur function 102G, and the B-image blur function 102B show unbiased blur shapes.

When the subject 88 is in front of the focal distance df (d<df), the image captured by the image sensor 54 is blurred. A blur function indicating the shape of blur on this image is different for each of the R image, the G image, and the B image. The blur function 103R for the R image shows the shape of the blur skewed to the right, the blur function 103G for the G image shows the shape of the unbiased blur, and the blur function 103B for the B image shows the shape of the blur skewed to the left. there is

In the present embodiment, using such characteristics, depth information to the subject is calculated for each pixel of the image.
[Functional Configuration of Embodiment]
FIG. 11 is a block diagram showing an example of the functional configuration of the smart phone 40. As shown in FIG. An arrow from the color filter 10 to the image sensor 54 indicates the path of light. Arrows pointing from the image sensor 54 indicate the paths of the electrical signals.

The red light that has passed through the first filter (Y) region 52a of the filter 52 is incident on the R pixel (referred to as the first sensor) 54a of the image sensor 54 via the lens 42, and passes through the first filter (Y) of the filter 52. Green light that has passed through the area 52 a is incident on the G pixels (referred to as the second sensor) 54 b of the image sensor 54 via the lens 42 . The green light that has passed through the second filter (C) region 52b of the filter 52 is incident on the second sensor 54b of the image sensor 54 via the lens 42, and the blue light that has passed through the second filter (C) region 52b of the filter 52. of light enters a B pixel (referred to as a third sensor) 54c of the image sensor 54 through the lens 42. As shown in FIG.

Output signals from the first sensor 54 a , the second sensor 54 b and the third sensor 54 c are input to the image input unit 102 . The image input unit 102 acquires, as a reference image, the G image whose point spread function (PSF) indicates a shape of unbiased blur, and one of the R and B images whose point spread function (PSF) indicates a shape of blur whose blur function is biased. Or acquire both as target images. The target image and the reference image are images captured at the same time by the same camera.

An output signal of the image input unit 102 is supplied to the display 60 and also supplied to the depth calculation unit 104 and the effective area determination unit 106 . The outputs of depth calculator 104 and valid area determiner 106 are also provided to display 60 .
[Depth calculation]
A depth calculation unit 104 calculates depth information up to a subject in an image for each pixel using a blur correction filter. Note that instead of calculating depth information for each pixel, depth information may be calculated for each pixel block, which is a set of a plurality of pixels. A blur correction filter is a filter that, when operated on a target image, makes the correlation between the target image and the reference image after the operation higher than the correlation between the target image and the reference image before the operation. A method of calculating the correlation between the corrected image and the reference image will be described later. When calculated with the blur correction filter, the shape of the blur of the target image approaches the shape of the blur of the reference image.

In the embodiment, a plurality of blur correction filters are prepared for each of a plurality of distances, which are created assuming a plurality of distances to the subject in the image. A corrected image obtained by correcting the blur shape of the target image is generated, and a blur correction filter having the highest correlation between the corrected image and the reference image is searched from among a plurality of blur correction filters. The distance corresponding to the blur correction filter with the highest correlation is determined as the depth information to the subject. However, instead of selecting one blur correction filter from a plurality of blur correction filters prepared in advance, a blur correction filter that satisfies the above conditions may be calculated.

The depth calculation unit 104 may further output a distance image in which each pixel of the image is colored according to the distance from the calculated depth.
The blurring function of the captured image is determined by the shape of the filter 52 (mode of area division) and the distance between the position of the subject and the focus position. FIG. 12 is a diagram illustrating an example of a blurring function of a reference image according to the embodiment; As shown in FIG. 12, the shape of the aperture through which the wavelength region of green light corresponding to the second sensor 54b is transmitted is a point-symmetric circular shape. There is no change before and after, and the width of the blur changes depending on the size of the distance between the subject and the focus position. A blur function indicating the shape of such blur can be expressed as a Gaussian function in which the width of blur changes depending on the distance between the position of the object and the focus position. Note that the blur function may be expressed as a pillbox function in which the width of blur changes depending on the distance between the position of the object and the focus position.

On the other hand, FIG. 13 is a diagram showing an example of the blurring function of the target image according to the embodiment. Note that the center of each figure is (x, y)=(0, 0). As shown in FIG. 13, the blur function of the target image (for example, the R image) can be expressed as a Gaussian function that attenuates the width of blur at x>0 when d>df where the subject is farther than the focus position. Also, the blur function of the target image (for example, the R image) can be expressed as a Gaussian function that attenuates the width of the blur at x<0 when d<df where the subject is closer than the focus position. The blur function of the other target image (for example, B image) can be expressed as a Gaussian function in which the width of blur attenuates at x<0 when d>df where the subject is farther than the focus position. Also, the blur function of the B image can be expressed as a Gaussian function in which the width of the blur attenuates at x>0 when d<df where the subject is closer than the focus position.

By analyzing the blur function of the reference image and the blur function of the target image, it is possible to obtain a plurality of blur correction filters for correcting the blur shape of the target image to the blur shape of the reference image.
FIG. 14 is a diagram illustrating an example of a blur correction filter according to the embodiment; The blur correction filters are distributed on a straight line (horizontal direction) passing through the center point of the line segment (vertical direction) on the boundary between the first filter region 52a and the second filter region 52b and orthogonal to this line segment. The distribution becomes a mountain-like distribution as shown in the figure, in which the peak points (positions and heights on the straight line) and how they spread from the peak points differ for each assumed distance. The blur shape of the target image can be corrected to various blur shapes assuming an arbitrary distance using a blur correction filter. That is, it is possible to generate a corrected image assuming an arbitrary distance.

The depth calculation unit 104 obtains, for each pixel of the captured image, the distance at which the blur shapes of the generated corrected image and the reference image are most similar or match. The degree of matching of the blur shape can be obtained by calculating the correlation between the corrected image and the reference image in a rectangular area of arbitrary size centered on each pixel. An existing similarity evaluation method may be used to calculate the degree of matching between blur shapes. The depth calculation unit 104 obtains the distance corresponding to the correction filter with the highest correlation between the corrected image and the reference image, and calculates the depth information up to the subject captured in each pixel. For example, existing similarity evaluation methods include SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross-Correlation), ZNCC (Zero-mean Normalized Cross-Correlation), Color Alignment Measure, etc. You should use it. In this embodiment, Color Alignment Measure is used, which utilizes the fact that the color components of a natural image locally have a linear relationship. In Color Alignment Measure, an index representing the correlation is calculated from the variance of the color distribution of the local border around the target pixel of the captured image.

[Depth Information Reliability Notification]
The embodiment has a function of determining the reliability of the depth information calculated by the depth calculation unit 104 and notifying the user of information according to the reliability. As an example of information notification according to reliability, a valid area made up of pixels with higher reliability than a reference reliability is obtained, and the valid area is notified to the user. Some examples of the effective area determination unit 106 shown in FIG. 11 regarding this notification are shown in FIGS. 15(a), (b) and (c).

The effective area determination unit 106a shown in FIG. 15A includes an effective area calculation unit 112 and an imaging parameter input unit 114. FIG. Since the effective area is determined by the imaging parameters, the effective area calculation section 112 calculates the effective area according to the imaging parameters input from the imaging parameter input section 114 .

The shooting parameters include lens brightness (F number), focal length, distance between filter and lens aperture, size of image sensor, and the like. The imaging parameter input unit 114 may display a parameter input/selection screen on the display 60 and allow the user to input each imaging parameter using a touch panel provided on the display 60 . Since the brightness of the lens, the focal length, and the size of the image sensor are values specific to the smartphone 40, they may be automatically input when the effective area notification program 64c is installed on the smartphone 40. good. Further, when the lens 42 has an optical zoom function, the focal length is variable, so the focal length information may be input from the zoom mechanism. Since the distance between the filter and the lens aperture is a unique value for each attachment 44, it must be entered for each attachment 44. FIG. However, the photographing parameter input unit 114 may be provided with a table storing this distance for each type of attachment 44, and the user may input the type of attachment 44 instead of the distance itself.

When a filter 52 having a vertically divided configuration similar to that of the filter 10 shown in FIG. The coordinates of the left and right edges of the effective area (for example, the number of pixels from the center in the left and right direction of the screen) are calculated based on .

An example of effective area calculation will be described. When a subject whose pixel values are known for all pixels on the screen, for example, a pure white background is photographed using the filter 52 as shown in FIG. is white, becomes bluish toward the right end, and becomes reddish toward the left end. FIG. 16(b) shows the cross-sectional profile of the pixel values (luminance) of the captured image in the dashed line portion of FIG. 16(a). Knowing the imaging parameters, the cross-sectional profile can be calculated by an optical simulation program. Therefore, the effective area calculator 112 calculates a cross-sectional profile as shown in FIG. Then, the effective area calculation unit 112 calculates the position coordinates of the left and right ends of the effective area where the R pixel value and the B pixel value are equal to or higher than a predetermined luminance. The predetermined brightness is related to the reference reliability. Note that the reference reliability for determining the effective area may be varied according to the allowable value of the measurement error of the depth information (referred to as the required depth level). That is, when the required depth level is high, the predetermined luminance is high and the effective area is narrowed, and when the required depth level is low, the predetermined luminance is low and the effective area is widened. The requested depth level is also input from the imaging parameter input unit 114 . Information about the determined effective area is displayed on the display 60 or output as sound via the audio output section 68 .

The effective area determination unit 106b shown in FIG. 15B includes an imaging parameter input unit 114, an effective area acquisition unit 116, and an effective area memory 118. FIG. A cross-sectional profile as shown in FIG. A valid area table indicating the valid area of is stored in the valid area memory 118 . Note that the required depth level is not essential, and when the effective area is held in advance as a table, information indicating the pre-calculated effective area corresponding to only the imaging parameters is held, and the effective area is variable according to the required depth level. You don't have to. It should be noted that without performing calculations using an optical simulation program, an object whose pixel values are known for all pixels on the screen, for example, a photographed image of a pure white background is analyzed, and the R pixel value and the B pixel value are equal to or greater than a predetermined luminance. The position coordinates of the left and right ends of the effective area may be obtained and stored in the memory 118 .

An example of the valid area table in the valid area memory 118 is shown in FIG. The effective area acquisition unit 116 reads from the effective area memory 118 the effective area information corresponding to the imaging parameters and the requested depth level input from the imaging parameter input unit 114 . If the lens has an optical zoom function, the focal length is variable, but the effective area is the same for a certain focal length range. and register it in the table. Information about the effective area read out from the effective area memory 118 is displayed on the display 60 or output as sound via the audio output section 68 .
The effective area can also be statistically determined by machine learning. For a scene and/or pixel pattern for which correct depth information is known, the estimation error can be obtained by estimating the depth with certain shooting parameters. Obtaining the estimated error for a sufficient number of scenes and/or pixel patterns provides statistics of the estimated error between the imaging parameters and the scene and/or image patterns. Therefore, once the shooting parameters and the scene and/or image pattern are specified, the effective area calculator 112 in FIG. 15A can obtain the statistics of the estimation error.
For example, when variance is used as a statistic, it can be determined that a region with a large variance of estimation errors is a region with low reliability. A statistical model that expresses the relationship between statistics and reliability can be obtained by machine learning such as a neural network if training data with known correct depth information are prepared. The obtained statistical model is held in the imaging parameter input unit 114 and input to the effective area calculation unit 112 when calculating the effective area. As a result, the effective area calculation unit 112 can obtain the reliability based on the shooting parameters, the scene and/or the image pattern, and the statistical model based on machine learning, and obtain the effective area with the reliability higher than the reference reliability.

The effective area calculation unit 112 shown in FIG. 15A and the effective area memory 118 shown in FIG. FIG. 15(c) shows an example of the valid area determining section 106c in this case. The effective area determination unit 106 c includes an imaging parameter input unit 114 and an effective area acquisition unit 120 . The imaging parameters input from the imaging parameter input section 114 to the effective area acquisition section 120 are transmitted to the server 94 via the communication section 62 . The valid area is calculated by the server 94 or read from the valid area memory, the information about the valid area is returned to the communication section 62, and the valid area obtaining section 120 obtains the information about the valid area. Information about the effective area received by the communication unit 62 is displayed on the display 60 or output as sound via the audio output unit 68 .

FIG. 18 shows an example of information on the effective area displayed on the display 60. As shown in FIG. An image acquired by the image input unit 102 is displayed on the display 60 . A pair of guide lines 124 indicating the left and right ends of the effective area are displayed on this image. The user can see the image on which the guideline 124 is displayed and confirm whether or not the measurement target for which the depth is to be determined is located within the effective area. When the object to be measured is positioned outside the effective area, the object to be measured can be positioned within the effective area by changing the composition or the like. Therefore, even if a color shift occurs due to a filter located outside the lens aperture, the measurement error of the depth information of the measurement target can be kept below the allowable value. Note that since the direction in which the filter 10 is arranged (the direction of the boundary line between the first filter area and the second filter area) is vertical, the effective area is a vertically elongated area with a certain width centered on the center of the screen. The effective area differs according to the arrangement direction of 10 . For example, since the filter 10 is arranged in the horizontal direction, the effective area is a horizontally long area with a constant width centered on the center of the screen.

As a modification of FIG. 18, instead of displaying the guideline 124, an image obtained by clipping only the effective area may be displayed. That is, instead of the images on the right side of the right guideline and on the left side of the left guideline in FIG. 18, a constant luminance, for example, a black background may be displayed. Furthermore, the image may be displayed by changing the color, brightness or contrast between the effective area and the non-effective area. When an image is displayed by changing the color, brightness, or contrast of the image, the effective area and the non-effective area are not limited to being binary-changed, and may be changed continuously according to the reliability. The reliability itself can be obtained based on the cross-sectional profile of the pixel values (brightness) of the captured image, as described with reference to FIG. For example, it can be determined that a cross-sectional position where the difference between the R pixel value and the B pixel value is large has a color shift and the reliability is low, and a cross-sectional position where the difference between the R pixel value and the B pixel value is small has no color shift and the reliability is high. . Furthermore, a reliability map in which each pixel of the captured image is colored according to the reliability may be displayed and saved, or the reliability (numerical value) of each pixel may be displayed and saved in the form of a table corresponding to the pixel. Also good. The reliability map and reliability table may be displayed together with the captured image.
The effective area may be presented to the user by displaying it as an image on the display 60, or if the effective area and the position of the guideline 124 are determined in advance, the user cannot see the image other than the effective area on the display 60. The effective area may be presented to the user by physical means such as putting a sticker on the area other than the effective area.

FIG. 19 is a flowchart showing an example of notification of information according to reliability according to the first embodiment. At block 152, the user launches the depth calculation program 64b and the coverage area notification program 64c. At block 154 , the user attaches attachment 44 to the front of lens 42 of smartphone 40 . As shown in FIG. 18, when the object to be measured for which depth information is to be obtained is an object whose edge direction is vertical, the user determines the orientation of the attachment 44 so that the marker 46a of the first lens barrel 46 matches the LED 40a of the smartphone 40. , the orientation of the second lens barrel 48 is determined so that the marker 48a of the second lens barrel 48 is aligned with the marker 46a. When the edge direction of the object to be measured is horizontal, the user determines the orientation of the attachment 44 so that the marker 46 a of the first lens barrel 46 matches the LED 40 a of the smartphone 40 , and then the marker of the second lens barrel 48 . The orientation of the second barrel 48 is determined so that the marker 48a is 90 degrees off the marker 46a.

At block 156, the user launches a camera application program (not shown but stored in non-volatile memory 64). When the camera application program is started, at block 158 the display 60 displays a through image output from the image sensor 54 .

At block 162, the user enters imaging parameters (type of attachment 44) using the touchpad. It should be noted that the imaging parameters may be input before execution of block 164 rather than after execution of block 158 . For example, when the depth calculation program 64b and the effective area notification program 64c are activated in block 152, a setting screen may be displayed and the imaging parameters may be input at that stage. Note that the lens brightness, focal length, and image sensor size, which are photographing parameters specific to the smartphone 40, are values specific to the smartphone 40 and need not be input by the user, but these may also be input by the user. .

At block 164, the effective area determination unit 106 calculates the effective area based on the imaging parameters (and possibly also the depth requirement level) (in the case of FIG. 15(a)) and reads it from the effective area memory 118 (FIG. 15 ( b)) or acquired from the server 94 via the communication unit 62 (in the case of FIG. 15(c)). At block 166, the coordinate information of the left and right ends of the effective information is supplied from the effective area determination unit 106 to the display 60, and a pair of guide lines 124 indicating the left and right ends of the effective area are displayed on the through image as shown in FIG. be done.

At block 168, the active area determiner 106 performs image analysis to identify background and objects in the through image. The background is a region of constant brightness value without contrast, and the object is a region with contrast and a difference in brightness value. Upon recognizing the object, the effective area determination unit 106 takes the object as a depth measurement target. Alternatively, an object-like region may be extracted by image recognition technology or object detection technology. Note that the depth measurement target is not limited to being automatically recognized using image analysis, and may be selected by the user on the touch panel.

At block 172, the effective area determination unit 106 determines whether or not the depth measurement target is located within the effective area. If the depth measurement target is not located within the effective area (NO determination in block 172), in block 174, the effective area determining unit 106 notifies the user of imaging guidance for positioning the depth measurement target within the effective area. . Examples of guidance include displaying on the display 60 a text that changes the composition, such as "more right" and "more left," and voice guidance that changes the composition, such as "more right" and "more left." Including outputting from the audio output section 68 .

During the image analysis in block 168, the edge direction of the subject is also recognized from the contrast, and in block 172, the edge direction of the subject is horizontal. If so, the guidance in block 174 may be to rotate the orientation of the filter 52 by 90 degrees.
By notifying the user of such a guide, the user can reliably position the depth measurement object within the effective area. After the depth measurement object is positioned within the effective area, the user operates the shutter. Therefore, at block 176 it is determined whether the shutter has been pressed. If the shutter is not pressed, the process from block 164 is repeated. If the shutter is pressed, it can be determined that the depth measurement object is located within the active area, and depth information is calculated for each pixel at block 178 and the depth information is displayed or saved. The depth information may be displayed and saved as a distance image in which each pixel of the image is colored according to the distance, or the depth information (numerical value) for each pixel may be displayed and saved in the form of a table corresponding to the pixel. good.

In addition, a photographed image corresponding to the distance image may also be stored, but the photographed image may have color shift, and may be unsightly if displayed as it is. Since the color shift range corresponds to the effective area, it is possible to compensate for the color shift based on the photographing parameters for determining the effective area and obtain a photographed image free of color shift. If a pixel-by-pixel reliability is desired, the reliability can also be compensated for color drift. Therefore, when storing a photographed image corresponding to a distance image, it may be stored after compensating for color deviation.

The depth information may be calculated for each frame of the through image in block 164 or the like in parallel with acquisition of the effective area regardless of the shutter operation. In this case, the depth measurement object of block 168 may be recognized based on depth information rather than contrast by image analysis. This is because depth information can only be obtained up to a certain finite distance, and depth information of an object at infinite distance cannot be obtained.

According to the first embodiment, the filter 52 that blurs a plurality of color components differently depending on the distance is attached to the front surface of the lens 42 of the smartphone 40 . Filter 52 has at least two filter regions. Depending on the installation method of the filter 52 and the photographing parameters of the camera optical system, the light of a plurality of color components does not pass uniformly at the end of the lens opening, and the light of one color component is more than the light of the other color components. In some cases, the depth information passes through many times, causing undesirable color shift (bias of color information) that is unnecessary for depth estimation, and depth information cannot be obtained with high accuracy for the entire screen. In the first embodiment, the user is notified of an effective region in which depth information can be obtained with high accuracy. Further, when the measurement target is not located within the effective area, guidance for positioning the measurement target within the effective area is also notified. For this reason, it is expected that the imaging will be performed so that the depth information of the object to be measured can be obtained with high accuracy. According to the first embodiment, there is provided an electronic device and a notification method that can easily obtain depth information with a single camera without the need to custom order a camera lens or modify a commercially available camera lens.

Other embodiments will be described below. In the following description, constituent elements corresponding to those of the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof are omitted.
[Second embodiment]
FIG. 20 shows the appearance of the electronic device of the second embodiment. FIG. 20(a) is a perspective view, and FIG. 20(b) is a side view showing a cross-sectional structure. A camera 202 can be used as the electronic device of the second embodiment.

A first lens barrel 206 is screwed into a screw for attaching a filter at the tip of the lens 204 of the camera 202 . Alternatively, the first lens barrel 206 may be attached to the tip of the lens 204 by bayonet-type fixing means. A second lens barrel 208 is inserted inside the first lens barrel 206 . A filter 212 is arranged at the tip of the second lens barrel 208 . Similar to the filter 52 of the first embodiment, the filter 212 consists of a yellow first filter area 212a and a cyan second filter area 212b. When the user applies a rotational force to the second barrel 208 , the second barrel 208 is rotated with respect to the first barrel 206 . Thereby, the orientation of the filter 212 (the orientation of the area dividing line by the first filter area 212a and the second filter area 212b) can be adjusted. The second barrel 208 remains stationary with respect to the first barrel 206 when the user does not apply a rotational force. A set screw (not shown) or the like may be provided on the first lens barrel 206 to prevent rotation of the second lens barrel 208 after the orientation of the filter 212 is adjusted.

Although the second lens barrel 208 is provided inside the first lens barrel 206 , it may be provided outside the first lens barrel 206 .
Also, if the edge direction of the object is different from the direction of the area dividing line of the filter 212, the camera 202 itself may be rotated so that the direction of the area dividing line is aligned with the edge direction. In this case, since the filter 212 does not have to be rotatable, the second lens barrel 208 is unnecessary and the filter 212 may be provided on the first lens barrel 206 .

The electrical configuration of the camera 202 is almost the same as an example of the electrical configuration of the smartphone 40 of the first embodiment, but differs from FIG. 7 in that the depth calculation program 64b and effective area notification program 64c are not installed. The depth calculation program and coverage area notification program are provided on the second device.

The second device may be server 94 or a personal computer owned by the owner of camera 202 . FIG. 21 shows an example of cooperation between a personal computer 230 having a depth calculation program and an effective area notification program and the camera 202 . Images (still images) captured by the camera 202 are stored in the memory card 222 in the memory card slot 220 . A memory card 222 removed from the memory card slot 220 of the camera 202 is inserted into a memory card slot 232 of the personal computer 230 . The personal computer 230 reads images from the memory card 222 in the memory card slot 232 . Accordingly, an image captured by the image sensor 214 of the camera 202 is sent to the personal computer 230 . If the camera 202 has a communication function, the image captured by the image sensor 214 of the camera 202 is communicated to the personal computer 230 without using the memory card 222 .

The personal computer 230 performs depth calculation and effective area notification processing in the same manner as the smart phone 40 of the first embodiment.
FIG. 22 is a flow chart showing an example of notification of information according to reliability according to the twenty-first embodiment. The flowchart of FIG. 22 is started with the memory card 222 storing the captured images of the camera 202 inserted into the memory card slot 232 of the personal computer 230 .

At block 302 , the user launches the depth calculation program and coverage area notification program on personal computer 230 . At block 304 , personal computer 230 reads an image from memory card 222 .
At block 306, the user enters imaging parameters and depth requirement levels. In some cases, it is not necessary to input the depth request level. Furthermore, if the camera 202 has a function of recording the shooting parameters along with the image, the shooting parameters are read into the personal computer 230 along with the image, so no user input is required.

At block 308, the personal computer 230 displays the image. A list of images of several frames may be displayed, or the images may be displayed by switching one frame at a time. At block 312, the user selects an image frame for depth calculation.
At block 314, the personal computer 230 determines the effective area with the effective area notification program. As in the first embodiment, the personal computer 230 may calculate the effective area based on the imaging parameters, or may read effective area information from the effective area memory provided in the personal computer 230 based on the imaging parameters. However, the effective area information may be obtained from another device such as the server 94 via the communication unit.

At block 316, the personal computer 230 displays the selected frame image on the display 236 and a pair of guidelines 124 indicating the left and right edges of the effective area, as shown in FIG. 18, as in the first embodiment. do.
In block 318, the personal computer 230 identifies the background, which is an area with a constant luminance value without contrast, and the object, which is an area with contrast and a difference in luminance value, through image analysis, and recognizes the object as a depth measurement target. . Note that the depth measurement target may be selected by the user on the touch panel instead of being recognized based on contrast by image analysis.

At block 322, personal computer 230 determines whether the depth measurement target is located within the effective area. If the depth measurement target is not located within the valid area (NO determination at block 312), block 324 provides an alert that depth information cannot be measured for the frame. The alert may be one that prompts re-shooting. The alert may be displayed on the display 236, or may be output as a voice from an audio output section (not shown). Furthermore, the alert may be stored in association with the frame image. By doing so, when selecting frames for which depth information is to be calculated in block 212, it is possible to eliminate wasteful selection of frames once it has been determined that the measurement target is located outside the effective area. In addition, if information on which side of the effective area the depth measurement target is located outside of the effective area is stored together with the image as alert information, the frame in which the alert is stored is displayed by the camera 202 and the alert is also output when re-capturing. By doing so, it can also be used as a guide when deciding the composition.

Thereafter, at block 326, personal computer 230 determines whether another frame has been selected. If another frame is not selected, the operation ends. If another frame is selected, personal computer 230 executes the processing of blocks after block 208 again.

If the depth measurement object is located within the effective area (determination of YES at block 312), at block 328 the personal computer 230 calculates depth information for each pixel according to the depth calculation program and displays or stores the depth information. . As in the first embodiment, the depth information may be a distance image in which each pixel of the image is colored according to the distance, or may be in the form of a table in which the depth information (numerical value) for each pixel corresponds to the pixel.

According to the second embodiment, in addition to the effects of the first embodiment, there is an effect that it is not necessary to install the depth calculation program and the effective area notification program in the camera 202 which is an electronic device. These programs need only be installed in a second electronic device, such as a server or a personal computer, to which camera images are transferred, and can perform depth calculation and effective area notification at a higher speed than processing by individual cameras. .

[Application example]
Next, an application example of an electronic device that obtains depth information will be described.
The electronic device of the embodiment can be applied to a monitoring system that detects an intruder in a space captured by a camera and issues an alarm. In a monitoring system, for example, the flow of people and cars in a store, a parking lot, etc. is grasped, and the intrusion of a person into a specific area is monitored. Further, when it is detected that a person moves in front of the automatic door, the door may be opened.

The electronic device of the embodiment can also be applied to a mobile body control system. In this system, moving objects such as autonomously moving robots such as mobile robots (Automated Guided Vehicles), cleaning robots, and communication robots, vehicles including automobiles, flying objects such as drones and airplanes, and moving objects such as ships is imaged, and acceleration, deceleration, stopping, collision avoidance, turning, and operation of safety devices such as airbags are controlled. This system also includes a so-called front camera that images the front of the automobile and a so-called rear camera that images the rear when backing up. Furthermore, a camera may be installed at the tip of a stationary robot arm or the like to control the gripping of an object by the robot arm.

When applied to drones, when inspecting cracks, broken wires, etc. from the sky, the inspection target is photographed with a camera and the distance to the inspection target is detected. It is determined whether it is inside. Based on this detection result or determination result, the thrust of the drone is controlled so that the distance to the inspection target is constant. This allows the drone to fly parallel to the inspection target.

Also, when the drone is flying, the camera captures the direction of the ground, detects the height from the ground to the drone, or determines whether the height from the ground is in front of or behind the reference distance. be done. Based on this detection result or determination result, the thrust of the drone is controlled so that the height from the ground becomes the specified height. This allows the drone to fly at a specified height.

It should be noted that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying constituent elements without departing from the scope of the present invention at the implementation stage. Also, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components may be omitted from all components shown in the embodiments. Furthermore, constituent elements of different embodiments may be combined as appropriate.

42... Lens, 52... Filter, 54... Image sensor, 60... Display, 62... Communication unit, 64b... Depth calculation program, 64c... Effective area notification program.

Claims (14)

An image captured by a camera including a filter having a first region that transmits a first wavelength band and a second region that transmits a second wavelength band, wherein the second wavelength band is transmitted through the first region. input means for inputting a photographed image including a one-color component image and a second color component image based on the second wavelength band transmitted through the second region;
a processing means for outputting the photographed image in a state in which an effective area for depth information calculation based on bias of color information in the first color component image and the second color component image is identified,
The electronic device, wherein the processing means acquires or calculates the effective area based on a distance between an aperture of a lens of the camera and the filter, which is at least one shooting parameter among a plurality of shooting parameters. 2. The electronic device according to claim 1, wherein said depth information is obtained based on a shape of blur of said first color component image and a shape of blur of said second color component image. The processing means transmits to an external device an effective area acquisition request including a distance between the aperture of the lens of the camera and the filter, which is at least one of the plurality of imaging parameters, and performs the at least one imaging. 2. The electronic device according to claim 1, wherein said effective area according to a parameter is received from said external device. storing the effective area obtained from the result of simulating the optical characteristic distribution of the camera based on the distance between the aperture of the lens of the camera and the filter, which is at least one of the plurality of shooting parameters; 2. The electronic device according to claim 1, further comprising a memory for 5. The electronic device according to any one of claims 1 to 4, wherein said plurality of shooting parameters further include a sensor size of said camera, a focal length of said lens, or a brightness of said lens. 6. The filter according to any one of claims 1 to 5, wherein the filter is arranged at the front end of the lens of the camera or closer to the subject than the front end, or arranged at the rear end of the lens or closer to the image sensor than the rear end. electronics. 7. When an object for which depth information is to be obtained in the photographed image is specified, the processing means detects the direction of the edge component of the object and outputs information corresponding to the direction of the detected edge component. Electronics as described. 2. The processing means according to claim 1, wherein said processing means outputs guidance for photographing such that an object to be measured for which depth information is to be obtained in said photographed image is positioned within said effective area by at least one of voice, text, and video. Electronics. further specifying means for specifying an object to be measured for obtaining depth information in the photographed image displayed on the display unit by user operation, distribution of depth information within the effective area, or object recognition from the photographed image; The electronic device of claim 1, comprising: 2. The electronic device according to claim 1, wherein said processing means outputs guidance recommending re-imaging when determining that a measurement target for which depth information is to be obtained in said photographed image is not within said effective area. 2. The electronic device according to claim 1, wherein said processing means expresses a guideline indicating said effective area on said photographed image displayed on said display unit by at least one of color, brightness and contrast. further comprising a display unit for displaying the captured image output by the processing means;
2. The electronic device according to claim 1, wherein said processing means compensates for color shift of said photographed image according to said effective area, and causes said display unit to display the photographed image after compensation. further comprising a display unit for displaying the captured image output by the processing means;
2. The electronic device according to claim 1, wherein the processing means causes the display unit to display at least one of the effective area and the reliability of depth estimation used to obtain the effective area together with the captured image. An image taken by a camera including a filter having a first region that transmits a first wavelength band and a second region that transmits a second wavelength band, the first image based on the first wavelength band transmitted through the first region inputting a photographed image including a color component image and a second color component image based on a second wavelength band transmitted through the second region;
color in the first color component image and the second color component image based on the distance between the aperture of the lens of the camera and the filter, which is at least one shooting parameter among a plurality of shooting parameters; Outputting in a state in which an effective area for calculation of depth information based on information bias is identified;
A notification method comprising:

Images