WO2019078339A1

WO2019078339A1 - Electronic apparatus

Info

Publication number: WO2019078339A1
Application number: PCT/JP2018/038947
Authority: WO
Inventors: 明徳世; 佳孝宮谷; 法明幸塚
Original assignee: ソニー株式会社
Priority date: 2017-10-19
Filing date: 2018-10-19
Publication date: 2019-04-25
Also published as: JPWO2019078339A1; US11368609B2; US20200314306A1; EP3700182A1; KR20200062220A; CN111201771A; JP7247890B2; EP3700182A4; CN111201771B

Abstract

The present disclosure relates to an electronic apparatus that makes it possible to reduce the size of an electronic apparatus that functions to capture images of the surroundings of a user. An electronic apparatus that a user wears or uses, wherein the electronic apparatus comprises an imaging part: that is arranged in a position from which the surroundings of the user that is wearing or using the electronic apparatus appear; that receives, from a subject, incident light that does not enter through an imaging lens or a pinhole; and that comprises a plurality of pixel output units that output single detection signals that indicate output pixel values that have been modulated on the basis of the angle of incidence of the incident light. The present disclosure can be applied, for example, to wearable devices.

Description

Electronics

The present disclosure relates to an electronic device, and more particularly to an electronic device having a function of imaging the surroundings of a user.

Conventionally, light from a subject is modulated and imaged without using an imaging lens by a grating-like optical filter covering the light receiving surface of an imaging element or an optical filter consisting of a diffraction grating, and an image of the subject is formed by predetermined arithmetic processing. There has been proposed an imaging device which restores an imaged image (see, for example, Non-Patent Document 1, Patent Documents 1 and 2).

Japanese Patent Publication No. 2016-510910 gazette WO 2016/123529

By the way, the imaging device which does not use an imaging lens as shown by a nonpatent literature 1 and

patent documents

1 and 2 can be miniaturized as there is no imaging lens, and expansion of an application range is expected.

The present disclosure has been made in view of such a situation, and enables downsizing of an electronic device having a function of imaging the surroundings of a user.

An imaging apparatus according to one aspect of the present disclosure is an imaging unit disposed in a position where the surroundings of a user who is wearing or using the electronic device are photographed in an electronic device worn or used by the user, and the imaging lens and The imaging unit includes a plurality of pixel output units that receive incident light from an object incident without passing through any of the pinholes and output one detection signal indicating an output pixel value modulated by the incident angle of the incident light. .

In one aspect of the present disclosure, the surroundings of the user wearing or using the electronic device are imaged by a plurality of pixel output units, and a detection signal is output.

According to one aspect of the present disclosure, it is possible to miniaturize an electronic device having a function of capturing an image of a user's surroundings.

In addition, the effect described here is not necessarily limited, and may be any effect described in the present disclosure.

It is a figure explaining the principle of the imaging in the imaging device to which the art of this indication is applied. It is a block diagram showing an example of basic composition of an imaging device to which art of this indication is applied. It is a figure which shows the structural example of the pixel array part of the image pick-up element of FIG. It is a figure explaining the 1st structural example of the image pick-up element of FIG. It is a figure explaining the 2nd structural example of the image pick-up element of FIG. It is a figure explaining the principle of generation | occurrence | production of incident angle directivity. It is a figure explaining the change of the incident angle directivity using an on-chip lens. It is a figure which shows the example of the type of a light shielding film. It is a figure explaining design of incidence angle directivity. It is a figure explaining the difference between an on-chip lens and an imaging lens. It is a figure explaining the difference between an on-chip lens and an imaging lens. It is a figure explaining the difference between an on-chip lens and an imaging lens. It is a figure explaining the relationship between a to-be-photographed object distance and the coefficient which shows incident angle directivity. It is a figure explaining the relationship between a narrow view angle pixel and a wide view angle pixel. It is a figure explaining the relationship between a narrow view angle pixel and a wide view angle pixel. It is a figure explaining the relationship between a narrow view angle pixel and a wide view angle pixel. It is a figure for demonstrating the difference in the image quality of a narrow view angle pixel and a wide view angle pixel. It is a figure for demonstrating the difference in the image quality of a narrow view angle pixel and a wide view angle pixel. It is a figure explaining the example which combines the pixel of a several view angle. It is a flowchart explaining the imaging process by the imaging device of FIG. It is a figure explaining the reduction method of processing load. It is a figure explaining the reduction method of processing load. It is a figure explaining the reduction method of processing load. It is a figure explaining the reduction method of processing load. It is a figure explaining the reduction method of processing load. FIG. 21 is a block diagram illustrating a configuration example of an electronic device to which the technology of the present disclosure is applied. It is a figure which shows the example of arrangement | positioning of the image pick-up element in a wearable device. It is a figure which shows the example of the incident angle directivity of an image pick-up element. It is a figure which shows the example of arrangement | positioning of the image pick-up element in a camera. It is a figure which shows the example of arrangement | positioning of the image pick-up element in a head mounted display. It is a figure which shows the example of arrangement | positioning of the image pick-up element in PC. It is a flow chart explaining user imaging control processing. It is a figure which shows the example of arrangement | positioning of the image pick-up element in a wearable device. It is a figure which shows the example of arrangement | positioning of the image pick-up element in a camera. It is a figure which shows the example of arrangement | positioning of the image pick-up element in a head mounted display. It is a figure which shows the example of arrangement | positioning of the image pick-up element in an overhead type headphone. It is a figure which shows the example of arrangement | positioning of the image pick-up element in an overhead type headphone. It is a figure which shows the example of arrangement | positioning of the image pick-up element in the headphone of a neck band type | mold. It is a flow chart explaining user circumference imaging control processing. It is a figure explaining the response | compatibility method to the deformation | transformation of the shape of the installation position of an image pick-up element. It is a block diagram showing an example of composition of an information processing system to which art of this indication is applied. It is a figure which shows the modification of the image pick-up element of FIG. It is a figure explaining the modification of a pixel output unit. It is a figure which shows the modification of an image pick-up element. It is a figure which shows the modification of an image pick-up element. It is a figure which shows the modification of an image pick-up element. It is a figure which shows the modification of an image pick-up element. It is a figure which shows the modification of an image pick-up element. It is a figure which shows an example of a schematic structure of an endoscopic surgery system. It is a block diagram which shows an example of a function structure of the camera head shown in FIG. 49, and CCU. It is a block diagram showing an example of rough composition of an internal information acquisition system.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description will be appropriately omitted.

Also, the description will be made in the following order.
1. Overview of Imaging Device of Present Disclosure Basic configuration example of imaging device of the present disclosure 3. First Embodiment: Case of Imaging a User Second Embodiment: Case of Imaging around User Modifications 6. Application example 7. Other

<< 1. Overview of Imaging Device of the Present Disclosure >>
First, an overview of the imaging device of the present disclosure will be described.

In the imaging device of the present disclosure, as shown in the upper left of FIG. 1, the imaging device 51 in which the detection sensitivity of each pixel is made to have incident angle directivity is used. Here, to impart incident angle directivity to the detection sensitivity of each pixel means that the light receiving sensitivity characteristic according to the incident angle of incident light to each pixel is different for each pixel. However, the light receiving sensitivity characteristics of all the pixels do not have to be completely different, and the light receiving sensitivity characteristics of some pixels may be the same.

Here, for example, it is assumed that all objects are a set of point light sources, and light is emitted from each point light source in all directions. For example, the object plane 31 of the upper left object in FIG. 1 is composed of the point light source PA to the point light source PC, and the point light source PA to the point light source PC respectively emit a plurality of light rays of light intensity a to light intensity c around It shall be. In addition, hereinafter, the imaging device 51 includes pixels (hereinafter, referred to as pixels Pa to Pc) having different incident angle directivity at positions Pa to Pc.

In this case, as shown in the upper left of FIG. 1, rays of the same light intensity emitted from the same point light source are made incident on the respective pixels of the image sensor 51. For example, a light beam of light intensity a emitted from the point light source PA is incident on the pixels Pa to Pc of the imaging device 51, respectively. On the other hand, rays emitted from the same point light source are incident at different incident angles for each pixel. For example, light rays from the point light source PA are incident on the pixels Pa to Pc at different incident angles.

Here, since the incident angle directivity of the pixels Pa to Pc is different from each other, light rays of the same light intensity emitted from the same point light source are detected with different sensitivities in each pixel. As a result, rays of the same light intensity are detected at different detection signal levels for each pixel. For example, detection signal levels for light rays of light intensity a from the point light source PA have different values in the pixels Pa to Pc.

The light receiving sensitivity level of each pixel with respect to the light beam from each point light source is determined by multiplying the light intensity of the light beam by a coefficient indicating the light receiving sensitivity (that is, the incident angle directivity) with respect to the incident angle of the light beam. For example, the detection signal level of the pixel Pa with respect to the light beam from the point light source PA multiplies the light intensity a of the light beam of the point light source PA by a coefficient indicating the incident angle directivity of the pixel Pa with respect to the incident angle of the light beam to the pixel Pa It is determined by

Therefore, the detection signal levels DA, DB, DC of the pixels Pc, Pb, Pa are expressed by the following formulas (1) to (3), respectively.

DA = α1 × a + β1 × b + γ1 × c
... (1)
DB = α2 × a + β2 × b + γ2 × c
... (2)
DC = α3 × a + β3 × b + γ3 × c
... (3)

Here, the coefficient α1 is a coefficient indicating the incident angle directivity of the pixel Pc with respect to the incident angle of the ray from the point light source PA to the pixel Pc, and is set according to the incident angle. Further, α1 × a indicates the detection signal level of the pixel Pc with respect to the light beam from the point light source PA.

The coefficient β1 is a coefficient indicating the incident angle directivity of the pixel Pc with respect to the incident angle of a ray from the point light source PB to the pixel Pc, and is set according to the incident angle. Further, β1 × b represents the detection signal level of the pixel Pc with respect to the ray from the point light source PB.

The coefficient γ1 is a coefficient indicating the incident angle directivity of the pixel Pc with respect to the incident angle of the light beam from the point light source PC to the pixel Pc, and is set according to the incident angle. Further, γ1 × c indicates the detection signal level of the pixel Pc with respect to the ray from the point light source PC.

As described above, the detection signal level DA of the pixel Pa includes the light intensities a, b and c of the light rays from the point light sources PA, PB and PC at the pixel Pc, and the incident angle directivity according to the respective incident angles. It is obtained by product-sum with coefficients α1, β1, and γ1.

Similarly, the detection signal level DB of the pixel Pb is, as shown in equation (2), the respective light intensities a, b and c of the light rays from the point light sources PA, PB and PC in the pixel Pb It is obtained by the product-sum with the coefficients α2, β2, and γ2 indicating the incident angle directivity according to the angle. Further, the detection signal level DC of the pixel Pc is, as shown in the equation (3), the respective light intensities a, b and c of the light rays from the point light sources PA, PB and PC at the pixel Pa and the respective incident angles It is determined by product-sum with the coefficients α 2, β 2 and γ 2 indicating the incident angle directivity according to

However, the detection signal levels DA, DB, DC of the pixels Pa, Pb, Pc are light of the light beams emitted from the point light sources PA, PB, PC as shown in the equations (1) to (3). The strengths a, b and c are mixed. Therefore, as shown in the upper right of FIG. 1, the detection signal level in the imaging device 51 is different from the light intensity of each point light source on the object plane 31. Therefore, the image obtained by the imaging device 51 is different from that on which the image of the object plane 31 is formed.

On the other hand, by creating simultaneous equations of equations (1) to (3) and solving the created simultaneous equations, light intensities a to c of the rays of the point light sources PA to PC can be obtained. Then, as shown in the lower right of FIG. 1, pixels having pixel values corresponding to the determined light intensity a to light intensity c are arranged in accordance with the arrangement (relative position) of the point light source PA to the point light source PC. The restored image on which the image of the object plane 31 is formed is restored.

In the following, a set of coefficients (for example, coefficients α1, β1, γ1) for each of the equations constituting the simultaneous equations is referred to as a coefficient set. Also, hereinafter, a set of a plurality of coefficient sets corresponding to a plurality of equations included in the simultaneous equations (for example, coefficient set α1, β1, γ1, coefficient set α2, β2, γ2, coefficient set α3, β3, γ3) Is called a coefficient set group.

In this way, imaging with incident angle directivity is performed in each pixel without the need for an imaging lens, a pinhole, and the optical filters shown in Patent Document 1 and Non-Patent Document 1 (hereinafter referred to as Patent Document etc.) It is possible to realize an imaging device in which the element 51 is an essential component. As a result, since the imaging lens, the pinhole, and the optical filter described in the patent document and the like do not become an essential component, the height of the imaging device can be reduced, that is, the thickness in the incident direction of light in the configuration realizing the imaging function. It becomes possible to make it thinner.

In addition, since the essential configuration is only the imaging device 51, it is possible to improve the degree of freedom in design. For example, in an imaging apparatus using a conventional imaging lens, it is necessary to arrange the pixels of the imaging element in a two-dimensional array according to the position where the image of the subject is formed by the imaging lens. There is no need for that in an imaging device using. Therefore, the degree of freedom of arrangement of each pixel is improved, and for example, each pixel can be freely arranged in a range where light from a subject is incident. For example, it becomes possible to arrange the pixels in a circular area, arrange them in a hollow rectangular area, or distribute them in a plurality of areas.

Then, regardless of the arrangement of each pixel, it is expressed by the above-mentioned equations (1) to (3) using coefficients corresponding to the incident angles of light rays from each point light source on the object plane 31 to each pixel. By creating and solving such simultaneous equations, it is possible to determine the light intensity of the ray from each point light source. Then, the restored image on which the image of the object plane 31 is formed is restored by arranging the pixels having pixel values according to the light intensity of each point light source determined according to the arrangement of the point light sources on the object plane 31 can do.

<< 2. Basic Configuration Example of Imaging Device of the Present Disclosure >>
Next, with reference to FIGS. 2 to 25, a basic configuration example of the imaging device of the present disclosure will be described.

<Configuration Example of Imaging Device 101>
FIG. 2 is a block diagram showing a configuration example of an imaging device 101 which is a basic imaging device to which the technology of the present disclosure is applied.

The imaging device 101 includes an imaging element 121, a restoration unit 122, a control unit 123, an input unit 124, a detection unit 125, an association unit 126, a display unit 127, a storage unit 128, a recording and reproduction unit 129, a recording medium 130, and a communication unit. 131 is provided. Further, signal processing and imaging are performed by the restoration unit 122, the control unit 123, the input unit 124, the detection unit 125, the association unit 126, the display unit 127, the storage unit 128, the recording and reproducing unit 129, the recording medium 130, and the communication unit 131. A signal processing control unit 111 that controls the device 101 is configured. Note that the imaging device 101 does not include an imaging lens (imaging lens free).

The imaging device 121, the restoration unit 122, the control unit 123, the input unit 124, the detection unit 125, the association unit 126, the display unit 127, the storage unit 128, the recording and reproduction unit 129, and the communication unit 131 are connected via the bus B1. Are connected to each other, and transmit and receive data via the bus B1. In the following, in order to simplify the description, the description of the bus B1 in the case where each unit of the imaging apparatus 101 transmits and receives data via the bus B1 is omitted. For example, when the input unit 124 supplies data to the control unit 123 via the bus B1, it is described that the input unit 124 supplies data to the control unit 123.

The image pickup device 121 corresponds to the image pickup device 51 described with reference to FIG. 1 and includes a pixel having incident angle directivity, and an image formed of a detection signal indicating a detection signal level corresponding to the amount of incident light. Are output to the restoration unit 122 or the bus B1.

More specifically, the imaging device 121 may have the same basic structure as that of a general imaging device such as a complementary metal oxide semiconductor (CMOS) image sensor. However, the imaging device 121 has a configuration in which the incident angle directivity is given, as will be described later with reference to FIGS. ing. The image sensor 121 varies (changes) the light reception sensitivity in accordance with the incident angle of incident light for each pixel, and has incident angle directivity with respect to the incident angle of incident light in pixel units.

In addition, since the image output from the imaging element 121 is an image constituted by a detection signal in which the image of the subject is not formed as shown in the upper right of FIG. 1 described above, the subject can be visually recognized. Can not. That is, although the detection image which consists of a detection signal which image sensor 121 outputs is a set of pixel signals, it is an image which can not recognize a subject even if a user looks at a subject (it can not see a subject).

Therefore, in the following, as shown in the upper right of FIG. 1, an image composed of a detection signal on which the image of the subject is not formed, that is, an image captured by the imaging element 121 is referred to as a detected image. Do.

The imaging device 121 may not be configured as a pixel array, and may be configured as, for example, a line sensor. In addition, the incident angle directivity does not necessarily have to be all different for each pixel, and the incident angle directivity may include the same pixels.

The restoration unit 122 corresponds to the subject distance corresponding to the distance from the imaging device 51 to the subject plane 31 (the subject plane corresponding to the restored image) in FIG. 1, for example, and the coefficients α1 to α3, β1 to β3, γ1 described above. The coefficient set group corresponding to γ3 is acquired from the storage unit 128. In addition, the restoration unit 122 uses the detection signal level of each pixel of the detection image output from the imaging device 121 and the acquired coefficient set group to be represented by the above-described Equations (1) to (3). Make simultaneous equations. Then, the reconstruction unit 122 solves the created simultaneous equations to obtain the pixel values of the respective pixels constituting the image on which the image of the subject shown in the lower right of FIG. 1 is formed. As a result, an image from which the user can visually recognize and recognize the subject (and can visually recognize the subject) is restored from the detected image. Hereinafter, an image restored from the detected image will be referred to as a restored image. However, when the image sensor 121 is sensitive to only light other than visible wavelength bands such as ultraviolet light, the restored image is not an image that can identify the subject as a normal image, but also in this case the restored image It is called.

Further, hereinafter, a restored image which is an image in a state in which an image of a subject is formed, and an image before color separation such as demosaicing processing or synchronization processing is referred to as a RAW image The detected image is distinguished as an image according to the array of color filters but not a RAW image.

Note that the number of pixels of the imaging device 121 and the number of pixels of the pixels forming the restored image do not necessarily have to be the same.

Further, the restoration unit 122 performs demosaicing processing, γ correction, white balance adjustment, conversion processing to a predetermined compression format, and the like on the restored image as necessary. Then, the restoration unit 122 outputs the restored image to the bus B1.

The control unit 123 includes, for example, various processors, and controls each unit of the imaging apparatus 101.

The input unit 124 includes an input device (for example, a key, a switch, a button, a dial, a touch panel, a remote controller, and the like) for performing an operation of the imaging apparatus 101, an input of data used for processing, and the like. The input unit 124 outputs an operation signal, input data, and the like to the bus B1.

The detection unit 125 includes the imaging device 101 and various sensors used to detect the state of the subject and the like. For example, the detection unit 125 may be an acceleration sensor or a gyro sensor that detects the posture or movement of the imaging device 101, a position detection sensor (for example, a GNSS (Global Navigation Satellite System) receiver) that detects the position of the imaging device 101, an object A distance measuring sensor or the like for detecting a distance is provided. The detection unit 125 outputs a signal indicating the detection result to the bus B1.

The associating unit 126 associates the detection image obtained by the imaging device 121 with the metadata corresponding to the detection image. The metadata includes, for example, a coefficient set group for restoring a restored image using a target detection image, an object distance, and the like.

The method of associating the detection image with the metadata is not particularly limited as long as the correspondence between the detection image and the metadata can be specified. For example, metadata is added to image data including a detected image, the same ID is added to a detected image and metadata, or the detected image and metadata are recorded on the same recording medium 130. Metadata is associated.

The display unit 127 is, for example, a display, and displays various types of information (for example, a restored image and the like). Note that the display unit 127 can also be provided with an audio output unit such as a speaker to output audio.

The storage unit 128 includes one or more storage devices such as a read only memory (ROM), a random access memory (RAM), and a flash memory, and stores, for example, programs and data used for processing of the imaging device 101. For example, the storage unit 128 stores coefficient set groups corresponding to the above-described coefficients α1 to α3, β1 to β3, and γ1 to γ3 in association with various object distances. More specifically, for example, the storage unit 128 stores, for each subject plane 31 at each subject distance, a coefficient set group including a coefficient for each pixel 121a of the image sensor 121 for each point light source set on the subject plane 31. doing.

The recording and reproduction unit 129 performs recording of data on the recording medium 130 and reproduction (reading) of data recorded on the recording medium 130. For example, the recording / reproducing unit 129 records the restored image on the recording medium 130 or reads the restored image from the recording medium 130. Also, for example, the recording / reproducing unit 129 records the detection image and the corresponding metadata on the recording medium 130 or reads out from the recording medium 130.

The recording medium 130 is made of, for example, any of a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or a combination thereof.

The communication unit 131 communicates with another device (for example, another imaging device, a signal processing device, and the like) by a predetermined communication method. The communication method of the communication unit 131 may be wired or wireless. Further, the communication unit 131 can also correspond to a plurality of communication methods.

<First Configuration Example of Imaging Element 121>
Next, with reference to FIG. 3 and FIG. 4, a first configuration example of the imaging element 121 of the imaging device 101 of FIG. 2 will be described.

FIG. 3 shows a front view of a part of the pixel array portion of the imaging element 121. FIG. Although FIG. 3 shows an example in which the number of pixels in the pixel array portion is 6 vertical pixels × 6 horizontal pixels, the number of pixels in the pixel array portion is not limited to this.

In the image sensor 121 of FIG. 3, a light shielding film 121b, which is one of the modulation elements, is provided for each pixel 121a so as to cover a part of the light receiving area (light receiving surface) of the photodiode. Incident incident light is optically modulated according to the incident angle. Then, for example, by providing the light shielding film 121b in a different range for each pixel 121a, the light receiving sensitivity with respect to the incident angle of incident light differs for each pixel 121a, and each pixel 121a has different incident angle directivity .

For example, the light shielding film 121b-1 and the light shielding film 121b-2 provided in the pixel 121a-1 and the pixel 121a-2 have different ranges for shielding the light receiving region of the photodiode (a light shielding region (position), And at least one of the shaded areas). That is, in the pixel 121a-1, the light shielding film 121b-1 is provided so as to shield a part of the left side of the light receiving region of the photodiode by a predetermined width. On the other hand, in the pixel 121a-2, a light shielding film 121b-2 is provided so as to shield a part of the right side of the light receiving region by a predetermined width. The width at which the light shielding film 121b-1 shields the light receiving area of the photodiode may be different from or the same as the width at which the light shielding film 121b-2 shields the light receiving area of the photodiode. . Similarly, in the other pixels 121a, the light shielding films 121b are randomly arranged in the pixel array so as to shield a different range of the light receiving area for each pixel.

The amount of light that can be received by the photodiode decreases as the rate at which the light shielding film 121b obscures the light receiving area of each pixel increases. Therefore, it is desirable that the area of the light shielding film 121b be such an area that a desired light quantity can be secured, and for example, a restriction of up to about 3/4 of the light receiving area may be added. By doing this, it is possible to secure a light amount of a desired amount or more. However, it is possible to receive a minimum amount of light if each pixel is provided with an unshielded range having a width corresponding to the wavelength of light to be received. That is, for example, in the case of the B pixel (blue pixel), the wavelength is about 500 nm, but it is possible to receive the minimum light amount unless the light is blocked beyond the width corresponding to this wavelength.

The upper part of FIG. 4 is a side cross-sectional view of the first configuration example of the imaging device 121, and the middle part of FIG. 4 is a top view of the first configuration example of the imaging device 121. Further, the side cross-sectional view of the upper stage of FIG. 4 is an AB cross section in the middle stage of FIG. 4. Furthermore, the lower part of FIG. 4 is a circuit configuration example of the imaging element 121.

In the imaging element 121 in the upper part of FIG. 4, incident light is incident from the top to the bottom in the drawing. The adjacent pixels 121a-1 and 121a-2 are so-called back side illumination type in which the wiring layer Z12 is provided in the lowermost layer in the drawing and the photoelectric conversion layer Z11 is provided thereon.

When it is not necessary to distinguish the pixels 121a-1 and 121a-2, the description of the numbers at the end of the reference numerals is omitted and simply referred to as the pixel 121a. Hereinafter, in the description, the numbers at the end of the reference numerals may be omitted in the same manner for the other configurations.

Further, FIG. 4 shows only the side view and the top view of two pixels constituting the pixel array of the image sensor 121, and needless to say, although more pixels 121a are arranged, Is omitted.

Further, the pixels 121a-1 and 121a-2 respectively include photodiodes 121e-1 and 121e-2 in the photoelectric conversion layer Z11. In addition, on-chip lenses 121c-1 and 121c-2 and color filters 121d-1 and 121d-2 are stacked on the photodiodes 121e-1 and 121e-2, respectively.

The on-chip lenses 121c-1 and 121c-2 condense incident light on the photodiodes 121e-1 and 121e-2.

The color filters 121d-1 and 121d-2 are optical filters that transmit light of a specific wavelength, such as red, green, blue, infrared, and white. In the case of white, the color filters 121d-1 and 121d-2 may or may not be transparent filters.

Light shielding films 121g-1 to 121g-3 are formed at the boundaries between the pixels in the photoelectric conversion layer Z11 of the pixels 121a-1 and 121a-2, and for example, as shown in FIG. L is incident on an adjacent pixel to suppress the occurrence of crosstalk.

Further, as shown in the upper and middle parts of FIG. 4, the light shielding films 121b-1 and 121b-2 shield a part of the light receiving surface S as viewed from the top. In the light receiving surfaces S of the photodiodes 121e-1 and 121e-2 in the pixels 121a-1 and 121a-2, different ranges are shielded by the light shielding films 121b-1 and 121b-2, respectively, whereby different incident angles are obtained. The directivity is set independently for each pixel. However, the light shielding range does not have to be different for all the pixels 121 a of the imaging device 121, and some pixels 121 a may have the same light shielding region.

As shown in the upper part of FIG. 4, the light shielding film 121b-1 and the light shielding film 121g-1 are connected to each other, and are L-shaped when viewed from the side. Similarly, the light shielding film 121b-2 and the light shielding film 121g-2 are connected to each other, and are configured in an L shape when viewed from the side. The light shielding film 121b-1, the light shielding film 121b-2, and the light shielding films 121g-1 to 121g-3 are made of metal, and for example, tungsten (W), aluminum (Al), or Al and copper Composed of an alloy with (Cu). The light shielding film 121b-1, the light shielding film 121b-2, and the light shielding films 121g-1 to 121g-3 are simultaneously formed of the same metal as the wiring in the same process as the wiring in the semiconductor process. It may be done. The film thicknesses of the light shielding film 121b-1, the light shielding film 121b-2, and the light shielding films 121g-1 to 121g-3 may not be the same depending on the position.

Further, as shown in the lower part of FIG. 4, the pixel 121 a includes a photodiode 161 (corresponding to the photodiode 121 e), a transfer transistor 162, an FD (Floating Diffusion: floating diffusion) portion 163, a selection transistor 164, and an amplification transistor 165. And a reset transistor 166, and is connected to the current source 168 via the vertical signal line 167.

The anode electrode of the photodiode 161 is grounded, and the cathode electrode is connected to the gate electrode of the amplification transistor 165 via the transfer transistor 162.

The transfer transistor 162 is driven according to the transfer signal TG. For example, when the transfer signal TG supplied to the gate electrode of the transfer transistor 162 becomes high level, the transfer transistor 162 is turned on. Thereby, the charge accumulated in the photodiode 161 is transferred to the FD unit 163 via the transfer transistor 162.

The amplification transistor 165 is an input portion of a source follower which is a readout circuit for reading out a signal obtained by photoelectric conversion in the photodiode 161, and a pixel signal of a level corresponding to the charge stored in the FD portion 163 is a vertical signal line 167. Output to That is, the amplification transistor 165 has a drain terminal connected to the power supply VDD and a source terminal connected to the vertical signal line 167 via the selection transistor 164, thereby connecting the current source 168 connected to one end of the vertical signal line 167. Configure a source follower.

The FD unit 163 is a floating diffusion region having a charge capacity C1 provided between the transfer transistor 162 and the amplification transistor 165, and temporarily accumulates the charge transferred from the photodiode 161 via the transfer transistor 162. The FD unit 163 is a charge detection unit that converts a charge into a voltage, and the charge stored in the FD unit 163 is converted into a voltage by the amplification transistor 165.

The selection transistor 164 is driven according to the selection signal SEL, and is turned on when the selection signal SEL supplied to the gate electrode becomes high level, and connects the amplification transistor 165 and the vertical signal line 167.

The reset transistor 166 is driven according to the reset signal RST. For example, the reset transistor 166 is turned on when the reset signal RST supplied to the gate electrode becomes high level, discharges the charge stored in the FD unit 163 to the power supply VDD, and resets the FD unit 163.

For example, the pixel circuit shown in the lower part of FIG. 4 operates as follows.

That is, as a first operation, the reset transistor 166 and the transfer transistor 162 are turned on, the charge stored in the FD unit 163 is discharged to the power supply VDD, and the FD unit 163 is reset.

As a second operation, the reset transistor 166 and the transfer transistor 162 are turned off, and during the exposure period, the photodiode 161 accumulates a charge according to the amount of incident light.

As a third operation, the reset transistor 166 is turned on and the FD section 163 is reset, and then the reset transistor 166 is turned off. By this operation, the FD unit 163 is set to the reference potential.

As a fourth operation, the potential of the FD section 163 in a reset state is output from the amplification transistor 165 as a reference potential.

As a fifth operation, the transfer transistor 162 is turned on, and the charge accumulated in the photodiode 161 is transferred to the FD portion 163.

As a sixth operation, the potential of the FD section 163 to which the charge of the photodiode is transferred is output from the amplification transistor 165 as a signal potential.

Then, a signal obtained by subtracting the reference potential from the signal potential by CDS (correlated double sampling) is output as a detection signal (pixel signal) of the pixel 121a. The value (output pixel value) of the detection signal is modulated according to the incident angle of incident light from the subject, and the characteristic (directivity) differs depending on the incident angle (having incident angle directivity).

<Second Configuration Example of Imaging Element 121>
FIG. 5 is a diagram showing a second configuration example of the imaging device 121. As shown in FIG. The upper side of FIG. 5 shows a side cross-sectional view of the pixel 121a of the imaging device 121 which is the second configuration example, and the middle stage of FIG. 5 shows a top view of the imaging device 121. Further, the side cross-sectional view of the upper stage of FIG. Furthermore, the lower part of FIG. 5 is a circuit configuration example of the imaging element 121.

In the imaging device 121 of FIG. 5, four photodiodes 121f-1 to 121f-4 are formed in one pixel 121a, and a light shielding film 121g is formed in a region separating the photodiodes 121f-1 to 121f-4. The configuration is different from that of the imaging device 121 of FIG. That is, in the imaging element 121 of FIG. 5, the light shielding film 121 g is formed in a “+” shape when viewed from the upper surface. In addition, about the structure common to them, the code | symbol same as FIG. 4 is attached | subjected, and detailed description is abbreviate | omitted.

In the image pickup device 121 of FIG. 5, the photodiodes 121 f-1 to 121 f-4 are separated by the light shielding film 121 g, so that electrical and optical crosstalk between the photodiodes 121 f-1 to 121 f-4 occurs. It is prevented. That is, the light shielding film 121g of FIG. 5 is for preventing crosstalk similarly to the light shielding film 121g of the imaging device 121 of FIG. 4, and is not for providing incident angle directivity.

Further, in the imaging device 121 of FIG. 5, one FD unit 163 is shared by the four photodiodes 121f-1 to 121f-4. The lower part of FIG. 5 shows an example of a circuit configuration in which one FD portion 163 is shared by four photodiodes 121f-1 to 121f-4. In the lower part of FIG. 5, the description of the same configuration as the lower part of FIG. 4 is omitted.

The lower part of FIG. 5 differs from the circuit configuration of the lower part of FIG. 4 in that photodiodes 161-1 to 161- are used instead of the photodiode 161 (corresponding to the photodiode 121e in the upper part of FIG. 4) and the transfer transistor 162. 4 (corresponding to the photodiodes 121f-1 to 121f-4 in the upper part of FIG. 5) and the transfer transistors 162-1 to 162-4 and share the FD section 163.

With such a configuration, the charge accumulated in the photodiodes 121f-1 to 121f-4 has a predetermined capacitance provided at the connection portion between the photodiodes 121f-1 to 121f-4 and the gate electrode of the amplification transistor 165. It is transferred to the common FD unit 163. Then, a signal corresponding to the level of the charge held in the FD unit 163 is read as a detection signal (pixel signal) (however, the CDS processing is performed as described above).

Therefore, the charges accumulated in the photodiodes 121f-1 to 121f-4 can be selectively contributed to the output of the pixel 121a, that is, the detection signal in various combinations. That is, the photodiodes 121f-1 to 121f-4 are configured to be able to read out the charge independently, and the photodiodes 121f-1 to 121f-4 (the photodiodes 121f-1 to 121f-4 output Different incident angle directivity can be obtained.

For example, by transferring the electric charges of the photodiode 121f-1 and the photodiode 121f-3 to the FD portion 163 and adding the signals obtained by reading out each, the incident angle directivity in the left-right direction can be obtained. Similarly, by transferring the electric charges of the photodiode 121f-1 and the photodiode 121f-2 to the FD portion 163 and adding the signals obtained by reading the respective, it is possible to obtain the incident angle directivity in the vertical direction.

Further, a signal obtained based on the charges selectively read out independently from the four photodiodes 121f-1 to 121f-4 is a detection signal corresponding to one pixel constituting the detection image.

The contribution of (the charge of) each photodiode 121f to the detection signal is not limited to, for example, whether or not the charge (the detected value) of each photodiode 121f is transferred to the FD unit 163, but also using an electronic shutter function. This can also be realized by resetting the charge accumulated in the photodiode 121 f before transfer to the FD unit 163. For example, if the charge of the photodiode 121f is reset immediately before transfer to the FD portion 163, the photodiode 121f does not contribute to the detection signal at all. On the other hand, by giving time to the charge of the photodiode 121f between the reset and the transfer of the charge to the FD portion 163, the photodiode 121f partially contributes to the detection signal.

As described above, in the case of the imaging device 121 of FIG. 5, the combination of the four photodiodes 121 f-1 to 121 f-4 used for the detection signal is changed to have different incident angle directivity for each pixel. You can The detection signal output from each pixel 121a of the image sensor 121 in FIG. 5 has a value (output pixel value) modulated according to the incident angle of the incident light from the subject, and the characteristic (directivity) according to the incident angle Are different (having incident angle directivity).

Hereinafter, a unit that outputs a detection signal corresponding to one pixel of a detected image is referred to as a pixel output unit. The pixel output unit includes at least one or more photodiodes, and generally, each pixel 121a of the imaging device 121 corresponds to one pixel output unit.

For example, in the imaging device 121 of FIG. 4, since one photodiode 121 e is provided for each pixel 121 a, one pixel output unit includes one photodiode 121 e. In other words, one pixel 121e constitutes one pixel output unit.

Then, by making the light shielding state by the light shielding film 121b of each pixel 121a different, the incident angle directivity of each pixel output unit can be made different. Then, in the imaging device 121 of FIG. 4, incident light to each pixel 121a is optically modulated using the light shielding film 121b, and as a result, the incident angle is directed by the signal output from the photodiode 121e of each pixel 121a. A detection signal for one pixel of the detected image reflecting the polarity is obtained. That is, the image pickup device 121 of FIG. 4 includes a plurality of pixel output units for receiving incident light from an object incident without any of the imaging lens and the pinhole, and each pixel output unit includes one photodiode 121e. The characteristic (incident angle directivity) with respect to the incident angle of incident light from the subject is set for each pixel output unit.

On the other hand, in the image pickup device 121 of FIG. 5, since four photodiodes 121f-1 to 121f-4 are provided in one pixel 121a, one pixel output unit includes four photodiodes 121e. Become. In other words, one pixel output unit is configured by the four photodiodes 121 f. On the other hand, individual pixel output units are not configured by each photodiode 121e alone.

Then, among the four photodiodes 121f-1 to 121f-4 as described above, by making the photodiode 121f contributing to the detection signal different for each pixel 121a, the incident angle directivity for each pixel output unit is different. It becomes a thing. That is, in the imaging device 121 of FIG. 5, the range not contributing to the output (detection signal) among the four photodiodes 121f-1 to 121f-4 functions in the same manner as the light-shielded region. Then, by combining the signals output from the photodiodes 121f-1 to 121f-4, a detection signal for one pixel of the detection image reflecting the incident angle directivity is obtained. That is, the imaging device 121 of FIG. 5 includes a plurality of pixel output units for receiving incident light from an object incident without any of the imaging lens and the pinhole, and each pixel output unit includes a plurality of photodiodes (for example, Characteristics (incident angle directivity) of each pixel output unit with respect to the incident angle of incident light from the subject by providing the photodiodes 121f-1 to 121f-4) and varying (the degree of) the photodiodes contributing to the output Are different from each other.

In the image sensor 121 of FIG. 5, incident light is incident on all the photodiodes 121f-1 to 121f-4 without being optically modulated, so that the detection signal is a signal obtained by optical modulation. Absent. Further, hereinafter, the photodiode 121 f not contributing to the detection signal is also referred to as a photodiode 121 f not contributing to a pixel output unit or an output.

In FIG. 5, the light receiving surface of the pixel output unit (pixel 121a) is equally divided into four, and photodiodes 121f having the same size as the light receiving surface are arranged in each region, that is, four photodiodes, etc. Although the divided example is shown, the division number and division position of the photodiode can be set arbitrarily.

For example, the photodiodes do not necessarily have to be equally divided, and the division positions of the photodiodes may be different for each pixel output unit. As a result, for example, even when the photodiodes 121 f at the same position contribute to the output among the plurality of pixel output units, the incident angle directivity differs between the pixel output units. Also, for example, by setting the number of divisions to be different between pixel output units, it is possible to more freely set the incident angle directivity. Furthermore, for example, both the division number and the division position may be made different between pixel output units.

Further, each of the image sensor 121 of FIG. 4 and the image sensor 121 of FIG. 5 has a configuration in which each pixel output unit can independently set the incident angle directivity. On the other hand, in the imaging device shown by the above-mentioned nonpatent literature 1 and

patent documents

1 and 2, each pixel output unit of an image sensor does not have the composition which can set up incidence angle directivity independently. In the image sensor 121 of FIG. 4, the incident angle directivity of each pixel output unit is set by the light shielding film 121 b at the time of manufacture. On the other hand, in the image sensor 121 of FIG. 5, the division number and division position of the photodiode of each pixel output unit are set at the time of manufacture, but the incident angle directivity of each pixel output unit (combination of photodiodes contributing to output) Can be set at the time of use (for example, at the time of imaging). In addition, in any of the imaging device 121 of FIG. 4 and the imaging device 121 of FIG. 5, it is not necessary to have a configuration in which all the pixel output units have incident angle directivity.

As described above, normally, each pixel of the imaging device corresponds to one pixel output unit, but as described later, there may be a case where one pixel output unit is formed by a plurality of pixels. . The following description will be made assuming that each pixel of the imaging device corresponds to one pixel output unit unless otherwise noted.

<Principle of generating incident angle directivity>
The incident angle directivity of each pixel of the imaging element 121 is generated, for example, according to the principle shown in FIG. The upper left part and the upper right part of FIG. 6 are diagrams for explaining the generation principle of the incident angle directivity in the imaging element 121 of FIG. 4, and the lower left part and the lower right part of FIG. 6 are for the imaging element 121 of FIG. It is a figure explaining the generation | occurrence | production principle of incident angle directivity.

Each of the upper left and upper right pixels in FIG. 6 includes one photodiode 121 e. On the other hand, the lower left and lower right pixels in FIG. 6 each include two photodiodes 121 f. Although an example in which one pixel includes two photodiodes 121 f is shown here, this is for convenience of explanation, and the number of photodiodes 121 f provided in one pixel may be another number. .

In the upper left pixel in FIG. 6, a light shielding film 121b-11 is formed to shield the right half of the light receiving surface of the photodiode 121e-11. Further, in the pixel at the upper right portion of FIG. 6, a light shielding film 121b-12 is formed so as to shield the left half of the light receiving surface of the photodiode 121e-12. Note that the alternate long and short dash line in the drawing is an auxiliary line which passes through the horizontal center of the light receiving surface of the photodiode 121e and is perpendicular to the light receiving surface.

For example, in the pixel at the upper left of FIG. 6, incident light from the upper right direction forming the incident angle θ1 with respect to the alternate long and short dash line in the drawing is left unshielded by the light shielding film 121b-11 of the photodiode 121e-11. It is easy to receive light by the half range. On the other hand, incident light from the upper left direction forming the incident angle θ2 with respect to the alternate long and short dash line in the figure is hard to be received by the left half range not blocked by the light shielding film 121b-11 of the photodiode 121e-11. . Therefore, the pixel in the upper left part of FIG. 6 has high incidence sensitivity with respect to incident light from the upper right in the figure and low sensitivity to incident light from the upper left. .

On the other hand, for example, in the pixel in the upper right part of FIG. 6, incident light from the upper right direction forming the incident angle θ1 is received by the left half range shielded by the light shielding film 121b-12 of the photodiode 121e-12. Hateful. On the other hand, incident light from the upper left direction forming the incident angle θ2 is likely to be received by the right half range which is not blocked by the light shielding film 121b-12 of the photodiode 121e-12. Therefore, the pixel in the upper right part of FIG. 6 has low incident sensitivity to incident light from the upper right in the figure and high incident sensitivity to incident light from the upper left. .

Further, in the pixel at the lower left part of FIG. 6, photodiodes 121f-11 and 121f-12 are provided on the left and right in the drawing, and a light shielding film 121b is provided by reading out one of the detection signals. It is set as the structure which has incident angle directivity.

That is, by reading out only the signal of the photodiode 121f-11 provided on the left side in the lower left part of FIG. 6, incident angle directivity similar to that of the upper left part in FIG. You can get it. That is, incident light from the upper right direction forming the incident angle θ1 with respect to the alternate long and short dash line in the figure is incident on the photodiode 121f-11 and a signal corresponding to the amount of light received is read out from the photodiode 121f-11. It contributes to the detection signal output from the pixel. On the other hand, incident light from the upper left direction forming the incident angle θ2 with respect to the alternate long and short dash line in the figure is incident on the photodiode 121f-12 but is not read out from the photodiode 121f-12. It does not contribute to the output detection signal.

Similarly, when two photodiodes 121f-13 and 121f-14 are provided as in the pixel at the lower right part of FIG. 6, only the signal of the photodiode 121f-14 provided on the right side in the drawing is read out By doing this, the incident angle directivity similar to that of the pixel in the upper right part of FIG. 6 can be obtained. That is, incident light from the upper right direction forming the incident angle θ1 is incident on the photodiode 121f-13, but since no signal is read out from the photodiode 121f-13, it does not contribute to the detection signal output from this pixel. On the other hand, incident light from the upper left direction forming the incident angle θ2 is incident on the photodiode 121f-14, and a signal corresponding to the amount of light received is read out from the photodiode 121f-14, so that it is output from this pixel Contribute to the detection signal.

In the pixel in the upper part of FIG. 6, an example is shown in which the range shielded from light and the range not shielded from light are separated at the center position in the horizontal direction of the pixel (the light receiving surface of the photodiode 121e). May be Further, in the lower pixel in FIG. 6, an example in which the two photodiodes 121 f are separated at the center position in the horizontal direction of the pixel is shown, but may be separated at other positions. Thus, different incident angle directivity can be produced by changing the light shielding range or the position where the photodiode 121f is divided.

<Incidence angle directivity in a configuration including an on-chip lens>
Next, referring to FIG. 7, the incident angle directivity in the configuration including the on-chip lens 121 c will be described.

The upper graph in FIG. 7 shows the incident angle directivity of the middle and lower pixels in FIG. The horizontal axis represents the incident angle θ, and the vertical axis represents the detection signal level. Note that the incident angle θ is 0 degrees when the direction of incident light coincides with the alternate long and short dash line in the middle left of FIG. 7, and the middle left incident angle θ21 in FIG. 7 is a positive direction. The right incident angle θ22 side is a negative direction. Therefore, with respect to the on-chip lens 121c, the incident angle of the incident light entering from the upper right is larger than the incident angle of the incident light entering from the upper left. That is, the incident angle θ becomes larger as the traveling direction of the incident light leans to the left (larger in the positive direction) and smaller as it leans to the right (larger in the negative direction).

Also, the pixels in the middle left part of FIG. 7 are the on-chip lens 121 c-11 for condensing incident light and the color filters 121 d-11 for transmitting light of a predetermined wavelength to the pixels in the upper left part of FIG. Is added. That is, in this pixel, the on-chip lens 121c-11, the color filter 121d-11, the light shielding film 121b-11, and the photodiode 121e-11 are laminated in order from the incident direction of the upper light in the drawing.

Similarly, the pixel at the middle right of FIG. 7, the pixel at the lower left of FIG. 7, and the pixel at the lower right of FIG. 7 are respectively the pixel at the upper right of FIG. 6 and the lower left of FIG. The on-chip lens 121 c-11 and the color filter 121 d-11 or the on-chip lens 121 c-12 and the color filter 121 d-12 are added to the pixel of FIG.

In the pixel at the center left part of FIG. 7, as shown by the solid line waveform in the upper part of FIG. 7, the detection signal level (light receiving sensitivity) of the photodiode 121e-11 changes according to the incident angle θ of incident light. That is, the larger the incident angle θ which is the angle formed by the incident light with respect to the alternate long and short dash line in the figure (the larger the incident angle θ in the positive direction (the more inclined in the right direction in the figure)) When the light is collected in the range in which 11 is not provided, the detection signal level of the photodiode 121e-11 is increased. Conversely, the smaller the incident angle θ of the incident light (the larger the incident angle θ in the negative direction (the more inclined in the left direction in the figure)), the light is collected in the range in which the light shielding film 121b-11 is provided. By being illuminated, the detection signal level of the photodiode 121e-11 decreases.

Further, in the pixel at the middle right part of FIG. 7, as indicated by the dotted line waveform in the upper part of FIG. 7, the detection signal level (light receiving sensitivity) of the photodiode 121e-12 changes according to the incident angle θ of incident light. Do. That is, as the incident angle θ of the incident light is larger (the incident angle θ is larger in the positive direction), the light is condensed in the range in which the light shielding film 121b-12 is provided, thereby the photodiode 121e-12. The detection signal level of Conversely, as the incident angle θ of incident light is smaller (as the incident angle θ is larger in the negative direction), light is incident in a range in which the light shielding film 121b-12 is not provided. The detection signal level increases.

The waveforms of the solid line and the dotted line shown in the upper part of FIG. 7 can be changed according to the range of the light shielding film 121 b. Therefore, it is possible to give different incident angle directivity to each other in pixel units by the range of the light shielding film 121b.

As described above, the incident angle directivity is a characteristic of the light receiving sensitivity of each pixel according to the incident angle θ, but this is a characteristic of the light blocking value according to the incident angle θ in the middle pixel of FIG. It can be said that That is, although the light shielding film 121 b blocks incident light in a specific direction at a high level, incident light from other directions can not be blocked sufficiently. The change of the light shieldable level produces different detection signal levels according to the incident angle θ as shown in the upper part of FIG. Therefore, if the direction in which light can be blocked at the highest level in each pixel is defined as the light blocking direction of each pixel, having different incident angle directivity in pixel units means, in other words, light blocking mutually different in pixel units It means that it has a direction.

Also, in the lower left pixel in FIG. 7, as in the lower left pixel in FIG. 6, the middle left pixel in FIG. It is possible to obtain incident angle directivity similar to that of the pixels of a part. That is, when the incident angle θ of the incident light increases (when the incident angle θ increases in the positive direction), the light is collected in the range of the photodiode 121f-11 from which the signal is read, and the detection signal level is large. Become. Conversely, the smaller the incident angle θ of the incident light (the larger the incident angle θ in the negative direction), the more light is collected in the range of the photodiode 121 f-12 from which the signal is not read out, and thus the detection signal level Becomes smaller.

Similarly, in the lower right pixel in FIG. 7, as in the lower right pixel in FIG. 6, the signal of only the photodiode 121 f-14 in the right in FIG. An incident angle directivity similar to that of the pixel in the middle right part of can be obtained. That is, when the incident angle θ of incident light increases (when the incident angle θ increases in the positive direction), light is collected in the range of the photodiode 121 f-13 that does not contribute to the output (detection signal). The level of the unit detection signal decreases. Conversely, the smaller the incident angle θ of the incident light (the larger the incident angle θ in the negative direction), the more light is collected in the range of the photodiode 121f-14 that contributes to the output (detection signal), The level of the detection signal in units of pixels is increased.

Note that as in the lower pixel in FIG. 7, a plurality of photodiodes are provided in the pixel, and in the pixels that can change the photodiodes contributing to the output, each photodiode has directivity with respect to the incident angle of incident light The on-chip lens 121c is an essential component of each pixel in order to generate incident angle directivity in pixel units.

As for the incident angle directivity, it is desirable that the randomness be high in pixel units. For example, when the same incident angle directivity is provided between adjacent pixels, the above-described equations (1) to (3) or equations (4) to (6) described later may be the same as each other. As a result, the number of equations is insufficient for the unknowns that are solutions of simultaneous equations, which may make it impossible to obtain the pixel values that make up the restored image.

In the following description, as in the case of the pixel 121a in FIG. 4, an example in the case of using the pixel 121a which realizes the incident angle directivity using the light shielding film 121b will be mainly described. However, except for the case where the light shielding film 121b is essential, it is also possible to basically use the pixel 121a which divides the photodiode to realize the incident angle directivity.

<About the composition of the light shielding film>
In the above, as shown in FIG. 3, as the configuration of the light shielding film 121b of each pixel 121a of the imaging element 121, the entire light receiving surface is shielded in the vertical direction, and the light shielding width and position in the horizontal direction are changed. Although an example is shown, naturally, the entire light receiving surface is shielded in the horizontal direction, and the width (height) and the position in the vertical direction are changed, so that each pixel 121a has incident angle directivity. You may make it

In the following, as in the example of FIG. 3, the light shielding film 121b that shields the entire light receiving surface of the pixel 121a in the vertical direction and shields the light receiving surface with a predetermined width in the horizontal direction It is called a band type light shielding film 121b. A light shielding film 121b that shields the entire light receiving surface of the pixel 121a in the horizontal direction and shields the light receiving surface at a predetermined height in the vertical direction is referred to as a vertical band type light shielding film 121b.

In addition, as shown in the left part of FIG. 8, combining the vertical band type and horizontal band type light shielding films 121 b, for example, an L-shaped light shielding film 121 b is formed for each pixel of the Bayer arrangement. It may be provided.

In addition, in FIG. 8, the black range represents the light shielding film 121b, and the same applies to the following drawings unless otherwise noted. Further, in the example of FIG. 8, the pixels 121a-21 and 121a-24 of G (green) pixels, the pixels 121a-22 of R (red) pixels, and the pixels 121a-23 of B (blue) pixels, which are in the Bayer array, are provided. L-shaped light shielding films 121b-21 to 121b-24 are provided for each of them.

In this case, each pixel 121a has incident angle directivity as shown in the right part of FIG. That is, in the right part of FIG. 8, the distribution of the light receiving sensitivity of each pixel 121a is shown, the horizontal axis represents the incident angle θx in the horizontal direction (x direction) of the incident light, and the vertical axis is the vertical of the incident light The incident angle θy in the direction (y direction) is represented. The light receiving sensitivity in the range C4 is higher than the outside of the range C4, the light receiving sensitivity in the range C3 is higher than the outside of the range C3, and the light receiving sensitivity in the range C2 is higher than the outside of the range C2. The light reception sensitivity in the range C1 is higher than the outside of the range C1.

Therefore, in each pixel 121a, the detection signal level for the incident light in which the incident angle θx in the horizontal direction (x direction) and the incident angle θy in the vertical direction (y direction) fall within the range C1 is the highest. Then, the detection signal level decreases in the order of incident light in which the incident angle θx and the incident angle θy fall within the range C2, within the range C3, within the range C4, and outside the range C4. The intensity distribution of the light receiving sensitivity shown in the right part of FIG. 8 is determined by the range shielded by the light shielding film 121b in each pixel 121a regardless of the Bayer arrangement.

In the following, like the L-shaped light shielding films 121 b-21 to 121 b-24 in FIG. 8, the shape in which the vertical band type light shielding film and the horizontal band type light shielding film are joined at their respective end portions The light shielding film 121b is generically referred to as an L-shaped light shielding film 121b.

<Setting method of incident angle directivity>
Next, with reference to FIG. 9, an example of the setting method of the incident angle directivity will be described.

For example, as shown in the upper part of FIG. 9, the light shielding range in the horizontal direction of the light shielding film 121b is from the left end of the pixel 121a to the position A, and the light shielding range in the vertical direction is from the upper end of the pixel 121a Consider the case where the range is up to B.

In this case, a weight Wx of 0 to 1 is set, which is a weight corresponding to the incident angle θx (deg) from the center position in the horizontal direction of each pixel, which is an index of the incident angle directivity in the horizontal direction. More specifically, assuming that the weight Wx is 0.5 at the incident angle θx = θa corresponding to the position A, the weight Wx is 1 at the incident angle θx <θa−α, and θa−α ≦ incident angle θx The weight Wx is set such that the weight Wx becomes (− (θx−θa) /2α+0.5) at ≦ θa + α and becomes 0 at the incident angle θx> θa + α.

Similarly, a weight Wy of 0 to 1 which is a weight according to the incident angle θy (deg) from the center position in the vertical direction of each pixel and which is an index of the incident angle directivity in the vertical direction is set. More specifically, assuming that the weight Wy is 0.5 at the incident angle θy = θb corresponding to the position B, the weight Wy is 0 at the incident angle θy <θb−α, and θb−α ≦ incident angle θy The weight Wy is set such that the weight Wy becomes ((θy−θb) /2α+0.5) at ≦ θb + α and becomes 1 at the incident angle θy> θb + α.

The weights Wx and Wy change as in the graph of FIG. 9 when ideal conditions are satisfied.

Then, by using the weights Wx and Wy obtained in this manner, the incident angle directivity of each pixel 121a, that is, the coefficient corresponding to the light receiving sensitivity characteristic can be obtained. For example, a value obtained by multiplying the weight Wx corresponding to the incident angle θx of incident light from a certain point light source on the object surface 31 and the weight Wy corresponding to the incident angle θy is set as the coefficient for the point light source.

Further, at this time, the inclination (1 / 2α) indicating the change of the weight in the range in which the weight Wx in the horizontal direction and the weight Wy in the vertical direction are around 0.5 is obtained by using the on-chip lens 121c having different focal lengths. It can be set.

For example, as shown by the solid line in the lower part of FIG. 9, when the focal length of the on-chip lens 121c matches the surface of the light shielding film 121b, the inclination of the weight Wx in the horizontal direction and the weight Wy in the vertical direction (1 / 2α ) Is steep. That is, the weight Wx and the weight Wy rapidly change to 0 or 1 near the boundary of the incident angle θx = θa in the horizontal direction and the incident angle θy = θb in the vertical direction, the values of which become around 0.5.

For example, as shown by the dotted line at the bottom of FIG. 9, when the focal length of the on-chip lens 121c matches the surface of the photodiode 121e, the inclination of the weight Wx in the horizontal direction and the weight Wy in the vertical direction (1 / 2α) becomes moderate. That is, the weight Wx and the weight Wy gradually change to 0 or 1 near the boundary of the incident angle θx = θa in the horizontal direction and the incident angle θy = θb in the vertical direction, the values of which become around 0.5.

For example, the focal length of the on-chip lens 121c changes with the curvature of the on-chip lens 121c. Therefore, different incident angle directivity, that is, different light receiving sensitivity characteristics can be obtained by changing the focal length of the on-chip lens 121 c using the on-chip lens 121 c having different curvatures.

Therefore, the incident angle directivity of the pixel 121a can be adjusted by the combination of the range in which the photodiode 121e is shielded by the light shielding film 121b and the curvature of the on-chip lens 121c. The curvature of the on-chip lens may be the same for all the pixels 121 a of the imaging device 121 or may be different for some of the pixels 121 a.

For example, based on the position of each pixel 121a, the shape, position and range of the light shielding film 121b of each pixel 121a, and the curvature of the on-chip lens 121c as an index indicating the incident angle directivity of each pixel 121a of the image sensor 121. The characteristics of the weight Wx and the weight Wy as in the graph of FIG. 9 are set for each pixel 121a. Further, based on the positional relationship between a certain point light source on the object plane 31 of a predetermined object distance and the pixel 121a of the imaging device 121, the incident angle of the light beam from the point light source to the pixel 121a is obtained. Then, based on the determined incident angle and the characteristics of the weight Wx and the weight Wy of the pixel 121a, the coefficient of the pixel 121a with respect to the point light source is determined.

Similarly, for the combination of each point light source on the object plane 31 and each pixel 121a of the imaging device 121, the coefficients are obtained as described above to obtain the coefficient sets α1 and α1 of the equations (1) to (3) described above. The coefficient set group of the image sensor 121 with respect to the object plane 31 can be obtained, such as β1, γ1, coefficient set α2, β2, γ2, coefficient set α3, β3, γ3.

As described later with reference to FIG. 13, when the subject distance from the subject surface 31 to the light receiving surface of the imaging device 121 is different, the incident angle of the light beam from each point light source of the subject surface 31 to the imaging device 121 is Because they are different, different coefficient set groups are required for each subject distance.

Further, even in the case of the object plane 31 having the same object distance, when the number and arrangement of the point light sources to be set are different, the incident angles of the light rays from the respective point light sources to the imaging device 121 are different. Therefore, a plurality of coefficient set groups may be required for the object plane 31 having the same object distance. In addition, the incident angle directivity of each pixel 121 a needs to be set so as to ensure the independence of the above-described simultaneous equations.

<The difference between on-chip lens and imaging lens>
In the imaging device 101 of the present disclosure, the imaging element 121 does not require an optical block or a pinhole formed of an imaging lens, but as described above, the on-chip lens 121 c is provided. Here, the on-chip lens 121c and the imaging lens have different physical actions.

For example, as shown in FIG. 10, of diffused light emitted from the point light source P101, light incident on the imaging lens 152 is condensed at the pixel position P111 on the imaging element 151. That is, the imaging lens 152 is designed to condense diffused light incident at different angles from the point light source P101 at the pixel position P111 and form an image of the point light source P101. The pixel position P111 is specified by the chief ray L101 passing through the point light source P101 and the center of the imaging lens 152.

Further, for example, as shown in FIG. 11, of diffused light emitted from a point light source P102 different from the point light source P101, light incident on the imaging lens 152 is a pixel different from the pixel position P111 on the imaging element 151 Light is collected at position P112. That is, the imaging lens 152 is designed to condense diffused light incident at different angles from the point light source P102 at the pixel position P112 and form an image of the point light source P102. The pixel position P112 is specified by the chief ray L102 passing through the point light source P102 and the center of the imaging lens 152.

As described above, the imaging lens 152 forms images of point light sources P101 and P102 having different chief rays at different pixel positions P111 and P112 on the imaging element 151.

Furthermore, as shown in FIG. 12, when the point light source P101 is at infinity, a part of the diffused light emitted from the point light source P101 is incident on the imaging lens 152 as parallel light parallel to the chief ray L101. . For example, parallel light, which is a light ray between the light ray L121 and the light ray L122 parallel to the chief ray L101, enters the imaging lens 152. Then, the parallel light incident on the imaging lens 152 is condensed at the pixel position P111 on the imaging element 151. That is, the imaging lens 152 is designed to condense parallel light from the point light source P101 existing at infinity into the pixel position P111 and form an image of the point light source P101.

Therefore, for example, the imaging lens 152 causes diffused light from a point light source having a chief ray incident angle θ1 to enter into the pixel (pixel output unit) P1 and has a chief ray incident angle θ2 different from the chief ray incident angle θ1. It has a light collecting function of causing diffused light from a light source to be incident on a pixel (pixel output unit) P2 different from the pixel P1. That is, the imaging lens 152 has a light collecting function for causing diffused light from light sources with different incident angles of principal rays to a plurality of pixels (pixel output units) adjacent to each other. However, for example, light from point light sources close to each other or point light sources existing at infinity and substantially close to each other may be incident on the same pixel (pixel output unit).

On the other hand, for example, as described with reference to FIGS. 4 and 5, the light passing through the on-chip lens 121c is received by the photodiode 121e or the photodiode 121f constituting the corresponding pixel (pixel output unit). It is incident only on the surface. In other words, the on-chip lens 121 c is provided for each pixel (pixel output unit), and condenses incident light incident thereon to only the corresponding pixel (pixel output unit). That is, the on-chip lens 121c does not have a light collecting function for causing light from different point light sources to be incident on different pixels (pixel output units).

When a pinhole is used, the relationship between the position of each pixel (pixel output unit) and the incident angle of light is uniquely determined. Therefore, in the case of the configuration using the pinhole and the conventional imaging device, the incident angle directivity can not be set freely and freely for each pixel.

<Relationship between the subject plane and the image sensor>
Next, with reference to FIG. 13, the relationship between the distance between the object plane and the image sensor 121 will be described.

As shown in the upper left of FIG. 13, when the subject distance between the image sensor 121 (similar to the image sensor 51 in FIG. 1) and the subject surface 31 is the distance d1, the pixels Pc, Pb, It is assumed that the detection signal levels DA, DB, and DC at Pa are represented by the same equations as the equations (1) to (3) described above.

Further, as shown in the lower left of FIG. 13, when viewed from the image pickup element 121, the object surface 31 ′ at which the object distance to the image pickup element 121 is a distance d 2 larger by d than the distance d 1, Also in the case of the far object plane 31 ', the detection signal levels at the pixels Pc, Pb and Pa on the image sensor 121 are the same as the detection signal levels DA, DB and DC as shown in the lower center of FIG. become.

However, in this case, light rays of light intensities a ', b' and c 'from the point light sources PA', PB 'and PC' on the object plane 31 'are received by the respective pixels of the image sensor 121. Further, since the incident angles of the light beams of the light intensities a ', b' and c 'to the imaging element 121 are different (change), different coefficient set groups are required. Therefore, the detection signal levels DA, DB, and DC at the pixels Pa, Pb, and Pc are expressed, for example, by the following formulas (4) to (6).

DA = α11 × a ′ + β11 × b ′ + γ11 × c ′
... (4)
DB = α12 × a ′ + β12 × b ′ + γ12 × c ′
... (5)
DC = α13 × a ′ + β13 × b ′ + γ13 × c ′
... (6)

Here, the coefficient set group consisting of the coefficient sets α11, β11, γ11, the coefficient sets α12, β12, γ12, the coefficient sets α13, β13, γ13 is the coefficient sets α1, β1, γ1 and the coefficient sets α2, β2 for the object surface 31. , Γ2, and coefficient sets α3, β3, and γ3 for the object plane 31 ′.

Therefore, the object plane can be obtained by solving the simultaneous equations of equations (4) to (6) using preset coefficient set groups α11, β11, γ11, α12, β12, γ12, α13, β13, γ13. The point light source PA ′ of the object surface 31 ′, as shown in the lower right of FIG. 13, in the same manner as in the case of obtaining the light intensities a, b and c of the light from the 31 point light sources PA, PB and PC. The light intensities a ', b' and c 'of the rays from PB' and PC 'can be determined. As a result, it is possible to restore the restored image of the subject surface 31 '.

Therefore, in the imaging apparatus 101 of FIG. 2, a coefficient set group for each distance (subject distance) from the imaging element 121 to the subject surface is prepared in advance, and the coefficient set group is switched for each subject distance to perform simultaneous equations. By creating and solving the simultaneous equations created, it is possible to obtain a restored image of the object plane at various object distances based on one detected image. For example, after capturing and recording a detected image once, the coefficient set group is switched according to the distance to the object surface using the recorded detected image, and the restored image is restored. It is possible to generate a restored image of the object plane.

Further, in the case where the subject distance and the angle of view can be specified, the pixels having incident angle directivity suitable for imaging the subject surface corresponding to the specified subject distance and angle of view without using all the pixels. The detection signal may be used to generate a restored image. As a result, a restored image can be generated using detection signals of pixels suitable for imaging the object plane corresponding to the specified object distance and angle of view.

For example, as shown in the upper part of FIG. 14, the pixel 121a shielded by the light shielding film 121b from the end of each of the four sides by the width d1 and each of the four sides as shown in the lower part of FIG. Consider a pixel 121 a ′ that is shielded by the light shielding film 121 b by a width d 2 (> d 1) from the end.

FIG. 15 shows an example of the incident angle of incident light from the object plane 31 to the center position C1 of the image sensor 121. Although FIG. 15 shows an example of the incident angle of incident light in the horizontal direction, the same applies to the vertical direction. In the right part of FIG. 15, the

pixels

121a and 121a 'in FIG. 14 are shown.

For example, when the pixel 121a of FIG. 14 is disposed at the center position C1 of the imaging element 121, the range of the incident angle of incident light from the object plane 31 to the pixel 121a is an angle as shown in the left part of FIG. It becomes A1. Accordingly, the pixel 121a can receive incident light for the width W1 in the horizontal direction of the object plane 31.

On the other hand, when the pixel 121a 'in FIG. 14 is disposed at the center position C1 of the imaging device 121, the pixel 121a' has a wider range of light shielding than the pixel 121a. The range of the incident angle of the incident light is the angle A2 (<A1) as shown in the left part of FIG. Therefore, the pixel 121 a ′ can receive incident light corresponding to the width W 2 (<W 1) in the horizontal direction of the object surface 31.

That is, while the pixel 121 a having a narrow light shielding range is a wide angle of view pixel suitable for imaging a wide range on the object surface 31, the pixel 121 a ′ having a wide light shielding range is narrow on the object surface 31. It is a narrow angle-of-view pixel suitable for imaging a range. Here, the wide angle of view pixel and the narrow angle of view pixel are expressions for comparing both of the

pixels

121a and 121a 'in FIG. 14 and are not limited to the case of comparing pixels of other angles of view.

Thus, for example, pixel 121a is used to restore image I1 of FIG. The image I1 is an image of an angle of view SQ1 corresponding to the subject width W1 including the whole of the person H101 as the subject in the upper stage of FIG. On the other hand, for example, the pixel 121a 'is used to restore the image I2 of FIG. The image I2 is an image of an angle of view SQ2 corresponding to the subject width W2 in which the periphery of the face of the person H101 in the upper part of FIG. 16 is zoomed up.

Further, for example, as shown in the lower part of FIG. 16, the pixel 121a of FIG. 14 is specified in the range ZA surrounded by the dotted line of the imaging device 121, and the pixel 121a ′ is specified in the range ZB surrounded by the dashed dotted line. It is conceivable to collect and arrange each pixel number. Then, for example, when the image of the angle of view SQ1 corresponding to the subject width W1 is restored, the image of the angle of view SQ1 is appropriately restored by using the detection signal of each pixel 121a in the range ZA. it can. On the other hand, when the image of the angle of view SQ2 corresponding to the subject width W2 is restored, the image of the angle of view SQ2 can be appropriately restored by using the detection signal of each pixel 121a 'in the range ZB. .

Since the angle of view SQ2 is narrower than the angle of view SQ1, when the images of the angle of view SQ2 and the angle of view SQ1 are restored with the same number of pixels, the image of the angle of view SQ2 is restored rather than the image of the angle of view SQ1. It is possible to obtain a higher quality restored image.

That is, when it is considered to obtain a restored image using the same number of pixels, a higher quality restored image can be obtained by restoring an image with a narrower angle of view.

For example, the right part of FIG. 17 shows a configuration example within the range ZA of the imaging device 121 of FIG. The left part of FIG. 17 shows a configuration example of the pixel 121a in the range ZA.

In FIG. 17, the range shown in black is the light shielding film 121b, and the light shielding range of each pixel 121a is determined, for example, according to the rule shown in the left part of FIG.

The main light-shielding portion Z101 in the left part of FIG. 17 (black part in the left part in FIG. 17) is a range in which light is shielded in common in each pixel 121a. Specifically, the main light shielding portion Z101 has a height in the range of dx1 from the left side and the right side of the pixel 121a and in the range from the upper side and the lower side of the pixel 121a to the inside of the pixel 121a. It is the range of dy1. Then, in each pixel 121a, a rectangular opening Z111 which is not shielded by the light shielding film 121b is provided within the range Z102 inside the main light shielding portion Z101. Therefore, in each pixel 121a, the range other than the opening Z111 is shielded by the light shielding film 121b.

Here, the openings Z111 of the pixels 121a are regularly arranged. Specifically, the horizontal position of the opening Z111 in each pixel 121a is the same in the pixels 121a in the same vertical direction. Further, the position in the vertical direction of the opening Z111 in each pixel 121a is the same in the pixels 121a in the same horizontal row.

On the other hand, the horizontal position of the opening Z111 in each pixel 121a is shifted at a predetermined interval according to the horizontal position of the pixel 121a. That is, as the position of the pixel 121a advances to the right, the left side of the opening Z111 moves to the position shifted to the right from the left side of the pixel 121a by the widths dx1, dx2,. The distance between the width dx1 and the width dx2, the distance between the width dx2 and the width dx3, ..., the distance between the width dxn-1 and the width dxn is the length obtained by subtracting the width of the opening Z111 from the width of the range Z102 Divided by the number of pixels n-1 in the horizontal direction.

Further, the position in the vertical direction of the opening Z111 in each pixel 121a is shifted at a predetermined interval according to the position in the vertical direction of the pixel 121a. That is, as the position of the pixel 121a moves downward, the upper side of the opening Z111 moves downward from the upper side of the pixel 121a by heights dy1, dy2,. The distance between the height dy1 and the height dy2, the distance between the height dy2 and the height dy3, ..., the distance between the height dyn-1 and the height dyn are respectively from the vertical height of the range Z102 to the opening Z111 The length obtained by subtracting the height of the image is divided by the number of pixels m-1 in the vertical direction.

The right part of FIG. 18 shows a configuration example within the range ZB of the imaging device 121 of FIG. The left part of FIG. 18 shows a configuration example of the pixel 121a 'in the range ZB.

In FIG. 18, the range shown in black is the light shielding film 121b ', and the light shielding range of each pixel 121a' is determined, for example, according to the rule shown in the left part of FIG.

The main light shielding portion Z151 at the left part of FIG. 18 (black part at the left part in FIG. 18) is a range in which light is shielded in common in each pixel 121a '. Specifically, the main light shielding portion Z151 goes from the left side and the right side of the pixel 121a 'to the inside of the pixel 121a' and in the range of the width dx1 'and from the upper side and the lower side of the pixel 121a' And the height dy1 '. In each pixel 121a ', a rectangular opening Z161 which is not shielded by the light shielding film 121b' is provided in a range Z152 inside the main light shielding portion Z151. Therefore, in each pixel 121a ', the range other than the opening Z161 is shielded by the light shielding film 121b'.

Here, the opening Z161 of each pixel 121a 'is regularly arranged similarly to the opening Z111 of each pixel 121a of FIG. Specifically, the horizontal position of the opening Z161 in each pixel 121a 'is the same in the pixels 121a' in the same vertical direction. Further, the vertical position of the opening Z 161 in each pixel 121 a ′ is the same in the pixels 121 a ′ in the same horizontal row.

On the other hand, the horizontal position of the opening Z161 in each pixel 121a 'is shifted at a predetermined interval in accordance with the horizontal position of the pixel 121a'. That is, as the position of the pixel 121a 'proceeds to the right, the left side of the opening Z161 moves to the position shifted to the right by the widths dx1', dx2 ', ..., dxn' from the left side of the pixel 121a '. Do. The distance between the width dx1 'and the width dx2', the distance between the width dx2 'and the width dx3',..., The distance between the width dxn-1 'and the width dxn' The length obtained by subtracting the width is divided by the number of pixels n-1 in the horizontal direction.

Further, the position in the vertical direction of the opening Z161 in each pixel 121a 'is shifted at a predetermined interval according to the position in the vertical direction of the pixel 121a'. That is, as the position of the pixel 121a 'moves downward, the upper side of the opening Z161 is shifted downward by height dy1', dy2 ', ..., dyn' from the upper side of the pixel 121a '. Moving. The distance between the height dy1 ′ and the height dy2 ′, the distance between the height dy2 ′ and the height dy3 ′,..., The distance between the height dyn−1 ′ and the height dyn ′ are in the vertical direction of the range Z152 The length obtained by subtracting the height of the opening Z161 from the height is divided by the number of pixels m-1 in the vertical direction.

Here, the length obtained by subtracting the width of the opening Z111 from the width of the range Z102 of the pixel 121a in FIG. 17 is the width of the opening Z161 from the width of the range Z152 of the pixel 121a ′ in FIG. It becomes larger than the width which I pulled. Therefore, the intervals of change in the widths dx1, dx2,..., Dxn in FIG. 17 are larger than the intervals of changes in the widths dx1 ', dx2',.

The length obtained by subtracting the height of the opening Z111 from the height in the vertical direction of the range Z102 of the pixel 121a of FIG. 17 is the height of the opening Z161 in the vertical direction of the range Z152 of the pixel 121a ′ of FIG. It is larger than the length minus the height. Accordingly, the change intervals of the heights dy1, dy2... Dyn in FIG. 17 are larger than the change intervals of the heights dy1 ', dy2'.

Thus, the interval of change of the position in the horizontal direction and the vertical direction of the opening Z111 of the light shielding film 121b of each pixel 121a of FIG. 17 and the horizontal direction of the opening Z161 of the light shielding film 121b ′ of each pixel 121a ′ of FIG. It is different from the interval of change of position in the direction and vertical direction. Then, the difference in this interval is the difference in object resolution (angular resolution) in the restored image. That is, the change in the horizontal and vertical positions of the opening Z161 of the light shielding film 121b 'of each pixel 121a' in FIG. 18 corresponds to the horizontal direction of the opening Z111 of the light shielding film 121b in each pixel 121a of FIG. It becomes narrower than the interval of change of position in the direction and vertical direction. Therefore, the restored image restored using the detection signal of each pixel 121a 'in FIG. 18 has a higher subject resolution than the restored image restored using the detection signal of each pixel 121a in FIG. Become.

As described above, by changing the combination of the light shielding range of the main light shielding portion and the opening range of the opening, the imaging device 121 including pixels (having various incident angle directivity) of various angles of view is realized. It becomes possible.

Although the example in which the pixel 121a and the pixel 121a ′ are divided into the range ZA and the range ZB is described above, this is to simplify the description, and the pixels 121a corresponding to different angles of view are the same. It is desirable to be mixedly arranged in the area.

For example, as shown in FIG. 19, four pixels of 2 pixels × 2 pixels shown by dotted lines are set as one unit U, and each unit U is a pixel 121a-W with a wide angle of view, a middle angle of view Four pixels of a pixel 121a-M, a pixel 121a-N with a narrow angle of view, and a pixel 121a-AN with a very narrow angle of view are used.

In this case, for example, when the number of pixels of all the pixels 121a is X, it is possible to restore a restored image using detection images of X / 4 pixels for each of four types of angle of view. At this time, four different sets of coefficient sets are used for each angle of view, and restored images of different angles of view are restored by four different simultaneous equations.

Therefore, by restoring the restored image using the detected image obtained from the pixels suitable for capturing the angle of view of the restored image to be restored, it is possible to obtain an appropriate restored image corresponding to the four types of angle of view. .

In addition, images of intermediate angles of view of four types of angle of view and images of angles of view before and after that may be generated by interpolation from images of four types of angle of view, and images of various angles of view are seamlessly generated By doing this, a pseudo optical zoom may be realized.

For example, when obtaining an image with a wide angle of view as a restored image, all wide angle of view pixels may be used, or part of the wide angle of view pixels may be used. Further, for example, when obtaining an image with a narrow angle of view as a restored image, all narrow angle of view pixels may be used, or a part of narrow angle of view pixels may be used.

<Imaging Process by Imaging Device 101>
Next, imaging processing by the imaging device 101 in FIG. 2 will be described with reference to the flowchart in FIG.

In step S1, the imaging element 121 captures an image of a subject. As a result, a detection signal indicating a detection signal level corresponding to the light amount of incident light from the subject is output from each pixel 121a of the imaging device 121 having different incident angle directivity, and the imaging device 121 detects each pixel 121a. The detection image composed of a signal is supplied to the restoration unit 122.

In step S2, the restoration unit 122 obtains a coefficient used for image restoration. Specifically, the restoration unit 122 sets the distance to the object plane 31 to be restored, that is, the object distance. Note that any method can be adopted as a method of setting the subject distance. For example, the restoration unit 122 sets the subject distance input by the user via the input unit 124 or the subject distance detected by the detection unit 125 as the distance to the subject plane 31 to be restored.

Next, the restoration unit 122 reads out from the storage unit 128 the coefficient set group associated with the set subject distance.

In step S3, the restoration unit 122 restores the image using the detected image and the coefficients. Specifically, the restoration unit 122 uses the detection signal level of each pixel of the detection image and the coefficient set group acquired in the process of step S2 to set the above-mentioned equation (1) to equation (3) or The simultaneous equations described with reference to equations (4) to (6) are created. Next, the restoration unit 122 calculates the light intensity of each point light source on the object plane 31 corresponding to the set object distance by solving the created simultaneous equations. Then, the restoration unit 122 arranges the pixels having pixel values according to the calculated light intensity according to the arrangement of the point light sources of the object plane 31 to generate a restored image on which the image of the object is formed.

In step S4, the imaging apparatus 101 performs various processes on the restored image. For example, the restoration unit 122 performs demosaicing processing, γ correction, white balance adjustment, conversion processing to a predetermined compression format, and the like on the restored image as necessary. In addition, the restoration unit 122 supplies, for example, a restored image to the display unit 127 for display, or supplies the restored image to the recording and reproduction unit 129, and records the restored image on the recording medium 130 as needed. , Output to other devices.

Thereafter, the imaging process ends.

In the above, the example of restoring the restored image from the detected image using the image sensor 121 and the coefficient set group associated with the subject distance has been described, but, for example, as described above in addition to the subject distance Alternatively, the set of coefficients corresponding to the angle of view of the restored image may be further prepared, and the set of coefficients corresponding to the subject distance and the angle of view may be used to restore the restored image. The resolution for the subject distance and the angle of view depends on the number of coefficient set groups prepared.

Further, in the description of the process using the flowchart of FIG. 20, the example using detection signals of all the pixels included in the detected image has been described, but the subject distance specified among the pixels constituting the image sensor 121 And a detection image consisting of detection signals of pixels having incident angle directivity corresponding to the angle of view may be generated and used to restore a restored image. Such processing enables the restored image to be restored with a detected image suitable for the subject distance and angle of view of the restored image to be obtained, and the restoration accuracy and image quality of the restored image are improved. That is, when the image corresponding to the specified subject distance and angle of view is, for example, an image corresponding to the angle of view SQ1 in FIG. 16, the pixel 121a having the incident angle directivity corresponding to the angle of view SQ1 is selected. By restoring the restored image with the detected image obtained from these, it is possible to restore the image of the angle of view SQ1 with high accuracy.

By the above-described processing, it is possible to realize the imaging device 101 having the imaging device 121 as an essential component in which each pixel is made to have incident angle directivity.

As a result, since the imaging lens, the pinhole, and the optical filter described in the above-mentioned patent documents and the like become unnecessary, it is possible to increase the degree of freedom in the design of the device, and configured separately from the imaging device 121 In addition, since it becomes unnecessary to use an optical element assumed to be mounted together with the imaging device 121 at the stage of configuring as an imaging device, it becomes possible to realize miniaturization of the device with respect to the incident direction of incident light It is possible to reduce the cost. Further, a lens corresponding to an imaging lens for forming an optical image, such as a focus lens, is not necessary. However, a zoom lens for changing the magnification may be provided.

In addition, although the process which decompress | restores the decompression | restoration image corresponding to a predetermined | prescribed object distance immediately after imaging a detection image was demonstrated above, a detection image is recorded, for example, without performing a decompression | restoration process immediately. After recording on the medium 130 or outputting to another device via the communication unit 131, the restored image may be restored using a detected image at a desired timing. In this case, restoration of the restored image may be performed by the imaging device 101 or may be performed by another device. In this case, for example, a restored image is obtained by solving simultaneous equations created using coefficient set groups according to an arbitrary subject distance and an angle of view, thereby obtaining an restored object for the object plane of an arbitrary subject distance and an angle of view. It is possible to realize refocusing and the like.

For example, in the case of using an imaging device including an imaging lens and a conventional imaging element, it is necessary to perform imaging while varying the focal length and the angle of view in order to obtain images of various focal lengths and angles of view. There is. On the other hand, in the imaging apparatus 101, it is possible to restore the restored image of an arbitrary subject distance or angle of view by switching the coefficient set group in this manner, so the focal length (that is, object distance) or angle of view changes variously. This eliminates the need for processing such as repeated imaging while making it happen.

In this case, for example, the user switches the coefficient set group corresponding to different subject distance and angle of view, and displays the restored image on the display unit 127 while restoring the restored image of the desired subject distance and angle of view. It is also possible to get.

When the detected image is recorded, when the subject distance and the angle of view at the time of restoration are determined, metadata used for the restoration may be associated with the detected image. For example, metadata is added to image data including a detected image, the same ID is added to a detected image and metadata, or the detected image and metadata are recorded on the same recording medium 130. Metadata is associated.

When the same ID is assigned to the detected image and the metadata, the detected image and the metadata can be recorded on different recording media, or can be output individually from the imaging apparatus 101.

Also, the metadata may or may not include the coefficient set group used for restoration. In the latter case, for example, the subject distance and the angle of view at the time of restoration are included in the metadata, and at the time of restoration, a coefficient set group corresponding to the subject distance and the angle of view is acquired from the storage unit 128 or the like.

Furthermore, when the restored image is immediately restored at the time of imaging, it is possible to select, for example, an image to be recorded or output to the outside from among the detected image and the restored image. For example, both images may be recorded or output to the outside, or only one of the images may be recorded or output to the outside.

In addition, also in the case of capturing a moving image, it is possible to select whether to restore the restored image at the time of capturing or to select an image to be recorded or output to the outside. For example, while capturing a moving image, it is possible to immediately restore the restored image of each frame, and to record or output the restored image and / or the detected image before restoration. In this case, it is also possible to display a restored image of each frame as a through image at the time of imaging. Alternatively, for example, it is possible to record a detected image of each frame or output the image outside without performing restoration processing at the time of imaging.

Furthermore, at the time of capturing a moving image, for example, the presence or absence of restoration of a restored image, and selection of an image to be recorded or output to the outside can be performed for each frame. For example, it is possible to switch the presence or absence of restoration of a restored image for each frame. In addition, for example, it is possible to individually switch the presence / absence of recording of the detected image and the presence / absence of recording of the restored image for each frame. Also, for example, while adding metadata to a detection image of a useful frame that may be used later, the detection images of all the frames may be recorded.

Further, it is also possible to realize an autofocus function as in an imaging apparatus using an imaging lens. For example, the autofocus function can be realized by determining the optimum subject distance by the hill climbing method similar to the contrast AF (Auto Focus) method based on the restored image.

Furthermore, in comparison with an imaging apparatus and the like including the optical filter described in the above-mentioned patent documents and the like and a conventional imaging element, using a detection image captured by the imaging element 121 having incident angle directivity in pixel units Since a restored image can be generated, it is possible to realize the increase in the number of pixels and obtain a restored image of high resolution and high angular resolution. On the other hand, in an imaging device including an optical filter and a conventional imaging device, it is difficult to miniaturize the optical filter even if the pixels are miniaturized, so it is difficult to realize high resolution of the restored image.

Further, in the imaging device 101 of the present disclosure, the imaging device 121 is an essential component, and for example, the optical filter described in the above-mentioned patent documents and the like are not required. It is possible to realize an imaging device with high environmental resistance without distortion.

Furthermore, in the imaging device 101 of the present disclosure, the imaging lens, the pinhole, and the optical filter described in the above-mentioned patent documents and the like are not required, so the degree of freedom in design of a configuration having an imaging function is improved. It becomes possible.

<Method of reducing processing load>
By the way, when randomness is given to the light shielding range (that is, incident angle directivity) of the light shielding film 121b of each pixel 121a of the image sensor 121, the processing by the restoration unit 122 is performed as the randomness of the difference in the light shielding range increases. The load on the Therefore, the processing load may be reduced by reducing the randomness by setting a part of the change in the light shielding range of the light shielding film 121b of each pixel 121a to be regular.

For example, the L-shaped light shielding film 121b is formed by combining the vertical band type and the horizontal band type, and the horizontal band type light shielding film 121b having the same width is combined in a predetermined column direction. In the row direction of, the vertical band type light shielding films 121b of the same height are combined. As a result, the light blocking range of the light blocking film 121b of each pixel 121a changes randomly in pixel units while having regularity in the column direction and the row direction. As a result, the difference in light blocking range of the light blocking film 121b of each pixel 121a, that is, the randomness of the difference in incident angle directivity can be reduced, and the processing load of the restoration unit 122 can be reduced.

Specifically, for example, as illustrated in the imaging element 121 ′ ′ of FIG. 21, the horizontal band type light shielding film 121b having the same width X0 is used for all pixels in the same column indicated by the range Z130. For the pixels in the same row indicated by the range Z150, a vertical band type light shielding film 121b of the same height Y0 is used. As a result, for the pixels 121a specified in each row and column, an L-shaped light shielding film 121b in which these are combined is used.

Similarly, for pixels in the same row indicated by the range Z131 adjacent to the range Z130, the light shielding film 121b of the horizontal band type having the same width X1 is used, and the same row indicated by the range Z151 adjacent to the range Z150. The vertical band type light shielding film 121b of the same height Y1 is used for the pixel of. As a result, for the pixels 121a specified in each row and column, an L-shaped light shielding film 121b in which these are combined is used.

Furthermore, for pixels in the same column indicated by the range Z132 adjacent to the range Z131, the light shielding film 121b of the horizontal band type having the same width X2 is used for all pixels in the same row indicated by the range Z152 adjacent to the range Z151. For the pixels, a vertical band type light shielding film 121b of the same height Y2 is used. As a result, for the pixels 121a specified in each row and column, an L-shaped light shielding film 121b in which these are combined is used.

By doing this, it is possible to set the range of the light shielding film to a different value for each pixel while giving regularity to the horizontal width and position of the light shielding film 121b and the height and position in the vertical direction. Because it can, the randomness of the change in incident angle directivity can be suppressed. As a result, it is possible to reduce the pattern of coefficient sets, and to reduce the processing load of the arithmetic processing in the restoration unit 122.

More specifically, as shown in the upper right part of FIG. 22, when obtaining a restored image of N × N pixels from a detected image Pic of N pixels × N pixels, the restored image of (N × N) rows × 1 column A vector X having the pixel value of each pixel as an element, a vector Y having an element value of each pixel of the (N × N) rows × 1 column detected image as an element, and an (N × N) row consisting of a coefficient set group By the matrix A of (N × N) columns, the relationship shown in the left part of FIG. 22 is established.

That is, in FIG. 22, each element of matrix N of (N × N) rows × (N × N) rows of coefficient sets, and (N × N) rows × 1 row vector X representing a restored image It is shown that the result of multiplication of is a vector Y of (N × N) rows × 1 column representing a detected image. Then, from this relationship, simultaneous equations corresponding to, for example, the above-described Equation (1) to Equation (3) or Equation (4) to Equation (6) are configured.

Note that, in FIG. 22, each element of the first column indicated by the range Z201 of the matrix A corresponds to an element of the first row of the vector X, and an N × N column indicated by the range Z202 of the matrix A Indicates that each element of X corresponds to an element of the N × Nth row of the vector X.

In the case where a pinhole is used, and in the case where a condensing function for making incident light incident from the same direction, such as an imaging lens, incident on both of adjacent pixel output units, position and light of each pixel are used. Therefore, the matrix A is a diagonal matrix in which all the down-tilting diagonal components become one. Conversely, when neither a pinhole nor an imaging lens is used as in the imaging device 101 of FIG. 2, the relationship between the position of each pixel and the incident angle of light is not uniquely determined, so the matrix A does not form a diagonal matrix. .

In other words, the restored image is obtained by solving the simultaneous equations based on the determinant shown in FIG. 22 and finding each element of the vector X.

However, in general, the determinant in Fig. 22, by multiplying the inverse matrix A ^-1 of the matrix A in both sides from the left, is deformed as shown in FIG. 23, the inverse matrix to a vector Y of the detected image A ^{- By} multiplying ¹ from the left, each element of the vector X which is a detected image can be obtained.

However, in reality, the matrix A can not be accurately determined, the matrix A can not be accurately measured, the basis vector of the matrix A can not be solved in the case near linear dependence, and each element of the detected image It contains noise. And it may not be possible to solve simultaneous equations by any of those reasons or their combination.

Therefore, for example, the following equation (7) using the concept of the regularized least squares method is used in consideration of a configuration that is robust to various errors.

... (7)

(7) where x is preceded by “^” is a vector X, A is a matrix A, Y is a vector Y, γ is a parameter, || A || is an L 2 norm It represents (square root of sum of squares). Here, the first term on the right side is the norm when minimizing both sides in FIG. 22, and the second term on the right side is a regularization term.

If this equation (7) is solved for x, the following equation (8) is obtained.

... (8)

Here, A ^t is the transposed matrix of the matrix A, I is the identity matrix.

However, since the matrix A has an enormous size, the amount of calculation and the required memory amount become large.

Therefore, for example, as shown in Figure 24, the matrix A, the matrix AL of N rows × N columns, decomposed into matrix AR ^T of N rows × N columns, N rows × N columns, each representing a restored image The result obtained by multiplying the former stage and the latter stage of the matrix X of X is made to be an N-row × N-column matrix Y representing a detected image. Thus, with respect to the matrix A of the number of elements (N × N) × (N × N), matrix AL of number of elements is (N × N), AR ^T, and the number of elements of each matrix is 1 / (N × N). As a result, it is possible to reduce the amount of computation and the amount of memory required.

Determinant shown in FIG. 24, for example, a matrix matrix AL in parentheses of formula (8) is realized by an inverse matrix of the transposed matrix of the matrix A and matrix AR ^T.

In the calculation as shown in FIG. 24, as shown in FIG. 25, the element group Z222 is obtained by multiplying each element group Z221 of the corresponding column of the matrix AL by the element of interest Xp in the matrix X Be Further, by multiplying the elements in the row corresponding to the element of interest Xp the element group Z222 matrix AR ^T, 2-dimensional response Z224 that corresponds to the element of interest Xp is calculated. Then, the two-dimensional response Z 224 corresponding to all elements of the matrix X is integrated to obtain the matrix Y.

Therefore, for example, in the element group Z 221 of each column of the matrix AL, a coefficient corresponding to the incident angle directivity of the horizontal band type pixel 121 a set to the same width for each column of the imaging device 121 shown in FIG. Is used.

Similarly, for example, a matrix in the element group Z223 of each row of the AR ^T, corresponding to the incident angle directivity of vertical band type pixel 121a to be set at the same height in each row of the image sensor 121 shown in FIG. 21 Coefficients are used.

As a result, since it is possible to reduce the matrix used when restoring the restored image based on the detected image, the amount of calculation is reduced, the processing speed is improved, and the power consumption related to the calculation is reduced. Is possible. In addition, since the matrix can be made smaller, the capacity of the memory used for calculation can be reduced, and the device cost can be reduced.

Note that FIG. 21 shows an example in which the light shielding range (light receiving range) is changed in pixel units while giving predetermined regularity in the horizontal direction and the vertical direction, but in the present disclosure, such an example is shown. Although the light blocking range (light receiving range) is not completely randomly set in a pixel unit, it is considered that the light blocking range (light receiving range) is also set randomly at some random settings. In other words, in the present disclosure, not only when the light blocking range (light receiving range) is completely set randomly on a pixel basis, but also to a certain degree random (for example, regularity for some of all pixels). The range which is given is included, but the other range is random) or something which seems to have some degree of regularity (of all the pixels, they are arranged according to the rule described with reference to FIG. 21) In the case of the arrangement where it can not be confirmed, it is regarded as random.

<< 3. First Embodiment >>
Next, a first embodiment of the present disclosure will be described with reference to FIGS. 26 to 32. FIG.

As described above, in the imaging device 121 using pixels having incident angle directivity, an imaging lens, an optical filter, and the like are not necessary, so the degree of freedom of arrangement of the pixels 121a is high. In the first embodiment of the present disclosure, at least a part of a user wearing or using an electronic device is photographed in various electronic devices by using the degree of freedom of arrangement of each pixel 121a of the imaging device 121. An imaging element 121 is provided at a position, at least a part of the user is imaged, and various application processes are performed using a restored image obtained by the restoration process.

Here, at least a part of the user is, for example, any part of the user's body, such as the user's entire body, face, eyes, head, torso, hands, feet and the like. In addition, for example, in the case of a medical device or the like, at least a part of the user may be not only outside the user but also inside the user (for example, in the oral cavity, in the viscera, etc.).

<Configuration Example of Electronic Device 301>
FIG. 26 is a block diagram illustrating a configuration example of the electronic device 301 according to the first embodiment of the present disclosure. In the figure, parts corresponding to those of the imaging apparatus 101 in FIG. 2 are assigned the same reference numerals, and the description thereof will be omitted as appropriate.

The electronic device 301 includes, for example, a wearable device, a portable information terminal worn or carried by a user such as a smartphone, a tablet, or a portable telephone, a personal computer, a wearable device, a game machine, a video playback device, a music playback device, and the like. For the wearable device, for example, various methods such as a head-mounted type, a watch type, a bracelet type, a necklace type, and a neck band type mounted on the head of the user can be adopted. The head-mounted wearable devices include, for example, glasses, goggles, head mounts, earphones, headsets, masks, and hats. Note that, for example, depending on the shape of the wearable device, one wearable device may correspond to a plurality of methods (for example, a goggle type and a head mount type).

The electronic device 301 includes an imaging unit 311 and a signal processing control unit 312.

The imaging unit 311 includes one or more n imaging devices 121-1 to 121-n.

Each imaging element 121 supplies a detection signal set including a detection signal output from each pixel 121 a to the restoration unit 321 or outputs it to the bus B2.

A detection image is generated by the detection signal set from each imaging element 121. Therefore, when the imaging unit 311 includes only one imaging element 121, a detection image is generated by one detection signal set from the imaging element 121.

In addition, each imaging element 121 may be installed in the same housing or may be installed in a different housing.

The signal processing control unit 312 includes a restoration unit 321, a control unit 322, an input unit 323, a detection unit 324, an association unit 325, an output unit 326, a storage unit 327, a recording and reproduction unit 328, a recording medium 329, and a communication unit 330. Prepare.

The restoration unit 321, the control unit 322, the input unit 323, the detection unit 324, the association unit 325, the output unit 326, the storage unit 327, the recording and reproduction unit 328, and the communication unit 330 are mutually connected via the bus B2. And transmit and receive data via the bus B2. In the following, in order to simplify the description, the description of the bus B2 in the case where each part of the electronic device 301 transmits and receives data via the bus B2 is omitted.

The restoration unit 321 performs restoration processing and the like of a restored image by the same processing as the restoration unit 122 of the imaging apparatus 101 in FIG. 2 using the detection signal set acquired from each of the imaging elements 121. The restoration unit 321 outputs the restored image to the bus B2.

The control unit 322 includes, for example, various processors, and performs control of each unit of the electronic device 301, various application processing, and the like.

The input unit 323 includes an input device (for example, a key, a switch, a button, a dial, a touch panel, a remote controller, and the like) for performing an operation of the electronic device 301, an input of data used for processing, and the like. The input unit 323 outputs an operation signal, input data, and the like to the bus B2.

The associating unit 325 associates the detection signal set of each imaging element 121 with the metadata corresponding to each detection signal set.

The output unit 326 includes, for example, an output device that outputs an image such as a display, a speaker, a lamp, a buzzer, and a vibration element, sound, light, vibration, and the like, and outputs various information and data.

The storage unit 327 includes one or more storage devices such as a ROM, a RAM, and a flash memory, and stores, for example, programs and data used for processing of the electronic device 301. For example, the storage unit 327 stores a coefficient set group corresponding to each imaging device 121. The coefficient set group is prepared, for example, for each assumed subject distance and angle of view.

The recording and reproducing unit 328 performs recording of data on the recording medium 329 and reproduction (reading) of data recorded on the recording medium 329. For example, the recording / reproducing unit 328 records the restored image on the recording medium 329 or reads the restored image from the recording medium 329. Also, for example, the recording / reproducing unit 328 records the detection signal set and the corresponding metadata on the recording medium 329 or reads out from the recording medium 329.

The recording medium 329 is made of, for example, an HDD, an SSD, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like, or a combination thereof.

The communication unit 330 communicates with other devices by a predetermined communication method. The communication method of the communication unit 330 may be wired or wireless. Further, the communication unit 330 can also correspond to a plurality of communication methods.

<Arrangement Example of Imaging Element 121>
Next, with reference to FIG. 27 to FIG. 31, an arrangement example of the imaging device 121 will be described while giving a specific example of the electronic device 301.

As an example of the electronic device 301, a part of the glasses-type wearable device 401 worn so as to cover the user's eyes is schematically shown in A of FIG. 27A to 27C show an example in which (the respective pixels 121a of) the imaging element 121 are arranged around the left lens 411L and the right lens 411R which are the eyepieces of the wearable device 401. It is done.

Here, the eyepiece portion is a portion close to the eyes of the user who wears or uses the electronic device 301, and includes, for example, an eyepiece lens such as the left lens 411L and the right lens 411R. When the eyepiece unit is a proximity lens, for example, the user can view an image (for example, an image or an image of an object) through the proximity lens.

Specifically, in the example of FIG. 27A, the imaging element 121-1 is on the back of the frame 412, that is, above the left lens 411L on the surface facing the user's face when the user wears the wearable device 401. The imaging element 121-2 is disposed above the right lens 411R on the back surface of the frame 412.

In the example of FIG. 27B, the imaging device 121 is disposed in the vicinity of the bridge on the back surface of the frame 412.

In the example of FIG. 27C, the imaging device 121-1 is disposed to surround the periphery of the left lens 411L on the back surface of the frame 412, and the imaging device 121-2 to surround the periphery of the right lens 411R on the back surface of the frame 412. Is arranged.

As described above, the imaging element 121 can be disposed in the empty space of the frame 412 of the wearable device 401 while suppressing an increase in size and a decrease in design. In addition, the imaging element 121 is disposed at a position where the vicinity of both eyes of the user wearing the wearable device 401 is captured, and imaging around both eyes of the user can be performed.

The incident angle directivity of each pixel 121a of each imaging element 121 of each of FIGS. 27A to 27C is the eye of the user who is in proximity to the left lens 411L and the right lens 411R when the wearable device 401 is attached. It is desirable that the light receiving sensitivity with respect to the direction be set to be high. As a result, in each pixel 121a, the light reception sensitivity to incident light from the direction of the eye of the user who is the main imaging target becomes high, and it is possible to capture the eye of the user more clearly.

For example, A of FIG. 28 and B of FIG. 28 schematically show an example of the incident angle directivity of the imaging device 121-1 of C of FIG. A of FIG. 28 and B of FIG. 28 schematically show the positional relationship between the left eye 421 L of the user with the wearable device 401 attached, the left lens 411 L, and the imaging element 12-1. FIG. 28A shows a view of the left lens 411L as viewed from the inside (the side facing the user's eyes), and FIG. 28B shows a cross-sectional view of the left lens 411L as viewed from the side.

The arrows in the figure indicate the tendency of the incident angle directivity of the pixels 121a in the respective regions of the imaging element 121-1. Further, hereinafter, in A of FIG. 28 and B of FIG. 28, the direction from the left lens 421 L to the left eye 421 (the right direction of B of FIG. 28) is taken as the backward direction.

The incidence angle directivity of the pixel 121a in the upper area of the imaging element 121-1 is set to the rear and downward direction, and the light reception sensitivity to the direction of the left eye 421L is high. The incident angle directivity of the pixel 121a in the region on the left side of the imaging element 121-1 is set to the diagonally right backward direction, and the light reception sensitivity with respect to the direction of the left eye 421L is high. The incident angle directivity of the pixel 121a in the area on the right side of the image pickup device 121-1 is set to the left diagonal backward direction, and the light reception sensitivity with respect to the direction of the left eye 421L is high. The incident angle directivity of the pixel 121a in the lower area of the imaging element 121-1 is set to the rear upper side, and the light receiving sensitivity with respect to the direction of the left eye 421L is high.

As a result, in the imaging element 121-1, the light reception sensitivity to incident light from the direction of the left eye 421L of the user who is the main imaging target becomes high, and it becomes possible to pick up the left eye 421L more clearly.

Although illustration is omitted, also in the imaging element 121-2 arranged around the right lens 411R, the incident angle directivity of each pixel 121a is set in the direction of the right eye 421R of the user who is the main imaging target Ru. As a result, the light reception sensitivity to the incident light from the right eye 421R of the user becomes high, and it becomes possible to capture the right eye 421R more clearly.

Of the examples in A to C in FIG. 27, from the viewpoint of the image quality of the restored image, the imaging element 121 is disposed as in C in FIG. 27, and the incident angle directivity of each pixel 121a is uneven. It is desirable to use it. When the incident angle directivity of each pixel 121a varies, the diversity of coefficient sets with respect to simultaneous equations used for restoration of a restored image is enhanced, and restoration accuracy is improved.

Note that the incident angle directivity of all the pixels 121 a does not necessarily have to be set so as to increase the light reception sensitivity with respect to the direction of the user's eyes. For example, the number of pixels 121a having incident angle directivity, in which the light reception sensitivity with respect to the direction of the user's eyes is high, may be larger than the number of pixels 121a having other incident angle directivity.

A camera 431 is schematically shown as an example of the electronic device 301 in A of FIG. 29 to C of FIG. Although only the periphery of the finder 441 of the camera 431 is illustrated in A of FIG. 29 to C of FIG. 29, for example, a display or a touch panel may be provided on the back of the camera 431 (the same side as the finder 441). And a user operation unit.

Also, an example in which each of the imaging elements 121 (each pixel 121a of the imaging device 121) is disposed around a finder 441 which is an eyepiece of the camera 431 is shown in A of FIG.

Specifically, in the example of FIG. 29A, the imaging element 121-1 is disposed in the vertically long rectangular area on the left side of the finder 441, and the imaging element 121-2 is disposed in the vertically long rectangular area on the right side of the finder 441. It is arranged.

In the example of FIG. 29B, the imaging device 121 is disposed in a horizontally long rectangular area above the finder 441.

In the example of C in FIG. 29, the imaging element 121 is disposed in the L-shaped area above and to the left of the finder 441.

In the example of FIG. 29D, the imaging element 121-1 is disposed in the horizontally long rectangular area above the finder 441, and the imaging element 121-2 is disposed in the horizontally long rectangular area below the finder 441.

In the example of E of FIG. 29, the imaging device 121 is disposed so as to surround the finder 441.

As described above, the image pickup device 121 can be disposed in the empty space around the finder 441 of the camera 431 while suppressing the size increase and the design decrease. In addition, when the user looks into the finder 441 to take a picture using the camera 431, the image pickup element 121 is disposed at a position where the eye periphery of the user is captured, and performs imaging of the eye periphery of the user be able to.

Note that the incident angle directivity of each pixel 121a of each imaging element 121 of each of A to E in FIG. 29 is the direction of the eye of the user approaching the finder 441 when looking through the finder 441 (for example, in front of the finder 441). It is desirable that the light receiving sensitivity to the above be set to be high.

As an example of the electronic device 301, a part of a goggle type head mounted display 461 mounted so as to cover the user's eyes is schematically shown in A of FIG. In A to C of FIG. 30, the inner surface of the head mounted display 461, that is, the surface facing the user's face when worn by the user is shown. An example is shown in which 121a) is disposed around the left lens 471L and the right lens 471R which are the eyepieces of the head mounted display 461.

Specifically, in the example of A in FIG. 30, the imaging element 121-1 is disposed in the rectangular area on the left side of the left lens 471L, and the imaging element 121-2 is disposed in the rectangular area on the right side of the right lens 471R. ing.

In the example of FIG. 30B, the imaging element 121-1 is disposed in the horizontally long rectangular area above the left lens 471L, and the imaging element 121-2 is disposed in the horizontally long rectangular area above the right lens 471R. There is.

In the example of C in FIG. 30, the imaging element 121-1 is disposed in the horizontally long rectangular area below the left lens 471L, and the imaging element 121-2 is disposed in the rectangular area below the right lens 471R.

As described above, the imaging element 121 can be disposed in the free space around the left lens 471L and the right lens 471R of the head mounted display 461 while suppressing an increase in size and a decrease in design. Then, for example, it is possible to perform imaging around both eyes of the user wearing the head mounted display 461.

The incident angle directivity of each pixel 121a of each imaging element 121 of each of FIGS. 30A to 30C is the same as that of the wearable device 401 of FIG. It is desirable that the light receiving sensitivity with respect to the direction be set to be high.

A notebook PC (personal computer) 491 is schematically shown as an example of the electronic device 301 in A of FIG. 31 to C of FIG.

In the PC 491, a cover provided with the display 501, which is a display unit, and a base provided with a keyboard 503, which is a user operation unit, are rotatably connected by a hinge to open and close the cover. be able to. Then, the display 501 and the keyboard 503 are exposed to the outside in a state in which the lid is opened as illustrated in A to 31C of FIG. 31, and the user can use the PC 491. Note that, for example, by providing a touch panel on the display 501, the display 501 (strictly speaking, the touch panel) can be used as a user operation unit.

31A to 31C show an example in which (the respective pixels 121a of) the imaging device 121 are arranged around the display 501. FIG.

In the example of FIG. 31A, the imaging device 121 is disposed on the upper side of the bezel 502 around the display 501.

In the example of FIG. 31B, the imaging device 121-1 is disposed on the left side of the bezel 502, and the imaging device 121-2 is disposed on the right side of the bezel 502.

In the example of C in FIG. 31, the imaging element 121 is disposed on four sides of the bezel 502 so as to surround the display 501.

As described above, the imaging element 121 can be disposed in the empty space of the bezel 502 while suppressing increase in size and reduction in design, and narrowing of the bezel can be realized. Further, the imaging element 121 is disposed at a position facing the user using the keyboard 503 while looking at the display 501 of the PC 491 and at a position where the periphery of the user's face is captured. It can be performed.

The incident angle directivity of each pixel 121a of each image sensor 121 of each of FIGS. 31A to 31C corresponds to the direction of the face of the user who is using the keyboard 503 while looking at the display 501 of the PC 491 (for example, display It is desirable that the light receiving sensitivity with respect to the front of 501) be set to be high.

<User imaging control processing>
Next, user imaging control processing executed by the electronic device 301 will be described with reference to the flowchart in FIG.

In step S101, each imaging element 121 of the electronic device 301 performs imaging of the user by the same processing as that of step S1 in FIG.

For example, in the case of the wearable device 401 of FIG. 27, imaging of the area around both eyes of the user wearing the wearable device 401 is performed.

In the case of the camera 431 of FIG. 29, imaging around the eyes of the user looking through the finder 441 is performed.

In the case of the head mounted display 461 of FIG. 30, imaging around the eyes of the user wearing the head mounted display 461 is performed.

In the case of the PC 491 of FIG. 31, imaging around the face of the user looking at the display 501 is performed.

Each imaging element 121 supplies the detection signal set including the detection signal of each pixel 121a to the associating unit 325.

In step S102, the restoration unit 321 obtains a coefficient used for image restoration. Specifically, the restoration unit 321 sets the subject distance by the same process as the process performed by the restoration unit 122 of the imaging apparatus 101 in step S2 of FIG. Then, the restoration unit 321 reads out, from the storage unit 327, the coefficient set group associated with the set subject distance.

In step S103, the reconstruction unit 321 reconstructs an image using the detection signal set and the coefficients. That is, the restoration unit 321 performs the same processing as the processing performed by the restoration unit 122 of the imaging apparatus 101 in step S3 in FIG. 20, the detection signal set output from each imaging element 121, and the coefficient obtained in the processing in step S102. The set group is used to restore the restored image.

In step S104, the electronic device 301 performs various processes on the restored image. For example, the restoration unit 321 performs demosaicing processing, γ correction, white balance adjustment, conversion processing to a predetermined compression format, and the like on the restored image as necessary. In addition, the restoration unit 321 supplies, for example, a restored image to the output unit 326 for display, or supplies the restored image to the recording and reproduction unit 328, and records the image on the recording medium 329 as needed. , Output to other devices.

In step S105, the electronic device 301 executes an application process using the restored image (that is, the image of the user).

For example, in the case of the wearable device 401 in FIG. 27, the control unit 322 performs the gaze detection of the user based on the image of the user's eyes in the restored image. In addition, arbitrary methods can be used for gaze detection. Then, for example, the control unit 322 generates an operation command corresponding to the movement of the user's line of sight, and transmits the operation command to another electronic device (not shown) via the communication unit 330. Thus, the user can operate another electronic device simply by wearing the wearable device 401 and moving his / her eyes.

For example, in the case of the camera 431 of FIG. 29, the control unit 322 performs the gaze detection of the user based on the image of the user's eyes in the restored image. In addition, arbitrary methods can be used for gaze detection. Then, for example, the control unit 322 controls each unit of the camera 431 to move the position (focus point) of the subject to be focused according to the movement of the user's line of sight, or performs various processes of the camera 431. To As a result, the user can set the focus point or perform various operations on the camera 431 simply by looking at the finder 441 and moving the line of sight.

Alternatively, for example, the control unit 322 performs user recognition processing or authentication processing based on the inside of the restored image.

Here, the recognition process is, for example, a process of specifying a user or recognizing a feature of the user. On the other hand, in the authentication process, for example, a user whose user is registered in advance or a legitimate user by, for example, collating a restored image with an image (for example, a face image, an eye image, etc.) registered in advance. It is a process which performs judgment etc. whether it is etc. In addition, recognition processing and authentication processing may be partially overlapped without being clearly distinguished. Further, any method can be used for the user recognition process and authentication process. For example, various biometric authentication such as face authentication and iris authentication can be used for recognition processing.

Then, for example, the control unit 322 causes the display of the output unit 326 to display a user interface screen corresponding to the recognized or authenticated user based on the recognition result or authentication result of the user, or changes the setting of the camera 431. , Authorize the use of a specific function (eg, playback, etc.).

For example, in the case of the head mounted display 461 of FIG. 30, the control unit 322 performs the gaze detection of the user based on the image of the user's eyes in the restored image. In addition, arbitrary methods can be used for gaze detection. Then, for example, the control unit 322 controls each unit of the head mount display 461 to perform various processes of the head mounted display 461 according to the movement of the user's line of sight. As a result, the user can perform various operations on the head mounted display 461 simply by mounting the head mounted display 461 and moving the line of sight.

For example, in the case of the PC 491 of FIG. 31, the control unit 322 performs user recognition processing or authentication processing based on the image of the user's face in the restored image. Then, for example, the control unit 322 displays a user interface screen corresponding to the recognized or authenticated user on the display of the output unit 326 based on the recognition result or the authentication result of the user, or the setting of the PC 491 (for example, custom setting , Image quality setting, parental control setting, etc., login to a specific account, access to a specific folder or file, and permission to use a specific function.

Thereafter, the user imaging control process ends.

In addition, although the example which decompress | restores a decompression | restoration image from a detection signal set using the coefficient set group matched with the image pick-up element 121 and subject distance above was demonstrated, for example, in addition to subject distance, it mentioned above As described above, a coefficient set group corresponding to the angle of view of the restored image may be further prepared, and the restored image may be restored using the coefficient set group corresponding to the subject distance and the angle of view.

As described above, it is possible to dispose the imaging element 121 while suppressing an increase in size and a decrease in design of the electronic device 301, and to image a user who uses the electronic device. Then, the user's image can be restored, and various application processes can be executed based on the restored image.

<< 4. Second embodiment >>
Next, a second embodiment of the present disclosure will be described with reference to FIGS. 33 to 41.

In the first embodiment described above, an example is shown in which imaging of a user who uses the electronic device 301 is performed, and various application processing is executed using an image of the user obtained by the restoration processing. On the other hand, in the second embodiment, imaging around the user using the electronic device 301 is performed, and various application processes are executed using images around the user obtained by the restoration process.

In the second embodiment, the electronic device 301 of FIG. 26 is used as in the first embodiment. On the other hand, in the second embodiment, unlike the first embodiment, the imaging device 121 is disposed at a position where the surroundings of the user wearing or using the electronic device 301 are captured.

<Arrangement Example of Imaging Element 121>
Here, with reference to FIG. 33 to FIG. 40, an arrangement example of the imaging device 121 will be described by giving a specific example of the electronic device 301.

In A of FIG. 33 to C of FIG. 33, a glasses-type wearable device 601 worn to cover the eyes of the user is schematically shown as an example of the electronic device 301. 33A to 33C show an example in which the imaging element 121 (each pixel 121a of the imaging element 121) is disposed on the surface of the frame 612 exposed to the outside when the wearable device 601 is worn by the user. It is done.

Specifically, in the example of A of FIG. 33, the imaging element 121-1 is disposed above the left lens 611L on the surface of the frame 612, and the imaging element 121-2 is disposed above the right lens 611R on the surface of the frame 612. It is arranged.

In the example of FIG. 33B, the imaging device 121-1 is disposed on the right side of the left lens 611L on the surface of the frame 612, and the imaging device 121-2 is disposed on the left side of the right lens 611R on the surface of the frame 612.

In the example of FIG. 33C, the imaging device 121-1 is disposed to surround the left lens 611L of the surface of the frame 612, and the imaging device 121-2 is disposed to surround the right lens 611R of the surface of the frame 612. Is arranged.

As described above, the image sensor 121 can be disposed in the empty space of the frame 612 of the wearable device 601 while suppressing the increase in size and the decrease in design. Then, for example, imaging in front of the user wearing the wearable device 601 can be performed.

A camera 631 is schematically shown as an example of the electronic device 301 in A of FIG. 34 to C of FIG. Further, an example in which (the respective pixels 121 a of) the imaging element 121 is disposed on the front surface of the casing of the camera 631 is shown.

Specifically, in the example of A of FIG. 34, the imaging element 121 is disposed in a vertically long rectangular area on the left side of the mount 641 and near the left end of the camera 631 in the front surface of the main body of the camera 631 There is.

In the example of FIG. 34B, the imaging elements 121-1 to 121-4 are respectively disposed in four rectangular areas near the four corners of the mount 641 in the front surface of the main body of the camera 631.

In the example of C in FIG. 34, the imaging device 121 is disposed on the front surface of the flash built-in portion 642 in which the flash of the camera 631 is built.

In the example of FIG. 34D, the imaging device 121 is disposed in a ring-shaped area along the outer periphery of the mount 641 of the camera 631.

As described above, the imaging element 121 can be disposed in the vacant space on the front surface of the housing of the camera 631 while suppressing the increase in size and the reduction in design. Then, for example, imaging in the imaging direction of the camera 631 can be performed.

As an example of the electronic device 301, a goggle type head mounted display 661 mounted so as to cover the user's eyes is schematically shown in A of FIG. 35 to D of FIG. Also, in A to D of FIG. 35, there is an example in which the imaging element 121 (each pixel 121a of the image pickup element 121) is disposed on the front surface of the housing exposed to the outside when the user mounts the head mounted display 661. It is shown.

Specifically, in the example of FIG. 35A, the imaging device 121 is disposed in a horizontally long rectangular area below the front surface of the main body 671.

In the example of FIG. 35B, the imaging element 121 is disposed in a horizontally long region at the upper end of the front surface of the main body 671.

In the example of C in FIG. 35, the imaging element 121 is disposed in the rectangular area on the front surface of the head pad 672.

In the example of D in FIG. 35, the imaging device 121-1 and the imaging device 121-2 are disposed in the left and right rectangular regions on the front surface of the main body portion 671.

Thus, the imaging element 121 can be disposed in the empty space on the front surface of the housing of the head mounted display 661 while suppressing an increase in size and a decrease in design. Then, for example, imaging in front of the user wearing the head mounted display 661 can be performed.

An overhead type headphone 691 is schematically shown as an example of the electronic device 301 in A to D of FIG. A of FIG. 36 and B of FIG. 36 show perspective views of the headphones 691 as viewed obliquely from the front, and C of FIGS. 37 and D of FIG. 37 show perspective views of the headphones 691 as viewed obliquely from the rear.

In the example of FIG. 36A, the imaging device 121-1 is disposed near the front center of the side surface of the left housing 701L, and the imaging device 121-2 is disposed near the front center of the side surface of the right housing 701R.

In the example of FIG. 36B, the imaging device 121 is disposed in the area along the front surface of the headset 702.

In the example of C of FIG. 37, the imaging element 121-1 is disposed near the rear center of the side surface of the left housing 701L, and the imaging element 121-2 is disposed near the rear center of the side surface of the right housing 701R.

In the example of FIG. 37D, the imaging device 121 is disposed in the area along the back surface of the headset 702.

As described above, the image sensor 121 can be disposed in a vacant space on the surface exposed to the outside in a state where the user wears the headphones 691 while suppressing an increase in size and a decrease in design. Then, for example, imaging in front of or behind the user wearing the headphones 691 can be performed.

In A of FIG. 38 and B of FIG. 38, a neck band type headphone 721 is schematically shown as an example of the electronic device 301.

In the example of FIG. 38A, the imaging device 121-1 is disposed in front of the side surface of the left housing 731L. Although not shown, the imaging device 121-2 is disposed in front of the side surface of the right housing 731R.

In the example of B of FIG. 38, the imaging element 121 is disposed in the vicinity of the rear of the neck band 732.

As described above, the image sensor 121 can be disposed in a vacant space on the surface exposed to the outside in a state where the user wears the headphones 721 while suppressing an increase in size and a decrease in design. Then, for example, imaging in front of or behind the user wearing the headphones 721 can be performed.

In the above example, an example of imaging the front or back of the user is described, but each imaging element 121 is configured to image other directions (for example, side, upper, lower, etc.) around the user. You may install it.

<User ambient imaging control processing>
Next, with reference to a flowchart of FIG. 39, a user surrounding imaging control process executed by the electronic device 301 will be described.

In step S201, each imaging element 121 of the electronic device 301 performs imaging of the surroundings of the user by the same processing as step S1 in FIG.

For example, in the case of the wearable device 601 of FIG. 33, imaging in front of the user wearing the wearable device 601 is performed.

In the case of the camera 631 of FIG. 34, imaging in the imaging direction of the camera 631 is performed.

In the case of the head mounted display 661 of FIG. 35, imaging in front of the user wearing the head mounted display 661 is performed.

In the case of the headphones 691 of FIGS. 36 and 37, imaging in front of or behind the user wearing the headphones 691 is performed.

In the case of the headphone 721 of FIG. 38, imaging in front of or behind the user wearing the headphone 721 is performed.

In step S202, the restoration unit 321 obtains a coefficient used for image restoration. Specifically, the restoration unit 321 sets the subject distance by the same process as the process performed by the restoration unit 122 of the imaging apparatus 101 in step S2 of FIG. Then, the restoration unit 321 reads out, from the storage unit 327, the coefficient set group associated with the set subject distance.

When the image sensor 121 is disposed at a deformable portion, such as the headset 702 of the headphone 691 of FIGS. 36 and 37 and the neck band 732 of the headphone 721 of FIG. The relative position between 121a changes.

On the other hand, the coefficient set group of simultaneous equations used for restoring the restored image described above is set on the premise that the relative position of each pixel 121a does not change.

Therefore, when the imaging device 121 is disposed at a deformable portion of the electronic device 301, a coefficient set group corresponding to the degree of deformation is prepared in advance, and the degree of deformation is detected, and the coefficient corresponding to the detected degree of deformation A set group may be used.

Here, the deformable portion is a portion that can change its shape when, for example, the electronic device 301 is used or mounted. The type of deformation is not particularly limited, and includes, for example, elongation, contraction, bending, deviation, twisting, separation, and the like. Further, the deformable portion includes, for example, a portion which is deformed by an externally applied force, and a portion which changes its shape by an actuator or the like.

For example, a bending sensor may be provided in the headset 702 of the headphone 691 of FIGS. 36 and 37, and a coefficient set group corresponding to the bending condition of the headset detected by the bending sensor may be used.

For example, coefficient set groups corresponding to the degree of bending of the three patterns of the bending sensor 751 of FIG. 40A to FIG. 40C are prepared in advance. Then, restoration of a restored image is performed using a coefficient set group corresponding to the bending condition closest to the detection result of the bending sensor 751 when the user wears the headphone 691.

In step S203, as in the process of step S103 of FIG. 32, the image is restored using the detection signal set and the coefficients.

In step S204, various processes are performed on the restored image, as in the process of step S104 in FIG.

In step S205, the electronic device 301 executes application processing using a restored image (that is, an image around the user).

For example, in the case of the wearable device 601 of FIG. 33, the control unit 322 performs recognition processing of the surroundings of the user based on the restored image. In addition, arbitrary methods can be used for the recognition process of a user's circumference. Then, for example, the control unit 322 controls the output unit 326 so that the user visually recognizes an image or information corresponding to the environment around the user obtained by the recognition processing through the left lens 611L and the right lens 611R. Superimpose in the field of view to realize AR (Augmented Reality).

Alternatively, for example, the control unit 322 controls the recording and reproducing unit 328 to record the restored image on the recording medium 329 as the user's life log.

For example, in the case of the camera 631 of FIG. 34, the control unit 322 performs tracking processing of the subject, recognition of a scene, and the like by performing recognition processing of the surroundings of the user based on the restored image. In addition, arbitrary methods can be used for the recognition process of a user's circumference. Thereby, for example, it is possible to follow the subject even during the shutter operation of an imaging device (not shown) which takes an image through the lens 643 of the camera 631. In addition, for example, a wider range than the lens 643 can be sensed to follow an object or recognize a scene.

For example, in the case of the head mounted display 661 of FIG. 35, the control unit 322 controls the output unit 326 to superimpose part or all of the restored image in the image viewed by the user.

For example, in the case of the headphones 691 of FIG. 36 and FIG. 37 or the headphones 721 of FIG. 38, the control unit 322 performs recognition processing of the user's surroundings based on the restored image. In addition, arbitrary methods can be used for the recognition process of a user's circumference. Then, based on the recognition result, the control unit 322 controls the output unit 326 and the like to perform user assistance and the like. For example, when a danger such as the approach of a vehicle is detected, the danger is notified by vibration, voice, etc., assistance is given by voice to people with impaired eyes, the name of the person who recognized is voiced Be notified.

Thereafter, the user surrounding imaging control process ends.

As described above, it is possible to dispose the imaging element 121 while suppressing the increase in the size and the design of the electronic device 301, and to capture the surroundings of the user who uses the electronic device. Then, the image around the user can be restored, and various application processes can be executed based on the restored image.

Note that although an example in which a restored image is recorded as a life log has been described above, a detection signal set may be recorded instead of the restored image, and the image may be restored as needed.

In this case, for example, the detection signal set acquired by each imaging element 121 is associated with the metadata corresponding to each detection signal set.

The metadata may or may not include, for example, a coefficient set group at the time of restoration. In the latter case, the metadata includes, for example, a subject distance used at the time of restoration, an angle of view, deformation information of a portion where the imaging device 121 is disposed, and the like.

Further, the method of associating the detection signal set with the metadata is not particularly limited as long as the correspondence between the detection signal set and the metadata can be specified. For example, detection is performed by adding metadata to data including a detection signal set, assigning the same ID to the detection signal set and metadata, or recording the detection signal set and metadata on the same recording medium 329. Signal sets and metadata are associated. Also, metadata may be individually associated with each detection signal set, or metadata may be associated with data in which the detection signal sets are combined into one.

<< 5. Modified example >>
Hereinafter, modifications of the embodiment of the present disclosure described above will be described.

<Modification of system configuration example>
Although FIG. 26 shows an example in which one electronic device 301 executes imaging processing of a user or his surroundings, restoration processing of a restored image, and application processing using a restored image, two or more devices perform these processing. You may share and go.

For example, the information processing system 801 in FIG. 41 includes an electronic device 811 and a signal processing device 812. Then, the electronic device 811 and the signal processing device 812 may share the imaging process, the restoration process, and the application process.

For example, the electronic device 811 may perform imaging processing, and the signal processing device 812 may perform restoration processing and application processing. Alternatively, for example, the electronic device 811 may perform imaging processing and application processing, and the signal processing device 812 may perform restoration processing. Alternatively, the electronic device 811 and the signal processing device 812 may jointly perform some of the imaging process, the restoration process, and the application process.

In addition, the present disclosure can be applied to an electronic device having a function of imaging the user or the surroundings of the user other than those described above. In an electronic device having a function of imaging a user, for example, as in the above-described example, the imaging element 121 is disposed around the eyepiece unit and the display unit. Further, in the electronic device having a function of imaging the surroundings of the user, for example, as in the example described above, the imaging element 121 is disposed on the surface exposed to the outside in a state of being worn by the user. Note that the arrangement of the imaging element 121 is not limited to the above-described example, and can be appropriately changed according to the main imaging target and the like.

Note that, for example, the imaging element 121 may be disposed in the eyepiece unit or the display unit, not around the eyepiece unit or the display unit of the electronic device. For example, the imaging device 121 may be provided on the surface of the left lens 411L and the right lens 411R in FIG. 27, the surface of the finder 441 in FIG. 29, the surface of the left lens 471L and the right lens 471R in FIG. It is possible to arrange.

<Modification Example Regarding Imaging Device and Imaging Element>
In addition, for example, as the shape of the light shielding film 121b of each pixel 121a, it is possible to adopt a shape other than the above-described horizontal band type, vertical band type, L-shaped type, and a type provided with a rectangular opening.

Furthermore, for example, in the image pickup device 121 described above with reference to FIG. 5, an example in which four photodiodes 121 f of 2 rows × 2 columns are provided in one pixel 121 a is shown. It is not limited to this example.

For example, as illustrated in FIG. 42, nine photodiodes 121 f-111 to 121 f-119 arranged in 3 rows × 3 columns are provided to one on-chip lens 121 c in one pixel 121 a. You may do so. That is, one pixel output unit may include nine photodiodes 121 f.

Then, for example, by not reading out the signals of five pixels of the photodiodes 121f-11, 121f-114, 121f-117 to 121f-119, the photodiodes 121f-11, 121f-114, 121f are substantially made. An incident angle characteristic similar to that of the pixel 121a provided with the L-shaped light shielding film 121b in which the light shielding film 121b is set in the range of -117 to 121f -119 can be obtained.

In this way, it is possible to obtain the same incident angle characteristics as in the case where the light shielding film 121 b is provided without providing the light shielding film 121 b. In addition, by switching the pattern of the photodiode 121 f which does not read the signal, the incident angle directivity can be changed as in the case where the position and the range shielded by the light shielding film 121 b are changed.

In the above description, one pixel output unit is configured by one pixel 121a. However, one pixel output unit may be configured by a plurality of pixels 121a.

For example, as shown in FIG. 43, it is possible to form one pixel output unit 851 b by the pixels 121 a-111 to the pixels 121 a-119 arranged in 3 rows × 3 columns. Each of the pixels 121 a-111 to 121 a-119 includes, for example, one photodiode and does not include an on-chip lens.

For example, by adding pixel signals from each pixel 121a, a detection signal for one pixel of a detected image is generated, and output of pixel signals from some pixels 121a is stopped or addition is not performed. Thus, the incident angle directivity of the pixel output unit 851 b can be realized. For example, the pixel signals of the pixels 121a-112, 121a-113, 121a-115, and 121a-116 are added to generate a detection signal, and the pixels 121a-111, 121a-114, and It is possible to obtain the same incident angle directivity as in the case where the L-shaped light shielding film 121b is provided in the range of the pixels 121a-117 to the pixels 121a-119.

Further, by switching the pattern of the pixel 121a for adding the pixel signal to the detection signal, the incident angle directivity can be set to different values as in the case where the position and the range shielded by the light shielding film 121b are changed. .

In this case, for example, the range of the pixel output unit can be changed by changing the combination of the pixels 121a. For example, a pixel output unit 851 s can be configured of pixels 121 a-111, pixels 121 a-112, pixels 121 a-114, and pixels 121 a of 2 rows × 2 columns including the pixels 121 a-115.

Furthermore, for example, it is possible to set the range of the pixel output unit later by recording the pixel signals of all the pixels 121a and setting the combination of the pixels 121a later. Furthermore, it is possible to set the incident angle directivity of a pixel output unit later by selecting the pixel 121a which adds a pixel signal to a detection signal among the pixels 121a in the set pixel output unit.

Further, FIG. 4 illustrates an example in which different incident angle directivity is given to each pixel by using the light shielding film 121b as the modulation element or changing the combination of photodiodes contributing to the output. Then, for example, as shown in FIG. 44, it is also possible to use an optical filter 902 covering the light receiving surface of the imaging element 901 as a modulation element so that each pixel has incident angle directivity.

Specifically, the optical filter 902 is disposed to cover the entire surface of the light receiving surface 901A at a predetermined interval from the light receiving surface 901A of the imaging element 901. The light from the subject surface 31 is modulated by the optical filter 902, and then enters the light receiving surface 901 </ b> A of the image sensor 901.

For example, as the optical filter 902, it is possible to use an optical filter 902BW having a black and white lattice pattern shown in FIG. In the optical filter 902BW, a white pattern portion that transmits light and a black pattern portion that blocks light are randomly disposed. The size of each pattern is set independently of the size of the pixel of the image sensor 901.

FIG. 46 shows the light receiving sensitivity characteristics of the image sensor 901 with respect to light from the point light source PA and the point light source PB on the object plane 31 when the optical filter 902BW is used. The light from the point light source PA and the light from the point light source PB are respectively modulated by the optical filter 902BW, and then enter the light receiving surface 901A of the imaging element 901.

For example, the light reception sensitivity characteristic of the image pickup device 901 with respect to the light from the point light source PA becomes like a waveform Sa. That is, since a shadow is generated by the black pattern portion of the optical filter 902BW, a light and dark pattern is generated in the image on the light receiving surface 901A for the light from the point light source PA. Similarly, the light receiving sensitivity characteristic of the image sensor 901 with respect to the light from the point light source PB becomes like a waveform Sb. That is, since a shadow is generated by the black pattern portion of the optical filter 902BW, a light and dark pattern is generated in the image on the light receiving surface 901A for the light from the point light source PB.

Since the light from the point light source PA and the light from the point light source PB have different incident angles with respect to the white pattern portions of the optical filter 902BW, deviation occurs in how the light and dark patterns appear on the light receiving surface. Therefore, each pixel of the image sensor 901 has incident angle directivity with respect to each point light source of the object plane 31.

Details of this scheme are disclosed in, for example, Non-Patent Document 1 mentioned above.

The optical filter 902HW of FIG. 47 may be used instead of the optical filter 902BW. The optical filter 902HW includes a linear polarization element 911A and a linear polarization element 911B having the same polarization direction, and a half wave plate 912. The half wave plate 912 is between the linear polarization element 911A and the linear polarization element 911B. It is pinched. In the half-wave plate 912, instead of the black pattern portion of the optical filter 902BW, a polarized portion shown by oblique lines is provided, and the white pattern portion and the polarized portion are randomly arranged.

The linear polarization element 911A transmits only light of a predetermined polarization direction out of substantially non-polarized light emitted from the point light source PA. Hereinafter, it is assumed that the linear polarization element 911A transmits only light whose polarization direction is parallel to the drawing. Among the polarized light transmitted through the linear polarizing element 911A, the polarized light transmitted through the polarizing portion of the half-wave plate 912 is changed in polarization direction in a direction perpendicular to the drawing as the plane of polarization is rotated. On the other hand, of the polarized light transmitted through the linear polarization element 911A, the polarized light transmitted through the white pattern portion of the half-wave plate 912 does not change its polarization direction as it is parallel to the drawing. Then, the linear polarization element 911 B transmits the polarized light transmitted through the white pattern portion, and hardly transmits the polarized light transmitted through the polarized portion. Therefore, the quantity of polarized light transmitted through the polarizing section is smaller than that of polarized light transmitted through the white pattern section. As a result, a pattern of shading similar to that in the case of using the optical filter BW is generated on the light receiving surface 901A of the imaging element 901.

In addition, as shown in A of FIG. 48, it is possible to use an optical interference mask as the optical filter 902LF. The light emitted from the point light sources PA and PB of the object plane 31 is irradiated onto the light receiving surface 901A of the imaging element 901 through the optical filter 902LF. As shown in the enlarged view of the lower part of A of FIG. 48, the unevenness | corrugation of a wavelength grade is provided in the light-incidence surface of optical filter 902LF, for example. The optical filter 902LF maximizes transmission of light of a specific wavelength emitted from the vertical direction. When the change of the incident angle (the inclination with respect to the vertical direction) to the optical filter 902LF of the light of the specific wavelength emitted from the point light sources PA and PB of the object plane 31 increases, the optical path length changes. Here, the light weakens when the optical path length is an odd multiple of a half wavelength, and the light strengthens when an even multiple of a half wavelength. That is, the intensity of the transmitted light of the specific wavelength emitted from the point light sources PA and PB and transmitted through the optical filter 902LF is modulated according to the incident angle with respect to the optical filter 902LF as shown in B of FIG. The light is incident on the light receiving surface 901A. Therefore, the detection signal output from each pixel output unit of the imaging element 901 is a signal obtained by combining the light intensity after modulation of each point light source for each pixel output unit.

Details of this method are disclosed, for example, in the above-mentioned Patent Document 1.

In the methods of Patent Document 1 and Non-patent Document 1, as in the image sensor 121 using the pixel 121a of FIG. 4 or the pixel 121a of FIG. 5 described above, light is incident in units of pixels 121a without affecting adjacent pixels. Angle directivity can not be set independently. Therefore, for example, when the pattern of the optical filter 902BW or the pattern of the diffraction grating of the optical filter 902LF is different, the incident angle directivity of at least a plurality of adjacent pixels of the imaging device 901 is different. In addition, adjacent pixels 121a have incident angle directivity close to each other.

The present disclosure can also be applied to an imaging device and an imaging element that perform imaging of light of wavelengths other than visible light such as infrared light. In this case, the restored image is not an image that allows the user to visually recognize the subject, but is an image in which the user can not visually recognize the subject. In addition, since it is difficult for the normal imaging lens to transmit far infrared light, the present technology is effective, for example, when imaging far infrared light.

<Other Modifications>
In the above description, an example in which biometric authentication is performed based on an image obtained by capturing the face and eyes of the user has been described, but in the present disclosure, biometric authentication is performed based on an image obtained by capturing another part of the user such as fingerprint authentication. It can be applied to the case where it is performed.

In addition, the present disclosure can be applied to the case of detecting a movement or a state of the user's eyes other than the line of sight of the user. For example, the present disclosure can be applied to the case where blink detection, sleep detection, or the like is performed based on an image obtained by capturing an eye or a face of a user. Furthermore, the present disclosure can also be applied to, for example, detection of attachment of an electronic device 301 such as a wearable device.

Further, for example, by applying machine learning such as deep learning, it is possible to perform image recognition and the like using a detection image before restoration and a detection signal set without using a restoration image after restoration. is there.

<< 6. Application example >>
The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure is applicable to an imaging unit such as a medical device, for example, an endoscopic surgery system, a capsule endoscope, and the like.

FIG. 49 is a diagram showing an example of a schematic configuration of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied. In FIG. 49, an operator (doctor) 5067 is illustrated operating a patient 5071 on a patient bed 5069 using the endoscopic surgery system 5000. As shown, the endoscopic surgery system 5000 includes an endoscope 5001, other surgical instruments 5017, a support arm device 5027 for supporting the endoscope 5001, and various devices for endoscopic surgery. And a cart 5037 on which the

In endoscopic surgery, instead of cutting and opening the abdominal wall, a plurality of cylindrical opening tools called trockers 5025a to 5025d are punctured in the abdominal wall. Then, the barrel 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient 5071 from the trocars 5025 a to 5025 d. In the illustrated example, an insufflation tube 5019, an energy treatment instrument 5021 and a forceps 5023 are inserted into the body cavity of the patient 5071 as the other surgical instruments 5017. In addition, the energy treatment tool 5021 is a treatment tool that performs incision and peeling of tissue, sealing of a blood vessel, or the like by high-frequency current or ultrasonic vibration. However, the illustrated surgical tool 5017 is merely an example, and various surgical tools generally used in endoscopic surgery, such as forceps and retractors, may be used as the surgical tool 5017, for example.

An image of a surgical site in a body cavity of a patient 5071 captured by the endoscope 5001 is displayed on the display device 5041. The operator 5067 performs a treatment such as excision of the affected area using the energy treatment tool 5021 and the forceps 5023 while viewing the image of the operative part displayed on the display device 5041 in real time. Although illustration is omitted, the insufflation tube 5019, the energy treatment instrument 5021 and the forceps 5023 are supported by the operator 5067 or an assistant during the operation.

(Support arm device)
The support arm device 5027 includes an arm portion 5031 extending from the base portion 5029. In the illustrated example, the arm unit 5031 includes

joints

5033 a, 5033 b, 5033 c, and

links

5035 a, 5035 b, and is driven by control from the arm control device 5045. The endoscope 5001 is supported by the arm unit 5031 and the position and posture thereof are controlled. In this way, stable position fixation of the endoscope 5001 can be realized.

(Endoscope)
The endoscope 5001 includes a lens barrel 5003 whose region of a predetermined length from the tip is inserted into a body cavity of a patient 5071 and a camera head 5005 connected to the proximal end of the lens barrel 5003. In the illustrated example, the endoscope 5001 configured as a so-called rigid endoscope having a rigid barrel 5003 is illustrated, but the endoscope 5001 is configured as a so-called flexible mirror having a flexible barrel 5003 It is also good.

At the tip of the lens barrel 5003, an opening into which an objective lens is fitted is provided. A light source device 5043 is connected to the endoscope 5001, and light generated by the light source device 5043 is guided to the tip of the lens barrel by a light guide extended inside the lens barrel 5003, and an objective The light is emitted to the observation target in the body cavity of the patient 5071 through the lens. In addition, the endoscope 5001 may be a straight endoscope, or may be a oblique endoscope or a side endoscope.

An optical system and an imaging device are provided inside the camera head 5005, and the reflected light (observation light) from the observation target is condensed on the imaging device by the optical system. The observation light is photoelectrically converted by the imaging element to generate an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image. The image signal is transmitted as RAW data to a camera control unit (CCU: Camera Control Unit) 5039. The camera head 5005 has a function of adjusting the magnification and the focal length by driving the optical system appropriately.

A plurality of imaging devices may be provided in the camera head 5005 in order to cope with, for example, stereoscopic vision (3D display). In this case, a plurality of relay optical systems are provided inside the lens barrel 5003 in order to guide observation light to each of the plurality of imaging elements.

(Various devices installed in the cart)
The CCU 5039 is configured by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and centrally controls the operation of the endoscope 5001 and the display device 5041. Specifically, the CCU 5039 subjects the image signal received from the camera head 5005 to various types of image processing for displaying an image based on the image signal, such as development processing (demosaicing processing). The CCU 5039 provides the display device 5041 with the image signal subjected to the image processing. Also, the CCU 5039 transmits a control signal to the camera head 5005 to control the driving thereof. The control signal may include information on imaging conditions such as magnification and focal length.

The display device 5041 displays an image based on the image signal subjected to the image processing by the CCU 5039 under the control of the CCU 5039. In the case where the endoscope 5001 corresponds to high resolution imaging such as 4K (horizontal pixel number 3840 × vertical pixel number 2160) or 8K (horizontal pixel number 7680 × vertical pixel number 4320), and / or 3D display In the case where the display device 5041 corresponds to each of the display devices, those capable of high-resolution display and / or those capable of 3D display may be used. In the case of high-resolution imaging such as 4K or 8K, the display device 5041 can have a size of 55 inches or more to obtain a further immersive feeling. Further, a plurality of display devices 5041 different in resolution and size may be provided depending on the application.

The light source device 5043 is composed of a light source such as an LED (light emitting diode), for example, and supplies the endoscope 5001 with irradiation light when imaging the surgical site.

The arm control device 5045 is constituted by a processor such as a CPU, for example, and operates in accordance with a predetermined program to control driving of the arm portion 5031 of the support arm device 5027 according to a predetermined control method.

The input device 5047 is an input interface to the endoscopic surgery system 5000. The user can input various instructions and input instructions to the endoscopic surgery system 5000 via the input device 5047. For example, the user inputs various information related to surgery, such as physical information of a patient and information on an operation procedure of the surgery, through the input device 5047. Further, for example, the user instructs, via the input device 5047, an instruction to drive the arm unit 5031 or an instruction to change the imaging condition (type of irradiated light, magnification, focal length, etc.) by the endoscope 5001. , An instruction to drive the energy treatment instrument 5021, and the like.

The type of the input device 5047 is not limited, and the input device 5047 may be any of various known input devices. For example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5057, and / or a lever may be applied as the input device 5047. When a touch panel is used as the input device 5047, the touch panel may be provided on the display surface of the display device 5041.

Alternatively, the input device 5047 is a device mounted by the user, for example, a wearable device of glasses type or an HMD (Head Mounted Display), and various types of input according to the user's gesture or line of sight detected by these devices. Is done. Further, the input device 5047 includes a camera capable of detecting the motion of the user, and various inputs are performed in accordance with the user's gesture and the line of sight detected from the image captured by the camera. Furthermore, the input device 5047 includes a microphone capable of collecting the user's voice, and various inputs are performed by voice via the microphone. In this manner, the user (for example, the operator 5067) belonging to the clean area operates the device belonging to the unclean area in a non-contact manner by the input device 5047 being configured to be able to input various information in a non-contact manner. Is possible. In addition, since the user can operate the device without releasing his / her hand from the operating tool, the convenience of the user is improved.

The treatment tool control device 5049 controls the drive of the energy treatment instrument 5021 for ablation of tissue, incision, sealing of a blood vessel, and the like. The insufflation apparatus 5051 has a gas in the body cavity via the insufflation tube 5019 in order to expand the body cavity of the patient 5071 for the purpose of securing a visual field by the endoscope 5001 and securing a working space of the operator. Send The recorder 5053 is a device capable of recording various types of information regarding surgery. The printer 5055 is a device capable of printing various types of information regarding surgery in various types such as texts, images, and graphs.

Hereinafter, the characteristic features of the endoscopic surgery system 5000 will be described in more detail.

(Support arm device)
The support arm device 5027 includes a base portion 5029 which is a base and an arm portion 5031 extending from the base portion 5029. In the illustrated example, the arm unit 5031 includes a plurality of

joints

5033 a, 5033 b and 5033 c and a plurality of

links

5035 a and 5035 b connected by the joints 5033 b, but in FIG. The structure of the arm unit 5031 is simplified and illustrated. In practice, the shapes, the number and arrangement of the joints 5033a to 5033c and the

links

5035a and 5035b, and the direction of the rotation axis of the joints 5033a to 5033c are appropriately set so that the arm 5031 has a desired degree of freedom. obtain. For example, the arm unit 5031 may be preferably configured to have six or more degrees of freedom. Thus, the endoscope 5001 can be freely moved within the movable range of the arm unit 5031. Therefore, the lens barrel 5003 of the endoscope 5001 can be inserted into the body cavity of the patient 5071 from a desired direction. It will be possible.

The joints 5033 a to 5033 c are provided with an actuator, and the joints 5033 a to 5033 c are configured to be rotatable around a predetermined rotation axis by driving the actuators. The drive of the actuator is controlled by the arm control device 5045, whereby the rotation angles of the joint portions 5033a to 5033c are controlled, and the drive of the arm portion 5031 is controlled. Thereby, control of the position and posture of the endoscope 5001 can be realized. At this time, the arm control device 5045 can control the drive of the arm unit 5031 by various known control methods such as force control or position control.

For example, when the operator 5067 appropriately inputs an operation via the input device 5047 (including the foot switch 5057), the driving of the arm unit 5031 is appropriately controlled by the arm control device 5045 according to the operation input, and The position and attitude of the endoscope 5001 may be controlled. According to the control, after the endoscope 5001 at the tip of the arm unit 5031 is moved from an arbitrary position to an arbitrary position, the endoscope 5001 can be fixedly supported at the position after the movement. The arm unit 5031 may be operated by a so-called master slave method. In this case, the arm unit 5031 can be remotely controlled by the user via the input device 5047 installed at a location distant from the operating room.

Also, when force control is applied, the arm control device 5045 receives the external force from the user, and the actuator of each joint 5033 a to 5033 c is moved so that the arm 5031 moves smoothly following the external force. So-called power assist control may be performed. Accordingly, when the user moves the arm unit 5031 while directly touching the arm unit 5031, the arm unit 5031 can be moved with a relatively light force. Therefore, it is possible to move the endoscope 5001 more intuitively and with a simpler operation, and the convenience of the user can be improved.

Here, in general, in endoscopic surgery, the endoscope 5001 is supported by a doctor called scopist. On the other hand, by using the support arm device 5027, the position of the endoscope 5001 can be more reliably fixed without manual operation, so that an image of the operation site can be stably obtained. , Can be performed smoothly.

The arm control device 5045 may not necessarily be provided in the cart 5037. Also, the arm control device 5045 may not necessarily be one device. For example, the arm control device 5045 may be provided at each joint 5033a to 5033c of the arm 5031 of the support arm device 5027, and the arm control devices 5045 cooperate with one another to drive the arm 5031. Control may be realized.

(Light source device)
The light source device 5043 supplies the endoscope 5001 with irradiation light for imaging the operative part. The light source device 5043 is composed of, for example, a white light source configured of an LED, a laser light source, or a combination thereof. At this time, when a white light source is configured by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high accuracy. Adjustments can be made. Further, in this case, the laser light from each of the RGB laser light sources is irradiated on the observation target in time division, and the drive of the imaging device of the camera head 5005 is controlled in synchronization with the irradiation timing to cope with each of RGB. It is also possible to capture a shot image in time division. According to the method, a color image can be obtained without providing a color filter in the imaging device.

In addition, the drive of the light source device 5043 may be controlled to change the intensity of the light to be output at predetermined time intervals. The drive of the imaging element of the camera head 5005 is controlled in synchronization with the timing of the change of the light intensity to acquire images in time division, and by combining the images, high dynamic without so-called blackout and whiteout is obtained. An image of the range can be generated.

The light source device 5043 may be configured to be able to supply light of a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the mucous membrane surface layer is irradiated by irradiating narrow band light as compared with irradiation light (that is, white light) at the time of normal observation using the wavelength dependency of light absorption in body tissue. The so-called narrow band imaging (Narrow Band Imaging) is performed to image a predetermined tissue such as a blood vessel with high contrast. Alternatively, in special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiation with excitation light. In fluorescence observation, a body tissue is irradiated with excitation light and fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue while being locally injected. What irradiates the excitation light corresponding to the fluorescence wavelength of the reagent, and obtains a fluorescence image etc. can be performed. The light source device 5043 can be configured to be able to supply narrow band light and / or excitation light corresponding to such special light observation.

(Camera head and CCU)
The functions of the camera head 5005 and the CCU 5039 of the endoscope 5001 will be described in more detail with reference to FIG. FIG. 50 is a block diagram showing an example of a functional configuration of the camera head 5005 and the CCU 5039 shown in FIG.

Referring to FIG. 50, the camera head 5005 has a lens unit 5007, an imaging unit 5009, a drive unit 5011, a communication unit 5013, and a camera head control unit 5015 as its functions. The CCU 5039 also has a communication unit 5059, an image processing unit 5061, and a control unit 5063 as its functions. The camera head 5005 and the CCU 5039 are communicably connected in both directions by a transmission cable 5065.

First, the functional configuration of the camera head 5005 will be described. The lens unit 5007 is an optical system provided at the connection with the lens barrel 5003. The observation light taken in from the tip of the lens barrel 5003 is guided to the camera head 5005 and enters the lens unit 5007. The lens unit 5007 is configured by combining a plurality of lenses including a zoom lens and a focus lens. The optical characteristic of the lens unit 5007 is adjusted so as to condense the observation light on the light receiving surface of the imaging element of the imaging unit 5009. Further, the zoom lens and the focus lens are configured such that the position on the optical axis can be moved in order to adjust the magnification and the focus of the captured image.

The imaging unit 5009 includes an imaging element and is disposed downstream of the lens unit 5007. The observation light which has passed through the lens unit 5007 is condensed on the light receiving surface of the imaging device, and an image signal corresponding to the observation image is generated by photoelectric conversion. The image signal generated by the imaging unit 5009 is provided to the communication unit 5013.

As an imaging element which comprises the imaging part 5009, it is an image sensor of a CMOS (Complementary Metal Oxide Semiconductor) type, for example, and a color imaging | photography thing which has Bayer arrangement is used. In addition, as the said image pick-up element, what can respond | correspond to imaging | photography of the high resolution image of 4K or more may be used, for example. By obtaining a high resolution image of the operation site, the operator 5067 can grasp the situation of the operation site in more detail, and can proceed the surgery more smoothly.

In addition, an imaging element constituting the imaging unit 5009 is configured to have a pair of imaging elements for acquiring image signals for right eye and left eye corresponding to 3D display. The 3D display enables the operator 5067 to more accurately grasp the depth of the living tissue at the operation site. When the imaging unit 5009 is configured as a multi-plate type, a plurality of lens units 5007 are also provided corresponding to each imaging element.

In addition, the imaging unit 5009 may not necessarily be provided in the camera head 5005. For example, the imaging unit 5009 may be provided inside the lens barrel 5003 immediately after the objective lens.

The drive unit 5011 is configured by an actuator, and moves the zoom lens and the focus lens of the lens unit 5007 along the optical axis by a predetermined distance under the control of the camera head control unit 5015. Thereby, the magnification and the focus of the captured image by the imaging unit 5009 may be appropriately adjusted.

The communication unit 5013 is configured of a communication device for transmitting and receiving various types of information to and from the CCU 5039. The communication unit 5013 transmits the image signal obtained from the imaging unit 5009 to the CCU 5039 via the transmission cable 5065 as RAW data. At this time, it is preferable that the image signal be transmitted by optical communication in order to display the captured image of the surgical site with low latency. During the operation, the operator 5067 performs the operation while observing the condition of the affected area by the captured image, so for safer and more reliable operation, the moving image of the operation site is displayed in real time as much as possible It is because that is required. In the case where optical communication is performed, the communication unit 5013 is provided with a photoelectric conversion module which converts an electrical signal into an optical signal. The image signal is converted into an optical signal by the photoelectric conversion module, and then transmitted to the CCU 5039 via the transmission cable 5065.

The communication unit 5013 also receives, from the CCU 5039, a control signal for controlling the drive of the camera head 5005. The the control signal, for example, information that specifies the frame rate of the captured image, information that specifies the exposure value at the time of imaging, and / or magnification and information, etc. indicating that specifies the focal point of the captured image, captured Contains information about the condition. The communication unit 5013 provides the received control signal to the camera head control unit 5015. The control signal from CCU 5039 may also be transmitted by optical communication. In this case, the communication unit 5013 is provided with a photoelectric conversion module that converts an optical signal into an electric signal, and the control signal is converted into an electric signal by the photoelectric conversion module and is then provided to the camera head control unit 5015.

Note that the imaging conditions such as the frame rate, the exposure value, the magnification, and the focus described above are automatically set by the control unit 5063 of the CCU 5039 based on the acquired image signal. That is, so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function are installed in the endoscope 5001.

The camera head control unit 5015 controls the drive of the camera head 5005 based on the control signal from the CCU 5039 received via the communication unit 5013. For example, the camera head control unit 5015 controls the drive of the imaging element of the imaging unit 5009 based on the information to specify the frame rate of the captured image and / or the information to specify the exposure at the time of imaging. Also, for example, the camera head control unit 5015 appropriately moves the zoom lens and the focus lens of the lens unit 5007 via the drive unit 5011 based on the information indicating that the magnification and the focus of the captured image are designated. The camera head control unit 5015 may further have a function of storing information for identifying the lens barrel 5003 and the camera head 5005.

Note that by disposing the lens unit 5007, the imaging unit 5009, and the like in a sealed structure with high airtightness and waterproofness, the camera head 5005 can have resistance to autoclave sterilization.

Next, the functional configuration of the CCU 5039 will be described. The communication unit 5059 is configured of a communication device for transmitting and receiving various types of information to and from the camera head 5005. The communication unit 5059 receives an image signal transmitted from the camera head 5005 via the transmission cable 5065. At this time, as described above, the image signal can be suitably transmitted by optical communication. In this case, the communication unit 5059 is provided with a photoelectric conversion module that converts an optical signal into an electrical signal in response to optical communication. The communication unit 5059 provides the image processing unit 5061 with the image signal converted into the electrical signal.

Further, the communication unit 5059 transmits a control signal for controlling driving of the camera head 5005 to the camera head 5005. The control signal may also be transmitted by optical communication.

An image processing unit 5061 performs various types of image processing on an image signal that is RAW data transmitted from the camera head 5005. As the image processing, for example, development processing, high image quality processing (band emphasis processing, super-resolution processing, NR (noise reduction) processing and / or camera shake correction processing, etc.), and / or enlargement processing (electronic zoom processing) And various other known signal processings. The image processing unit 5061 also performs detection processing on the image signal to perform AE, AF, and AWB.

The image processing unit 5061 is configured by a processor such as a CPU or a GPU, and the image processing and the detection processing described above can be performed by the processor operating according to a predetermined program. When the image processing unit 5061 is configured by a plurality of GPUs, the image processing unit 5061 appropriately divides the information related to the image signal, and performs image processing in parallel by the plurality of GPUs.

The control unit 5063 performs various types of control regarding imaging of the surgical site by the endoscope 5001 and display of the imaged image. For example, the control unit 5063 generates a control signal for controlling the drive of the camera head 5005. At this time, when the imaging condition is input by the user, the control unit 5063 generates a control signal based on the input by the user. Alternatively, when the endoscope 5001 is equipped with the AE function, the AF function, and the AWB function, the control unit 5063 determines the optimum exposure value, focal length, and the like according to the result of the detection processing by the image processing unit 5061. The white balance is appropriately calculated to generate a control signal.

Further, the control unit 5063 causes the display device 5041 to display an image of the operative site based on the image signal subjected to the image processing by the image processing unit 5061. At this time, the control unit 5063 recognizes various objects in the surgical site image using various image recognition techniques. For example, the control unit 5063 detects a shape, a color, and the like of an edge of an object included in an operation part image, thereby enabling a surgical tool such as forceps, a specific living part, bleeding, mist when using the energy treatment tool 5021, and the like. It can be recognized. When the control unit 5063 causes the display device 5041 to display the image of the operation unit, the control unit 5063 superimposes and displays various types of surgery support information on the image of the operation unit, using the recognition result. The operation support information is superimposed and presented to the operator 5067, which makes it possible to proceed with the operation more safely and reliably.

A transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electrical signal cable compatible with communication of electrical signals, an optical fiber compatible with optical communication, or a composite cable of these.

Here, in the illustrated example, communication is performed by wire communication using the transmission cable 5065, but communication between the camera head 5005 and the CCU 5039 may be performed wirelessly. When the communication between the two is performed wirelessly, it is not necessary to lay the transmission cable 5065 in the operating room, so that the movement of the medical staff in the operating room can be eliminated by the transmission cable 5065.

Heretofore, an example of the endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied has been described. In addition, although the endoscopic surgery system 5000 was demonstrated as an example here, the system to which the technique which concerns on this indication can be applied is not limited to this example. For example, the technology according to the present disclosure may be applied to a flexible endoscopic system for examination or a microsurgical system.

Among the configurations described above, the technology according to the present disclosure can be suitably applied to the imaging unit 5009. By applying the technology according to the present disclosure to the imaging unit 5009, the imaging unit 5009 can be miniaturized. In addition, since the degree of freedom of arrangement of each pixel of the imaging device is high, it is easy to obtain a surgical site image at a desired position, and it is possible to perform surgery more safely and more reliably.

FIG. 51 is a diagram showing an example of a schematic configuration of an in-vivo information acquiring system 5400 to which the technology according to the present disclosure can be applied. Referring to FIG. 51, the in-vivo information acquisition system 5400 includes a capsule endoscope 5401 and an external control device 5423 that comprehensively controls the operation of the in-vivo information acquisition system 5400. At the time of examination, the capsule endoscope 5401 is swallowed by the patient. The capsule endoscope 5401 has an imaging function and a wireless communication function, and moves inside the organ such as the stomach and intestine by peristaltic movement and the like while being naturally discharged from the patient, Images (hereinafter also referred to as in-vivo images) are sequentially captured at predetermined intervals, and information on the in-vivo images is sequentially wirelessly transmitted to the external control device 5423 outside the body. The external control device 5423 generates image data for displaying the in-vivo image on a display device (not shown) based on the received information about the in-vivo image. Thus, in the in-vivo information acquiring system 5400, it is possible to obtain an image obtained by imaging the appearance of the inside of the patient's body at any time during the period from when the capsule endoscope 5401 is swallowed until it is discharged.

The configurations and functions of the capsule endoscope 5401 and the external control device 5423 will be described in more detail. As illustrated, the capsule endoscope 5401 includes a light source unit 5405, an imaging unit 5407, an image processing unit 5409, a wireless communication unit 5411, a power feeding unit 5415, a power supply unit 5417, and a state detection in a capsule casing 5403. The functions of the unit 5419 and the control unit 5421 are installed.

The light source unit 5405 is formed of, for example, a light source such as a light emitting diode (LED), and emits light to the imaging field of the imaging unit 5407.

The imaging unit 5407 includes an imaging device and an optical system including a plurality of lenses provided in front of the imaging device. Reflected light is irradiated to the body tissue to be observed light (hereinafter, referred to as observation light) is condensed by the optical system and is incident on the imaging element. The imaging device receives the observation light and performs photoelectric conversion to generate an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image. The image signal generated by the imaging unit 5407 is provided to the image processing unit 5409. Note that, as an imaging element of the imaging unit 5407, various known imaging elements such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor may be used.

The image processing unit 5409 is configured by a processor such as a central processing unit (CPU) or a graphics processing unit (GPU), and performs various signal processing on the image signal generated by the imaging unit 5407. The signal processing may be minimum processing for transmitting the image signal to the external control device 5423 (eg, compression of image data, conversion of frame rate, conversion of data rate, and / or conversion of format, etc.) . By configuring the image processing unit 5409 to perform only the minimum necessary processing, the image processing unit 5409 can be realized with smaller size and lower power consumption. It is suitable. However, if space in the housing 5403 or the power consumption is sufficient, the image processing unit 5409 may perform additional signal processing (for example, noise removal processing or other high image quality processing). Good. The image processing unit 5409 supplies the image signal subjected to the signal processing to the wireless communication unit 5411 as RAW data. When the state detection unit 5419 acquires information on the state (motion, posture, etc.) of the capsule endoscope 5401, the image processing unit 5409 associates the information with the information to wirelessly transmit the image signal. The communication unit 5411 may be provided. Thereby, the captured image can be associated with the position in the body in which the image is captured, the imaging direction of the image, and the like.

The wireless communication unit 5411 is configured of a communication device capable of transmitting and receiving various information to and from the external control device 5423. The communication device includes an antenna 5413 and a processing circuit that performs modulation for transmitting and receiving a signal. The wireless communication unit 5411 performs predetermined processing such as modulation processing on the image signal subjected to the signal processing by the image processing unit 5409, and transmits the image signal to the external control device 5423 through the antenna 5413. In addition, the wireless communication unit 5411 receives a control signal related to drive control of the capsule endoscope 5401 from the external control device 5423 via the antenna 5413. The wireless communication unit 5411 provides the received control signal to the control unit 5421.

The feeding portion 5415 includes an antenna coil for receiving power, a power regeneration circuit which regenerates power from current generated in the antenna coil, a booster circuit, and the like. In the feeding portion 5415, power is generated using a so-called contactless charging principle. Specifically, when a magnetic field (electromagnetic wave) of a predetermined frequency is applied to the antenna coil of the feeding portion 5415 from the outside, an induced electromotive force is generated in the antenna coil. The electromagnetic wave may be, for example, a carrier wave transmitted from the external control device 5423 via the antenna 5425. Electric power is regenerated from the induced electromotive force by the electric power regenerating circuit, and the electric potential is appropriately adjusted in the booster circuit to generate electric power for storage. The power generated by the power supply unit 5415 is stored in the power supply unit 5417.

The power supply unit 5417 is formed of a secondary battery, and stores the power generated by the power supply unit 5415. In FIG. 51, an arrow or the like indicating the supply destination of the power from the power supply unit 5417 is omitted to avoid the drawing being complicated, but the power stored in the power supply unit 5417 is the light source unit 5405. The image pickup unit 5407, the image processing unit 5409, the wireless communication unit 5411, the state detection unit 5419, and the control unit 5421 can be used to drive them.

The state detection unit 5419 includes a sensor for detecting the state of the capsule endoscope 5401 such as an acceleration sensor and / or a gyro sensor. The state detection unit 5419 can obtain information on the state of the capsule endoscope 5401 from the detection result of the sensor. The state detection unit 5419 provides the image processing unit 5409 with information on the acquired state of the capsule endoscope 5401. In the image processing unit 5409, as described above, information on the state of the capsule endoscope 5401 may be associated with the image signal.

The control unit 5421 is configured by a processor such as a CPU, and centrally controls the operation of the capsule endoscope 5401 by operating according to a predetermined program. Control unit 5421 drives light source unit 5405, imaging unit 5407, image processing unit 5409, wireless communication unit 5411, power supply unit 5415, power supply unit 5417, and state detection unit 5419 according to a control signal transmitted from external control device 5423. By appropriately controlling, the functions of the respective units as described above are realized.

The external control device 5423 can be a processor such as a CPU or a GPU, or a microcomputer or a control board on which memory elements such as a processor and a memory are mixed. The external control device 5423 has an antenna 5425, and can transmit and receive various types of information to and from the capsule endoscope 5401 via the antenna 5425. Specifically, the external control device 5423 controls the operation of the capsule endoscope 5401 by transmitting a control signal to the control unit 5421 of the capsule endoscope 5401. For example, with the control signal from the external control device 5423, the irradiation condition of light to the observation target in the light source unit 5405 can be changed. In addition, an imaging condition (for example, a frame rate in the imaging unit 5407, an exposure value, and the like) can be changed by a control signal from the external control device 5423. Further, the contents of processing in the image processing unit 5409 and conditions (for example, transmission interval, number of transmission images, etc.) under which the wireless communication unit 5411 transmits an image signal may be changed by a control signal from the external control device 5423. .

Further, the external control device 5423 performs various types of image processing on the image signal transmitted from the capsule endoscope 5401 and generates image data for displaying the captured in-vivo image on the display device. As the image processing, for example, development processing (demosaicing processing), high image quality processing (band emphasis processing, super-resolution processing, NR (noise reduction) processing and / or camera shake correction processing, etc.), and / or enlargement processing ( Various known signal processing may be performed, such as electronic zoom processing). The external control device 5423 controls the drive of the display device (not shown) to display the in-vivo image captured based on the generated image data. Alternatively, the external control device 5423 may cause the recording device (not shown) to record the generated image data, or may cause the printing device (not shown) to print out.

Heretofore, an example of the in-vivo information acquiring system 5400 to which the technology according to the present disclosure can be applied has been described. Among the configurations described above, the technology according to the present disclosure can be suitably applied to the imaging unit 5407. By applying the technology according to the present disclosure to the imaging unit 5407, the capsule endoscope 5401 can be further miniaturized, and thus the burden on the patient can be further reduced.

<< 7. Other >>
The series of processes described above can be performed by hardware or software. When the series of processes are performed by software, a program that configures the software is installed on a computer. Here, the computer includes a computer (for example, the control unit 123 or the like) incorporated in dedicated hardware.

The program executed by the computer can be provided by being recorded in, for example, a recording medium (eg, the recording medium 130 or the like) as a package medium or the like. Also, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

Note that the program executed by the computer may be a program that performs processing in chronological order according to the order described in this specification, in parallel, or when necessary, such as when a call is made. It may be a program to be processed.

Further, in the present specification, a system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same case. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device housing a plurality of modules in one housing are all systems. .

Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present technology.

For example, the present technology can have a cloud computing configuration in which one function is shared and processed by a plurality of devices via a network.

Further, each step described in the above-described flowchart can be executed by one device or in a shared manner by a plurality of devices.

Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step can be executed by being shared by a plurality of devices in addition to being executed by one device.

The present disclosure can also have the following configurations.

(1)
In the electronic device worn or used by the user,
The imaging unit is disposed at a position where the surroundings of a user wearing or using the electronic device are captured, and receives incident light from an object that is incident without passing through either the imaging lens or the pinhole, An electronic apparatus comprising: an imaging unit including a plurality of pixel output units for outputting one detection signal indicating an output pixel value modulated by an incident angle of incident light.
(2)
The electronic device according to (1), wherein each of the pixel output units is disposed on a surface exposed to the outside of the electronic device when the electronic device is attached to the user.
(3)
The electronic device according to (2), wherein the electronic device is mounted on a head of the user.
(4)
The electronic device is a glasses type or a goggle type. The electronic device according to (3).
(5)
The electronic device is a headphone The electronic device according to (3).
(6)
The electronic device is a camera. The electronic device according to (1).
(7)
The electronic device according to any one of (1) to (6), further including a restoration unit that restores a restored image using a plurality of the detection signals from each of the pixel output units.
(8)
The electronic device according to (7), wherein the restoration unit restores the restored image using a plurality of coefficients indicating the incident angle directivity of each pixel output unit.
(9)
At least a part of the pixel output units are arranged in the deformable part of the electronic device;
The electronic device according to (8), wherein the restoration unit selects the coefficient used for restoration of the restored image based on a degree of deformation of the deformable portion.
(10)
The electronic device according to (7) or (8), further including: a control unit that executes a predetermined process based on the restored image.
(11)
The electronic device according to (10), wherein the control unit performs recognition processing of the surroundings of the user based on the restored image.
(12)
The electronic device according to (11), further including: a recording unit configured to control recording of the restored image.
(13)
Any one of the above (1) to (11), further comprising: an association unit that associates the detection signal set including a plurality of detection signals with metadata used when recovering a restored image using the detection signal set. Electronic device described in.
(14)
The imaging unit includes one or more imaging devices.
The electronic device according to any one of (1) to (13), wherein the plurality of pixel output units are provided in the imaging device.
(15)
The electronic apparatus according to any one of (1) to (14), wherein each of the pixel output units is disposed in two or more distant regions.
(16)
The electronic apparatus according to any one of (1) to (15), wherein the plurality of pixel output units have a configuration capable of independently setting the incident angle directivity.
(17)
The plurality of pixel output units are
With one photodiode,
An electronic device according to (16), further comprising: a light shielding film configured to block incidence of a part of the incident light to the photodiode.
(18)
Among the plurality of pixel output units, at least two pixel output units include a plurality of photodiodes, and the photodiodes contributing to the output are different from each other, so that the incident angle directivity differs from each other (16) Electronic device described.
(19)
The imaging unit has a configuration for causing the incident angle directivity to the incident light of output pixel values of at least two pixel output units among the plurality of pixel output units to be different from each other. The electronic device according to any one of (1) to (18).

In addition, the effect described in this specification is an illustration to the last, is not limited, and may have other effects.

DESCRIPTION OF SYMBOLS 101 imaging apparatus, 111 signal processing control part, 121 imaging element, 121a, 121a 'pixel, 121A light receiving surface, 121b light shielding film, 121c on-chip lens, 121e, 121f photodiode, 122 restoration part, 123 control part, 125 detection part , 126 association unit, 301 electronic device, 311 imaging unit, 312 signal processing control unit, 321 restoration unit, 322 control unit, 325 association unit, 326 output unit, 328 recording and reproduction unit, 401 wearable device, 411L, 411R lens, 412 Frame, 431 camera, 441 finder, 461 head mounted display, 471L, 471R lens, 491 PC, 501 display Ray, 502 bezel, 601 wearable device, 611L, 611R lens, 612 frame, 631 camera, 641 mount, 642 flash unit, 643 lens, 661 head mounted display, 671 main unit, 672 head pad, 691 headphones, 701L, 701R Housing, 702 Headset, 721 Headphones, 731L, 731R Housing, 732 Neckband, 751 Bending Sensor, 801 Information Processing System, 811 Electronics, 812 Signal Processor, 901b, 901s Pixel Output Unit

Claims

In the electronic device worn or used by the user,
The imaging unit is disposed at a position where the surroundings of a user wearing or using the electronic device are captured, and receives incident light from an object that is incident without passing through either the imaging lens or the pinhole, An electronic apparatus comprising: an imaging unit including a plurality of pixel output units for outputting one detection signal indicating an output pixel value modulated by an incident angle of incident light.
The electronic device according to claim 1, wherein each of the pixel output units is disposed on a surface exposed to the outside of the electronic device when the electronic device is attached to the user.
The electronic device according to claim 2, wherein the electronic device is attached to a head of the user.
The electronic device according to claim 3, wherein the electronic device is a glasses type or a goggle type.
The electronic device according to claim 3, wherein the electronic device is a headphone.
The electronic device according to claim 1, wherein the electronic device is a camera.
The electronic device according to claim 1, further comprising a restoration unit that restores a restored image using a plurality of the detection signals from each of the pixel output units.
The electronic device according to claim 7, wherein the restoration unit restores the restored image using a plurality of coefficients indicating the incident angle directivity of each pixel output unit.
At least a part of the pixel output units are arranged in the deformable part of the electronic device;
The electronic device according to claim 8, wherein the restoration unit selects the coefficient used for restoration of the restored image based on a degree of deformation of the deformable portion.
The electronic device according to claim 7, further comprising: a control unit that executes a predetermined process based on the restored image.
The electronic device according to claim 10, wherein the control unit performs recognition processing of the surroundings of the user based on the restored image.
The electronic device according to claim 11, further comprising: a recording unit that controls recording of the restored image.
The electronic device according to claim 1, further comprising: an association unit that associates the detection signal set including a plurality of the detection signals with metadata used in restoring a restored image using the detection signal set.
The imaging unit includes one or more imaging devices.
The electronic device according to claim 1, wherein a plurality of the pixel output units are provided in the imaging device.
The electronic device according to claim 1, wherein each of the pixel output units is disposed in two or more distant regions.
The electronic device according to claim 1, wherein the plurality of pixel output units have a configuration capable of independently setting the incident angle directivity.
The plurality of pixel output units are
With one photodiode,
The electronic device according to claim 16, further comprising: a light shielding film configured to block incidence of a part of the incident light to the photodiode.
The incident angle directivity is different from each other by providing at least two pixel output units among the plurality of pixel output units with a plurality of photodiodes and making the photodiodes contributing to the output different from each other. Electronic devices.
The imaging unit has a configuration for causing the incident angle directivity to the incident light of output pixel values of at least two pixel output units among the plurality of pixel output units to be different from each other. Item 1. The electronic device according to item 1.