JP6724560B2 - Imaging device and image processing device - Google Patents

Description

The present invention relates to an imaging apparatus and an image processing apparatus.

There is known an imaging device that acquires information (so-called light field information) indicating an incident position, an incident direction, and an intensity of an incident light ray (for example, Patent Document 1). An image (for example, an image focused on an arbitrary image plane within a predetermined range) is generated by performing a predetermined calculation on the acquired light field information. No consideration is given to the handling of information regarding the microlenses and the image pickup device included in the image pickup apparatus.

Japanese Patent Publication No. 2008-515110

According to the first aspect, the imaging device includes a plurality of two-dimensionally arranged microlenses and a plurality of pixel groups including a plurality of pixels, and collects light passing through each microlens of the plurality of microlenses. An image sensor that receives light from each pixel group, a storage unit that stores first data regarding at least one of the shapes and sizes of the plurality of microlenses, and second data regarding the arrangement of the plurality of microlenses. The plurality of microlenses include a first microlens and a second microlens different in at least one of shape and size from the first microlens .
According to the second aspect, the imaging device has a plurality of microlenses arranged two-dimensionally and a plurality of pixel groups including a plurality of pixels, and outputs the light passing through each microlens of the plurality of microlenses. A data acquisition unit that externally acquires an image sensor that receives light in each pixel group, first data regarding at least one of the shapes and sizes of the plurality of microlenses, and second data regarding the arrangement of the plurality of microlenses. And the plurality of microlenses includes a first microlens and a second microlens different in at least one of shape and size from the first microlens .
According to the third aspect, the image processing device is arranged two-dimensionally , including the first microlens and the second microlens having a shape and a size different from those of the first microlens. A first input section for receiving first data relating to at least one of the shapes and sizes of the plurality of microlenses, and second data relating to the arrangement of the plurality of microlenses; and light that has passed through the plurality of microlenses. A photoelectric conversion output data output by a plurality of pixel groups into which a light is incident, a second input portion to which the photoelectric conversion output data is input, and a predetermined value using the first data and the second data for the photoelectric conversion output data. An image processing unit that performs calculation to create image data .

Block diagram showing the configuration of the image processing system Sectional drawing which shows the structure of a 1st camera and a 2nd camera typically. The figure which shows the structure of a 1st image sensor typically. The figure which shows the structure of a 2nd image sensor typically. Schematic diagram of the structure of the original image file stored by the control unit Sectional drawing which showed typically the luminous flux from the light spot on the image surface of a synthetic|combination object, and the image sensor. Illustration of image composition processing The figure which shows typically the structure of the 3rd imaging device which a 3rd camera has. Schematic diagram of the structure of the original image file The top view which shows typically the structure of the 4th imaging element which a 4th camera has. Schematic diagram of the structure of the original image file The top view which shows typically the structure of the 5th image pick-up element which a 5th camera has. Schematic diagram of the structure of the original image file The top view which shows typically the structure of the 6th image pick-up element which a 6th camera has. Schematic diagram of the structure of the original image file The figure which shows the modification of the size, shape, and arrangement of microlenses

(First embodiment)
FIG. 1 is a block diagram showing the configuration of the image processing system according to the first embodiment. The image processing system 1 includes a first camera 2a, a second camera 2b, and an image processing device 3. In the following description, the first camera 2a and the second camera 2b are collectively referred to as the camera 2.

The camera 2 captures a subject image and stores the original image file in a portable storage medium (so-called memory card or the like). The original image file includes original image data. The original image data is so-called light field information, which is image data used for computational photography such as refocusing. That is, the original image data is image data that is not suitable for viewing as it is, and image data of the subject image can be generated by performing a predetermined calculation. The camera 2 and the image processing device 3 have a function (image combining function) of generating image data of a subject image from original image data. The image composition function will be described later in detail.

FIG. 2A is a sectional view schematically showing the configuration of the first camera 2a. The first camera 2a includes an image pickup optical system 21, a first image pickup element 22a, a display unit 23, a control unit 24, and a ROM 25.

The imaging optical system 21 forms a subject image. The first image pickup device 22a outputs a photoelectric conversion signal (image pickup signal) obtained by photoelectrically converting subject light that has passed through the image pickup optical system 21. The display unit 23 is a display device having a display screen such as a liquid crystal panel. The ROM 25 is a non-volatile storage medium. The ROM 25 stores a control program and image sensor data described later. Although details will be described later, the contents of the image pickup device data stored in the ROM 25 are contents corresponding to the first image pickup device 22a.

The control unit 24 includes a CPU, peripheral circuits, and the like. The control unit 24 controls each unit of the first camera 2a by reading and executing the control program from the ROM 25. Note that instead of the control unit 24 that reads and executes the control program, a control unit that has an equivalent function configured by an electronic circuit or the like may be used.

The first camera 2a has an unillustrated attachment/detachment mechanism to/from which the memory card 26 can be attached/detached. The memory card 26 is a portable storage medium. The control unit 24 stores in the memory card 26 an original image file based on the image pickup signal output by the first image pickup device 22a and the image pickup device data stored in the ROM 25. The original image file may be stored in a storage device of the server via a built-in memory (non-volatile memory) (not shown) of the camera 2 or a network. Although the method of accessing the original image file by the control unit 24 differs depending on the location where the original image file is stored, there is no difference in the method of generating a refocus image using the image combining function described later.

FIG. 2B is a sectional view schematically showing the configuration of the second camera 2b. The second camera 2b has a configuration in which the first image sensor 22a of the first camera 2a is replaced with the second image sensor 22b. Although details will be described later, the contents of the image pickup device data stored in the ROM 25 are contents corresponding to the second image pickup device 22b. In the following description, the first image sensor 22a and the second image sensor 22b are collectively referred to as the image sensor 22.

FIG. 3 is a diagram schematically showing the configuration of the first image sensor 22a. 3A is a perspective view, FIG. 3B is a plan view seen from the image pickup optical system 21, and FIG. 3C is a sectional view. The first image sensor 22a includes a first microlens array 221a and a light receiving element array 222.

The first microlens array 221a has a plurality of first microlenses 223a. The plurality of first microlenses 223a have a circular shape. The plurality of first microlenses 223a are squarely arranged in a two-dimensional form. In the following description, the diameter of the first microlens 223a will be denoted by w1 and the focal length will be denoted by f1.

The light receiving element array 222 has a plurality of light receiving elements 224. The plurality of light receiving elements 224 are two-dimensionally arranged in a square array on the light receiving surface of the light receiving element array 222. In the following description, the arrangement interval (pitch) of the plurality of light receiving elements 224 is expressed as dp.

The light beam from the subject that has passed through the imaging optical system 21 (FIG. 2) passes through any of the plurality of first microlenses 223a and is incident on any of the plurality of light receiving elements 224. The light receiving element 224 is arranged near the focal position of the first microlens 223a. In other words, the light receiving element 224 is arranged near the position away from the main plane of the first microlens 223a by the focal length f1 of the first microlens 223a.

The plurality of light receiving elements 224 and the plurality of first microlenses 223a have a correspondence relationship with each other. Specifically, as shown in FIG. 3B and FIG. 3C, the plurality of light receiving elements 224 arranged in the region 225 covered by one first microlens 223a includes the first microlenses. Only the light flux that has passed through the lens 223a is incident. In the following description, the plurality of light receiving elements 224 corresponding to one first microlens 223a will be referred to as a first light receiving element group 225a. That is, one first microlens 223a and one first light receiving element group 225a have a one-to-one correspondence.

Although details will be described later, in order to realize the image combining function, relative positions of one first microlens 223a and one first light receiving element group 225a corresponding to the one first microlens 223a. You need to understand the relationship. The relative positional relationship referred to here is to identify which light receiving element 224 included in the first light receiving element group 225a the light entering from a direction at a certain position of the first microlens 223a enters. Required information. That is, the relative positional relationship means what kind of positional relationship each light receiving element 224 included in the first light receiving element group 225a has with respect to the first microlens 223a. For example, when it is known from the intersection P1 of the main plane 226 of the microlens 223 and the optical axis O shown in FIG. 3C, it is known in which direction and how far each light receiving element 224 exists. It is possible to know which light receiving element 224 the light incident from a certain position of the microlens 223 enters. The distance from the main plane 226 to each light receiving element 224 is equal to the focal length f1 of the first microlens 223a. Since the intersection of the optical axis O passing through the intersection P1 and the first light receiving element group 225a is known, the direction of each light receiving element 224 seen from the intersection P1 can be specified from the pitch dp of the light receiving elements 224. .. That is, the relative positional relationship between the first microlens 223a and each light receiving element 224 in the corresponding first light receiving element group 225a is specified by the focal length f1 of the first microlens 223a and the pitch dp of the light receiving elements 224. be able to.

FIG. 4 is a diagram schematically showing the configuration of the second image sensor 22b. 4A is a perspective view, FIG. 4B is a plan view seen from the imaging optical system 21, and FIG. 4C is a cross-sectional view, which are respectively FIG. 3A, FIG. 3B, and FIG. It corresponds to 3(c). The second image sensor 22b has a second microlens array 221b and a light receiving element array 222.

The second microlens array 221b has a plurality of second microlenses 223b. The plurality of second microlenses 223b have a circular shape. The plurality of second microlenses 223b are squarely arranged in a two-dimensional form. In the following description, the diameter of the second microlens 223b is written as w2, and the focal length is written as f2, and the plurality of light receiving elements 224 corresponding to one second microlens 223b are set to the second It is called a light receiving element group 225b. The first microlens array 221a and the second microlens array 221b are the microlens array 221, the first microlens 223a and the second microlens 223b are the microlens 223, and the first light receiving element group 225a and the second light receiving element group 225b. Are collectively referred to as a light receiving element group 225.

The diameter w2 of the second microlens 223b is larger than the diameter w1 of the first microlens 223a. Therefore, the number of light receiving elements 224 included in one second light receiving element group 225b is larger than the number of light receiving elements 224 included in one first light receiving element group 225a.

FIG. 5 is a schematic diagram showing an example of the structure of an original image file stored by the control unit 24. In the following description, first, the original image file 40 stored by the control unit 24 of the first camera 2a will be described with reference to FIG.

The original image file 40 includes a header part 41 and an image data part 42. In the image data section 42, the photoelectric conversion signal (image pickup signal) output from the image pickup device 22 is stored as original image data 45. The original image data 45 is data in which so-called light field information is recorded. The original image data 45 is data in which the position of each light receiving element 224 and the amount of light received by the light receiving element 224 are recorded. For example, the original image data 45 is so-called RAW data in which the photoelectric conversion signals output from the plurality of light receiving elements 224 are stored without being processed. If the RAW data is data obtained by raster-scanning the received light output of each light receiving element 224, the number X of light receiving elements 224 included in the light receiving element array 222 arranged in the x direction and the number Y arranged in the y direction (light receiving element array It is possible to specify the position of the corresponding light receiving element 224 in the form of x-coordinates and y-coordinates from the number of vertical and horizontal light receiving elements 222) and the storage positions of those light receiving outputs in the RAW data. The number of light receiving elements in the vertical and horizontal directions of the light receiving element array 222 may be included in the header portion 41 as a part of the image pickup element data 43. Further, even if the RAW data is data in which the received light output of each light receiving element 224 is stored in an order other than the raster scan, the corresponding position of the light receiving element 224 can be similarly specified in the form of x-coordinates and y-coordinates. it can. That is, such RAW data can be said to be data in which the position of each light receiving element 224 and the amount of light received by the light receiving element 224 are recorded. The original image data 45 may be JPEG data obtained by subjecting the photoelectric conversion signal to known image processing.

The header section 41 can include image sensor data 43 together with various metadata such as shooting settings and shooting date and time. The image pickup element data 43 includes size data 46 regarding the size of the first microlens 223a and distance data 47 regarding the distance between the first microlens 223a and the light receiving element array 222 (light receiving element 224). The size data 46 can be diameter data indicating the diameter w1 of the microlens 223a, for example. The size data may use a radius. Further, as will be described later, the shape (circular shape) of the microlens 223a may be included in the size data. In addition, the distance data 47 can be focal length data indicating the focal length f1 of the first microlens 223a. It should be noted that the image sensor data 43 includes the number X of the light receiving elements 224 included in the light receiving element array 222 arranged in the x direction and the number Y arranged in the y direction (the number of light receiving elements in the vertical and horizontal directions of the light receiving element array 222). Good.

If the microlenses 223 included in the microlens array 221 are sequentially numbered, by referring to the image sensor data 43, the light receiving elements 225 corresponding to the microlenses 223 to which a certain number is assigned are included in the light receiving elements 225. The element 224 can be identified. That is, it is possible to extract the received light output of the light receiving element 224 included in the light receiving element group 225 corresponding to the microlens 223 to which a certain number is given, from the RAW data.

As described above, the distance between the main plane of the first microlens 223a and the light receiving element 224 is equal to the focal length f1 of the first microlens 223a. Further, the relative positional relationship between the first microlens 223a and the respective light receiving elements 224 in the corresponding first light receiving element group 225a is, as shown in FIG. 3C, the focal length f1 of the first microlens 223a. It can be specified from the pitch dp of the light receiving elements 224.

For example, the focus of the first microlens 223a can be determined by deciding in which direction and in what direction a certain light receiving element 224 in the first light receiving element group 225a is located from the intersection of the main plane of the first microlens 223a and the optical axis. It can be specified from the distance f1 and the pitch dp of the light receiving elements 224.

That is, the distance data 47 is data relating to the positional relationship between the first microlens 223a and the first light receiving element group 225a. More specifically, the distance data 47 is data regarding the positional relationship between the intersection of the main plane of the first microlens 223a and the optical axis and the first light receiving element group 225a.

The values stored in the ROM 25 and the original image file 40 as the size data 46 and the distance data 47 may be design values (ideal values) or may be actual measurement values considering individual differences. In addition to the design value, the amount of deviation from the design value may be stored. Further, the positional shift amount between the microlens array 221 and the light receiving element array 222 may be stored.

Next, the original image file 40 stored by the control unit 24 of the second camera 2b will be described. The original image file 40 stored by the control unit 24 of the second camera 2b is the same as the original image file 40 stored by the control unit 24 of the first camera 2a except that the numerical values of the size data 46 and the distance data 47 are different. It has a similar structure.

The controller 24 of the second camera 2b stores size data 46 indicating the diameter w2 of the second microlens 223b. The control unit 24 of the second camera 2b stores the distance data 47 indicating the focal length f2 of the second microlens 223b.

The image combining function of the camera 2 and the image processing device 3 will be described. The image synthesizing function in the present embodiment is a function of creating an image of an arbitrary image plane (so-called refocus image) from original image data. An example of a known method of creating a refocus image will be described with reference to FIG.

FIG. 6 is a cross-sectional view schematically showing the light flux from the light spot on the image plane to be combined and the image sensor 22. In FIG. 6, consider a light spot P provided on the image surface S to be combined. The spread angle θ of the light traveling from the light point P to the image pickup element 22 is defined by the size of the pupil of the image pickup optical system 21 (that is, the aperture value of the image pickup optical system 21). The aperture value of the micro lens 223 is configured to be equal to or smaller than the aperture value of the image pickup optical system 21. Therefore, the light flux emitted from the light point P and incident on a certain microlens 223 does not spread outside the region of the light receiving element group 225 covered by the microlens 223. It is not necessary to provide a diaphragm for the microlenses 223 by providing partition walls that block light between the microlenses 223. Specifically, for example, a partition is provided in a portion where the microlenses 223(1) and the microlenses 223(2) adjacent to each other are connected. By providing the partition wall, the light flux incident on a certain microlens 223 does not spread outside the light receiving element group 225 region covered by the microlens 223.

Here, as shown in FIG. 6, if the light flux from the light spot P is incident on the five microlenses 223(1) to 223(5), these microlenses 223(1) to 223(5) enter. By integrating the amount of incident light (photoelectric conversion output of the light receiving elements 224(1) to 224(5)) on the light receiving surfaces of the incident light beams 30(1) to 30(5), the pupil from the light point P is limited. The total incident light amount is obtained. That is, the light amount of the light spot P (pixel to be combined) on the image surface S to be combined is obtained.

The control unit 24 sets a plurality of light spots P on the designated image plane S, and specifies, for each light spot P, the microlens 223 on which the light flux from the light spot P is incident. The control unit 24 specifies which of the light receiving elements 224 the light flux from the light spot P enters for each of the specified microlenses 223. The control unit 24 calculates the pixel value of the light spot P by integrating the photoelectric conversion outputs of the specified light receiving elements 224.

That is, in the image combining process by the control unit 24, (1) a plurality of light spots P are set on the image plane S, (2) one unselected light spot P is selected, and (3) the selected light spot P is selected. The microlens 223 on which the light flux from P enters is specified, and (4) the light receiving element 224 on which the light flux from the light point P enters from the light receiving element group 225 corresponding to each specified microlens 223, ( 5) The received light output of the specified light receiving element 224 is integrated to calculate the pixel value of the light spot P, and (6) until all of the plurality of light spots P set on the image surface S are selected (2) to ( This is performed by executing the steps (1) to (6), which is to repeat the step 5).

The control unit 24 must know the shape and size of the microlens 223 in order to execute the step (3) described above. For example, as illustrated in FIG. 7A, when a light beam from a certain light point P spreads into a range 32 indicated by a broken line and enters the microlens array 221, the microlens 223 arranged in the range 32 is removed. Need to be identified.

In the present embodiment, both the first image pickup device 22a and the second image pickup device 22b have circular microlenses 223 which are squarely arranged in a two-dimensional manner. Therefore, the shape of the microlenses 223 is circular. Is known. The control unit 24 specifies the size of the microlens 223 by reading the size data 46 from the original image file 40. As a result, the control unit 24 specifies that the light beam spreading in the range 32 illustrated in FIG. 7A is incident on the nine microlenses 223(1) to 223(9) illustrated in FIG. 7A. can do.

The control unit 24 must know the relative positional relationship between the microlens 223 and the light receiving element group 225 in order to specify the steps (4) and (5) described above. For example, FIG. 7B shows how a light beam from a certain light point P passes through the microlenses 223(2) to 223(4) and enters the light receiving elements 224(2) to 224(4). ..

In order to identify the light receiving elements 224(2) to 224(4), how far the light receiving element 224 is from the main plane 226 of the microlens 223, and how the light receiving element 224 is below the microlens 223 are specified. It is necessary to know how many of them are arranged, that is, the relative positional relationship between the microlens 223 and the light receiving element group 225.

The distance between the main plane 226 of the microlens 223 and the light receiving element 224 is equal to the focal length f of the microlens 223. Further, how much the light receiving elements 224 are arranged below the microlenses 223 can be specified from the shape of the microlenses 223, the size of the microlenses 223, and the pitch dp of the light receiving elements 224. That is, the relative positional relationship between the microlens 223 and the light receiving element group 225 can be specified from the focal length f of the microlens 223, the shape and size of the microlens 223, and the pitch of the light receiving elements 224.

In the present embodiment, both the first image pickup device 22a and the second image pickup device 22b have circular microlenses 223 which are squarely arranged in a two-dimensional manner. Therefore, the shape of the microlenses 223 is circular. Is known. The pitch dp of the light receiving elements 224 is also known. The control unit 24 identifies the size of the microlens 223 and the distance between the main plane 226 of the microlens 223 and the light receiving element 224 by reading the size data 46 and the distance data 47 from the original image file 40. The control unit 24 identifies the relative positional relationship between the microlens 223 and the light receiving element group 225 from the above various information. As a result, the control unit 24 identifies the light receiving elements 224(2) to 224(4) to which the light flux passing through the three microlenses 223(2) to 223(4) illustrated in FIG. 7B is incident. can do.

The processing executed by the image processing apparatus 3 for the image synthesizing function is the same as the processing performed by the control unit 24 described above, and thus description thereof will be omitted. That is, the image processing device 3 reads out and inputs the size data 46, the distance data 47, and the original image data 45 from the original image file stored in the memory card 26 or the like, and inputs the size data 46 to the original image data 45. Then, the refocus calculation described above is performed using the distance data 47 to create image data of an arbitrary image plane.

According to the above-mentioned embodiment, the following effects can be obtained.
(1) The light receiving element array 222 has a plurality of light receiving element groups 225 including a plurality of light receiving elements 224, and the light passing through each microlens 223 of the plurality of microlenses 223 is received by each light receiving element group 225. The control unit 24 stores size data 46 regarding the sizes of the plurality of microlenses 223 and distance data 47 regarding the positional relationship between the plurality of microlenses 223 and the plurality of light receiving element groups 225 in the original image file 40. Since this is done, it is possible to correctly create image data without knowing the microlenses 223 and the like used for imaging.

(2) The distance data 47 is data relating to the positional relationship between the intersections of the main planes 226 of the plurality of microlenses 223 and the optical axis O and the plurality of light receiving element groups 225. Since this is done, it is possible to correctly create image data without knowing the microlenses 223 and the like used for imaging.

(3) The control unit 24 stores the photoelectric conversion output of the light receiving element array 222 in the original image file 40 in association with the size data 46 and the distance data 47. Since this is done, it is possible to correctly create image data without knowing the microlenses 223 and the like used for imaging.

(4) The control unit 24 has an image synthesizing function of creating image data of an arbitrary image plane by the size data 46, the distance data 47, and the photoelectric conversion output of the light receiving element array 222. Since it did in this way, the image data of an arbitrary image surface can be obtained from the imaging result once.

(5) Data regarding the shape (circular shape) of the microlens 223 and the pitch dp of the light receiving elements 224 are common to all the cameras 2 and are not stored in the original image file 40. Since this is done, the file size of the original image file 40 can be minimized.

(Second embodiment)
The image processing system according to the second embodiment has a third camera 2c instead of the first camera 2a described in the first embodiment. Along with this, an original image file different from that of the first embodiment is stored. Hereinafter, first, the third image sensor 22c included in the third camera 2c will be described, and then the original image file will be described.

FIG. 8 is a diagram schematically showing the configuration of the third image sensor 22c included in the third camera 2c according to the second embodiment. 8A is a plan view seen from the image pickup optical system 21, and FIG. 8B is a sectional view. The third image sensor 22c has a third microlens array 221c and a light receiving element array 222.

The third microlens array 221c includes a third microlens 223c having a diameter w3 and a fourth microlens 223d having a diameter w4. The diameter w4 is twice the diameter w3. These two types of microlenses are two-dimensionally staggered as shown in FIG. That is, four third microlenses 223c are arranged in two rows and two columns at a certain position, and fourth microlenses 223d are arranged adjacent to each other vertically and horizontally. In other words, one fourth microlens 223d is arranged at a certain position, and four third microlenses 223c are arranged adjacent to each other in the vertical and horizontal directions in two rows and two columns. As shown in FIG. 8B, the third microlens 223c and the fourth microlens 223d have the same focal length f3.

FIG. 9 is a schematic diagram of the structure of the original image file 140 according to the second embodiment. FIG. 9 illustrates the configuration of the original image file 140 stored by the third camera 2c. The original image file 140 includes a header part 141 and an image data part 42. The structure of the image data section 42 is the same as that of the first embodiment.

The header part 141 includes image sensor data 143. The image sensor data 143 includes lens type data 148, first lens data 149a, second lens data 149b, and array data 150. It should be noted that the image sensor data 43 includes the number X of the light receiving elements 224 included in the light receiving element array 222 arranged in the x direction and the number Y arranged in the y direction (the number of light receiving elements in the vertical and horizontal directions of the light receiving element array 222). Good.

The lens type data 148 indicates the number of types of microlenses 223 included in the image pickup device 22 used for image pickup. For example, the third image pickup device 22c has two types of microlenses 223, which are the third microlens 223c and the fourth microlens 223d, and therefore the lens type data 148 indicates “2”.

The first lens data 149a and the second lens data 149b are data for each microlens 223 included in the image pickup device 22 used for image pickup. This lens data exists in the number corresponding to the lens type data 148. For example, the third image sensor 22c has two types of microlenses 223, which are the third microlens 223c and the fourth microlens 223d, and therefore, there are two lens data. Since the first image sensor 22a and the second image sensor 22b have only one type of microlens 223, only one lens data is stored.

The first lens data 149a is lens data corresponding to the third microlens 223c, and includes size data 146a and distance data 147a. The size data 146a indicates the diameter w3 of the third microlens 223c. The distance data 147a indicates the focal length f3 of the third microlens 223c.

The second lens data 149b is lens data corresponding to the fourth microlens 223d, and includes size data 146b and distance data 147b. The size data 146b indicates the diameter w4 of the fourth microlens 223d. The distance data 147b indicates the focal length f3 of the fourth microlens 223d.

The array data 150 is data indicating how the third microlenses 223c and the fourth microlenses 223d are arrayed. The third microlenses 223c and the fourth microlenses 223d are regularly arranged. Therefore, the array data 150 indicates that the third microlenses 223c and the fourth microlenses 223d are regularly arrayed in a predetermined pattern.

The control unit 24 uses the original image file 140 configured as described above to perform image composition processing. The control unit 24 recognizes that the microlens array 221 has two types of microlenses 223 by reading the lens type data 148 from the original image file 140. The control unit 24 recognizes how the two types of microlenses 223 are arranged by reading the arrangement data 150 from the original image file 140. The control unit 24 recognizes the sizes of the two types of microlenses 223 by reading the size data 146a and the size data 146b from the original image file 140. The control unit 24 reproduces, for example, the array of the microlenses 223 illustrated in FIG. 8A from the above information. Further, by referring to the image pickup element data 143, the light receiving elements 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be specified. That is, the light reception output of the light receiving element 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be extracted from the RAW data.

The control unit 24 uses the reproduced array of the microlenses 223 to perform the process described in the first embodiment for identifying the microlenses 223 on which the light flux from the light spot P is incident. As a result, as described in the first embodiment with reference to the example of FIG. 7A, the microlens 223 on which the light flux from the light spot P is incident can be specified.

The control unit 24 must know the relative positional relationship between the microlens 223 and the light receiving element group 225 in order to specify the steps (4) and (5) described above. In the present embodiment, it is known that the shape of the microlens 223 is circular. The pitch dp of the light receiving elements 224 is also known. The control unit 24 reads the size data 146a, the size data 146b, the distance data 147a, and the distance data 147b from the original image file 140 to recognize the distance between the main plane 226 of the microlens 223 and the light receiving element 224. The control unit 24 identifies the relative positional relationship between the microlens 223 and the light receiving element group 225 from the above various information. As a result, the control unit 24 can identify the light receiving element 224 on which the light flux passing through the microlens 223 is incident, as described in the example of FIG. 7B in the first embodiment.

In the present embodiment, the second camera 2b stores the original image file 140 having the same structure as the third camera 2c. In the second camera 2b, since the microlens 223 is of one type, the lens type data 148 is “1”. Further, only the first lens data 149a is stored, and the second lens data 149b is not stored. In the first lens data 149a, the size data 146a indicates “w2” and the distance data 147a indicates “f2”. The array data 150 indicates a square array of a single type of microlens 223.

According to the above-described embodiment, the same operational effect as that of the first embodiment can be obtained.

(Third Embodiment)
The image processing system according to the third embodiment has a fourth camera 2d instead of the first camera 2a described in the first embodiment. Along with this, an original image file different from that of the first embodiment is stored. Hereinafter, the fourth image sensor 22d included in the fourth camera 2d will be described first, and then the original image file will be described.

FIG. 10 is a plan view schematically showing the configuration of the fourth image sensor 22d included in the fourth camera 2d according to the third embodiment. The fourth image pickup element 22d has a fourth microlens array 221d and a light receiving element array 222.

The fourth microlens array 221d includes a fifth microlens 223e having a diameter w5 and a sixth microlens 223f having a diameter w6. The diameter w6 is twice the diameter w5. The fourth microlens array 221d includes a large number of fifth microlenses 223e and a smaller number of sixth microlenses 223f than the fifth microlenses 223e. The large number of fifth microlenses 223e are two-dimensionally squarely arranged. Some of the fifth microlenses 223e are replaced with sixth microlenses 223f. The sixth microlenses 223f are irregularly arranged in the large number of fifth microlenses 223e.

FIG. 11 is a schematic diagram of the structure of the original image file 240 according to the third embodiment. FIG. 11 illustrates the configuration of the original image file 240 stored by the fourth camera 2d. In the present embodiment, the first camera 2a, the second camera 2b, and the third camera 2c also store the original image file 240 having the structure shown in FIG. The original image file 240 includes a header part 241 and an image data part 42. The structure of the image data section 42 is the same as that of the first embodiment.

The header section 241 includes image sensor data 243. The image sensor data 243 includes lens type data 248, first lens data 249a, second lens data 249b, and array data 250. It should be noted that the image sensor data 43 includes the number X of the light receiving elements 224 included in the light receiving element array 222 arranged in the x direction and the number Y arranged in the y direction (the number of light receiving elements in the vertical and horizontal directions of the light receiving element array 222). Good.

The lens type data 248 indicates the number of types of microlenses 223 included in the image pickup device 22 used for image pickup. For example, the fourth image sensor 22d has two types of microlenses 223, a fifth microlens 223e and a sixth microlens 223f, and therefore the lens type data 248 indicates "2".

The first lens data 249a and the second lens data 249b are data for each microlens 223 included in the image pickup device 22 used for image pickup. This lens data exists in the number corresponding to the lens type data 248. For example, the fourth image sensor 22d has two types of microlenses 223, that is, a sixth microlens 223e and a sixth microlens 223f, and therefore, there are two lens data. Since the first image sensor 22a and the second image sensor 22b have only one type of microlens 223, only one lens data is stored.

The first lens data 249a is lens data corresponding to the fifth microlens 223e and includes size data 246a and distance data 247a. The size data 246a indicates the diameter w5 of the fifth microlens 223e. The distance data 247a indicates the focal length f5 of the fifth microlens 223e.

The second lens data 249b is lens data corresponding to the sixth microlens 223f, and includes size data 246b and distance data 247b. The size data 246b indicates the diameter w6 of the sixth microlens 223f. The distance data 247b indicates the focal length f6 of the sixth microlens 223f.

The array data 250 is data indicating how the fifth microlenses 223e and the sixth microlenses 223f are arrayed. In the fourth image sensor 22d, a part of the large number of fifth microlenses 223e arranged in a two-dimensional square array is irregularly replaced by the sixth microlenses 223f. Therefore, the array data 250 indicates that the fifth microlenses 223e are arrayed on the entire surface and also relates to the position where the sixth microlenses 223f are located (for example, the coordinate value with the fifth microlens 223e as a reference). Contains data.

The control unit 24 uses the original image file 240 configured as described above to perform image composition processing. The control unit 24 recognizes that the microlens array 221 has two types of microlenses 223 by reading the lens type data 248 from the original image file 240. The control unit 24 recognizes how the two types of microlenses 223 are arranged by reading the arrangement data 250 from the original image file 240. The control unit 24 recognizes the sizes of these two types of microlenses 223 by reading the size data 246a and the size data 246b from the original image file 140. The control unit 24 reproduces the array of the microlenses 223 shown in FIG. 10, for example, from the above information. Further, by referring to the image sensor data 243, the light receiving element 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be specified. That is, the light reception output of the light receiving element 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be extracted from the RAW data.

The control unit 24 uses the reproduced array of the microlenses 223 to perform the process described in the first embodiment for identifying the microlenses 223 on which the light flux from the light spot P is incident. As a result, as described in the first embodiment with reference to the example of FIG. 7A, the microlens 223 on which the light flux from the light spot P is incident can be specified.

The control unit 24 must know the relative positional relationship between the microlens 223 and the light receiving element group 225 in order to specify the steps (4) and (5) described above. In the present embodiment, it is known that the shape of the microlens 223 is circular. The pitch dp of the light receiving elements 224 is also known. The control unit 24 recognizes the distance between the light receiving element 224 and the main plane 226 of the microlens 223 by reading the size data 246a, the size data 246b, the distance data 247a, and the distance data 247b from the original image file 240. The control unit 24 identifies the relative positional relationship between the microlens 223 and the light receiving element group 225 from the above various information. As a result, the control unit 24 can identify the light receiving element 224 on which the light flux passing through the microlens 223 is incident, as described in the example of FIG. 7B in the first embodiment.

In the present embodiment, the second camera 2b stores the original image file 240 having the same structure as the fourth camera 2d. In the second camera 2b, since the microlens 223 is of one type, the lens type data 248 is “1”. Further, only the first lens data 249a is stored, and the second lens data 249b is not stored. In the first lens data 249a, the size data 246a indicates “w2” and the distance data 247a indicates “f2”. The array data 250 indicates that the second microlenses 223b are arrayed on the entire surface.

According to the above-described embodiment, the same operational effect as that of the first embodiment can be obtained.

(Fourth Embodiment)
The image processing system according to the fourth embodiment has a fifth camera 2e instead of the first camera 2a described in the first embodiment. Along with this, an original image file different from that of the first embodiment is stored. Hereinafter, the fifth image sensor 22e included in the fifth camera 2e will be described first, and then the original image file will be described.

FIG. 12 is a plan view schematically showing the configuration of the fifth image sensor 22e included in the fifth camera 2e according to the fourth embodiment. The fifth imaging element 22e has a fifth microlens array 221e and a light receiving element array 222.

The fifth microlens array 221e has a seventh microlens 223g and an eighth microlens 223h. The seventh microlens 223g has a regular hexagonal outer shape with a diameter w7. The eighth microlens 223h has an outer shape in which one seventh microlens 223g and six surrounding seventh microlenses 223g are joined.

The fifth microlens array 221e has a large number of seventh microlenses 223g and a smaller number of eighth microlenses 223h than the seventh microlenses 223g. The large number of seventh microlenses 223g are two-dimensionally arranged without a gap. Some of the seventh microlenses 223g are replaced with eighth microlenses 223h. The eighth microlenses 223h are irregularly arranged in the large number of seventh microlenses 223g.

FIG. 13 is a schematic diagram of the structure of the original image file 340 according to the fourth embodiment. FIG. 13 illustrates the configuration of the original image file 340 stored by the fifth camera 2e. In the present embodiment, the first camera 2a, the second camera 2b, the third camera 2c, and the fourth camera 2d also store the original image file 340 having the structure shown in FIG. The original image file 340 includes a header part 341 and an image data part 42. The structure of the image data section 42 is the same as that of the first embodiment.

The header section 341 includes image sensor data 343. The image sensor data 343 includes lens type data 348, first lens data 349a, second lens data 349b, and array data 350. Note that the image sensor data 343 includes the number X of light receiving elements 224 included in the light receiving element array 222 arranged in the x direction and the number Y arranged in the y direction (the number of light receiving elements in the vertical and horizontal directions of the light receiving element array 222). Good.

The lens type data 348 indicates the number of types of the microlenses 223 included in the image pickup device 22 used for image pickup. For example, the fifth image sensor 22e has two types of microlenses 223, the seventh microlens 223g and the eighth microlens 223h, and therefore the lens type data 348 indicates "2".

The first lens data 349a and the second lens data 349b are data for each microlens 223 included in the image pickup device 22 used for image pickup. This lens data exists in the number corresponding to the lens type data 248. For example, the fifth image pickup element 22e has two types of microlenses 223, that is, the seventh microlens 223g and the eighth microlens 223h, and therefore, two pieces of lens data exist. Since the first image sensor 22a and the second image sensor 22b have only one type of microlens 223, only one lens data is stored.

The first lens data 349a is lens data corresponding to the seventh micro lens 223g, and includes outer shape data 351a, size data 346a, and distance data 347a. The outer shape data 351a indicates the outer shape (regular hexagon) of the seventh microlens 223g. The size data 346a indicates the diameter w7 of the seventh microlens 223g. The distance data 347a indicates the focal length f7 of the seventh microlens 223g.

The second lens data 349b is lens data corresponding to the eighth microlens 223h and includes outer shape data 351b and distance data 347b. The outer shape data 351b indicates the outer shape of the eighth microlens 223h (the shape in which seven regular hexagonal seventh microlenses 223g are joined). The distance data 347b indicates the focal length f8 of the eighth microlens 223h. The outer shape data 351b indicates that the eighth microlens 223h has a shape in which seven seventh microlenses 223g are joined, and the diameter of the eighth microlens 223h is known. Therefore, in the example of FIG. 13, the second lens data 349b omits the data indicating the diameter of the eighth microlens 223h. The second lens data 349b may include size data indicating the diameter of the eighth microlens 223h).
Further, in FIG. 13, the outer shape data 351a and the size data 346a are described separately in the first lens data 349a, but the outer shape data 351a may be included in the size data 346a. That is, the size data 346a may have the diameter w7 of the seventh microlens 223g and the outer shape data 351a. On the contrary, the size data 346a may be included in the outer shape data 351a. In that case, the outer shape data 351a includes the outer shape (regular hexagon) of the seventh microlens 223g and the diameter w7 of the seventh microlens 223g.

The array data 350 is data indicating how the seventh microlens 223g and the eighth microlens 223h are arrayed. In the fifth imaging element 22e, a part of the large number of seventh microlenses 223g arranged in a two-dimensional manner with no gaps is irregularly replaced by the eighth microlenses 223h. Therefore, the array data 350 indicates that the seventh microlenses 223g are arrayed on the entire surface, and also relates to the position where the eighth microlenses 223h are located (for example, coordinate values with the seventh microlens 223g as a reference). Contains data.

The control unit 24 uses the original image file 340 configured as described above to perform image composition processing. The control unit 24 recognizes that the microlens array 221 has two types of microlenses 223 by reading the lens type data 348 from the original image file 340. The control unit 24 recognizes how the two types of microlenses 223 are arranged by reading the arrangement data 350 from the original image file 340. The control unit 24 reads the outer shape data 351a, the size data 346a, the outer shape data 351b, and the size data 346b from the original image file 340 to recognize the shapes and sizes of the two types of microlenses 223. The control unit 24 reproduces the array of the microlenses 223 shown in FIG. 12, for example, from the above information. Further, by referring to the image sensor data 343, the light receiving element 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be specified. That is, the light reception output of the light receiving element 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be extracted from the RAW data.

The control unit 24 uses the reproduced array of the microlenses 223 to perform the process described in the first embodiment for identifying the microlenses 223 on which the light flux from the light spot P is incident. As a result, as described in the first embodiment with reference to the example of FIG. 7A, the microlens 223 on which the light flux from the light spot P is incident can be specified.

The control unit 24 must know the relative positional relationship between the microlens 223 and the light receiving element group 225 in order to specify the steps (4) and (5) described above. In the present embodiment, the pitch dp of the light receiving elements 224 is known. The control unit 24 recognizes the distance between the main plane 226 of the microlens 223 and the light receiving element 224 by reading the distance data 347a and the distance data 347b from the original image file 340. The control unit 24 identifies the light receiving element group 225 corresponding to each microlens 223 by reading the outer shape data 351a, the outer shape data 351b, the size data 346a, and the size data 346b from the original image file 340. The control unit 24 identifies the relative positional relationship between the microlens 223 and the light receiving element group 225 from the above various information. As a result, the control unit 24 can identify the light receiving element 224 on which the light flux passing through the microlens 223 is incident, as described in the example of FIG. 7B in the first embodiment.

In the present embodiment, the second camera 2b stores the original image file 340 having the same structure as the fifth camera 2e. In the second camera 2b, since the microlens 223 is of one type, the lens type data 348 is “1”. Further, only the first lens data 349a is stored, and the second lens data 349b is not stored. In the first lens data 349a, the outer shape data 351a indicates "circle", the size data 346a indicates "w2", and the distance data 347a indicates "f2". The array data 350 indicates that the second microlenses 223b are arrayed on the entire surface.

According to the above-described embodiment, the same operational effect as that of the first embodiment can be obtained.

(Fifth Embodiment)
The image processing system according to the fifth embodiment has a sixth camera 2f instead of the first camera 2a described in the first embodiment. Along with this, an original image file different from that of the first embodiment is stored. Hereinafter, the sixth image sensor 22f included in the sixth camera 2f will be described first, and then the original image file will be described.

FIG. 14 is a plan view schematically showing the configuration of the sixth image sensor 22f included in the sixth camera 2f according to the fifth embodiment. The sixth image pickup element 22f has a sixth microlens array 221f and a light receiving element array 222.

The sixth microlens array 221f has a ninth microlens 223i and a tenth microlens 223h. The ninth microlens 223i has a regular hexagonal outer shape with a diameter w9. The tenth microlens 223j has a circular outer shape with a diameter w10.

The sixth microlens array 221f includes a large number of ninth microlenses 223i and a smaller number of tenth microlenses 223j than the ninth microlenses 223i. The large number of ninth microlenses 223i are two-dimensionally arranged without any gap. Some of the ninth microlenses 223i are replaced with tenth microlenses 223j. The tenth microlenses 223j are irregularly arranged in the large number of ninth microlenses 223i.

The diameter w10 of the tenth microlens 223j is larger than the diameter w9 of the ninth microlens 223i. Therefore, the ninth microlens 223i around the tenth microlens 223j has a shape in which a part of the original regular hexagon is hollowed out by the tenth microlens 223j.

FIG. 15 is a schematic diagram of the structure of the original image file 440 according to the fifth embodiment. FIG. 15 illustrates the configuration of the original image file 440 stored by the sixth camera 2f. In the present embodiment, the first camera 2a, the second camera 2b, the third camera 2c, the fourth camera 2d, and the fifth camera 2e also store the original image file 440 having the structure shown in FIG. The original image file 440 includes a header section 441 and an image data section 42. The structure of the image data section 42 is the same as that of the first embodiment.

The header section 441 includes image sensor data 443. The image sensor data 443 includes lens type data 448, first lens data 449a, second lens data 449b, and array data 450. It should be noted that the image sensor data 43 includes the number X of the light receiving elements 224 included in the light receiving element array 222 arranged in the x direction and the number Y arranged in the y direction (the number of light receiving elements in the vertical and horizontal directions of the light receiving element array 222). Good.

The lens type data 448 indicates the number of types of the microlens 223 included in the image pickup device 22 used for image pickup. For example, the sixth image pickup device 22f has two types of microlenses 223, namely, a ninth microlens 223i and a tenth microlens 223j, and thus the lens type data 448 indicates “2”.

The first lens data 449a and the second lens data 449b are data for each microlens 223 included in the image sensor 22 used for imaging. This lens data exists in the number corresponding to the lens type data 248. For example, the sixth image pickup device 22f has two types of microlenses 223, namely, a ninth microlens 223i and a tenth microlens 223j, and thus two pieces of lens data exist. Since the first image sensor 22a and the second image sensor 22b have only one type of microlens 223, only one lens data is stored.

The first lens data 449a is lens data corresponding to the ninth microlens 223i and includes outer shape data 451a, size data 446a, and distance data 447a. The outer shape data 451a indicates the outer shape (regular hexagon) of the ninth microlens 223i. The size data 446a indicates the diameter w9 of the ninth microlens 223i. The distance data 447a indicates the focal length f9 of the ninth microlens 223i.

The second lens data 449b is lens data corresponding to the tenth microlens 223j and includes outer shape data 451b, size data 446b, and distance data 447b. The outer shape data 451b indicates the outer shape (circle) of the tenth microlens 223j. The size data 446b indicates the diameter w10 of the tenth microlens 223j. The distance data 447b indicates the focal length f10 of the tenth microlens 223j.

The array data 450 is data indicating how the ninth microlenses 223i and the tenth microlenses 223j are arrayed. In the sixth image pickup device 22f, a part of the large number of ninth microlenses 223i arranged two-dimensionally with no gaps is irregularly replaced by the tenth microlenses 223j. Therefore, the array data 450 indicates that the ninth microlenses 223i are arrayed on the entire surface, and also relates to the position where the tenth microlenses 223j are located (for example, coordinate values with the ninth microlens 223i as a reference). Contains data. Further, it includes information indicating that the tenth microlens 223j has priority. Here, the priority lens is a lens that maintains the outer shape shown in the lens data 449 when the two types of lenses (the ninth microlens 223i and the tenth microlens 223j) are arranged to overlap each other. It is a thing. On the contrary, the non-priority lens (the ninth microlens 223i) has a shape obtained by scooping out by the lens (the tenth microlens 223j) in which a part of the outer shape (regular hexagon) shown in the lens data 449 is prioritized. Become.

The control unit 24 uses the original image file 440 configured as described above to perform image combination processing. The control unit 24 recognizes that the microlens array 221 has two types of microlenses 223 by reading the lens type data 448 from the original image file 440. The control unit 24 recognizes how the two types of microlenses 223 are arranged by reading the arrangement data 450 from the original image file 440. The control unit 24 reads the outer shape data 451a, the size data 446a, the outer shape data 451b, and the size data 446b from the original image file 440 to recognize the shapes and sizes of these two types of microlenses 223. The control unit 24 reproduces the array of the microlenses 223 shown in FIG. 14, for example, from the above information. Further, by referring to the image pickup element data 443, the light receiving elements 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be specified. That is, the light reception output of the light receiving element 224 included in the light receiving element group 225 corresponding to a certain microlens 223 can be extracted from the RAW data.

The control unit 24 uses the reproduced array of the microlenses 223 to perform the process described in the first embodiment for identifying the microlenses 223 on which the light flux from the light spot P is incident. As a result, as described in the first embodiment with reference to the example of FIG. 7A, the microlens 223 on which the light flux from the light spot P is incident can be specified.

The control unit 24 must know the relative positional relationship between the microlens 223 and the light receiving element group 225 in order to specify the steps (4) and (5) described above. In the present embodiment, the pitch dp of the light receiving elements 224 is known. The control unit 24 recognizes the distance between the main plane 226 of the microlens 223 and the light receiving element 224 by reading the distance data 447a and the distance data 447b from the original image file 440. The control unit 24 identifies the light receiving element group 225 corresponding to each microlens 223 by reading the outer shape data 451a, the outer shape data 451b, the size data 446a, and the size data 446b from the original image file 440. The control unit 24 identifies the relative positional relationship between the microlens 223 and the light receiving element group 225 from the above various information. As a result, the control unit 24 can identify the light receiving element 224 on which the light flux passing through the microlens 223 is incident, as described in the example of FIG. 7B in the first embodiment.

In the present embodiment, the second camera 2b stores the original image file 440 having the same structure as the sixth camera 2f. In the second camera 2b, since the microlens 223 is of one type, the lens type data 448 is “1”. Further, only the first lens data 449a is stored, and the second lens data 449b is not stored. In the first lens data 449a, the outer shape data 451a indicates "circle", the size data 446a indicates "w2", and the distance data 447a indicates "f2". The array data 450 indicates that the second microlenses 223b are arrayed on the entire surface.

According to the above-described embodiment, the same operational effect as that of the first embodiment can be obtained.

The following modifications are also within the scope of the present invention, and one or more modifications can be combined with the above-described embodiment.
(Modification 1)
In each of the above-described embodiments, various examples of the size, shape, and arrangement of the microlenses 223 have been described. On the other hand, it is possible to configure the original image file. For example, the shape of the microlens may be a square, a regular octagon, or the like. Further, for convenience of explanation, up to two types of lens type data have been described, but the original image file may be configured to have three types or four types of lens data. Further, for example, when three types of lenses are included, three types of lenses having the same shape but different sizes can be used, or three types of lenses having different shapes can be used. That is, the microlens array 221 (imaging device 22) including the microlenses 223 having an arbitrary size, shape, and arrangement can be described by the configuration of the original image file of each of the above-described embodiments.

FIG. 16 is a diagram showing a modification of the size, shape, and arrangement of the microlenses 223. FIG. 16B is a plan view schematically showing the configuration of the seventh image sensor 22g. The seventh microlens array 221g of the seventh image pickup device 22g is obtained by distorting the rectangular microlens array 60 shown in FIG. By applying the distortion, each eleventh microlens 223k forming the seventh microlens array 221g has a complicated shape. Ie. The seventh microlens array 221g has a huge number of eleventh microlenses 223k. Therefore, in the camera 2 having the seventh imaging element 22g, when the microlens 223 is specified by the shape and size as in the above-described embodiments, a huge amount of lens data and array data are required. In such a case, the original image file may store data indicating, for example, the shape of the base microlens array 60 shown in FIG. 16A and the distortion added thereto. Alternatively, it suffices to store data in which each of the boundary lines that divide the plurality of eleventh microlenses 223k shown in FIG. Regardless of which data is stored, the control unit 24 and the image processing device 3 can restore the outer shape of the plurality of eleventh microlenses 223k illustrated in FIG. 16B from the data, Correct image data can be created based on the outer shape of the eleventh microlens 223k. By adopting such an expression format, information regarding the microlenses 223 having a complicated shape or arrangement can be stored while keeping the size of the original image file small.

(Modification 2)
In each of the above-described embodiments, the image sensor data stored in the original image file is the data stored in the ROM 25 in advance. The control unit 24 reads the image pickup device data from the ROM 25 and writes it in the original image file. However, the image sensor data may be stored in the original image file by a different method.

For example, when the image sensor 22 of the camera 2 is configured to be replaceable, the image sensor data must be changed according to the replacement of the image sensor 22. The same applies to the case where the microlens array 221 or the light-receiving element array 222 is configured to be replaceable instead of the image pickup element 22. As described above, when the member related to the image sensor data is replaceable, a storage medium for storing the image sensor data is attached to the replaceable member, and the control unit 24 captures the image from the storage medium. The element data may be read out. Alternatively, in the ROM 25, a database including a plurality of combinations of IDs (for example, member model numbers) that specify replaceable members and corresponding image sensor data is stored, and the control unit 24 is currently mounted on the member. The image pickup device data corresponding to may be retrieved from the database and read. In addition, the database is stored in a device (for example, a database server) provided outside the camera 2 instead of in the ROM 25, and the camera 2 performs data communication with the device as necessary to obtain the image sensor data. It may be received from the device.

(Modification 3)
In each of the embodiments described above, the image sensor data is stored in the original image file, and when the camera 2 or the image processing device 3 creates a refocused image using the image combining function, the image is captured from the original image file. The element data was read and used. The image sensor data may be stored in a location different from the original image file. For example, the file may be stored in association with the original image file as a file different from the original image file. The information specifying the file in which the image sensor data is stored may be written in the header of the original image file, for example.

Alternatively, each time the camera 2 uses the image synthesizing function, the image sensor data may be read from the ROM 25 and used to create a refocused image.
Alternatively, each time the camera 2 or the image processing device 3 uses the image synthesizing function, the ID (for example, the model number of the camera 2 or the image sensor 22) for identifying the camera 2 or the image sensor 22 is associated with the corresponding image sensor data. The image sensor data may be read from a device (for example, a database server) provided outside the camera 2 and the image processing device 3, which stores a database including a plurality of combinations, and may be used to create a refocus image. In this case, when the camera 2 or the image processing device 3 creates a refocused image, if an ID specifying the camera 2 or the image sensor 22 is transmitted to the device, the database is stored using the ID received by the device. The image pickup device data may be specified by searching and the specified image pickup device data may be returned to the camera 2 or the image processing device 3. When the image pickup device 22 of the camera 2 mounted on the camera 2 is configured to be replaceable, the ID for identifying the image pickup device 22 may be used. The ID that identifies the camera 2 or the image sensor 22 may be stored in the original image file. By not storing the image sensor data in the original image file, the file size of the original image file can be reduced.

Further, the camera 2 may be provided with a mode for storing the image sensor data in the original image file and a mode for not storing the image sensor data in the original image file. In the former mode, since the image sensor data is included in the original image file, it is not necessary to read the image sensor data from another file or a device (for example, a database server) provided outside the camera 2 and the image processing device 3, The processing load can be reduced. In the latter mode, since the image sensor data is not included in the original image file, the file size of the original image file can be reduced. In this way, by making it possible to switch between the two modes, it becomes possible to handle the original image file more flexibly.

(Modification 4)
Although the pitch of the light receiving elements 224 is constant in each of the above-described embodiments, the pitch of the light receiving elements 224 may be different for each imaging element 22. For example, the microlenses 223 may have the same size and shape, but there may be two types of image pickup elements 22 having different pitches and shapes of the light receiving elements 224. In this case, the pitch of the light receiving elements 224 may be included as a part of the image pickup element data 443.

(Modification 5)
In each of the above-described embodiments, the distance data indicating the focal length of the microlens 223 is used as the data regarding the positional relationship between the plurality of microlenses 223 and the plurality of light receiving element groups 225. Instead of the distance data, for example, distance data indicating the F value of the microlens 223 or distance data indicating the effective aperture of the microlens 223 may be used. As is well known, since a predetermined relationship is established among the focal length of the lens, the effective aperture of the lens, and the F value of the lens, if two of these three pieces of information are known, the remaining It is possible to derive one piece of information.

Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other modes that are conceivable within the scope of the technical idea of the present invention are also included within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Image processing system, 2... Camera, 3... Image processing apparatus, 21... Imaging optical system, 22... Imaging element, 23... Display part, 24... Control part, 25... ROM, 26... Memory card, 221... Microlens Array, 222... Light receiving element array, 223... Micro lens, 224... Light receiving element, 225... Light receiving element group

Claims (11)

A plurality of microlenses arranged two-dimensionally,
An imaging sensor having a plurality of pixel groups including a plurality of pixels, each pixel group receiving light that has passed through each microlens of the plurality of microlenses,
A storage unit that stores first data regarding at least one of the shapes and sizes of the plurality of microlenses and second data regarding the arrangement of the plurality of microlenses ;
The plurality of microlenses include a first microlens and a second microlens having a shape and a size different from those of the first microlens . A plurality of microlenses arranged two-dimensionally,
An image sensor having a plurality of pixel groups including a plurality of pixels, each of which receives light having passed through each microlens of the plurality of microlenses,
A data acquisition unit that externally acquires first data relating to at least one of the shapes and sizes of the plurality of microlenses and second data relating to the arrangement of the plurality of microlenses ;
The plurality of microlenses include a first microlens and a second microlens having a shape and a size different from those of the first microlens . The imaging device according to claim 1 or 2,
The said 2nd data is an imaging device containing the information regarding the positional relationship of the intersection of the principal plane and optical axis of these microlenses, and these pixel groups. The imaging device according to any one of claims 1 to 3 ,
Before SL to the first data, an imaging device includes information relating to the first microlens and the second microlens. The imaging device according to any one of claims 1 to 4 ,
Before SL in the second data, the imaging device includes information on the arrangement of the first microlens and the second microlens. The imaging device according to any one of claims 1 to 5,
The image pickup apparatus further comprising an image generation unit that generates an image file including the first data, the second data, and a photoelectric conversion output of the image sensor. The imaging device according to claim 6,
A first mode in which the image generation unit generates an image file including the first data, the second data, and a photoelectric conversion output of the image sensor; and the image generation unit includes the first data and the second data. A second mode for generating an image file without including the image pickup device. The imaging device according to any one of claims 1 to 7,
The image pickup apparatus further comprising an image processing unit that performs a predetermined calculation on the photoelectric conversion output of the image pickup sensor using the first data and the second data to create image data. At least one of the shapes and sizes of a plurality of two-dimensionally arranged microlenses including a first microlens and a second microlens having a shape and a size different from that of the first microlens. A first input section for receiving first data regarding the above and second data regarding the arrangement of the plurality of microlenses;
A second input unit to which the photoelectric conversion output data output by the plurality of pixel groups to which the light passing through the plurality of microlenses respectively enters are input;
An image processing unit that performs a predetermined calculation on the photoelectric conversion output data using the first data and the second data to create image data,
An image processing apparatus including. The image processing apparatus according to claim 9,
A storage unit that stores a plurality of combinations of the first data and the second data, and identification information of the imaging device;
A third input unit to which the identification information is input,
A search unit that searches the storage unit for the first data and the second data corresponding to the identification information input to the third input unit, and inputs the first data and the second data to the first input unit;
An image processing apparatus further comprising: The image processing apparatus according to claim 9,
A third input unit for inputting identification information of the image pickup device that picked up the photoelectric conversion output data;
A transmission unit that transmits the identification information to a server that stores a plurality of combinations of the first data and the second data and the identification information of the imaging device; the first data corresponding to the identification information from the server; A receiver for receiving the second data,
An image processing apparatus further comprising:

Images