WO2014136392A1

WO2014136392A1 - Image processing device, image processing method and imaging device

Info

Publication number: WO2014136392A1
Application number: PCT/JP2014/000889
Authority: WO
Inventors: 壮功北田; 高山　淳
Original assignee: コニカミノルタ株式会社
Priority date: 2013-03-05
Filing date: 2014-02-20
Publication date: 2014-09-12
Also published as: JPWO2014136392A1

Abstract

In the case of the present image processing device, image processing method, and imaging device, the boundary position of an image circle of an imaging optical system is detected on the basis of an electric signal that is output from an imaging component upon receiving an optical image of a photographic object formed on a light-receiving surface by the imaging optical system, predefined parameters related to predefined image correction processing are determined on the basis of the detected image circle boundary position, and the image of the photographic object is corrected using the determined predefined parameters.

Description

Image processing apparatus, image processing method, and imaging apparatus

The present invention relates to an image processing apparatus and an image processing method for processing an image. The present invention relates to an image processing apparatus and an imaging apparatus using the image processing method.

In recent years, imaging devices (cameras) have been used as information acquisition means in portable devices, in-vehicle devices, medical devices, industrial devices, and the like. In particular, recent mobile phones and smartphones are typically equipped with an imaging device. In many cases, these imaging devices employ lenses made of resin materials such as plastics because they are relatively inexpensive and can be produced in large quantities. Although this resin material lens has the above-mentioned advantages, it has the disadvantage that it is easily affected by temperature changes, and the optical characteristics (focal length, image center, distortion coefficient, etc.) of the lens change due to the temperature changes. Yes. When an image of a subject is generated based on an electrical signal obtained by an imaging device via an imaging optical system including a lens, the image generation process is generally performed on the assumption that these optical characteristics do not change. Changes in the optical characteristics directly lead to degradation of image quality. If a change in optical characteristics can be detected and a parameter suitable for the change can be used, an image with reduced image quality can be obtained.

For example, Patent Document 1 discloses a technique for correcting such a change in optical characteristics due to a temperature change. The imaging device disclosed in Patent Document 1 is disposed at a position facing a subject, and is provided with a lens array in which a plurality of lenses are arranged in an array, an image plane side of the lens array, and the plurality of lenses. In an imaging apparatus comprising: an imaging element that captures a compound eye image that is a set of reduced images (single-eye images) of the imaged subject; and an arithmetic unit that processes the compound eye image captured by the imaging element. A light-shielding wall that prevents crosstalk of light rays between adjacent lenses constituting the lens array, and the computing unit includes a center position and a focal point of each lens based on luminance information incident on the image sensor. The internal parameters related to the distance and the lens distortion are determined, and the distortion amount due to the ambient temperature of the light shielding wall is calculated and corrected. And in this patent document 1, the method of Zhang disclosed in the nonpatent literature 1 is proposed as a method of calculating | requiring the internal parameter and external parameter intrinsic | native to the camera of each temperature.

By the way, the image pickup apparatus disclosed in Patent Document 1 removes the influence of temperature change by calculating the position of each lens based on the luminance information of the light shielding wall imaged by the image sensor. Therefore, in the imaging device disclosed in Patent Document 1, the light shielding wall is indispensable in order to remove the influence due to the temperature change. For this reason, in the imaging device disclosed in Patent Document 1, the number of manufacturing steps and costs for the light shielding wall are increased.

JP 2011-147079 A

The present invention has been made in view of the above circumstances, and its purpose is to detect a change in optical characteristics of a lens without providing a special optical member, and to correct an image in accordance with the change in the optical characteristics. An image processing apparatus, an image processing method, and an imaging apparatus capable of performing the above are provided.

In the image processing apparatus, the image processing method, and the imaging apparatus according to the present invention, the imaging optical system is based on the electrical signal output from the imaging element by receiving the optical image of the subject imaged on the light receiving surface by the imaging optical system. The image circle boundary position is detected, a predetermined parameter relating to a predetermined image correction process is determined based on the detected boundary position of the image circle, and using the determined predetermined parameter, The image is corrected. Therefore, since such an image processing apparatus, an image processing method, and an imaging apparatus use an image circle that is inherently included in the imaging optical system for determining parameters relating to image correction processing, a special optical member is provided. Without change, it is possible to detect a change in the optical characteristics of the lens and correct the image in accordance with the change in the optical characteristics.

The above and other objects, features and advantages of the present invention will become apparent from the following detailed description and the accompanying drawings.

It is a block diagram which shows the structure of the imaging device in embodiment. It is a figure for demonstrating the light-receiving surface (imaging surface) of the image pick-up element in the said imaging device. It is a figure which shows the structure of the parameter table memorize | stored in the memory | storage part in the said imaging device. It is a figure for demonstrating the change of the image circle by the temperature change in the said imaging device. 5 is a flowchart illustrating an operation related to image correction in the imaging apparatus. It is a figure for demonstrating the luminance distribution of an image circle in the said imaging device. It is a figure for demonstrating the determination method which determines the boundary position of an image circle in the said imaging device. It is a figure for demonstrating the data used for the determination of the boundary position of an image circle in the said imaging device. It is a figure for demonstrating the boundary position change and luminance distribution change of the image circle by a temperature change in the said imaging device.

Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted suitably. Further, in this specification, when referring generically, it is indicated by a reference symbol without a suffix, and when referring to an individual configuration, it is indicated by a reference symbol with a suffix.

FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment. FIG. 2 is a diagram for explaining a light receiving surface (imaging surface) of the image sensor in the imaging apparatus of the embodiment. FIG. 3 is a diagram illustrating a configuration of a parameter table stored in the storage unit in the imaging apparatus according to the embodiment.

The imaging apparatus according to the present embodiment is an apparatus that captures an image of a subject using an imaging optical system and generates an image of the subject. Such an imaging apparatus CA includes, for example, an imaging optical system 1 and an image processing unit 2 as shown in FIG. The imaging apparatus CA may further include at least one of the input unit 6, the display unit 7, and the interface unit (IF unit) 8 as necessary.

The imaging optical system 1 is an optical system that forms an optical image of a subject on a predetermined surface, and includes one or more optical elements. The imaging optical system 1 includes, for example, an optical aperture, one or a plurality of lenses, and a light shielding plate between the optical elements as necessary in order to prevent so-called stray light. The imaging optical system 1 may be a monocular that forms one optical image (one image circle) of the subject, but in this embodiment, the imaging optical system 1 includes a plurality of lenses (single eyes). A compound eye that forms a plurality of optical images (a plurality of image circles) of the subject. The plurality of lenses are arranged in an array in parallel so that their optical axes are parallel to each other. The lens array may be a one-dimensional lens array including a plurality of lenses arranged in a line on a straight line in parallel in a one-dimensional array, but in this embodiment, the lens array is linearly independent in two directions More specifically, it is a two-dimensional lens array including a plurality of lenses arranged in a two-dimensional array in parallel in two directions orthogonal to each other. The two-dimensional lens array may include a plurality of lenses having a honeycomb structure in the parallel direction (parallel direction). The single-lens lens may have a single lens configuration or a multiple lens configuration.

The image processing unit 2 is a device that generates an image of a subject. For example, as illustrated in FIG. 1, the image processing unit 2 includes an imaging device 3, a control calculation unit 4, and a storage unit 5. The image processing unit 2 is assembled so that the light receiving surface of the image pickup device 3 becomes a predetermined surface of the image pickup optical system 1 (position of the predetermined surface of the image pickup optical system 1 = position of the light receiving surface of the image pickup device).

The image pickup device 3 includes a plurality of photoelectric conversion elements (a plurality of pixels), and an optical image of a subject formed by the image pickup optical system 1 on the light receiving surface formed by the plurality of photoelectric conversion elements. An element that converts an electric signal using a photoelectric conversion element. More specifically, the image sensor 3 is connected to the control calculation unit 4. For example, R (red), G (green), and B according to the amount of light in the optical image of the subject imaged by the imaging optical system 1. Photo-electrically convert the image signal of each component of (blue) and output to the control calculation unit 4. Thus, the optical image of the subject is guided to the light receiving surface of the image sensor 3 along the optical axis by the imaging optical system 1 at a predetermined magnification, and the optical image of the subject is captured by the image sensor 3.

In the present embodiment, since the imaging optical system 1 includes a two-dimensional lens array, two optical images (single-eye images) of subjects equal to the number of lenses in the two-dimensional lens array are present on the light receiving surface of the imaging element 3. The images are formed in parallel in a two-dimensional array in a manner corresponding to the arrangement of a plurality of lenses in the two-dimensional array lens. For this reason, the light receiving surface (effective pixel region) of the image pickup device 3 is divided into a plurality of image pickup regions corresponding to the positions of the image circles formed on the light receiving surface by the lenses in the two-dimensional lens array. The imaging area is an imaging area for each lens (each eye). More specifically, for example, when the two-dimensional array lens is composed of 16 lenses arranged in parallel in a 4 × 4 two-dimensional array in two XY directions orthogonal to each other, As shown in FIG. 2, the light receiving surface (effective pixel region) 30 has 16 pieces corresponding to the positions of the 16 image circle ICs formed on the light receiving surface 30 by the 16 lenses in the two-dimensional lens array. Imaging regions (imaging regions for each eye) 31 (31-11 to 31-44), and an optical image of a subject is formed on each of these 16 imaging regions 31. In FIG. 2, only the image circle IC for the imaging region 31-11 in the first row and the first column is shown, and the others are omitted. These 16 imaging regions 31 are arranged in parallel in a 4 × 4 two-dimensional array in two directions in the XY directions orthogonal to each other in accordance with the arrangement of the 16 lenses described above. Each of these 16 imaging regions 31 is set in a rectangular shape inscribed in an image circle IC formed under a predetermined design condition (for example, room temperature 23 ° C.). That is, the imaging region 31 is set so that its diagonal line is the diameter of the image circle IC. One imaging region 31 includes a plurality of pixels (photoelectric conversion elements) whose number is 1/16 or less of the total number of pixels of the imaging element 3.

The image sensor 3 includes a boundary position detection unit for detecting the boundary position of the image circle IC of the imaging optical system 1. More specifically, the light receiving surface (effective pixel region) 30 of the image pickup device 3 has a size including the boundary position of the image circle IC in a region adjacent to the image pickup region 31 as a boundary position detection unit. Area 32 is set. In the example shown in FIG. 2, one boundary position detection region 32 is set in a region adjacent to the imaging region 31-11 in the first row and the first column. As described above, the boundary position detection region 32 uses a photoelectric conversion element that is not used as the imaging region 31 on the light receiving surface 30. A plurality of boundary position detection areas 32 may be set, for example, a plurality may be set for all the imaging areas 31, or a plurality may be set for the imaging area 31 in the first row, for example. In addition, for example, a plurality of settings may be set for the imaging region 31 in the first column, for example, a plurality of settings may be set for the imaging region 31 on the diagonal line, and for example, for the imaging region 31 every other row. For example, a plurality may be set for the imaging region 31 every other column. By providing a plurality of boundary position detection regions 32 as described above, the detection accuracy of the boundary position can be improved. Here, an integrally molded lens tends to be distorted due to a temperature change depending on its manufacturing method. In addition, when a plurality of lenses that are not integrally molded are used, there is often a tendency of distortion depending on the arrangement position of the lens due to the temperature distribution in the imaging apparatus (this temperature distribution is caused by the heat generated by the electronic circuit, for example). For example, in the case of an integrally molded lens, it is conceivable that the lens expands isotropically around the center position of the image sensor 3. In such a case, since the distortion amount of the lens due to temperature change is the largest at the peripheral (diagonal, four corners) position, by providing a plurality of detection regions for a plurality of individual eyes located in the periphery, Strain can be measured with high accuracy. Accordingly, it is not necessary to provide detection areas for all the eyes, and a plurality of detection areas can be arranged efficiently. Further, depending on the manufacturing method, the lens may expand only in the vertical direction or the horizontal direction. In such a case, the distortion amount can be accurately measured by arranging a plurality of detection areas in one row or one column.

In the above description, the light receiving surface of one image sensor 3 is divided according to the number of lenses of the two-dimensional lens array. However, an image sensor may be provided for each lens of the two-dimensional lens array. In the case of the 4 × 4 two-dimensional lens array described above, 16 image sensors corresponding to the 16 lenses are provided.

The storage unit 5 is connected to the control calculation unit 4 and includes various programs such as a control program for controlling the entire imaging apparatus CA, an image forming program for forming an image of the subject, and an image correction program for correcting the image of the subject. And an element for storing various data such as data necessary for executing these various programs and data generated during the execution. The storage unit 5 includes, for example, a ROM (Read Only Memory) that is a nonvolatile storage element that stores the various programs, and an EEPROM (Electrically Erasable Programmable Read) that is a rewritable nonvolatile storage element that stores the various data. It is configured to include a storage element such as an only memory (RAM), a volatile storage element such as a RAM (Random Access Memory), and peripheral circuits thereof. The RAM of the storage unit 5 also functions as a so-called working memory of a CPU described later of the control calculation unit 4.

The storage unit 5 functionally stores an image storage unit 51 that stores image data, and a parameter storage unit 52 that stores parameters corresponding to each temperature (each boundary position of the image circle) in the imaging optical system 1 in advance. Is provided.

The predetermined parameter relating to the predetermined image correction process is a value used when correcting the image of the subject. Examples of the predetermined parameter include an internal parameter, an external parameter, a blur coefficient, and the like. The internal parameters are, for example, the focal position, the center position, the distortion coefficient, and the like of the imaging optical system 1, and the external parameters are the translation coefficient, the rotation coefficient, and the like of the imaging optical system 1. The blur coefficient is a coefficient used in so-called super-resolution processing. Each value of such a predetermined parameter is obtained in advance for each temperature of the imaging optical system 1. In the present embodiment, the temperature of the imaging optical system 1 is obtained at the boundary position of the image circle IC. For this reason, each value of the predetermined parameter is obtained in advance for each boundary position of the image circle IC in the imaging optical system 1. For example, by considering the specification of the operating temperature range of the imaging apparatus CA, each boundary position of the image circle IC in the imaging optical system 1 for each of the temperatures 0 ° C., 20 ° C., 40 ° C., 60 ° C., and 80 ° C. Is determined in advance, and each value of the predetermined parameter is determined in advance. The internal parameter and the external parameter may be obtained in advance by, for example, the Zhang method disclosed in Non-Patent Document 1 described above. As a result, the correspondence between each boundary position of the image circle IC in the imaging optical system 1 and each value of the predetermined parameter is obtained in advance.

Thus, for example, a predetermined function expression indicating the correspondence between each boundary position of the image circle and each value of the predetermined parameter is obtained from the correspondence created in advance and stored in the parameter storage unit 52. The predetermined parameter relating to the predetermined image correction processing may be determined from the boundary position of the image circle by using the predetermined function formula created in advance. In the present embodiment, the predetermined parameter is generated in this way. In addition, a lookup table indicating the correspondence relationship is created and stored in the parameter storage unit 52, and a predetermined parameter relating to a predetermined image correction process is obtained by using the previously created lookup table (parameter table). It is determined from the boundary position of the circle.

Such a parameter table PT includes, for example, a boundary position x field 5211 for registering a boundary position x in the x direction and a boundary position y field 5212 for registering a boundary position y in the y direction, as shown in FIG. A temperature field 522 for registering the temperature, a focal length fx field 5231 for registering the focal length fx in the x direction, a focal length fy field 5232 for registering the focal length fy in the y direction, and the image center in the x direction. An image center cx field 5241 for registering the position cx, an image center cy field 5242 for registering the image center position cy in the y direction, a distortion coefficient k1 field 5251 for registering the first distortion coefficient k1, and a second distortion. Distortion coefficient k2 field 5252 for registering coefficient k2 and distortion for registering third distortion coefficient p1 The number p1 field 5253, and includes various fields of the distortion coefficient p2 field 5254 for registering a fourth strain coefficient p2, and a record for each temperature.

As described above, in the example shown in FIG. 3, the correspondence between each boundary position of the image circle and each value of the internal parameters (focal position, center position, and distortion coefficient) is registered in the parameter table PT.

For example, in the example shown in FIG. 3, in the record in which “20” is registered in the temperature field 522, “3” is registered in the boundary position x field 5211, and “ 4 ”is registered,“ fx2 ”is registered in the focal length fx field 5231,“ fy2 ”is registered in the focal length fy field 5232, and the image center cx field 5241 has “Cx1” is registered, “cy1” is registered in the image center cy field 5242, “k12” is registered in the distortion coefficient k1 field 5251, and the distortion coefficient k2 field 5252 is registered. , “K22” is registered, “p12” is registered in the distortion coefficient p1 field 5253, and the distortion coefficient p2 field is registered. The Rudo 5254, "p22" has been registered.

Of course, the correspondence between each boundary position of the image circle and each value of the external parameter (translation coefficient and rotation coefficient) may be registered in the parameter table PT, and each boundary position of the image circle and each value of the blur coefficient May be registered in the parameter table PT. In the above description, the parameter table PT may be prepared for each lens (single eye), which is applied to all lenses in the lens array (common to all eyes). Thus, by preparing the parameter table PT corresponding to the lens (single eye) for each lens (single eye), it is possible to appropriately set the parameter value according to the optical characteristic of the lens with respect to the temperature change.

The control calculation unit 4 controls the operation of the entire imaging apparatus CA by controlling each unit in accordance with the function, and corrects the image by using a predetermined parameter related to a predetermined image correction process, while correcting the image of the subject. It is an apparatus to form. The control calculation unit 4 includes, for example, a CPU (Central Processing Unit) and peripheral circuits thereof. The control calculation unit 4 functionally includes a control unit 41, an image generation unit 42, a boundary detection unit 43, a parameter determination unit 44, and an image correction unit 45 by executing various programs.

The control unit 41 controls the entire operation of the imaging apparatus CA by controlling each unit according to the function.

The image generation unit 42 generates an image of a subject based on the electrical signal output from the image sensor 3. The image data of the generated image is stored in the image storage unit 51 of the storage unit 5.

The boundary detection unit 43 detects the boundary position of the image circle of the imaging optical system 1 based on the electrical signal output from the imaging device 3.

The parameter determination unit 44 determines a predetermined parameter related to a predetermined image correction process based on the boundary position of the image circle detected by the boundary detection unit 43.

The image correction unit 45 corrects the image of the subject generated by the image generation unit 42 and stored in the image storage unit 51 using the predetermined parameter determined by the parameter determination unit 44. The image data of the corrected image is stored in the image storage unit 51 of the storage unit 5.

The input unit 6 is a device for inputting various instructions such as an instruction for imaging conditions and an instruction to start imaging to the imaging apparatus CA from the outside, and is, for example, a rotary switch or a button switch. The display unit 7 is a device for outputting instruction contents input from the input unit 6 and processing results of the control calculation unit 4 such as an image of a subject, and is an LCD (liquid crystal display), an organic EL display, or the like, for example. . The display unit 7 may function as a so-called live view finder. The input unit 6 may include, for example, a position input device that detects and inputs an operation position such as a resistance film method or a capacitance method, and the position input device and the display unit 7 constitute a so-called touch panel. Good.

In this touch panel, a position input device is provided on the display surface of the display unit, one or more input content candidates that can be input are displayed on the display unit, and the user touches the display position where the input content to be input is displayed. Then, the position is detected by the position input device, and the display content displayed at the detected position is input to the imaging device CA as the operation input content of the user. In such a touch panel, since the user can easily understand the input operation intuitively, an imaging apparatus CA that is easy for the user to handle is provided. And by changing the display content displayed on the display part 7, various input content can be input and it becomes possible to perform various operation with respect to imaging device CA.

The IF unit 6 is a circuit for inputting / outputting data to / from an external device. For example, an interface circuit using the IEEE802.11 series standard called Wi-Fi (Wireless fidelity), Bluetooth (registered trademark) ) An interface circuit using a standard, an interface circuit performing infrared communication such as an IrDA (Infrared Data Association) standard, and an interface circuit using a USB (Universal Serial Bus) standard.

Next, the operation of the imaging device in this embodiment will be described. FIG. 4 is a diagram for explaining a change in the image circle due to a temperature change in the imaging apparatus according to the embodiment. 4A is a diagram for explaining a change in the size of the image circle due to a temperature change, and FIG. 4B is a diagram for explaining a change in the center position of the image circle due to a temperature change.

In such an image pickup apparatus CA, the temperature of the image pickup optical system 1 changes due to a change in the ambient temperature used. For this reason, expansion / contraction or distortion may occur in the optical element of the imaging optical system 1. In particular, when the optical element of the imaging optical system 1 is a lens made of a resin material made of a resin material such as plastic, expansion and contraction and distortion are more likely to occur due to its temperature characteristics. When the expansion / contraction occurs in the optical element of the imaging optical system 1, as shown in FIG. 4A, the size (area) of the image circle IC expands and contracts, and the boundary position of the image circle IC changes accordingly. To do. In the example shown in FIG. 4A, when the temperature of the imaging optical system 1 changes from the temperature 20 ° C. to the temperature 60 ° C., the image circle IC increases from the image circle IC (20) to the image circle IC (60). IC (t) represents an image circle IC when the temperature is t ° C. Further, when the distortion occurs in the optical element of the imaging optical system 1, as shown in FIG. 4B, the center position of the image circle IC (image center position) is shifted, and the boundary position of the image circle IC changes accordingly. ,shift. In the example shown in FIG. 4B, when the temperature of the imaging optical system 1 changes from the temperature 20 ° C. to the temperature 60 ° C., the center position (cx, cy) of the image circle IC is the center position (cx2) of the image circle IC (20). , Xy2) is shifted to the center position (cx4, xy4) of the image circle IC (60).

Therefore, the boundary position of the image circle IC correlates with the temperature of the imaging optical system 1 and correlates with a predetermined parameter relating to image correction processing given for each temperature of the imaging optical system 1. For this reason, predetermined parameters relating to image correction processing for removing or reducing an error given to an image due to a temperature change are set by detecting the boundary position of the image circle IC by associating with each other in advance. It becomes possible to do.

More specifically, the imaging apparatus CA of the present embodiment detects the boundary position of the image circle IC by operating as follows, determines the value of a predetermined parameter related to the image correction process, and determines this The subject image is corrected by using predetermined parameters.

FIG. 5 is a flowchart showing an operation related to image correction in the imaging apparatus of the embodiment. FIG. 6 is a diagram for explaining the luminance distribution of the image circle in the imaging apparatus according to the embodiment. FIG. 6A is a diagram schematically showing the state of the light receiving surface of the image sensor when a subject having a substantially uniform luminance distribution is imaged, and FIG. 6B is a luminance distribution along line X1-X2 shown in FIG. 6A. FIG. The horizontal axis in FIG. 6B indicates the position on the X1-X2 line, and the vertical axis indicates the luminance value. FIG. 7 is a diagram for explaining a determination method for determining the boundary position of the image circle in the imaging apparatus according to the embodiment. The horizontal axis in FIG. 7 indicates the position, and the vertical axis indicates the luminance value. FIG. 8 is a diagram for explaining data used for determining the boundary position of the image circle in the imaging apparatus according to the embodiment. FIG. 8A shows the luminance distribution in the vicinity of the boundary of the image circle obtained when a subject having a uniform luminance distribution (uniform subject) is imaged. FIG. 8B shows the luminance distribution BA in the vicinity of the boundary of the image circle obtained when a general subject (general subject A) having a nonuniform luminance distribution is imaged. FIG. 8C shows the luminance distribution BB in the vicinity of the boundary of the image circle obtained when another general subject (general subject B) with a nonuniform luminance distribution is imaged. FIG. 8D shows a luminance distribution BC in the vicinity of the boundary of the image circle obtained when a general subject (general subject C) having a nonuniform luminance distribution is imaged. FIG. 8E shows an average luminance distribution Bave obtained by averaging the luminance distributions BA to BC shown in FIGS. 8B to 8D. FIG. 9 is a diagram for explaining a change in the boundary position of the image circle and a change in luminance distribution due to a temperature change in the imaging apparatus according to the embodiment. FIG. 9A shows a change in the boundary position of the image circle due to a temperature change, and FIG. 9B shows a change in the luminance distribution of the image circle due to a temperature change. The horizontal axis in FIG. 9B indicates the position, and the vertical axis indicates the luminance value.

For example, when a power switch (not shown) included in the input unit 6 of the imaging apparatus CA is operated and the imaging apparatus CA is activated, the control calculation unit 4 controls the control program and the image correction program stored in the storage unit 5. Etc. are executed. As a result, a control unit 41, an image generation unit 42, a boundary detection unit 43, a parameter determination unit 44, and an image correction unit 45 are functionally configured in the control calculation unit 4.

For example, when an unillustrated shutter button included in the input unit 6 in the imaging apparatus CA is operated, the imaging apparatus CA generates an image of the subject. More specifically, a light beam from the subject enters the imaging optical system 1, and an optical image of the subject is formed on the light receiving surface 30 of the imaging element 3 by the imaging optical system 1. In the present embodiment, since the imaging optical system 1 includes the two-dimensional lens array as described above, an optical image (single-eye image) of a plurality of subjects with the size of the image circle IC is formed on the light receiving surface 30 of the imaging element 3. Is imaged.

The image sensor 3 converts the optical image of the subject into an electric signal having a magnitude corresponding to the amount of light received by each photoelectric conversion element. These electrical signals are output from the image sensor 3 to the control calculation unit 4.

The image generation unit 42 of the control calculation unit 4 is an image of a subject image (single-eye image) based on each electric signal obtained by each photoelectric conversion element corresponding to each imaging region 31 for each imaging region 31. Generate data. Then, the image generation unit 42 stores the generated image data in the image storage unit 51 in association with the imaging region 31.

In FIG. 5, the boundary detection unit 43 of the control calculation unit 4 detects the boundary position of the image circle IC of the imaging optical system 1 based on each electrical signal obtained by each photoelectric conversion element corresponding to the boundary position detection region 32. To do. Then, the boundary detection unit 43 notifies the parameter determination unit 44 of the detected boundary position of the image circle IC (S1). The detection of the boundary position of the image circle IC is executed as follows, for example.

For example, when the subject has a substantially uniform luminance distribution, an optical image is formed on the light receiving surface 30 of the image sensor 3 as shown in FIG. 6A. Then it is almost uniformly dark. For this reason, the luminance distribution on the straight line X1-X2 including not only the image circle IC but also the pixels (photoelectric conversion elements) outside the image circle IC has a luminance value on one outer side of the image circle IC as shown in FIG. 6B. Is approximately 0, the brightness value suddenly increases near one boundary in the image circle IC, the brightness value is substantially constant at a predetermined value in the image circle IC, and suddenly near the other boundary in the image circle IC. The brightness value becomes small, and the profile having the brightness value of approximately 0 again has the outside outside the image circle IC. That is, the amount of light is large at the center (center) of the image circle IC, and the amount of light decreases toward the periphery of the image circle IC.

For this reason, the threshold value Th for determining whether or not it is the boundary position of the image circle IC is set to a predetermined value, for example, the first threshold value Th0 (Th0 = 0) as shown in FIG. The boundary position of the circle IC can be detected. That is, the boundary detection unit 43 first has the luminance value of each electric signal obtained by each photoelectric conversion element (each pixel) arranged along a predetermined straight line in the boundary position detection region 32 having a first luminance value. It is determined whether or not the threshold Th0 (= 0) is exceeded. When the boundary detection unit 43 determines that the luminance value exceeds the first threshold Th0, the position of the photoelectric conversion element (pixel) at which the luminance value exceeds the first threshold Th0 is set as the boundary position of the image circle IC. To detect. Further, for example, as shown in FIG. 7, by setting a threshold value Th for determining whether or not the boundary position of the image circle IC is, for example, a second threshold value Th1 (Th1 = intermediate value) as an intermediate value, the image circle The boundary position of the IC can be detected. That is, the boundary detection unit 43 first sets the luminance value to the second value for each electric signal obtained by each photoelectric conversion element (each pixel) arranged along a predetermined straight line in the boundary position detection region 32. It is determined whether or not the threshold Th1 (= intermediate value) is exceeded. When the boundary detection unit 43 determines that the luminance value exceeds the second threshold Th1, the position of the photoelectric conversion element (pixel) where the luminance value exceeds the second threshold Th1 is set as the boundary position of the image circle IC. To detect. The intermediate value is a central value between the maximum luminance value and the minimum luminance value. Such an intermediate value may be set in advance to a central value between the maximum luminance value and the minimum luminance value that can be taken by the photoelectric conversion element of the image sensor 3 (static setting). It may be set to a central value between the maximum luminance value and the minimum luminance value obtained from the actual luminance distribution of each photoelectric conversion element (each pixel) arranged along a predetermined straight line in the detection region 32 (motion Setting). Thus, by setting the threshold value Th to a predetermined value excluding 0, the influence of noise is reduced, and the detection accuracy of the boundary position of the image circle IC is improved.

In the present embodiment, as shown in FIGS. 2 and 7, when the XY coordinate system is set along two adjacent sides in the rectangular boundary position detection region 32, a straight line along the X axis Specifically, for each electric signal obtained by each photoelectric conversion element (each pixel) arranged along a straight line of Y = 4, the boundary detection unit 43 sets the luminance value to the second threshold Th1. When it is determined whether or not the luminance value exceeds the second threshold Th1, the position of the photoelectric conversion element (pixel) at which the luminance value exceeds the second threshold Th1 is set as the boundary position of the image circle IC. To detect.

In the above description, the luminance distribution of the subject is substantially uniform, but the luminance distribution of the subject is not necessarily uniform. For this reason, the boundary detection unit 43 is configured to detect the boundary position of the image circle IC of the imaging optical system 1 based on a plurality of electrical signals obtained by converting each of the plurality of optical images by the imaging device 3. May be. The plurality of optical images may be formed on the same subject at different points in time, or may be formed on different subjects.

In such a case, the boundary detection unit 43 may be configured to detect the boundary position of the image circle of the imaging optical system 1 based on the integrated signal obtained by integrating the plurality of electrical signals. More specifically, for example, as shown in FIGS. 8B to 8D, a plurality of electrical signals are first obtained by converting each of a plurality of optical images having different luminance distributions by the image sensor 3. The plurality of electrical signals are added for each photoelectric conversion element (each pixel) and averaged by the number of the plurality of optical images. Thereby, the integrated signal is obtained. Since this integrated signal is an average of a plurality of optical images having various luminance distributions, it becomes a substantially uniform luminance distribution as shown in FIG. 8E and approximates the luminance distribution shown in FIGS. 7 and 8A.

In such a case, the boundary detection unit 43 selects the maximum luminance value at the corresponding pixel from the plurality of electrical signals, and based on the selected maximum luminance value, the boundary of the imaging optical system is selected. It may be configured to detect the boundary position of the image circle. More specifically, for example, by converting each of a plurality of optical images having different luminance distributions by the imaging element 3, a plurality of electrical signals are first obtained. In the plurality of electric signals, the maximum luminance value is selected for each photoelectric conversion element (each pixel), and the selected maximum luminance value is set as the luminance value of the photoelectric conversion element (the pixel). The As a result, a signal composed of the maximum luminance value in each photoelectric conversion element (each pixel) is obtained, and the boundary position of the image circle of the imaging optical system is detected from the luminance distribution of this signal in the same manner as described above.

Next, the parameter determination unit 44 of the control calculation unit 4 determines a predetermined parameter related to a predetermined image correction process based on the boundary position of the image circle IC detected by the boundary detection unit 43. Then, the parameter determination unit 44 notifies the image correction unit 45 of the determined parameter value (S2). The determination of this parameter is executed as follows, for example.

For example, in the present embodiment, the parameter table PT of the lookup table stored in advance in the parameter storage unit 52 as described above is used when determining the parameters. More specifically, the parameter determination unit 44 searches the boundary position x field 5211 and the boundary position y field 5212 using the boundary position of the image circle IC detected by the boundary detection unit 43 as a key. A record in which the boundary position of the detected image circle IC is registered in the boundary position x field 5211 and the boundary position y field 5212 is found. Then, the parameter determination unit 44 performs the focal length fx field 5231, focal length fy field 5232, image center cx field 5241, image center cy field 5242, distortion coefficient k1 field 5251, distortion coefficient k2 field 5252, distortion in the found record. From the coefficient p1 field 5253 and the distortion coefficient p2 field 5254, the focal length fx, the focal length fy, the image center position cx, the image center position cy, the first distortion coefficient k1, the second distortion coefficient k2, and the third distortion, respectively. The parameter values of the coefficient p1 and the fourth distortion coefficient p2 are acquired. Thereby, each value of each parameter is determined.

Next, the image correction unit 45 corrects the image of the subject generated by the image generation unit 42 and stored in the image storage unit 51 using each value of each parameter determined by the parameter determination unit 44 (S3). .

For example, when correcting the focal length shift in the imaging optical system 1, the focal length shift appears in the image as a change in the size of the image. For this reason, the image correcting unit 45 uses a focal length fx and a focal length fy determined by the parameter determining unit 44, and uses a known correction method for enlarging / reducing the image with an enlargement ratio corresponding to the focal length shift amount. , Correct the image. If the enlargement ratio is greater than 1, the enlargement / reduction process is an enlargement process. If the enlargement ratio is 1, the enlargement / reduction process is an equal magnification process, and the enlargement ratio is less than 1. In this case, the enlargement / reduction process is a reduction process.

Also, for example, when correcting the shift of the center position in the imaging optical system 1, the shift of the center position appears as a shift in the baseline length in the stereo camera. For this reason, the image correction unit 45 uses the known correction method for correcting the baseline length according to the image center position cx determined by the parameter determination unit 44 and the shift amount of the center position using the image center position cy. to correct. When the baseline length is shifted longer than the original baseline length due to the shift of the center position, the baseline length is corrected to be shorter according to the shift amount of the center position, and the baseline length is longer than the original baseline length due to the shift of the center position. In the case of a short shift, the base line length is corrected to be long according to the shift amount of the center position. Further, the shift of the center position appears as a positional shift of the posture between the single-eye images (image position bias). Therefore, the image correction unit 45 corrects the image by correcting the position of the posture by a known correction method using the image center position cx and the image center position cy determined by the parameter determination unit 44.

Also, for example, when correcting distortion in the imaging optical system 1, the distortion appears as image distortion, particularly distortion in the peripheral portion of the image. For this reason, the image correction unit 45 uses the first to fourth distortion coefficients k1, k2, p1, and p2 determined by the parameter determination unit 44, and has an abbreviated shape similar to a scene in which a distorted image can be seen with the naked eye. The image is corrected by a known correction method for correcting a natural image without distortion.

Further, for example, when correcting the rotation, the image correction unit 45 corrects the image by a known correction method that rotates the image by a matrix for rotating the image using the parameter value determined by the parameter determination unit 44. To do. Further, for example, when correcting the translation, the image correction unit 45 corrects the image by a known correction method that translates the image by a matrix for translating the image using the parameter value determined by the parameter determination unit 44. To do.

Further, for example, when correcting blur, the image correction unit 45 corrects the blur estimation coefficient in the super-resolution processing using the blur coefficient determined by the parameter determination unit 44, and performs known super-resolution processing. By doing so, the image is corrected. This super-resolution processing is a technique for generating a single high-resolution image from a plurality of low-resolution images. For example, the high-resolution processing is performed by aligning a plurality of overlapping low-resolution images with sub-pixel accuracy. One pixel image value is complemented, and then one high-resolution image is generated by removing blur, noise, and the like. Such super-resolution processing is disclosed in, for example, Japanese Patent Application Laid-Open Nos. 2003-141529, 2009-65535, and 2011-182237. Note that the blur is also caused by the image correction processing related to the internal parameters such as the correction of the shift of the focal length, the correction of the shift of the center position, and the correction of the distortion. For this reason, it is preferable that the blur correction is performed after the image correction processing related to the internal parameters.

By performing such correction processing, it is possible to correct an image according to a change in optical characteristics in the imaging optical system 1.

In one specific example, when the temperature of the imaging optical system 1 changes from about 20 ° C. to about 60 ° C., the size of the image circle IC increases as shown in FIG. 9A, and the boundary position of the image circle IC shifts. To do. As a result, the luminance distribution is also shifted as shown in FIG. 9B. The boundary position (X, 4) of the shifted image circle IC is detected as, for example, coordinates (7, 4) by the boundary detection unit 43 in process S1. As a result, in the process S2, the parameter table PT is retrieved by the parameter determination unit 44 at the coordinates (7, 4), and the focal length fx and the focal length are obtained from the fields 5231 to 5254 of the record corresponding to the coordinates (7, 4). fy, image center position cx, image center position cy, first distortion coefficient k1, second distortion coefficient k2, third distortion coefficient p1 and fourth distortion coefficient p2 parameter values fx4, fy4, cx4, xy4, k14, k24 , P14 and p24 are obtained, and each value of each parameter is determined. In step S3, the parameter values fx4, fy4, cx4, xy4, k14, k24, p14, and p24 are used to correct the image of the subject.

In the above description, the parameter is determined for each formation of the image of the subject, but the parameter determination timing is not limited to this. For example, the parameter is determined when the imaging apparatus CA is activated, and this parameter may be used until the imaging apparatus CA is turned off. For example, the parameter may be determined and updated at a predetermined time interval set in advance. Further, for example, the parameter may be determined and updated every time a subject image formed by operating a shutter button (not shown) reaches a predetermined number.

As described above, the imaging device CA and the image processing unit 2 in the present embodiment detect the boundary position of the image circle IC of the imaging optical system 1 based on the electrical signal output from the imaging device 3, and the detected image circle IC. Based on the boundary position, a predetermined parameter relating to a predetermined image correction process is determined, and the image of the subject is corrected using the determined predetermined parameter. For this reason, since the imaging apparatus CA and the image processing unit 2 in this embodiment use the image circle IC inherently included in the imaging optical system 1 to determine parameters relating to image correction processing, It is possible to detect a change in the optical characteristic of the lens without providing a special optical member as in the technique, and to correct an image in accordance with the change in the optical characteristic.

In addition, the imaging apparatus CA and the image processing unit 2 according to the present embodiment have boundaries between the image circle ICs of the imaging optical system 1 based on a plurality of electrical signals obtained by converting each of the plurality of optical images by the imaging element 3. Since the position is detected, the boundary position of the image circle IC can be detected with higher accuracy. Therefore, the imaging apparatus CA and the image processing unit 2 in this embodiment can correct an image with higher accuracy.

Further, the imaging apparatus CA in the present embodiment includes a lens array in which the imaging optical system 1 is arranged with lenses having small diameters, so that the back focus of the imaging optical system 1 can be shortened and the height can be reduced. . Such an imaging apparatus CA is particularly suitable for mobile phones and smartphones that are becoming thinner.

This specification discloses various modes of technology as described above, and the main technologies are summarized below.

An image processing apparatus according to an aspect includes an imaging device that converts an optical image of a subject formed on a light receiving surface by an imaging optical system into an electrical signal, and an image of the subject based on the electrical signal output from the imaging device. An image generation unit that generates the image circle, a boundary detection unit that detects a boundary position of the image circle of the imaging optical system based on the electrical signal output from the imaging element, and the image circle detected by the boundary detection unit A parameter determining unit that determines a predetermined parameter related to a predetermined image correction process based on the boundary position, and an image of the subject generated by the image generating unit using the predetermined parameter determined by the parameter determining unit. An image correction unit for correction.

The image circle of the imaging optical system is a circular range in which light passing through a lens included in the imaging optical system connects the images. For this reason, the boundary of the image circle is affected by the optical characteristics of the lens included in the imaging optical system. The image processing device detects a boundary position of an image circle of the imaging optical system based on an electric signal output from the image sensor, and performs a predetermined image correction process based on the detected boundary position of the image circle. The parameter is determined, and the image of the subject is corrected using the determined predetermined parameter. As described above, since the image processing apparatus uses the image circle inherent in the imaging optical system for determining the parameters relating to the image correction processing, for example, the light shielding wall disclosed in Patent Document 1 is used. Without providing a special optical member, a change in the optical characteristics of the lens can be detected, and an image can be corrected according to the change in the optical characteristics.

In another aspect, in the above-described image processing apparatus, the boundary detection unit includes an image circle of the imaging optical system based on a plurality of electrical signals obtained by converting each of a plurality of optical images by the imaging element. The boundary position of is detected. Preferably, the boundary detection unit detects a boundary position of an image circle of the imaging optical system based on an integrated signal obtained by integrating the plurality of electrical signals. Preferably, the boundary detection unit selects a maximum luminance value at a corresponding pixel from the plurality of electric signals, and based on the selected maximum luminance value, a boundary position of an image circle of the imaging optical system Is detected.

Such an image processing apparatus detects the boundary position of the image circle of the imaging optical system based on a plurality of electrical signals obtained by converting each of a plurality of optical images by the imaging device, so that it has higher accuracy. In addition, the boundary position of the image circle can be detected. Therefore, such an image processing apparatus can correct an image with higher accuracy.

In another aspect, in the above-described image processing apparatus, the predetermined parameter is one or more of a focus position, a center position, a distortion coefficient, a translation coefficient, a rotation coefficient, and a blur coefficient.

According to this, there is provided an image processing apparatus that corrects an image using one or more of a focal position, a center position, a distortion coefficient, a translation coefficient, a rotation coefficient, and a blur coefficient as the predetermined parameter.

An image processing method according to another aspect includes an imaging step of converting an optical image of a subject formed on a light receiving surface of an imaging element by an imaging optical system into an electrical signal, and an electrical signal obtained by the imaging step. An image generation step of generating an image of the subject based on the boundary, a boundary detection step of detecting a boundary position of an image circle of the imaging optical system based on the electrical signal obtained by the imaging step, and the boundary detection step A parameter determining step for determining a predetermined parameter relating to a predetermined image correction process based on the detected boundary position of the image circle, and the image generating unit using the predetermined parameter determined by the parameter determining step And an image correction process for correcting the image of the subject.

Such an image processing method detects a boundary position of an image circle of an imaging optical system based on an electric signal output from an image sensor, and relates to a predetermined image correction process based on the detected boundary position of the image circle. A predetermined parameter is determined, and the image of the subject is corrected using the determined predetermined parameter. As described above, since the image processing method uses the image circle inherent in the imaging optical system for determining the parameters relating to the image correction processing, the optical characteristics of the lens can be changed without providing a special optical member. Can be detected, and the image can be corrected in accordance with the change in the optical characteristics.

An imaging apparatus according to another aspect includes an imaging optical system that forms an optical image of a subject on a predetermined surface, and an image processing unit that generates the image of the subject, wherein the image processing unit includes the imaging unit. The image processing apparatus according to any one of the above, wherein the light receiving surface of the element is assembled so as to be a predetermined surface of the imaging optical system.

Since such an image pickup apparatus uses any of the above-described image processing apparatuses, it detects a change in the optical characteristics of the lens without providing a special optical member, and corrects the image in accordance with the change in the optical characteristics. be able to.

In another aspect, in the above-described imaging device, the imaging optical system includes a lens array including a plurality of lenses arranged in an array.

Such an image pickup apparatus includes a lens array in which lenses having small diameters are arranged in the image pickup optical system, so that the back focus of the image pickup optical system can be shortened and the height can be reduced. Such an imaging device is particularly suitable for mobile phones and smartphones that are becoming thinner.

This application is based on Japanese Patent Application No. 2013-42771 filed on March 5, 2013, the contents of which are included in the present application.

In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Therefore, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. To be construed as inclusive.

According to the present invention, an image processing device, an image processing method, and an imaging device can be provided.

Claims

An image sensor that converts an optical image of a subject imaged on a light receiving surface by an imaging optical system into an electrical signal;
An image generation unit that generates an image of the subject based on an electrical signal output from the image sensor;
A boundary detection unit that detects a boundary position of an image circle of the imaging optical system based on an electrical signal output from the imaging element;
A parameter determination unit that determines a predetermined parameter related to a predetermined image correction process based on a boundary position of the image circle detected by the boundary detection unit;
An image processing apparatus comprising: an image correction unit that corrects the image of the subject generated by the image generation unit using the predetermined parameter determined by the parameter determination unit.
The boundary detection unit detects a boundary position of an image circle of the imaging optical system based on a plurality of electrical signals obtained by converting each of a plurality of optical images by the imaging device. The image processing apparatus according to 1.
The image processing according to claim 2, wherein the boundary detection unit detects a boundary position of an image circle of the imaging optical system based on an integrated signal obtained by integrating the plurality of electrical signals. apparatus.
The boundary detection unit selects a maximum luminance value at a corresponding pixel from the plurality of electrical signals, and detects a boundary position of an image circle of the imaging optical system based on the selected maximum luminance value. The image processing apparatus according to claim 2.
The predetermined parameter is one or more of a focus position, a center position, a distortion coefficient, a translation coefficient, a rotation coefficient, and a blur coefficient. Image processing apparatus.
An imaging step of converting an optical image of a subject imaged on a light receiving surface of an image sensor by an imaging optical system into an electrical signal;
An image generation step of generating an image of the subject based on the electrical signal obtained by the imaging step;
A boundary detection step of detecting a boundary position of an image circle of the imaging optical system based on the electrical signal obtained by the imaging step;
A parameter determining step for determining a predetermined parameter related to a predetermined image correction process based on a boundary position of the image circle detected by the boundary detecting step;
An image processing method comprising: an image correction step of correcting the image of the subject generated by the image generation unit using the predetermined parameter determined by the parameter determination step.
An imaging optical system that forms an optical image of a subject on a predetermined surface;
An image processing unit for generating an image of the subject,
The image processing device according to any one of claims 1 to 5, wherein the image processing unit is assembled so that a light receiving surface of the imaging element is a predetermined surface of the imaging optical system. An imaging device.
The imaging apparatus according to claim 7, wherein the imaging optical system includes a lens array including a plurality of lenses arranged in an array.