JP2009017336A - Imaging device and imaging system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device, an imaging method, an imaging system, an image processing device, an image processing program, and an image processing method capable of feeding information for restoring the image before composite to the image after composite. <P>SOLUTION: The imaging system 100 includes: an imaging unit 10 for picking up an object with exposure time T1-T3(T1<T2<T3); a switch (SW) 11 for switching the output destination of a pixel signal for every type of exposure time; first to third memories 12-14 for storing pixel data corresponding to the exposure time T1-T3; a pixel data composite unit 15 for compounding the pixel data corresponding to the exposure time T1-T3; a restoring objective image selecting unit 16 for selecting the pixel data of a restoring objective image; and a restoring information feeding unit 17 for feeding restoration information of restoration objective pixel data to the pixel data after composite. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, and more particularly, to an imaging apparatus capable of expanding a dynamic range of a captured image by combining pixel signals obtained by imaging a subject with a plurality of types of exposure times.

  Conventionally, cameras for surveillance and crime prevention have been installed in various places. In general, since the dynamic range of a camera is narrower than that of a human, there is a problem that an object that can be confirmed by human eyes is not reflected in a camera image due to blackout or overexposure. Therefore, as a means for obtaining a wide dynamic range (HDR) image, there is an image composition method using multi-step exposure. As an imaging apparatus using an image composition method by multistage exposure, for example, there is an imaging apparatus and an imaging method described in Patent Document 1.

In the imaging apparatus and imaging method described in Patent Document 1, when an HDR image is generated by synthesizing multi-stage exposure images, the presence / absence of a moving object is determined for each pixel to change the image synthesis ratio. For example, an image having a small exposure time (short) is used in an area where a moving object is present, and an image having a large exposure time (long) is used in an area where there is no moving object and is not saturated.
JP 2004-254151 A

However, in the prior art of the above-mentioned patent document 1, when an important image portion is blurred and cannot be identified in the image after synthesis, information on pixels that are not used for synthesis is discarded. It was impossible to refer to the previous image or to identify the important image portion using the pre-combination image as a clue.
Therefore, the present invention has been made paying attention to such an unsolved problem of the conventional technology, and it is possible to add information for restoring the pre-combination image to the composited image. An object of the present invention is to provide an imaging device and an imaging system.

[Mode 1] In order to achieve the above object, an imaging apparatus according to mode 1
An imaging apparatus comprising: a photoelectric conversion unit having a configuration in which a plurality of photoelectric conversion elements that convert received light into electric charges and store them are arranged in a matrix; and a function of controlling an exposure time of the photoelectric conversion elements,
Pixel signal reading means for reading out pixel signals corresponding to the plurality of types of exposure times when exposed at a plurality of types of exposure times from each pixel constituting the photoelectric conversion element;
Pixel data synthesizing means for synthesizing pixel data, which is pixel data corresponding to the plurality of types of exposure times, read by the pixel signal reading means for each pixel;
Restoration information giving means for giving restoration information used when restoring the pre-combination pixel data corresponding to predetermined exposure times in the plurality of types of exposure times to the pixel data after the pixel data synthesis means has been combined; It is characterized by providing.

  With such a configuration, when the subject is imaged, for example, when the photoelectric conversion elements are exposed in the order of exposure times T1 to TN (T1 <T2 <... <T (N-1), TN), Pixel signals corresponding to the amount of charge accumulated in the photoelectric conversion element at each exposure time are read out. In this reading process, for example, it is possible to expose the photoelectric conversion element at each exposure time of T1 to TN and read out the pixel signal from the photoelectric conversion element exposed at each exposure time by destructive reading. Further, for example, it is possible to perform exposure up to the exposure time TN and to read out the pixel signal from the photoelectric conversion element by nondestructive reading at each exposure time T1 to TN. Hereinafter, the same applies to the imaging method according to the nineteenth aspect.

  When pixel signals corresponding to a plurality of types of exposure times (T1 to TN) are read out, pixel data, which is data of pixel signals corresponding to the read out types of exposure times, is displayed for each pixel. Is synthesized. For example, when exposure is performed at exposure times T1 to T3, pixel data corresponding to these three types of exposure times for the target pixel is synthesized, and synthesized pixel data of the target pixel is generated. Further, the synthesized pixel data constitutes captured image data with a wide dynamic range (HDR).

  Then, restoration information used when restoring the pre-combination pixel data corresponding to the predetermined exposure time among a plurality of types of exposure time is added to the post-combination pixel data. This restoration information can be used to restore the pre-combination pixel data only with this information, the restoration information can be used to restore the pre-combination pixel data using the post-combination pixel data information, and there is data corresponding to the restoration information. For example, the pixel data before synthesis can be restored by referring to the stored data table. Hereinafter, the same applies to the imaging method of form 19, the image processing apparatus of form 20, the image processing program of form 21, the image processing method of form 22, and the like.

As described above, restoration information used when restoring pixel data before composition for a predetermined exposure time can be added to the pixel data after composition. Therefore, composition is performed using the restoration information given to the pixel data after composition. Previous pixel data can be restored.
As a result, for example, when an important subject cannot be identified in the combined image, the pre-combination image is restored and identified from the restored image, or combined with the restored pre-combination image. For example, it is possible to perform identification processing using an image before composition after imaging, such as identification using a subsequent image.

  Here, the above-mentioned “photoelectric conversion unit” is an image sensor configured using, for example, a charge-coupled device (CCD), CMOS technology, or the like. For example, as a nondestructive readable image sensor using CMOS technology, there is a threshold modulation image sensor (for example, VMIS (Threshold Voltage Modulation Image Sensor)). Hereinafter, the same applies to the imaging method of aspect 19.

  The function for controlling the exposure time is, for example, a known electronic shutter function including a system such as a global shutter, a focal plane shutter (rolling shutter), or the aperture mechanism of the imaging apparatus, and the exposure amount is changed. Applicable functions. The electronic shutter function has a function of applying a voltage to an image sensor composed of a CCD, a CMOS, etc., and accumulating light as an electric charge in the image sensor. The diaphragm mechanism is a mechanism that adjusts the amount of light that enters the lens by opening and closing the wing, which is called a diaphragm blade, that blocks light. Hereinafter, the same applies to the imaging method of aspect 19.

The destructive readout is accompanied by a reset process for emptying the charge accumulated in the photoelectric conversion element when the charge (pixel signal) is read from the photoelectric conversion element.
The non-destructive reading is to read out charges (pixel signals) from a photoelectric conversion element while keeping the accumulated state without emptying the charge accumulated in the photoelectric conversion element. That is, since the reset process is not performed at the time of reading the charge, the charge can be read out repeatedly for different exposure times during the accumulation of the charge until reaching the set exposure time. Therefore, non-destructive readout has an advantage that multistage exposure can be easily realized. Hereinafter, the same applies to the imaging method according to the nineteenth aspect.

[Mode 2] Further, the imaging apparatus according to mode 2 is the imaging apparatus according to mode 1,
The restoration information includes at least one of single-color luminance information, color difference information, and color signal information included in the pixel data before synthesis.
With such a configuration, for example, if the pixel data is monochrome pixel data, the luminance information can be given as restoration information to the combined pixel data.

For example, if the pixel data is color pixel data composed of color signal information of a plurality of colors, the luminance information of any one color or the luminance information and color difference information of any one color, or the color signal information of each color is used as restoration information. It can be given to the combined pixel data.
Here, the color signal information is signal information including color information and luminance information thereof.

[Mode 3] Further, the imaging apparatus according to mode 3 is the imaging apparatus according to mode 2,
The restoration information includes information on a difference value between luminance information included in the pixel data before combination and luminance information included in the pixel data after combination, color difference information included in the pixel data before combination, and pixel data after combination. At least one of the difference value information with respect to the color difference information possessed by the color signal information and the difference value information between the color signal information possessed by the pixel data before composition and the color signal information possessed by the pixel data after composition. It is characterized by that.

With such a configuration, the difference information between the luminance information included in the pixel data before combining and the luminance information included in the pixel data after combining, the color difference information included in the pixel data before combining, and the pixel data after combining The difference information between the color difference information and the color signal information included in the pixel data before combination and the color signal information included in the pixel data after combination are used as restoration information. It can be given to subsequent pixel data.
Thereby, the effect that the data amount provided as restoration information can be reduced is acquired.

[Mode 4] Furthermore, the imaging device according to mode 4 is the imaging device according to any one of modes 1 to 3,
The restoration information adding means replaces a bit string of lower-order NC bits in the combined pixel data with an NC bit string (NC <N) obtained by compressing an N-bit bit string of the restoration information.

  With such a configuration, it is possible to reduce the amount of information to be given as restoration information, and to obtain the effect that restoration information can be given without changing the format of the image data after synthesis. . For example, when 8-bit luminance information is given as restoration information, it is compressed into upper 3 bits of information, and this 3-bit restoration information is embedded instead of the lower 3 bits of the combined pixel data, Both post-combination and pre-combination images can be obtained with image quality comparable to that of the original, and restoration information can be added without changing the format of the pixel data, so special processing is performed. In addition, it is possible to handle post-combination image data to which restoration information is added in general-purpose application software or the like.

[Embodiment 5] Furthermore, the imaging apparatus according to Embodiment 5 is the imaging apparatus according to any one of Embodiments 1 to 4,
The restoration information adding means corresponds to each color of the plurality of colors used when restoring the color signal information of the plurality of colors included in the pixel data before the synthesis when the pixel data has color signal information of a plurality of colors. The restoration information is added to the color signal information of each color included in the combined pixel data.

  With such a configuration, information for restoring luminance information corresponding to the three primary colors of R, G, and B, for example, constituting the pixel data of each pixel can be added as restoration information. It is possible to restore the pixel data before synthesis of each color from the restoration information given to the color signal information (pixel data of each color).

[Mode 6] Furthermore, the imaging device according to mode 6 is the imaging device according to any one of modes 1 to 3,
When the pixel data includes color signal information of a plurality of colors, the restoration information providing unit converts a bit string of color signal information of any one color of the plurality of colors of the pixel data before synthesis into the colors of the plurality of colors. The restoration information is added to the combined pixel data by dividing the pixel data into several minutes and replacing the bit string on the lower side of the combined pixel data with the divided bit strings.

With such a configuration, the color signal information of any one color among the color signal information of the plurality of colors included in the pixel data before synthesis can be added to the pixel data after synthesis as restoration information without loss of information. At the same time, since it is only necessary to provide color signal information for one color, an effect of reducing the amount of data to be provided can be obtained. Furthermore, there is an effect that restoration information can be given without changing the format of the combined pixel data.
In this case, for example, the pixel data before synthesis can be restored by using the color signal information of the restoration information and the color signal information of other colors included in the pixel data after synthesis.

[Mode 7] Furthermore, the imaging device according to mode 7 is the imaging device according to any one of modes 1 to 3,
When the pixel data includes R (Red), G (Green), and B (Blue) color signal information, the restoration information adding unit converts the color signal information included in the pixel data before combining into luminance information and Divided into color difference information, the luminance information is divided into three, and the lower-order bit string of the pixel data of each color after the combination is replaced with the divided bit string, whereby the restoration information is added to the pixel data after the combination. It is characterized by giving.

With such a configuration, the luminance information included in the pixel data before combining can be added to the pixel data after combining as restoration information without loss of information, and only the luminance information needs to be added. The effect that the amount of data can be reduced is obtained. Furthermore, there is an effect that restoration information can be given without changing the format of the combined pixel data.
In this case, monochrome data with no loss of luminance information can be restored to the original data by combining the restoration information (luminance information) that is divided and given to each color signal information after synthesis. .

[Embodiment 8] Furthermore, an imaging apparatus according to Embodiment 8 is the imaging apparatus according to Embodiment 3,
In correspondence with each bit string represented by a bit number L smaller than the number of bits of the pixel data, a difference information table storing information of the difference values of different numerical values, respectively,
The restoration information providing means is a bit string in which L bits on the lower side of the combined pixel data are associated with information on difference values closest to the restoration information of the pixel data before combining in the difference information table It is characterized by replacing with.

  With such a configuration, information on the difference value corresponding to the bit string assigned as restoration information can be obtained from the table and the pixel data before synthesis can be restored. While reducing, it is possible to restore the pre-combination pixel data with higher accuracy. Furthermore, since the information included in the table is used as the difference value information, the effect that the size of the difference information table can be reduced is also obtained.

Here, the difference value information is, for example, an absolute difference amount α “α = α = the absolute difference amount between the pixel data S and P, where S is pixel data before combining and P is pixel data after combining. S−P ”, or a relative difference amount β“ β = S / P ”, which is a relative amount of the difference value with respect to the pixel data P, corresponds. Hereinafter, the same applies to the imaging device according to the ninth embodiment.
Further, for example, when the difference information table is created by creating a difference information table in which difference value information is stored in association with each bit string composed of 3 bits, for example, “000”, “001” ,..., “110”, “111” are stored in association with the difference value information for the eight types of bit strings. For example, in the case of an absolute difference amount, +192, +128, +64..., -192, -255 are stored in association with the eight types of bit strings, while on the other hand, in the case of a relative difference amount, 175%. , 150%,..., 25%, 0%, etc. are stored in association with each other. Note that, by increasing the number of bits in the bit string, finer difference value information can be prepared. Hereinafter, the same applies to the imaging device according to the ninth embodiment.

[Embodiment 9] Furthermore, the imaging device according to Embodiment 9 is the imaging device according to any one of Embodiments 3 to 8,
The image processing apparatus includes an approximation unit that approximates the brightness information of the image generated from the pre-combination pixel data to the brightness information of the image generated from the post-combination pixel data.
With such a configuration, for example, when a difference information table is used for assigning restoration information, it is possible to associate finer difference value information with a bit string assigned as restoration information. As a result, the difference in the difference value between the bit strings can be reduced, so that it is possible to restore the pre-combination pixel data with higher accuracy. In addition, since the difference value can be reduced, an effect that the data amount (the number of bits) can be reduced when the difference value information is directly applied as restoration information can be obtained. In addition, even when the difference value is compressed and given, an effect that more accurate information can be given with the same number of bits can be obtained.

[Mode 10] Further, the imaging apparatus according to mode 10 is the imaging apparatus according to mode 9,
The approximating means adjusts the gamma value of the image generated from the pixel data before synthesis, thereby obtaining the brightness information of the image as the information on the brightness of the image generated from the pixel data after synthesis. It is characterized in that
With such a configuration, the pixel data before synthesis is adjusted by adjusting the gamma value, which is the ratio between the change in image brightness and the input / output voltage, for the image generated from the pixel data before synthesis. Can be appropriately approximated to the brightness information of the image generated from the combined pixel data.

  Thereby, for example, when a difference information table is used for giving restoration information, finer difference value information can be associated with a bit string given as restoration information. As a result, the difference in the difference value between the bit strings can be reduced, so that it is possible to restore the pre-combination pixel data with higher accuracy. In addition, since the difference value can be reduced, an effect that the data amount (the number of bits) can be reduced when the difference value information is directly applied as restoration information can be obtained. In addition, even when the difference value is compressed and given, an effect that more accurate information can be given with the same number of bits can be obtained.

[Mode 11] Furthermore, an imaging apparatus according to mode 11 is the imaging apparatus according to mode 9,
The approximating means adjusts the gain of the image generated from the pixel data before the synthesis to change the brightness information of the image into the information of the brightness of the image generated from the pixel data after the synthesis. It is characterized by approximation.
With such a configuration, by adjusting the gain of the output signal for the image generated from the pixel data before synthesis, information on the brightness of the image generated from the pixel data before synthesis is obtained. It is possible to appropriately approximate the brightness information of the image generated from the combined pixel data.

  Thereby, in the difference information table, finer difference value information can be associated with the bit string to be given as restoration information. As a result, the difference in the difference value between the bit strings can be reduced, so that it is possible to restore the pre-combination pixel data with higher accuracy.

[Mode 12] Furthermore, the imaging device according to mode 12 is the imaging device according to any one of modes 1 to 11,
Based on the pixel data corresponding to the plurality of types of exposure time, comprising restored image selection means for selecting pixel data of the exposure time to be restored from among the pixel data corresponding to the plurality of types of exposure time,
The restoration information adding unit adds the restoration information of the restoration target pixel data determined by the restoration image selection unit to the combined pixel data.

With such a configuration, based on pixel data of images corresponding to a plurality of types of exposure times, among the pixel data corresponding to each of these exposure times, the image quality is relatively good (the subject is relatively clear). The pixel data forming the displayed image can be selected as a restoration target.
Thereby, it is possible to refer to when the image quality of the combined image is poor, or to obtain appropriate pixel data restoration information by using it for subject identification processing.

[Mode 13] Further, the imaging apparatus according to mode 13 is the imaging apparatus according to mode 12,
The restoration image selection means selects pixel data corresponding to the exposure time to be restored based on a histogram of luminance information of an image for each exposure time constituted by the pixel data before synthesis. .
With such a configuration, for example, an ideal luminance information histogram shape is compared with a histogram of an image composed of pre-combination pixel data corresponding to each exposure time. It is possible to discriminate an image having a histogram having a shape closest to the histogram of accurate luminance information (luminance information with sharp image quality).

  Therefore, restoration information of pixel data that forms an image having a histogram closest to an ideal histogram among pixel data corresponding to a plurality of types of exposure times can be added to each combined pixel data. In addition, it can be referred to when the image quality of the combined image is poor, or can be used for subject identification processing, so that it is possible to provide appropriate pixel data restoration information.

[Mode 14] Further, the imaging apparatus according to mode 14 is the imaging apparatus according to mode 12,
The restored image selection means selects pixel data corresponding to the exposure time to be restored based on the frequency information of the image for each exposure time configured from the pixel data before synthesis.
With such a configuration, for example, the image quality of the image composed of pixel data corresponding to each exposure time can be determined from the high frequency component of the pixel data before synthesis, and so on. Thus, it is possible to determine an image with the best image quality (for example, an image that does not become blurred or relatively lightly blurred) or an image that includes the most information (object).

Therefore, among the images corresponding to each of the plurality of types of exposure times, for example, restoration information of pixel data that forms an image containing more high-frequency components (an image that is sharper and contains more objects) By adding to each pixel data, it is possible to refer to when the image quality of the synthesized image is poor, or to provide appropriate image restoration information when used for subject identification processing. can get.
Here, the frequency component extraction processing is performed by, for example, extracting an edge component (high frequency component) included in the pixel data using a known edge filter (high pass filter) such as a Laplacian filter.

[Mode 15] On the other hand, in order to achieve the above object, the imaging system of mode 15
The imaging apparatus according to any one of forms 1 to 14,
Pixel data restoring means for restoring the pre-combination pixel data based on the restoration information given to the combined pixel data.
With such a configuration, an operation equivalent to that of the imaging device according to any one of the first to thirteenth aspects can be obtained, and a pre-combination pixel corresponding to a predetermined exposure time using restoration information included in the post-combination pixel data. Data can be restored.
Thereby, an effect equivalent to that of the imaging device according to any one of the first to fourteenth aspects is obtained.

  Here, this system may be realized as a single device, terminal, or other device, or may be realized as a network system in which a plurality of devices, terminals, or other devices are communicably connected. . In the latter case, each component may belong to any one of a plurality of devices and the like as long as they are connected so as to communicate with each other.

[Mode 16] Furthermore, an imaging system according to mode 16 is the imaging system according to mode 15,
When the pixel data has color signal information of a plurality of colors and the restoration information of the combined pixel data is luminance information of any one of the colors, The pre-combination pixel data is restored based on the luminance information and the color difference information of the post-combination pixel data.
With such a configuration, there is almost no difference in color difference between the pixel data before and after the synthesis, and therefore, by restoring the pixel data before synthesis using the color difference information of the pixel data after synthesis, the color difference between the original and the original is almost the same. It is possible to restore an image that is not present.

[Mode 17] Furthermore, an imaging system according to mode 17 is the imaging system according to mode 15 or 16,
The pixel data restoration unit includes the pixel data including color signal information of R (Red), G (Green), and B (Blue), and the restoration information included in the combined pixel data is the R, G, When the color signal information of B is divided into luminance information and color difference information, the luminance information is divided into the number of colors, and the divided information given to the pixel data corresponding to each color after the combination is divided. The luminance information is combined to restore single-color luminance information of the pixel data before synthesis.

With such a configuration, it is possible to restore monochrome data with no loss of luminance information with respect to the original data by combining restoration information (luminance information) that is divided and added to each color signal information after synthesis. Can do.
Therefore, since only luminance data needs to be given as information for restoring the original data, there is an effect that the data amount of the restored information can be reduced, and the color is monochromatic, but the same luminance information as the original data is obtained. Therefore, it is possible to restore an image in which an important object in the image can be clearly visually recognized by adding luminance information of an image with good image quality as restoration information.

[Mode 18] Furthermore, an imaging system according to mode 18 is the imaging system according to any one of modes 15 to 17,
In correspondence with each bit string represented by a bit number L smaller than the number of bits of the pixel data, a difference information table storing information of the difference values of different numerical values, respectively,
When the restoration information included in the combined pixel data is a bit string corresponding to the difference value information, the pixel data restoration unit obtains difference value information corresponding to the bit string from the difference information table. The pixel data before synthesis is restored based on the acquired difference value information.

  With such a configuration, information on the difference value corresponding to the bit string assigned as restoration information can be acquired from the table and the pixel data before synthesis can be restored, so depending on the accuracy of the difference information table, An effect is obtained that an image having an image quality closer to the image to be restored can be restored.

[Mode 19] On the other hand, in order to achieve the above object, an imaging method according to mode 19 includes:
Imaging used in an imaging apparatus having a photoelectric conversion unit having a configuration in which a plurality of photoelectric conversion elements that convert received light into charges and store them in a matrix and a function for controlling the exposure time of the photoelectric conversion elements A method,
A pixel signal reading step of reading out pixel signals corresponding to the plurality of types of exposure times when exposed at a plurality of types of exposure times from each pixel constituting the photoelectric conversion element;
A pixel data synthesis step for synthesizing, for each pixel, pixel data that is pixel data corresponding to the plurality of types of exposure times read out in the pixel signal readout step;
A restoration information giving step for giving restoration information used when restoring the pre-combination pixel data corresponding to predetermined exposure times in the plurality of types of exposure times to the pixel data after the pixel data synthesis step; It is characterized by including.
Thereby, the same operation and effect as those of the imaging device of aspect 1 can be obtained.

[Mode 20] In order to achieve the above object, an image processing apparatus according to mode 20
Pixel data synthesizing means for synthesizing pixel data composed of pixel signals respectively corresponding to each exposure time obtained by imaging a subject with a plurality of types of exposure times;
Restoration information giving means for giving restoration information used when restoring pre-combination pixel data corresponding to predetermined exposure times in the plurality of types of exposure times to the pixel data after the pixel data synthesis means It is characterized by providing.
Thereby, the same operation and effect as those of the imaging device of aspect 1 can be obtained.

[Form 21] In order to achieve the above object, an image processing program of form 21 includes:
A pixel data synthesizing step for synthesizing pixel data composed of pixel signals respectively corresponding to each exposure time obtained by imaging a subject with a plurality of types of exposure times;
A restoration information adding step of adding restoration information used when restoring pixel data before synthesis corresponding to predetermined exposure times in the plurality of types of exposure time to the pixel data after synthesis in the pixel data synthesis step. A program for causing a computer to execute processing is included.
With such a configuration, when the program is read by the computer and the computer executes processing in accordance with the read program, the same operations and effects as those of the imaging apparatus according to the first aspect are obtained.

[Mode 22] In order to achieve the above object, an image processing method according to mode 22 includes:
A pixel data synthesizing step for synthesizing pixel data composed of pixel signals respectively corresponding to each exposure time obtained by imaging a subject with a plurality of types of exposure times;
A restoration information giving step for giving restoration information used when restoring pixel data before synthesis corresponding to predetermined exposure times in the plurality of types of exposure time to the pixel data after synthesis in the pixel data synthesis step; It is characterized by including.
Thereby, the same operation and effect as those of the imaging device according to the first aspect can be obtained.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. 1 to 7 are diagrams showing a first embodiment of an imaging apparatus, an imaging system, an imaging method, an image processing apparatus, an image processing program, and an image processing method according to the present invention.
First, the configuration of an imaging system according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of an imaging system 100 according to the present invention.

  As shown in FIG. 1, the imaging system 100 includes an imaging unit 10 that images a subject with three types of exposure times T1 to T3 (T1 <T2 <T3), and exposure times T1 to T3 output from the imaging unit 10. A switch (SW) 11 that switches the output destination of the corresponding pixel signal for each type of exposure time, and a first memory 12 that stores pixel signal data (hereinafter referred to as first pixel data) corresponding to the exposure time T1. A second memory 13 for storing pixel signal data corresponding to the exposure time T2 (hereinafter referred to as second pixel data), and pixel signal data corresponding to the exposure time T3 (hereinafter referred to as third pixel data). ) And a third memory 14 for storing.

  The imaging unit 10 includes a CCD image sensor configured to include a CCD and a light receiving element (phototransistor), condenses light from a subject on each pixel via a lens, and controls an exposure time by an electronic shutter function. Then, the light receiving element of each pixel is exposed at the exposure times T1, T2, and T3. A pixel signal (analog data) corresponding to each exposure time is read out from each pixel exposed at the exposure times T1, T2, and T3, and the read analog pixel signal is converted into an AFE (Analog Front End) (not shown). ) Are converted into digital pixel signals (pixel data) and then output to SW11. At this time, the imaging unit 10 also outputs a signal for identifying which of the exposure times T1, T2, and T3 corresponds to the pixel data to be output.

When the pixel data is input from the imaging unit 10, the SW 11 switches a switch based on the identification signal input corresponding thereto, and corresponds to the exposure time indicated by the identification signal among the first to third memories 12 to 14. The input pixel data is output to the memory.
The first to third memories 12 to 14 are composed of a frame memory and a memory controller, and store pixel data for one frame imaged at an exposure time corresponding to each of them. At this time, a two-dimensional storage address of each pixel data is generated by the memory controller based on a pixel clock, a vertical synchronization signal, and a horizontal synchronization signal from a timing control unit (not shown), and the data is stored in a two-dimensional memory space. to manage.

  The imaging system 100 further includes a pixel data synthesis unit 15 that synthesizes pixel data corresponding to the exposure times T1 to T3, a restoration target image selection unit 16 that selects a restoration target image, and a pixel data synthesis unit 15 after synthesis. To the pixel data (hereinafter referred to as post-combination pixel data), a restoration information adding unit 17 that adds restoration information for restoring the pixel data to be restored, and post-combination pixel data (hereinafter, referred to as restoration information). Image storage unit 18 for storing restoration information-added post-combination pixel data), display unit 19 for displaying an image based on post-combination information-added post-combination pixel data, and restoration information attached to post-combination-post-combination pixel data And a pixel data restoring unit 20 for restoring the pixel data to be restored.

  The pixel data combining unit 15 first normalizes the pixel data corresponding to the exposure times T1 to T3 according to T3, and then combines the normalized pixel data of T1 to T3 to obtain the combined pixel data. Generate. Since the combined pixel data is generated by combining three pieces of pixel data having different exposure times, an image constituted by the combined pixel data of each pixel is an image having a wide dynamic range (HDR image).

In addition, the pixel data combining unit 15 performs, for example, an external device when the display capability of the display unit does not correspond to the dynamic range of the combined pixel data (HDR pixel data). Thus, it is possible to compress the highlight portion by performing correction by gamma correction or LUT (Look-Up Table).
As illustrated in FIG. 2, the restoration target image selection unit 16 includes a histogram generation unit 16a, a comparison unit 16b, and a pixel data selection unit 16c.

Here, FIG. 2 is a block diagram illustrating a detailed configuration of the restoration target image selection unit 16.
The histogram generation unit 16a reads the first to third pixel data corresponding to the exposure times T1 to T3 from the first to third memories, and for each exposure time based on the read first to third pixel data. A histogram indicating the relationship between the pixel value (luminance value) and the number of pixels is generated. The generated histograms corresponding to the images of the exposure times T1 to T3 are output to the comparison unit 16b.

  The comparison unit 16b has a histogram having an ideal distribution shape (hereinafter referred to as an ideal histogram), and compares the ideal histogram with first to third histograms corresponding to images with exposure times T1 to T3, respectively. To do. Specifically, the ideal histogram and the first to third histograms compare the number of pixels with respect to each luminance level or a predetermined range of luminance levels, and use the exposure time information that best matches both as a comparison result to obtain a pixel data selection unit. To 16c.

When the information on the exposure time is input from the comparison unit 16b, the pixel data selection unit 16c reads the pixel data from the memory corresponding to the information on the exposure time among the first to third memories 12 to 14, and reads the read data. The pixel data is output to the restoration information adding unit 17 as restoration target pixel data (hereinafter referred to as restoration target pixel data).
The restoration information adding unit 17 includes a restoration information generation unit 17a and a restoration information embedding unit 17b.

Here, FIG. 3 is a block diagram illustrating a detailed configuration of the restoration information adding unit 17.
The restoration information generation unit 17 a generates restoration information for restoring the restoration target pixel data based on the restoration target pixel data input from the restoration target image selection unit 16.
In the present embodiment, specifically, if the pixel data is monochromatic data, the higher NC bits (NC <N) in the restoration target pixel data (N bits) are extracted. If the pixel data is data corresponding to each color such as R, G, B, etc., the upper NC bits of the pixel data (N bits) corresponding to each color are extracted. The restoration information generation unit 17a outputs the extracted higher NC bit data as restoration information to the restoration information embedding unit 17b.

The restoration information embedding unit 17b embeds the restoration information input from the restoration information generation unit 17a in the combined pixel data input from the pixel data synthesis unit 15.
Specifically, the data of the lower NC bits of the post-combination pixel data is replaced with the NC bit data of the restoration information. As a result, post-combination pixel data with restoration information added with restoration information is generated. The generated post-combination pixel data with restoration information added is output to the image storage unit 18 and the display unit 19, respectively.

The image storage unit 18 includes a storage medium such as a flash memory and a hard disk and a drive device thereof. The restored information-added combined pixel data to which the restoration information is added by the restoration information adding unit is stored in a storage medium in a predetermined file format. save.
The display unit 19 is composed of an HDR display (for example, a contrast ratio “40000: 1” or the like). The restored information-added combined image data (hereinafter referred to as restoration information-added HDR image) input from the restoration information adding unit 17. The image of the subject is displayed based on the restoration information-added HDR image data stored by the image storage unit 18. Further, the HDR display can display the image of the restoration information-added HDR image data without range compression.

In response to the restoration instruction, the pixel data restoration unit 20 reads restoration-information-added post-combination pixel data to be restored from the storage medium via the image storage unit 18 and includes the restoration-information-added post-combination pixel data. Based on the information, the pixel data to be restored is restored.
In the present embodiment, since the upper NC bits of the restoration target pixel data are given as the restoration information, the NC bits are returned to N-bit data (the lower (N-NC) bits are all 0). The restoration target pixel data is restored by converting to.

  Note that the functions of the above-described components of the imaging system 100 are realized by only hardware, only software, or a combination of hardware and software. Accordingly, the imaging system 100 is not shown in the case where there is a component that implements a function using software, but a processor for executing the software, a storage medium such as a ROM for storing the software, A storage medium such as a RAM used for execution and a bus for exchanging data with each component are provided.

  In addition, each of the above-described components may be configured so as to be entirely included in one device (in this case, an imaging device = an imaging system), or the imaging unit 10, SW11, first to third memories 12 to 14, the pixel data composition unit 15, the restoration target image selection unit 16, and the restoration information addition unit 17 are configured in one device (in this case, corresponding to the imaging device), and the image storage unit 18, the display unit 19, and the pixel The data restoration unit 20 may be composed of one or a plurality of devices. In the latter case, the imaging device, the image storage unit 18, the display unit 19, and the pixel data restoration unit 20 are connected by wire or wirelessly.

Next, the operation of the present embodiment will be described with reference to FIGS.
Here, FIG. 4 is a diagram illustrating a relationship between the combined pixel data obtained by linear combination and the combined pixel data obtained by compressing the highlight portion by gamma correction. FIGS. 5A to 5D are diagrams illustrating an example of an ideal histogram and an image histogram corresponding to exposure times T1 to T3. FIG. 6 is a diagram illustrating an example of a concept of restoration information generation processing and restoration information embedding processing. FIGS. 7A to 7F are diagrams showing a flow of a series of processing including pixel data synthesis processing, restoration target image selection processing, restoration information embedding processing, and restoration target image restoration processing.

  The imaging system 100 first controls the shutter speed (exposure time) by the electronic shutter function, and exposes each pixel of the imaging unit 10 in the order of the exposure times T1 (15H), T2 (100H), and T3 (500H). The pixel signal corresponding to the exposure amount at each exposure time is read out. Here, H represents time (s) in one horizontal period. Therefore, for example, if it is 15H, the exposure time is 15 × H (s).

  The read pixel signal (analog data) is temporarily stored in a line memory (not shown), then converted to digital pixel data via an AFE (not shown), and SW11 together with an identification signal for identifying exposure times T1 to T3. Is output. When the pixel data and the identification signal are input, the SW 11 switches the switch to the memory corresponding to the exposure time indicated by the identification signal among the first to third memories 12 to 14, and transfers the pixel data to the memory. Output. Here, it is assumed that the pixel data is data having an 8-bit gradation value (luminance value).

  On the other hand, when the pixel data is input, the first to third memories 12 to 14 are input by a memory controller based on a vertical synchronization signal, a horizontal synchronization signal, and a pixel clock input from a timing control unit (not shown). A two-dimensional address (x, y) for the pixel data is generated, and the pixel data is stored in a storage area corresponding to the address. The first to third memories 12 to 14 share the same pixel address value.

Here, it is assumed that first to third pixel data corresponding to images as shown in FIGS. 7A to 7C are obtained in the exposure times T1 to T3, respectively.
In this manner, when the pixel data corresponding to the exposure times T1 to T3 for one frame is stored in the first to third memories 12 to 14, the pixel data composition unit 15 stores the first to third memories 12 to 12. To 14, the first to third pixel data corresponding to the exposure times T1 to T3 are read out for each pixel. On the other hand, the restoration target image selection unit 16 reads out the first to third pixel data for each image from the first to third memories 12 to 14.
The pixel data combining unit 15 combines the first to third pixel data for the read pixels.
As a combining method, the first to third pixel data are linearly combined, and thereafter gamma correction or LUT correction is performed as necessary to compress the highlight portion.

Specifically, the first to third pixel data corresponding to the exposure times T1 to T3 are linearly synthesized according to the following expression (1).

IMG_HDR_Linear = (IMG_T1 × T3 / T1 + IMG_T2 × T3 / T2 + IMG_T3 × T3 / T3) / (255 × T3 / T1 + 255 × T3 / T2 + 255 × T3 / T3) (1)

In the above equation (1), IMG_HDR_Linear is the pixel value of the combined pixel data, and IMG_T1 to IMG_T3 are the pixel values of the first to third pixel data.
As shown in the above equation (1), the first to third pixel data are normalized in accordance with the exposure time T3 to obtain the sum, and the sum is used as the highlight brightness level to the first to third. Similar to the pixel data, the post-combination pixel data is obtained by dividing by the normalized sum.

Further, when the highlight portion is compressed, gamma correction is performed by the gamma value having the input / output relationship shown in FIG. 4 according to the following equation (2), and the highlight portion of the combined pixel data is compressed.

IMG_HDR_Gamma = 255 × IMG_HDR_Linear ^0.3 (2)

Although the highlight portion is compressed by gamma correction here, the present invention is not limited to this, and correction using an LUT can also be performed. By using the LUT, the highlight portion can be more reliably compressed.

Note that the combined pixel data is referred to as combined pixel data regardless of whether the highlight portion is compressed or not. In addition, since the above equation (1) is an equation based on the premise that the highlight portion is compressed by gamma correction, even if the gamma correction is not performed, the combined pixel value calculated by the above equation (1) is used. On the other hand, it is necessary to multiply “255”.
When the combined pixel data is generated for each pixel in this way, the generated combined pixel data is output to the restoration information adding unit 17.

At this time, a composite image as shown in FIG. 7D is obtained from the post-combination pixel data.
On the other hand, the restoration target image selection unit 16 counts the number of pixels corresponding to each luminance value (luminance level) for each image based on the pixel data group corresponding to each exposure time, and the relationship between the luminance level and the number of pixels. Is generated.

Here, it is assumed that the first to third histograms shown in FIGS. 5B to 5D are generated for the images of the exposure times T1 to T3.
The information of the generated first to third histograms is output to the comparison unit 16b.
When the first to third histogram information is input from the histogram generation unit 16a, the comparison unit 16b compares the first to third histograms with the ideal histogram shown in FIG.

As shown in FIG. 5A, the ideal histogram has a luminance value of 127 with luminance values “0” and “255” set to “0” with respect to a luminance level range of “0 to 255”. An isosceles triangle is formed by connecting these “0”, “127”, and “255” as vertices with the maximum number of pixels.
On the other hand, as shown in FIG. 5B, the first histogram corresponding to the exposure time T1 has a large number of pixels in a relatively low luminance level range (a luminance value of 85 or less). As shown in FIG. 5C, the second histogram corresponding to the exposure time T2 has a large number of pixels in the luminance level in the intermediate range (range of luminance values 86 to 169). As shown in FIG. 5D, the third histogram corresponding to the exposure time T3 has a large number of pixels in a relatively high luminance level range (a luminance value of 170 or more).

As described above, since the second histogram corresponding to the exposure time T2 having the largest number of pixels in the intermediate range can be determined to have the most similar shape to the ideal histogram, the comparison unit 16b uses the information on the exposure time T2 as the pixel. It outputs to the data selection part 16c.
When the information about the exposure time T2 is input, the pixel data selection unit 16c outputs a read command for pixel data to the second memory 13 corresponding to the exposure time T2.

When the second memory 13 receives the read command from the pixel data selection unit 16c, the memory controller sequentially reads out the second pixel data from the head address of the frame memory, and the read second pixel data is stored in the pixel data selection unit 16c. Forward to.
The pixel data selection unit 16c outputs the second pixel data transferred from the second memory 13 to the restoration information adding unit 17 as restoration target pixel data.

When the restoration target pixel data (here, the second pixel data) is input from the restoration target image selection unit 16 and the post-synthesis pixel data is input from the pixel data synthesis unit 15, the restoration information adding unit 17 first restores. In the information generation unit 17a, data of upper NC bits of the restoration target pixel data is extracted.
Here, assuming that the pixel data is monochrome data, the post-combination pixel data is P “10101010 (binary number)” and the restoration target pixel data is S “10101010 (binary number)” as shown in FIG.

Then, as shown in FIG. 6, the upper 3 bits “101” of the restoration target pixel data S “10101010” are extracted.
Specifically, the upper 3 bits of the restoration target pixel data S are shifted to the right (lower side) by 5 bits, and data “00000101” in which all the upper 5 bits after the shift are set to “0” is generated.

Then, the generated “00000101” is output to the restoration information embedding unit 17b as restoration information.
When the restoration information “00000101” is input from the restoration information generation unit 17 a, the restoration information embedding unit 17 b extracts the upper 5 bits “10101” of the post-combination pixel data P “10101010”.

Specifically, the internal logic circuit calculates the logical product of the combined pixel data P “10101010” and “11111000” to obtain “10101000” with the lower 3 bits set to “0”.
When the upper 5 bits of the combined pixel data P are extracted, the restoration information embedding unit 17b calculates the logical sum of the restoration information “00000101” and “10101000” by the internal logic circuit. As shown, the upper 3 bits “101” of the restoration target pixel data S are embedded in the lower 3 bits of the post-combination pixel data P. Thereby, post-combination pixel data “10101101” to which restoration information is added to the post-combination pixel data is generated.

  When the first to third pixel data is color data, the same processing as the monochrome data is performed on the pixel data corresponding to the color signal of each color. Specifically, when the first to third pixel data include, for example, three types of pixel data corresponding to the three primary colors R, G, and B, as in the case of the monochrome data, R, G, The upper 3 bits of the pixel data of each color B are extracted, and the extracted 3-bit data is embedded in the lower 3 bits of the pixel data of each color in the combined pixel data with the same colors.

By performing the above-described series of processing for all the pixels corresponding to the imaging area of the subject, image data (restoration information-added HDR image data) with a wide dynamic range and with restoration information is generated. The generated restoration information-added HDR image data is output to the display unit 19 and also output to the image storage unit 18.
The image storage unit 18 stores the restoration information addition HDR image data input from the restoration information addition unit 17 in a storage medium in a predetermined file format.

  The display unit 19 displays a wide dynamic range image of the subject in real time based on the restoration information addition HDR image data input from the restoration information addition unit 17. On the other hand, when the video is not displayed in real time, the display unit 19 reads the restoration information-added HDR image data stored by the image storage unit 18 from the storage medium in response to an instruction from the user, and adds the read restoration information to the storage medium. Based on the HDR image data, an image of a wide dynamic range of the subject is displayed as shown in FIG.

In the post-combination pixel data to which the restoration information is added, some or all of the lower 3 bits of data may be different from the post-combination pixel data before the addition, but the upper 5 bits are the same as the data before the addition. Therefore, the image quality after the application is almost the same as the image quality before the application.
On the other hand, in a wide dynamic range image of a subject, for example, when important information cannot be identified, and when it is desired to refer to an image before synthesis, the pixel data restoration unit 20 performs an image to be restored (before synthesis). Can be restored.

When restoring an image to be restored, a restoration instruction including information of restoration information-added HDR image data to be restored is given to the pixel data restoration unit 20.
When the restoration instruction is input, the pixel data restoration unit 20 reads the restoration information-added HDR image data to be restored from the storage medium via the image storage unit 18.
Then, the restoration target pixel data is restored for each restoration information addition pixel data (for each pixel) based on the read restoration information addition HDR image data.

  In the present embodiment, since the lower 3 bits in the restoration information-added pixel data (8 bits) are restoration information, the lower 3 bits are shifted to the left (upper side) by 5 bits, and the lower 5 bits after the shift are changed to “ As “0” (may be left as it is), the restoration target pixel data is restored. For example, when the restoration information addition pixel data is “10101101 (binary number)”, the lower 3 bits “101” are shifted to the left by 5 bits, and the lower 5 bits after the shift are all set to “0”. Is restored pixel data to be restored (hereinafter referred to as restoration data).

As shown in FIG. 7F, an image corresponding to the exposure time T2, which is an image to be restored, can be obtained from the data restored in this way.
Note that the restored data may differ in part or all of the lower 5 bits from the original data “10101010”, but at least the upper 3 bits have the same value as the original data. The image of the configured image data has an image quality almost the same as that of the original image data.

As described above, the imaging system 100 according to the present embodiment synthesizes pixel data obtained by exposing each pixel of the CCD image sensor with the three types of exposure times T1 to T3, and synthesizes the image corresponding to a wide dynamic range image. Post-pixel data can be generated.
Also, luminance histograms (first to third histograms) are generated for each exposure time from the first to third pixel data respectively corresponding to the exposure times T1 to T3, and the generated first to third histograms are generated. From the comparison result, the pixel data of the exposure time corresponding to the histogram closest to the ideal histogram can be selected as the target pixel data.

Further, restoration information is generated from the data of the upper NC bits of the selected restoration target pixel data, and the restoration information is given to the post-combination pixel data by replacing the lower NC bits of the post-combination pixel data with the restoration information. Can do.
Further, it is possible to restore the pixel data to be restored from the pixel data to which the restoration information is added.

As described above, it is possible to select pixel data having the best exposure time among the exposure times T1 to T3 as restoration target pixel data and to add restoration information for restoring the restoration target pixel data to the composite pixel data. it can.
In addition, since the data obtained by compressing the restoration target pixel data is given as restoration information by replacing it with lower bits of the synthesized pixel data, restoration information is given without changing the format of the synthesized pixel data. be able to. As a result, the data to be added as restoration information can be reduced, and can be used in a general-purpose application or the like without converting the format of restoration-information-added combined pixel data.

Further, since the restoration target pixel data can be restored from the restoration information-added synthetic pixel data, the restoration target pixel data is restored and restored when the image of an important subject is blurred in the synthesized image. It is possible to refer to an image composed of data and perform identification processing using image data of the image.
In the first embodiment, the CCD image sensor of the imaging unit 10 corresponds to the photoelectric conversion unit described in the first or nineteenth aspect, and the process of reading the pixel signal from each pixel in the imaging unit 10 is described in the first aspect. Corresponding to the pixel signal reading means described in the form 19 or the pixel signal reading step described in the form 19, the pixel data combining unit 15 corresponds to the pixel data combining means described in the form 1 or 20, and the restoration target image selecting unit 16 is configured in the form Corresponding to the restored image selecting means described in 12 or 13, the restored information adding unit 17 is the restored information adding means described in any one of the forms 1, 4, 5, 12, and 20, or the restored information adding means described in the forms 19, 21, and 22. Corresponding to any one of the restoration information adding steps described in the above, the pixel data restoring unit 20 corresponds to the pixel data restoring means described in the form 15.

In the first embodiment, the imaging unit 10, SW 11, first to third memories 12 to 14, pixel data composition unit 15, restoration target image selection unit 16, and restoration information adding unit 17 in the imaging system 100 are described. The block composed of 1 corresponds to the imaging device according to any one of modes 1, 2, 3, 4, 5, 12, and 13, and includes a pixel data synthesis unit 15, a restoration target image selection unit 16, and restoration information addition. The block configured by the unit 17 corresponds to the image processing device described in mode 20.
[Variation 1 of the first embodiment]
Next, a first modification of the first embodiment of the imaging apparatus, imaging system, imaging method, image processing apparatus, image processing program, and image processing method according to the present invention will be described with reference to FIGS.

In the first embodiment, the restoration target image selection unit 16 selects the pixel data to be restored based on the histogram of the image corresponding to each exposure time. The difference is that pixel data to be restored is selected based on the frequency component corresponding to the exposure time.
Furthermore, in the first embodiment, when the restoration target pixel data is color image data, the restoration information adding unit 17 extracts the upper 3 bits of the pixel data corresponding to each color and extracts the upper 3 While the bit data is embedded in the lower 3 bits of post-combination pixel data corresponding to each color, restoration information is given to the post-combination pixel data. Among the corresponding pixel data, the pixel data information corresponding to any one color is divided into the number of colors of a plurality of colors, and the divided data is used as restoration information in the lower bits of the pixel data of each color of the combined pixel data. The difference is that restoration information is added to post-combination pixel data by embedding.

Further, the restoration method of the pixel data to be restored in the pixel data restoration unit 20 is different from the first embodiment due to the difference.
Specifically, in the imaging system 100 of the first modification, the restoration target image selection unit 16 is changed to the restoration target image selection unit 16 ′, and the processing contents of the restoration information adding unit 17 and the pixel data restoration unit 20 are further changed. The other components are the same as those of the imaging system 100 of the first embodiment.

Hereinafter, the same components as those in the first embodiment are denoted by the same reference numerals, description thereof will be omitted as appropriate, and portions different from those in the first embodiment will be described in detail.
First, a detailed configuration of the restoration target image selection unit 16 ′ will be described with reference to FIG.
Here, FIG. 8 is a block diagram illustrating a detailed configuration of the restoration target image selection unit 16 ′.
As illustrated in FIG. 8, the restoration target image selection unit 16 ′ includes a frequency component extraction unit 16d, a comparison unit 16e, and a pixel data selection unit 16c.

The frequency components of the pixel signals of the exposure times T1 to T3 are extracted (calculated) by edge extraction using a known edge extraction filter.
The frequency component extraction unit 16d uses a known Laplacian filter (high-pass filter) to perform filter processing on the pixel data of the edge extraction target pixel and its surrounding pixels, and obtains an edge component (high frequency component) for the target pixel data. Extract.

The comparison unit 16e is based on the edge components corresponding to the pixel data of the exposure times T1 to T3 extracted by the frequency component extraction unit 16d, and the image having the largest edge component amount among the images corresponding to the exposure times T1 to T3. Is output to the pixel data selection unit 16c as a comparison result.
Next, the restoration information adding unit 17 in the first modification will be described.

In this embodiment, each pixel data corresponds to each color signal of the three primary colors of R, G, and B. Pixel data R (N bits), pixel data G (N bits), pixel data B (N bits) ) Of three types of pixel data.
In the present embodiment, when the restoration target pixel data RsGsBs is input from the restoration target image selection unit 16 ′, the restoration information generation unit 17a in the present first modification first sets Rs, which constitutes the restoration target pixel data RsGsBs. Pixel data Gs is selected from the pixel data Rs, Gs, and Bs corresponding to each color of Gs and Bs, and the pixel data Gs (N bits) is divided by the number of colors (3).

In the present embodiment, when the pixel data Gs (N bits) is data having a number of bits that cannot be equally divided into three, the pixel data Gs is doubled, for example, so that the number of bits can be equally divided into three. Convert. For example, when the pixel data Gs is 8 bits, it is doubled and converted to 9-bit data, and the converted pixel data Gs is divided into three.
The three types of divided data GsA to GsC (N / 3 bits) obtained by dividing into three in this way are output to the restoration information embedding unit 17b as restoration information. For example, when the pixel data Gs is “101010101 (binary number)”, the divided data GsA is “101”, the divided data GsB is “010”, and the divided data GsC is “101” in order from the higher bit side. .

When the division information GsA to GsC is input from the restoration information generation unit 17a, the restoration information embedding unit 17b according to the first modification uses the division data GsA to GsC as pixel data corresponding to each color in the combined pixel data RpGpBp. By embedding in the lower “N / 3” bits of Rp, Gp, and Bp, restoration information is added to the combined pixel data.
Specifically, the divided data GsA is embedded in the lower “N / 3” bits of the pixel data Rp, the divided data GsB is embedded in the lower “N / 3” bits of the pixel data Gp, and the lower “N / 3” of the pixel data Bp. The divided data GsC is embedded in the bit.

In this way, the pixel data Rp, Gp, and Bp in which the divided data GsA to GsC are embedded are output to the image storage unit 18 and the display unit 19 as restoration information-added combined pixel data, respectively.
Next, the pixel data restoration unit 20 in Modification 1 will be described.
When the pixel data restoration unit 20 in the first modification obtains the post-combination pixel data after restoration information addition from the image storage unit 18, the pixel data restoration unit 20 includes the pixel data Rp, Gp, Bp of each color included in the post-combination pixel data after restoration information addition. Based on the restored information, the restoration target pixel data Rs, Gs, and Bs are restored.

  Specifically, since the restoration information GsA, GsB, and GsC are respectively embedded in the lower “N / 3” bits of the restoration information-added combined pixel data Rp, Gp, and Bp, first, from Rp, Gp, and Bp, Each bit string of the restoration information GsA, GsB, and GsC is extracted, and data obtained by shifting the extracted bit string to a corresponding bit position of the pixel data Gs to be restored is generated. The pixel data Gs to be restored can be obtained by calculating the logical sum of these shifted data.

When the pixel data Gs is restored, pixel data Rs (“Rs = Rp × Gs / Gp”) to be restored is obtained by multiplying the pixel data Rp and Bp by the ratio between the pixel data Gs and the pixel data Gp, respectively. Pixel data Bs (“Bs = Bp × Gs / Gp”) can be obtained.
Next, the operation of the present embodiment will be described with reference to FIGS.
Here, FIG. 9A is a diagram illustrating an example of a high-pass filter, and FIG. 9B is a diagram illustrating an example of pixel values of a pixel to be processed and its surrounding eight pixels. FIG. 10 is a diagram showing an example of an image at each exposure time after the filter processing by the high-pass filter in FIG. 9A and an image after synthesis.

The operation up to the pixel data composition unit 15 is the same as that in the first embodiment, and a description thereof will be omitted.
Hereinafter, the operation of the restoration target image selection unit 16 ′ will be described.
In the frequency component extraction unit 16d, the restoration target image selection unit 16 uses a Laplacian filter (high-pass filter) for each pixel data for each image based on pixel data groups corresponding to the exposure times T1 to T3. Processing is performed to extract the edge component amount of the image corresponding to each exposure time.

Here, first, the pixel data (x, y) to be processed and the pixel data of the eight pixels around the pixel of the pixel data are filtered using a high-pass filter as shown in FIG. The edge component (high frequency component) of each pixel is extracted.
For example, when the pixel values of the processing target pixel and the surrounding eight pixels are as shown in FIG. 9B, the edge component of the processing target pixel is obtained using the high-pass filter of FIG. “60 × 1 + 60 × 1 + 60 × 1 + 60 × 1 + 60 × −9 + 60 × 1 + 60 × 1 + 60 × 1 + 60 × 1 = −60” is extracted.

  Such edge component extraction processing is performed on all pixels of the image corresponding to the exposure times T1 to T3, respectively, and the sum of absolute values of the edge components of each pixel extracted for each image is used as the edge component of the image. Calculate as a quantity. Here, the edge component amounts corresponding to the exposure times T1, T2, and T3 are E1, E2, and E3, respectively. The calculated edge component amounts E1 to E3 are output to the comparison unit 16e.

  When the edge component amounts E1 to E3 are input from the frequency component extraction unit 16d, the comparison unit 16e selects an image having an exposure time to which restoration information is provided as a restoration target based on the E1 to E3. In the present embodiment, it is defined that the sharpness of the image is higher as the amount of edge components is larger, and the values of the edge component amounts E1 to E3 are compared to correspond to the edge component amount having the largest value. An image having an exposure time is selected as an image to be restored, and information on the exposure time of the image is output to the pixel data selection unit 16c.

  Here, it is assumed that the image after the filter processing corresponding to the exposure times T1 to T3 is as shown in FIGS. The white part of each image is an edge component. As can be seen from FIGS. 10A to 10C, among these, the image corresponding to the exposure time T3 contains the most edge components. That is, since it is expected that more objects are reflected in the image corresponding to the exposure time T3, the time information of the exposure time T3 is output to the pixel data selection unit 16c.

When the information about the exposure time T3 is input, the pixel data selection unit 16c outputs a read command for pixel data to the third memory 14 corresponding to the exposure time T3.
When the third memory 14 receives the read command from the pixel data selection unit 16c, the third memory 14 sequentially reads the third pixel data from the start address of the frame memory by the memory controller, and the read third pixel data is stored in the pixel data selection unit 16c. Forward to.

The pixel data selection unit 16c outputs the third pixel data transferred from the third memory 14 to the restoration information adding unit 17 as restoration target pixel data.
When the restoration target pixel data (here, the third pixel data) is input from the restoration target image selection unit 16 and the post-synthesis pixel data is input from the pixel data synthesis unit 15, the restoration information adding unit 17 first restores. In the information generation unit 17a, the pixel data Gs is divided into three of the pixel data Rs, Gs, and Bs included in the third pixel data. Here, assuming that the pixel data Gs is 8 bits, the pixel data Gs is doubled and converted into 9-bit data, and then divided into three. The divided data GsA to GsC obtained by dividing into three are output to the restoration information embedding unit 17b as restoration information.

  The restoration information embedding unit 17b receives the post-combination pixel data from the pixel data synthesis unit 15 and receives the restoration information (GsA to GsC) from the restoration information generation unit 17a. The divided data GsA, GsB, and GsC are embedded in the lower 3 bits of the pixel data Rp, Gp, and Bp, respectively. For example, when the pixel data Rp, Gp, and Bp are all “10101010 (binary number)” and GsA, GsB, and GsC are “101”, “010”, and “101”, the first Similarly to the restoration information embedding unit 17b in the embodiment, GsA is embedded in the lower 3 bits of the pixel data Rp, GsB is embedded in the lower 3 bits of Gp, and GsC is embedded in the lower 3 bits of Bp. , Rp “10101101”, Gp “10101010”, and Bp “10101101” to generate post-combination combined pixel data.

Restoration information-added HDR image data is generated by performing the above-described series of processing on all the pixels corresponding to the imaging region of the subject. The generated restoration information-added HDR image data is output to the image storage unit 18 and the display unit 19, respectively.
On the other hand, the operation of the pixel data restoration unit 20 when restoring the restoration target image from the restoration information-added HDR image data as described above will be described.

When the restoration instruction is input, the pixel data restoration unit 20 reads the restoration information-added HDR image data to be restored from the storage medium via the image storage unit 18.
Then, the restoration target pixel data is restored for each restoration information addition pixel data (for each pixel) based on the read restoration information addition HDR image data.
In the present embodiment, for example, pixel data Rp, Gp, and Bp of each color of post-synthesis information-added combined pixel data are “10101101 (binary number)”, “10101010 (binary number)”, and “10101101 (binary number), respectively. ) ”, These lower 3 bits are used as restoration information, so that the restoration information GsA of Gs given to Rp is obtained by calculating a logical product of Rp“ 10101101 ”and“ 00000111 ”. Next, the lower 3 bits “101” of “00000101” are shifted to the left by 5 bits to obtain “10100000”.

In addition, Gs restoration information GsB given to Gp first calculates the logical product of Gp “10101010” and “00000111” to extract “00000010”, and then extracts the lower 3 bits of “00000010”. By shifting 2 bits to the left, “00001000” can be obtained.
Also, the Gs restoration information GsC assigned to Bp first calculates the logical product of Bp “10101101” and “00000111” to extract “00000101”, and then extracts the lower 3 bits of “00000101”. It is possible to obtain “00000010” by shifting 1 bit to the right. Here, the reason why the least significant bit is deleted by shifting 1 bit to the right is that the pixel data Gs is doubled by a multiplier or the like when the divided data is created and treated as 9-bit data.

Then, the pixel data Gs “10101010” to be restored can be obtained by calculating the logical sum of GsA, GsB, and GsC thus obtained.
When the pixel data Gs is obtained, the pixel data Rp and the pixel data Bp are multiplied by the ratio “Gs / Gp” between the pixel data Gs and the pixel data Gp, respectively, so that the pixel data Rs (“Rs = Rp × Gs / Gp ”) and pixel data Bs (“ Bs = Bp × Gs / Gp ”).

As described above, the frequency component amounts (edge component amounts E1 to E3) are extracted for each exposure time from the first to third pixel data corresponding to the exposure times T1 to T3, respectively. By comparing the sizes, pixel data having an exposure time corresponding to the edge component amount having the largest value can be selected as target pixel data.
Further, pixel data of any one of the pixel data corresponding to each of a plurality of colors of the selected restoration target pixel data (N bits) is divided by the number of colors C, and the divided data obtained by the division is ( By embedding N / C bits) in the lower “N / C” bits of the pixel data corresponding to each color of the combined pixel data, restoration information can be added to the combined pixel data.

Further, from the pixel data to which the restoration information is added, the pixel data of each color to be restored is obtained by using the pixel data corresponding to one color given as the restoration information and the pixel data corresponding to a plurality of colors of the combined pixel data. Can be restored.
As described above, the pixel data of the exposure time having the highest sharpness among the exposure times T1 to T3 is selected as the restoration target pixel data, and restoration information for restoring the restoration target pixel data is given to the composite pixel data. Can do.

  In addition, pixel data of any one color of pixel data of restoration target pixel data divided by the number of colors is used as restoration information, and is distributed and embedded in the lower bits of the pixel data of each color of the synthesized pixel data. Therefore, restoration information can be given without changing the format of post-combination pixel data. As a result, the data to be added as restoration information can be reduced, and the post-synthesis information-added combined pixel data can be used as it is in a general-purpose application.

Further, since the restoration target pixel data can be restored from the restoration information-added synthetic pixel data, the restoration target pixel data is restored and restored when the image of an important subject is blurred in the synthesized image. It is possible to refer to an image composed of data and perform identification processing using image data of the image.
In the first modification, the restoration target image selection unit 16 corresponds to the restoration image selection unit described in the form 12 or 14, and the restoration information addition unit 17 is described in any one of the forms 1, 6, 12, and 20. Corresponding to the restoration information adding means described in any one of Embodiments 19, 21, and 22, and the pixel data restoring unit 20 corresponds to the pixel data restoring means described in Embodiment 15 or 16.

In the first embodiment, the imaging unit 10, SW 11, first to third memories 12 to 14, pixel data composition unit 15, restoration target image selection unit 16, and restoration information adding unit 17 in the imaging system 100 are described. The block composed of 1 corresponds to the imaging device according to any one of Embodiments 1, 2, 3, 6, 12, and 14, and includes a pixel data synthesis unit 15, a restoration target image selection unit 16, and a restoration information addition unit 17. The block constituted by the image processing apparatus corresponds to the image processing apparatus according to mode 20.
[Modification 2 of the first embodiment]
Next, a second modification of the first embodiment of the imaging apparatus, imaging system, imaging method, image processing apparatus, image processing program, and image processing method according to the present invention will be described.

In the second modification, pixel data corresponding to each color of R, G, and B is divided into luminance data and color difference data, and further, the data obtained by dividing the luminance data into three is used as restoration information, and the combined pixel The point that restoration information is given to post-combination pixel data by embedding in the lower bits of pixel data of each color of data is different from the first embodiment and the first modification.
Further, in the pixel data restoration unit 20, the restoration method of the pixel data to be restored is different from that in the first embodiment and the first modification with the difference.

Specifically, the imaging system 100 of the first modification is the same as the imaging system 100 of the first embodiment, except that the processing contents of the restoration information adding unit 17 and the pixel data restoration unit 20 are different.
Hereinafter, the same components as those in the first embodiment are denoted by the same reference numerals, description thereof will be omitted as appropriate, and portions different from those in the first embodiment will be described in detail.

First, the restoration information adding unit 17 in the second modification will be described.
In this embodiment, each pixel data corresponds to each color signal of the three primary colors of R, G, and B. Pixel data R (N bits), pixel data G (N bits), pixel data B (N bits) ) Of three types of pixel data.
When the restoration target pixel data RsGsBs is input from the restoration target image selection unit 16, the restoration information generation unit 17a according to the second modification example has pixel data corresponding to the R, G, and B colors constituting the restoration target pixel data RsGsBs. Rs, Gs, and Bs are divided into luminance data Y and color difference data Cb and Cr, and luminance data Y is divided into three.

Here, since the information of the color difference data is not given as the restoration information, only the luminance data Y is obtained according to the following equation (4).

Y = 0.30 × Rs + 0.59 × Gs + 0.11 × Bs (4)

In this embodiment, when the luminance data Y (N bits) is data having a number of bits that cannot be equally divided into three, the bits that can be equally divided into three, for example, by doubling the coefficient used when obtaining the luminance data. Convert to number data. For example, when each pixel data is 8 bits, as shown in the following equation (5), the coefficient for obtaining the luminance data Y is doubled to obtain 9-bit luminance data, and the luminance data Y is set to 3 To divide.

Y = 0.60 × Rs + 1.17 × Gs + 0.23 × Bs (5)

Three types of divided data YA to YC (N / 3 bits) obtained by dividing the luminance data Y calculated according to the above formula (4) or (5) into three are output to the restoration information embedding unit 17b as restoration information. Is done. The divided data YA, YB, and YC become higher to lower than the luminance data Y.

When the division information YA to YC is input from the restoration information generation unit 17a, the restoration information embedding unit 17b in the second modification example uses the division data YA to YC as pixel data of each color constituting the post-synthesis pixel data RpGpBp. By embedding in the lower “N / 3” bits of Rp, Gp, and Bp, restoration information is added to the combined pixel data.
Specifically, the divided data YA is embedded in the lower “N / 3” bits of the pixel data Rp, the divided data YB is embedded in the lower “N / 3” bits of the pixel data Gp, and the lower “N / 3” of the pixel data Bp. The divided data YC is embedded in the bits.

In this way, the pixel data Rp, Gp, and Bp in which the divided data YA to YC are embedded are output to the image storage unit 18 and the display unit 19 as restoration-information-added combined pixel data, respectively.
Next, the pixel data restoration unit 20 in Modification 2 will be described.
When the pixel data restoration unit 20 in the second modification example acquires the post-combination-information-added combined pixel data from the image storage unit 18, the pixel-data restoration unit 20 includes the pixel data Rp, Gp, Bp of each color included in the post-combination-added-combination pixel data The luminance data Y to be restored is restored based on the restored information. In the present embodiment, the color image to be restored is restored as a monochrome image.

Specifically, the restoration information YA, YB, and YC are embedded in the lower “N / 3” bits of the restoration-information-added combined pixel data Rp, Gp, and Bp, respectively. Luminance data Y can be obtained in the same manner as in Example 1. This luminance data Y becomes the monochrome image data to be restored.
As described above, the pixel data (N bits) corresponding to the R, G, and B colors of the restoration target pixel data is divided into the luminance data Y and the color difference data Cb, Cr, and the luminance data Y is divided into three. By embedding the obtained divided data YA to YC (all N / C bits) in the lower “N / C” bits of the pixel data corresponding to each color of the combined pixel data, restoration information is added to the combined pixel data. Can be granted.

Further, by restoring the luminance data given as the restoration information from the pixel data to which the restoration information is given, it is possible to restore the monochrome pixel data to be restored.
As described above, since the luminance data obtained from the pixel data of each color of the restoration target pixel data is divided into three as restoration information, it is distributed and embedded in the lower bits of the pixel data of each color of the composite pixel data. The luminance information of the image can be given as restoration information without loss, and the post-synthesis information-added combined pixel data can be used as it is in a general-purpose application without format conversion.

Further, since the restoration target pixel data can be restored from the restoration information-added synthetic pixel data, the restoration target pixel data is restored and restored when the image of an important subject is blurred in the synthesized image. It is possible to refer to an image composed of data and perform identification processing using image data of the image.
In the second modification, the restoration information adding unit 17 is the restoration information adding unit according to any one of modes 1, 7, 12, and 20, or the restoration information adding step according to any one of modes 19, 21, and 22. The pixel data restoration unit 20 corresponds to the pixel data restoration unit described in the form 15 or 17.

  In the second modification, the imaging system 100 includes the imaging unit 10, SW 11, first to third memories 12 to 14, pixel data synthesis unit 15, restoration target image selection unit 16, and restoration information addition unit 17. The block corresponding to the imaging device according to any one of Embodiments 1, 2, 3, 7, 12, and 14 includes a pixel data synthesis unit 15, a restoration target image selection unit 16, and a restoration information addition unit 17. The block corresponding to the image processing apparatus according to the twentieth aspect.

[Second Embodiment]
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. 11 to 13 are diagrams showing a second embodiment of an imaging apparatus, an imaging system, an imaging method, an image processing apparatus, an image processing program, and an image processing method according to the present invention.

  In the first embodiment, the first modification, and the second modification, information obtained by compressing or dividing the information of the restoration target pixel data is given to the post-combination pixel data as restoration information. The present embodiment has a difference information table in which difference information is registered in association with a detection bit string made up of a predetermined bit string, and restores the detection bit string corresponding to the difference information for restoring the restoration target pixel data. The difference is that the information is added to the combined pixel data.

  Specifically, in the imaging system 200 of the present embodiment, in the imaging system 100 of the first embodiment, the restoration information adding unit 17 is replaced with the restoration information adding unit 21 corresponding to the operation process of the present embodiment. The difference information table storage unit 22 is added, and the processing contents of the pixel data restoration unit 20 differ according to the changes and additions. Otherwise, the imaging system 100 of the first embodiment. It will be the same.

Hereinafter, the same components as those in the first embodiment are denoted by the same reference numerals, description thereof will be omitted as appropriate, and portions different from those in the first embodiment will be described in detail.
First, based on FIG. 11, the structure of the imaging system 200 of this Embodiment is demonstrated.
Here, FIG. 11 is a block diagram illustrating a configuration of the imaging system 200.
As shown in FIG. 11, the imaging system 200 includes an imaging unit 10, a switch (SW) 11, first to third memories 12 to 14, a pixel data synthesis unit 15, a restoration target image selection unit 16, Based on the difference information table, the restoration information adding unit 21 that adds the restoration information of the pixel data to be restored to the combined pixel data, the difference information table storage unit 22 that stores the difference information table, the image storage unit 18, and the display unit 19 and a pixel data restoring unit 20 that restores the pixel data to be restored based on the difference information table.

As shown in FIG. 12, the restoration information adding unit 21 includes a restoration information generating unit 21a and a restoration information embedding unit 21b.
Here, FIG. 12 is a block diagram illustrating a detailed configuration of the restoration information adding unit 21.
The restoration information generation unit 21 a includes post-combination pixel data input from the pixel data synthesis unit 15, restoration target pixel data input from the restoration target image selection unit 16, and difference information stored in the difference information table storage unit 22. Based on the table, restoration information for restoring the restoration target pixel data is generated.

In the present embodiment, specifically, if the pixel data is monochromatic data, first, difference information is generated based on the restoration target pixel data S (N bits) and the combined pixel data P (N bits). To do.
Specifically, when the absolute difference amount between the restoration target pixel data S and the combined pixel data P is used as the difference information, “absolute difference amount α = S−P” is obtained by calculation. When the relative difference between the restoration target pixel data S and the combined pixel data P is used as difference information, “relative difference amount β = S / P” is obtained by calculation.

If the pixel data is data corresponding to each color such as R, G, B, etc., the pixel data Rs, Gs, Bs (N bits) corresponding to each color of the restoration target pixel data and each color of the combined pixel data Difference information for each color from corresponding pixel data Rp, Gp, Bp (N bits) is obtained.
Furthermore, the restoration information generation unit 21a compares each difference information in the difference information table with the difference information obtained by the above calculation, and detects a bit string (NC bit (NC bit (NC <N)). Then, the acquired detection bit string is output to the restoration information embedding unit 21b as restoration information.

The restoration information embedding unit 21 b embeds the restoration information input from the restoration information generation unit 21 a in the combined pixel data input from the pixel data synthesis unit 15.
Specifically, similarly to the restoration information embedding unit 17b of the first embodiment, the restoration information embedding unit 21b converts lower NC bit data of post-combination pixel data into NC bit data of restoration information. Replace. As a result, post-combination pixel data with restoration information added with restoration information is generated. The generated post-combination pixel data with restoration information added is output to the image storage unit 18 and the display unit 19, respectively.

Returning to FIG. 11, the difference information table storage unit 22 associates the difference information (discrete absolute difference amount or relative value corresponding to the gradation range of the pixel data) in association with each detection bit string formed of NC bit strings. The difference information table in which the difference amount information) is stored is stored.
The pixel data restoration unit 20 obtains post-combination pixel data after restoration information addition from the image storage unit 18 in response to the restoration instruction, extracts a detection bit string included in the obtained post-combination pixel data after restoration information addition, Difference information corresponding to the detected bit string is acquired from the difference information table stored in the difference information table storage unit 22.

Further, the pixel data restoration unit 20 restores the restoration target pixel data S from the acquired difference information and the combined pixel data.
Specifically, if the difference information is the absolute difference amount α, “S = P + α” is calculated to restore the restoration target pixel data S. If the difference information is the relative difference amount β, the restoration target pixel data S is restored by calculating “S = P × β”.

In general, an image before synthesis is darker than an image after synthesis. Therefore, the difference information table may have a negative coefficient as much as possible (a value for reducing luminance at the time of restoration). desirable.
Here, the function of each component of the imaging system 200 is realized by only hardware, only software, or a combination of hardware and software. Therefore, the imaging system 200 is not shown in the case where there is a component that implements a function using software, but a processor for executing the software, a storage medium such as a ROM for storing the software, A storage medium such as a RAM used for execution and a bus for exchanging data with each component are provided.

  In addition, each of the above-described components may be configured so as to be entirely included in one device (in this case, an imaging device = an imaging system), or the imaging unit 10, SW11, first to third memories 12 to 14, the pixel data composition unit 15, the restoration target image selection unit 16, the restoration information addition unit 21, and the difference information table storage unit 22 are configured in one device (in this case, corresponding to the imaging device), and an image storage unit 18, the display unit 19 and the pixel data restoration unit 20 may be configured by one or a plurality of devices. In the latter case, the imaging device, the image storage unit 18, the display unit 19, and the pixel data restoration unit 20 are connected by wire or wirelessly.

Next, the operation of the present embodiment will be described with reference to FIG.
Here, FIG. 13 is a diagram illustrating an example of the difference information table.
Since the operation up to the restoration target image selection unit 16 is the same as that in the first embodiment, description thereof is omitted.
Hereinafter, the operation of the restoration information adding unit 21 will be described.

Here, it is assumed that the pixel data is monochrome data.
When the restoration target pixel data (here, referred to as second pixel data) is input from the restoration target image selection unit 16 and the post-synthesis pixel data is input from the pixel data synthesis unit 15, In the restoration information generation unit 21a, difference information is generated based on the restoration target pixel data and the combined pixel data.

For example, when the pixel value of the post-combination pixel data P is “140” and the pixel value of the restoration target pixel data S is “85”, the absolute difference amount α is “α = 85−140 = −55”, The difference amount β is “β = 85 / 140≈61 [%]”.
Next, the restoration information generation unit 21a compares the generated difference information with the difference information in the difference information table, and acquires a detection bit string corresponding to the difference information having the closest value as restoration information.

Here, it is assumed that the difference information table has an absolute difference amount or a relative difference amount as shown in FIG. 13 with respect to a 3-bit detection bit string.
When α is “−55”, since “−64” is the closest value as shown in FIG. 13, the restoration information generation unit 21 a stores the difference information stored in the difference information table storage unit 22. From the table, the detection bit string “100” corresponding to “−64” is acquired as restoration information.

Further, when β is “61 [%]”, as shown in FIG. 13, “50 [%]” is the closest value, so that the restoration information generation unit 21 a performs the difference information table storage unit 22. “101” corresponding to “50 [%]” is acquired as restoration information from the difference information table stored in the table.
Accordingly, when the difference information is the absolute difference amount α, “100” is output to the restoration information embedding unit 21b as “100” as the restoration information when the difference information is the relative difference amount β.

When the restoration information is input from the restoration information generation unit 21a, the restoration information embedding unit 21b converts the lower 3 bits of the combined pixel data “10001100 (binary number)” into the inputted restoration information (3 bits). Replace. Since this specific method is the same as that of the first embodiment, description thereof is omitted.
When the first to third pixel data is color data, the same processing as the monochrome data is performed on the pixel data corresponding to the color signal of each color. Specifically, when the first to third pixel data include, for example, three types of pixel data corresponding to the three primary colors R, G, and B, as in the case of the monochrome data, R, G, Difference information between the pixel data of each B color and the pixel data of each R, G, B color in the combined pixel data is generated for each color, and a detection bit string (for example, 3 bits) corresponding to the generated difference information is set as a difference. Obtained from the information table, the obtained detection bit string is embedded in the lower 3 bits of the pixel data of each color in the post-combination pixel data with the same color.

By performing the above-described series of processing for all the pixels corresponding to the imaging region of the subject, image data (hereinafter referred to as restoration information-added HDR image data) with a wide dynamic range and the restoration information is generated. . The generated restoration information-added HDR image data is output to the display unit 19 and also output to the image storage unit 18.
On the other hand, the operation of the pixel data restoration unit 20 when restoring the restoration target image from the restoration information-added HDR image data as described above will be described.

When the restoration instruction is input, the pixel data restoration unit 20 reads the restoration information-added HDR image data to be restored from the storage medium via the image storage unit 18.
Then, the restoration target pixel data is restored for each restoration information addition pixel data (for each pixel) based on the read restoration information addition HDR image data.
Since the detection bits corresponding to the difference information table are embedded in the lower 3 bits of the restoration information-added pixel data, the pixel data restoration unit 20 first extracts the detection bits.

Here, it is assumed that the detected bit string “100” corresponding to the absolute difference amount α is extracted.
Next, using the extracted detection bit string “100”, a difference information table having an absolute difference amount as shown in FIG. 13 is searched. Thereby, as shown in FIG. 13, the absolute difference amount “−64” corresponding to “100” is searched.
When the absolute difference amount α corresponding to the restoration information is retrieved, the restoration target pixel data S “S = P + α is obtained from the retrieved absolute difference amount α“ −64 ”and the combined pixel data P“ 140 ”. = 140−64 = 76 ”is restored.

On the other hand, when the detected bit string “101” corresponding to the relative difference amount β is extracted, the difference information table having the absolute difference amount as shown in FIG. 13 is searched using the extracted detected bit string “101”. . Thereby, as shown in FIG. 13, the relative difference amount “50 [%]” corresponding to “101” is searched.
When the relative difference amount β corresponding to the restoration information is retrieved, the restoration target pixel data S “S” is obtained from the retrieved relative difference amount β “50 [%]” and the combined pixel data P “140”. = P × β = 140 × 0.5 = 70 ”is restored.

As described above, the difference information between the pixel data of the restoration target pixel data and the combined pixel data is generated, and the detection bit string corresponding to the difference information stored in the difference information table having the value closest to the generated difference information is obtained. By embedding the restoration information in the lower bits of the combined pixel data, the restoration information of the pixel data to be restored can be added to the combined pixel data.
Thereby, depending on the setting contents of the difference information table, restoration information that can restore the restoration target image with high accuracy can be given, and the table capacity can be reduced rather than using the pixel data itself by configuring the table with the difference information. be able to. Further, since the restoration information is embedded in the lower bits of the post-combination pixel data, the post-synthesis information-added post-combination pixel data can be used as it is in a general-purpose application without format conversion.

Further, using the restoration information (search bit string) as a search key, the difference information table is detected, the difference information corresponding to the search key is obtained, and the pixel data to be restored is obtained from the obtained difference information and the combined pixel data. Can be restored.
When the image of an important subject is blurred in the composite image, the restoration target pixel data is restored, an image composed of the restoration data is referred to, and an identification process is performed using the image data of the image. Can go.

In the second embodiment, the restoration information adding unit 21 is the restoration information adding unit according to any one of the first, eighth, twelfth, and twentyth aspects, or the restoration according to any one of the nineteenth, twenty-first, and twenty-second. Corresponding to the information providing step, the pixel data restoring unit 20 corresponds to the pixel data restoring means described in the form 15 or 17.
In the second embodiment, the imaging unit 10, SW 11, first to third memories 12 to 14, pixel data composition unit 15, restoration target image selection unit 16, and restoration information addition unit 17 in the imaging system 100 are described. The block composed of 1 corresponds to the imaging device according to any one of modes 1, 2, 3, 8, 12, and 14, and includes a pixel data synthesis unit 15, a restoration target image selection unit 16, and a restoration information addition unit 17. The block constituted by the image processing apparatus corresponds to the image processing apparatus according to mode 20.

[Modification of Second Embodiment]
Next, a modification of the second embodiment of the imaging apparatus, imaging system, imaging method, image processing apparatus, image processing program, and image processing method according to the present invention will be described with reference to FIGS.

In this modification, as preprocessing for generating and assigning restoration information, the brightness of an image to be restored (hereinafter referred to as a restoration target image) is approximated to the brightness of an image after synthesis (hereinafter referred to as a synthesized image). This is different from the second embodiment in that the absolute difference amount α and the relative difference amount β constituting the difference information table are made finer by performing the process.
Specifically, the imaging system 200 ′ according to the present modification adds an approximation unit 23 that performs processing for approximating the brightness to the imaging system 200 according to the second embodiment, and further includes a difference information table storage unit. The difference information table stored in 22 corresponds to the approximate contents of the approximating unit 23.

Hereinafter, the same components as those in the first and second embodiments are denoted by the same reference numerals, description thereof will be omitted as appropriate, and portions different from those in the first and second embodiments will be described in detail.
First, the configuration of the imaging system 200 ′ of the present embodiment will be described based on FIG.
Here, FIG. 14 is a block diagram showing a configuration of the imaging system 200 ′.
As illustrated in FIG. 14, the imaging system 200 ′ includes an imaging unit 10, a switch (SW) 11, first to third memories 12 to 14, a pixel data synthesis unit 15, and a restoration target image selection unit 16. The restoration information adding unit 21, the difference information table storage unit 22, the approximation unit 23 for approximating the brightness of the restoration target image to the brightness of the composite image, the image storage unit 18, the display unit 19, and the pixel data restoration. And the unit 20.

The approximating unit 23 adjusts the input / output characteristics of the image by correcting the restoration target pixel data input from the restoration target image selection unit 16, and approximates the brightness of the restoration target image to the brightness of the composite image.
In the present embodiment, as the first brightness approximation process, the approximating unit 23 adjusts the gamma value of the restoration target image generated from the restoration target pixel data, thereby adjusting the brightness of the restoration target image. A process for approximating the brightness can be executed.

Further, as the second brightness approximation process, the approximating unit 23 adjusts the gain of the restoration target image generated from the restoration target pixel data to approximate the brightness of the restoration target image to the brightness of the composite image. Processing can be executed.
Here, the function of each component of the imaging system 200 ′ is realized by only hardware, only software, or a combination of hardware and software. Accordingly, the imaging system 200 ′ is not shown in the case where there is a component that implements functions using software, but a processor for executing the software, a storage medium such as a ROM for storing the software, and software A storage medium such as a RAM used for executing the data, a bus for exchanging data with each component, and the like.

  In addition, each of the above-described components may be configured so as to be entirely included in one device (in this case, an imaging device = an imaging system), or the imaging unit 10, SW11, first to third memories 12 to 14, the pixel data composition unit 15, the restoration target image selection unit 16, the restoration information addition unit 21, the difference information table storage unit 22, and the approximation unit 23 are configured in one device (in this case, corresponding to the imaging device). The image storage unit 18, the display unit 19, and the pixel data restoration unit 20 may be configured by one or a plurality of devices. In the latter case, the imaging device, the image storage unit 18, the display unit 19, and the pixel data restoration unit 20 are connected by wire or wirelessly. In the latter case, the difference information table storage unit 22 may be provided separately.

Next, the flow of the first brightness approximation process in the approximation unit 23 will be described with reference to FIG. The first brightness approximation process is a process realized by a processor executing a dedicated program stored in a storage medium such as a ROM.
Here, FIG. 15 is a flowchart showing the first brightness approximation process.
When the first brightness approximation process is started, as shown in FIG. 15, first, the process proceeds to step S100, and the approximation unit 23 substitutes “0.1” for the variable γT corresponding to the provisional gamma, The process proceeds to step S102.

In step S102, the approximation unit 23 substitutes a value larger than the maximum value (maximum gradation value × number of pixels) that the minimum value can take in the variable min corresponding to the minimum value, and the process proceeds to step S104.
In step S104, the approximation unit 23 sets the restoration target pixel value S (before gamma adjustment) to the variable S ′ (x, y) corresponding to the restoration target pixel value after gamma adjustment for the image data of the restoration target image. A value (S ^γT ) obtained by raising x, y) to the power of ^γT is substituted, and the process ^proceeds to step S106.

In step S106, in the approximation unit 23, the value of the variable S ′ (x, y) corresponding to the pixel value to be restored after gamma adjustment for the image data of the restoration target image, and the combined pixel data P (x, y). ) Is calculated, and the calculated value is substituted into a variable dq corresponding to the difference amount, and the process proceeds to step S108.
In step S108, the approximation unit 23 determines whether or not the value of the variable dq corresponding to the difference amount is smaller than the value of the variable min. If it is determined that the value is smaller, the process proceeds to step S110. For (No), the process proceeds to step S114.

When the process proceeds to step S110, the approximation unit 23 substitutes the value of the variable dq for the variable min, and the process proceeds to step S112.
In step S112, the approximation unit 23 substitutes the value of the variable γT corresponding to the temporary gamma for the variable γ corresponding to gamma, and the process proceeds to step S114.
In step S114, the approximating unit 23 determines whether or not the value of the variable γT has become “10”. When it is determined that the value has become “10”, the process proceeds to step S116, and otherwise (No). The process proceeds to step S118.

When the process proceeds to step S116, the approximation unit 23 sets the pixel value S (x, y) to be restored before gamma adjustment to the variable S ′ (x, y) corresponding to the pixel value to be restored after gamma adjustment. A value (S ^γ ) obtained by multiplying the value of γ by the value of the variable γ is substituted, and the process ends.
On the other hand, when the process proceeds to step S118, the approximation unit 23 substitutes a value obtained by adding 0.1 to the current value of γT for the variable γT, and the process proceeds to step S104.

Next, the flow of the second brightness approximation process in the approximation unit 23 will be described with reference to FIG. The second brightness approximation process is a process realized by a processor executing a dedicated program stored in a storage medium such as a ROM.
FIG. 16 is a flowchart showing the second brightness approximation process.
When the second brightness approximation process is started, as shown in FIG. 16, the process first proceeds to step S200, and the approximation unit 23 substitutes “0.1” for the variable GT corresponding to the provisional gain, The process proceeds to step S202.

In step S202, the approximating unit 23 substitutes a value larger than the maximum value (maximum gradation value × number of pixels) that the minimum value can take in the variable min corresponding to the minimum value, and proceeds to step S204.
In step S204, the approximation unit 23 sets the restoration target pixel value S (before gain adjustment) to the variable S ′ (x, y) corresponding to the restoration target pixel value after gamma adjustment for the image data of the restoration target image. A value (S × GT) obtained by multiplying x, y) by the value of the variable GT is substituted, and the process proceeds to step S206.

In step S206, in the approximation unit 23, the value of the variable S ′ (x, y) corresponding to the pixel value to be restored after gain adjustment and the synthesized pixel data P (x, y) for the image data of the restoration target image. ) Is calculated, and the calculated value is substituted into the variable dq corresponding to the difference amount, and the process proceeds to step S208.
In step S208, the approximating unit 23 determines whether or not the value of the variable dq is smaller than the value of the variable min. If it is determined that the value is smaller, the process proceeds to step S210. If not (No), The process proceeds to step S214.

When the process proceeds to step S210, the approximation unit 23 substitutes the value of the variable dq for the variable min, and the process proceeds to step S212.
In step S212, the approximation unit 23 substitutes the value of the variable GT corresponding to the provisional gain for the variable G corresponding to the gain, and the process proceeds to step S214.
In step S214, the approximating unit 23 determines whether or not the value of the variable GT is “10”. When it is determined that the value is “10”, the process proceeds to step S216, and if not (No). The process proceeds to step S218.

When the process proceeds to step S216, the approximation unit 23 sets the restoration target pixel value S (x, y) before gain adjustment to the variable S ′ (x, y) corresponding to the restoration target pixel value after gain adjustment. A value (S × G) obtained by multiplying the value of G by the value of the variable G corresponding to the gain is substituted, and the process ends.
On the other hand, when the process proceeds to step S218, the approximation unit 23 substitutes a value obtained by adding 0.1 to the current GT value for the variable GT, and the process proceeds to step S204.

Next, the operation of the present embodiment will be described based on FIG.
Here, FIG. 17 is a diagram illustrating an example of the difference information table based on the brightness approximation process.
Here, since the operation up to the restoration target image selection unit 16 is the same as that in the first embodiment, the description thereof is omitted.

Hereinafter, the operation of the approximating unit 23 will be described.
First, the actual operation of the first approximation process will be described.
Here, it is assumed that the pixel data has an 8-bit gradation value (0 to 255).
When the synthesized pixel data is input from the pixel data synthesis unit 15 and the restoration target pixel data is input from the restoration target image selection unit 16, the approximation unit 23 receives the initial value “0.1” in the variable γT corresponding to the temporary gamma. ”Is substituted (step S100), and a value larger than“ 255 × the number of pixels of the restoration target image ”is substituted as an initial value for the variable min corresponding to the minimum value (step S102).

Next, a value (255 × (S (x, y) / 255) ^0.1 ) obtained by multiplying the value (luminance value) of each restoration target pixel data S (x, y) of the restoration target image by the value of γT is calculated. The calculated value is substituted into a variable S ′ (x, y) corresponding to restoration target pixel data after gamma adjustment (step S104). This process is performed for all the pixels of the restoration target image.
Once the pixel values after adjustment by the current provisional gamma for all the pixels of the restoration target image are calculated, next, the values of the restoration target pixel data S ′ (x, y) after the gamma adjustment of the restoration target image are combined. A difference value from the value of the rear pixel data P (x, y) is calculated. Specifically, a difference value (S ′ (x, y) −P (x, y)) between both pixel values at the same pixel position is calculated for all pixels. Then, the sum (absolute) of the absolute values of the difference values for all the pixels is calculated, and the calculated value is substituted into the variable dq as the difference amount (step S106).

  Then, it is determined whether or not the value of the variable dq is smaller than the value of the variable min (step S108). At present, since the initial value is assigned to the variable min, the value of the variable dq is always small (the branch of “Yes” in step S108). Accordingly, the value of the variable dq is substituted for the variable min (step S110), and the current value of the variable γT “0.1” is substituted for the variable γ (step S112).

Next, it is determined whether or not the value of the variable γT has become “10” (step S114). At the present time, the value of the variable γT is “0.1” (the branch of “No” in step S114), so the value of the variable γT is increased by 0.1 (step S118). At this time, the value of the variable γ is “0.1”, and the value of the variable γT is “0.2”.
By repeating the above series of processing, the difference between the previous difference amount (value of the variable min) and the difference amount after the current gamma adjustment (value of the variable dq) is repeatedly performed. While the amount is smaller, the value of the variable min is updated to the current difference value, and the value of the variable γ is updated to the value of the variable γT. On the other hand, if the current difference amount is equal to or greater than the previous difference amount, the updating process is not performed and the loop is repeated until the value of the variable γT becomes 10, and when the value of the variable γT becomes 10, A value (S (x, y) ^γ ) obtained by multiplying the value of the restoration target pixel data S (x, y) by the value of the variable γ (S (x, y) ^γ ) is substituted for S ′ (x, y) (step S116). .

As a result, the restoration target pixel data corresponding to all the pixels of the restoration target image is corrected with the adjusted gamma value.
In other words, by repeating the above series of processing, the gamma value is adjusted to a value in which the sum of absolute values of the difference values between the luminance value of the restoration target image and the luminance value of the composite image becomes the minimum value (as small as possible). Then, by correcting the restoration target pixel data with the adjusted gamma value, the brightness of the restoration target image is approximated to the brightness of the synthesized image.

Next, the operation of the second approximation process will be described.
When the synthesized pixel data is input from the pixel data synthesis unit 15 and the restoration target pixel data is input from the restoration target image selection unit 16, the approximation unit 23 receives the initial value “0.1” in the variable GT corresponding to the provisional gain. ”Is substituted (step S200), and a value larger than“ 255 × the number of pixels of the restoration target image ”is substituted as an initial value for the variable min corresponding to the minimum value (step S202).

Next, a value (S (x, y) × 0.1) obtained by multiplying the value (luminance value) of each restoration target pixel data S (x, y) of the restoration target image by the value of the variable GT is calculated. The calculated value is substituted into a variable S ′ (x, y) corresponding to the restoration target pixel data after gain adjustment (step S204). This process is performed for all the pixels of the restoration target image.
Once the pixel values after adjustment with the current provisional gain for all the pixels of the restoration target image are calculated, the value of the restoration target pixel data S ′ (x, y) after gain adjustment of the restoration target image is then combined with A difference value from the value of the rear pixel data P (x, y) is calculated. Then, the sum (absolute) of the absolute values of the difference values for all pixels is calculated, and the calculated value is substituted into the variable dq as the difference amount (step S206).

  Then, it is determined whether or not the value of the variable dq is smaller than the value of the variable min (step S208). At this time, since the initial value is assigned to the variable min, the value of the variable dq is always small (the “Yes” branch in step S208). Therefore, the value of the variable dq is substituted for the variable min (step S210), and the current value “0.1” of the variable GT is substituted for the variable G (step S212).

Next, it is determined whether or not the value of the variable GT has become “10” (step S214). At this time, since the value of the variable GT is “0.1” (the branch of “No” in Step S214), the value of the variable GT is increased by 0.1 (Step S218). At this time, the value of the variable G is “0.1”, and the value of the variable GT is “0.2”.
By repeating the above series of processing, the difference between the previous difference amount (value of the variable min) and the difference amount after the current gain adjustment (value of the variable dq) is repeatedly performed. While the amount is smaller, the value of the variable min is updated to the current difference value, and the value of the variable G is updated to the value of the variable GT. On the other hand, if the current difference amount is equal to or greater than the previous difference amount, the update process is not performed and the loop is repeated until the value of the variable GT becomes 10. When the value of the variable GT becomes 10, A value (S (x, y) × G) obtained by multiplying the value of the restoration target pixel data S (x, y) by the value of the variable G is substituted for S ′ (x, y) (step S4). S216).

Thereby, the restoration target pixel data corresponding to all the pixels of the restoration target image is corrected with the adjusted gain.
That is, by repeating the above series of processing, the gain is adjusted to a value at which the sum of absolute values of the difference values between the luminance value of the restoration target image and the luminance value of the composite image is the minimum value (as small as possible). By correcting the restoration target pixel data with the adjusted gain, the brightness of the restoration target image is approximated to the brightness of the composite image.

  By performing pre-processing for approximating the brightness of the restoration target image to the brightness of the synthesized image by either the first approximation process or the second approximation process, after the synthesis with the restoration target pixel data S ′ The difference value (S′−P) with respect to the pixel data P is a smaller value than the difference value when the preprocessing is not performed. Therefore, as shown in FIG. 17, the value of the difference information table can be made finer (each value is smaller) than the value shown in FIG. 13 in the second embodiment.

For example, if the pixel value of the post-combination pixel data P is “140” and the pixel value of the restoration target pixel data S is “85”, the absolute difference amount α is “α = 85” when the preprocessing is not performed. −140 = −55 ”, and the relative difference amount β is“ β = 85 / 140≈61 [%] ”.
On the other hand, if the adjusted gamma value becomes “0.6” by the pre-processing, the pixel value “85” of the restoration target pixel data S is “S ′ = 255 × (85/255) ^0.6 ≈132. Will be corrected. Therefore, the absolute difference amount α is “α = 132−140 = −8”, and the relative difference amount β is “β = 132 / 140≈94 [%]”.

Therefore, when the difference information is set to the absolute difference amount α, the restoration information generating unit 21a corresponds to “0” closest to the absolute difference amount α “−8” from the difference information table having the value illustrated in FIG. The detection bit string “011” is acquired.
When the difference information is set to the relative difference amount β, the detection corresponding to “94 [%]” having the same value as the relative difference amount β “94 [%]” from the difference information table having the values shown in FIG. The bit string “100” is acquired.

Since the subsequent processing is the same as that of the second embodiment, description thereof is omitted.
As described above, in the imaging system 200 ′ of the present modification, the approximation unit 23 approximates the brightness of the restoration target image generated from the restoration target pixel data to the brightness of the synthesized image generated from the post-synthesis pixel data. be able to.
In addition, it is possible to generate and assign restoration information using the values in the difference information table corresponding to the brightness approximation process.

Thereby, the difference between the restoration target pixel data and the post-combination pixel data can be reduced, so that the value of the difference information table can be set to a fine value and the difference between the values of the table can also be reduced. Restoration information from which difference information that can restore the restoration target image can be obtained can be added.
In the modification of the second embodiment, the approximating unit 23 corresponds to the approximating means described in any one of the ninth, tenth, and eleventh aspects.

  In each of the above-described embodiments and modifications, the imaging unit 10 is configured to have a CCD image sensor. However, the configuration is not limited to this, and a configuration including a CMOS image sensor in which each pixel is configured by a CMOS element. Other configurations may be used. With a configuration having a CMOS image sensor, pixel signals can be read from each pixel in a non-destructive manner, so that pixel signals corresponding to a plurality of types of exposure times can be read during one frame period. Thus, for example, if the electronic shutter function of the rolling shutter method is used, pixel signals having different exposure times can be sequentially read out in units of image lines. Each of the above processes can be performed at a higher speed than when the process is performed after the minutes are acquired.

In the second modification of the first embodiment, the edge amount is extracted as the frequency component of each pixel data. However, the present invention is not limited to this, and the frequency component is extracted using FFT. It is good also as a structure of.
In each of the above embodiments and modifications, the pixel data to be restored is selected based on the luminance histogram and frequency components. However, the present invention is not limited to this, and pixels corresponding to a predetermined exposure time are used. Data may be selected as a restoration target, or pixel data corresponding to all exposure times may be selected as a restoration target.

  In each of the above embodiments and modifications, the exposure times T1, T2, and T3 are set to 15H, 100H, and 500H, respectively. However, the present invention is not limited to this, and the relationship of T1 <T2 <T3 can be maintained. Other exposure times may be used. The exposure time may be arbitrarily set by the user, or may be dynamically changed according to the type of subject and the imaging environment.

1 is a block diagram illustrating a configuration of an imaging system 100 according to the present invention. 3 is a block diagram illustrating a detailed configuration of a restoration target image selection unit 16. FIG. 4 is a block diagram illustrating a detailed configuration of a restoration information adding unit 17. FIG. It is a figure which shows the relationship between the post-combination pixel data which carried out the linear synthesis | combination, and the post-combination pixel data which compressed the highlight part by gamma correction. (A)-(d) is a figure which shows an example of the histogram of an ideal form and the histogram of the image corresponding to exposure time T1-T3. It is a figure which shows an example of the concept of the production | generation process of restoration information, and the embedding process of restoration information. (A)-(f) is a figure which shows the flow of a series of processes including pixel data synthetic | combination processing, restoration object image selection processing, restoration information embedding process, and restoration object image restoration processing. It is a block diagram which shows the detailed structure of restoration object image selection part 16 '. (A) is a figure which shows an example of a high-pass filter, (b) is a figure which shows an example of the pixel value of a process target pixel and its surrounding 8 pixels. It is a figure which shows an example of the image of each exposure time after a filter process with the high pass filter of Fig.9 (a), and the image after a synthesis | combination. 1 is a block diagram illustrating a configuration of an imaging system 200. FIG. 3 is a block diagram illustrating a detailed configuration of a restoration information adding unit 21. FIG. It is a figure which shows an example of a difference information table. It is a block diagram which shows the structure of imaging system 200 '. It is a flowchart which shows a 1st brightness approximation process. It is a flowchart which shows a 2nd brightness approximation process. It is a figure which shows an example of the difference information table on the assumption of the brightness approximation process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100,200,200 '... Imaging system, 10 ... Imaging part, 11 ... SW, 12 ... 1st memory, 13 ... 2nd memory, 14 ... 3rd memory, 15 ... Pixel data composition part, 16, 16' ... Restoration Target image selection unit, 17, 21 ... restoration information adding unit, 18 ... image storage unit, 19 ... display unit, 20 ... pixel data restoration unit, 22 ... difference information table storage unit, 23 ... approximation unit, 16a ... histogram generation unit 16b, 16e ... comparison unit, 16c ... pixel data selection unit, 17a, 21a ... restoration information generation unit, 17b, 21b ... restoration information embedding unit

Claims (15)

An imaging apparatus comprising: a photoelectric conversion unit having a configuration in which a plurality of photoelectric conversion elements that convert received light into electric charges and store them are arranged in a matrix; and a function of controlling an exposure time of the photoelectric conversion elements,
Pixel signal reading means for reading out pixel signals corresponding to the plurality of types of exposure times when exposed at a plurality of types of exposure times from each pixel constituting the photoelectric conversion element;
Pixel data synthesizing means for synthesizing pixel data, which is pixel data corresponding to the plurality of types of exposure times, read by the pixel signal reading means for each pixel;
Restoration information giving means for giving restoration information used when restoring the pre-combination pixel data corresponding to predetermined exposure times in the plurality of types of exposure times to the pixel data after the pixel data synthesis means has been combined; An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the restoration information includes at least one of single-color luminance information, color difference information, and color signal information included in the pixel data before synthesis.   The restoration information includes information on a difference value between luminance information included in the pixel data before combination and luminance information included in the pixel data after combination, color difference information included in the pixel data before combination, and pixel data after combination. At least one of the difference value information with respect to the color difference information possessed by the color signal information and the difference value information between the color signal information possessed by the pixel data before composition and the color signal information possessed by the pixel data after composition. The imaging apparatus according to claim 1.   The restoration information adding means replaces a bit string of lower-order NC bits in the combined pixel data with an NC bit string (NC <N) obtained by compressing an N-bit bit string of the restoration information. The imaging device according to any one of claims 1 to 3.   The restoration information adding means corresponds to each color of the plurality of colors used when restoring the color signal information of the plurality of colors included in the pixel data before the synthesis when the pixel data has color signal information of a plurality of colors. 5. The imaging apparatus according to claim 1, wherein restoration information to be added is added to color signal information of each color included in the combined pixel data. 6.   When the pixel data includes color signal information of a plurality of colors, the restoration information providing unit converts a bit string of color signal information of any one color of the plurality of colors of the pixel data before synthesis into the colors of the plurality of colors. The divided information is divided into several minutes, and the restoration information is added to the combined pixel data by replacing the bit string on the lower side of the combined pixel data with each divided bit string. The imaging device according to any one of claims 1 to 3.   When the pixel data includes R (Red), G (Green), and B (Blue) color signal information, the restoration information adding unit converts the color signal information included in the pixel data before combining into luminance information and Divided into color difference information, the luminance information is divided into three, and the lower-order bit string of the pixel data of each color after the combination is replaced with the divided bit string, whereby the restoration information is added to the pixel data after the combination. The imaging device according to any one of claims 1 to 3, wherein the imaging device is provided. In correspondence with each bit string represented by a bit number L smaller than the number of bits of the pixel data, a difference information table storing information of the difference values of different numerical values, respectively,
The restoration information providing means is a bit string in which L bits on the lower side of the combined pixel data are associated with information on difference values closest to the restoration information of the pixel data before combining in the difference information table The imaging apparatus according to claim 3, wherein the imaging apparatus is replaced with   4. An approximation means for approximating information on the brightness of an image generated from the pre-combination pixel data to information on the brightness of an image generated from the post-combination pixel data. The imaging device according to claim 8.   The approximating means adjusts the gamma value of the image generated from the pixel data before synthesis, thereby obtaining the brightness information of the image as the information on the brightness of the image generated from the pixel data after synthesis. The imaging apparatus according to claim 9, wherein   The approximating means adjusts the gain of the image generated from the pixel data before the synthesis to change the brightness information of the image into the information of the brightness of the image generated from the pixel data after the synthesis. The imaging apparatus according to claim 9, which is approximated. Based on the pixel data corresponding to the plurality of types of exposure time, comprising restored image selection means for selecting pixel data of the exposure time to be restored from among the pixel data corresponding to the plurality of types of exposure time,
12. The restoration information adding unit adds the restoration information of the restoration target pixel data determined by the restoration image selection unit to the combined pixel data. The imaging apparatus of Claim 1.   The restoration image selection means selects pixel data corresponding to the exposure time to be restored based on a histogram of luminance information of an image for each exposure time constituted by the pixel data before synthesis. The imaging device according to claim 12.   The restored image selection unit selects pixel data corresponding to an exposure time to be restored based on frequency information of an image for each exposure time configured from the pixel data before synthesis. 12. The imaging device according to 12. The imaging device according to any one of claims 1 to 14,
An imaging system comprising: pixel data restoring means for restoring the pixel data before synthesis based on restoration information given to the pixel data after synthesis.

Images