JP5720250B2 - Imaging apparatus and image processing control program - Google Patents

Description

  The present invention relates to an imaging apparatus and an image processing control program.

There is known an imaging apparatus that determines a flicker frequency using an electrical signal read from a preset area of an imaging element (for example, Patent Document 1). Further, a solid-state imaging device is known in which the exposure time in each pixel of the solid-state imaging device is read out separately at a timing shifted by half the flicker cycle of illumination, and the obtained two systems of signals are synchronized and added together ( For example, Patent Document 2).
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP 2009-77065 A [Patent Document 2] JP 2003-32551 A

  When not only the light intensity of the illumination light periodically changes but also the color temperature of the illumination light changes with time, there is a problem that not only luminance unevenness occurs but also the color covering changes for each image region.

  In order to solve the above-described problem, an imaging device according to a first aspect of the present invention includes an imaging device having a plurality of pixel lines that are sequentially exposed, and a plurality of images that are generated by being exposed multiple times by the imaging device. And an image processing unit that generates a superimposed image, and the image processing unit generates a superimposed image in which white balance correction is performed for each of a plurality of strip-shaped image regions parallel to a plurality of pixel lines.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 is a system configuration diagram of a camera 100 according to an embodiment. It is a figure which shows typically an example of the process in still image shooting mode. It is a figure explaining the color component of the image data. 3 is an example of a superimposed image 400 that has been subjected to white balance correction. It is a figure which shows typically the example of a process which performs white balance correction | amendment in a partial area. It is a processing flow figure in still picture photography mode. It is a figure which shows typically an example of the process in video recording mode. It is a figure which shows the processing flowchart in a moving image shooting mode. It is a flowchart which performs the process of imaging | photography and a frame production | generation.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a system configuration diagram of a camera 100 according to the present embodiment. The camera 100 includes an optical system 101. The optical system 101 includes a zoom lens, a focus lens, and the like. Subject light enters the optical system 101 along the optical axis, passes through a focal plane shutter 102 disposed in front of the image sensor 103, and forms an image as a subject image on the light receiving surface of the image sensor 103.

  The image sensor 103 is an element that photoelectrically converts an optical image that is a subject image, and for example, a CCD or a CMOS sensor is used. The subject image photoelectrically converted by the image sensor 103 is read out as electric charge, and gain adjustment or the like is performed by the amplifier unit 104. The signal amplified by the amplifier unit 104 is transferred as an analog signal to the A / D converter 105 and converted to a digital signal by the A / D converter 105. Controls such as charge readout of the image sensor 103 and driving of the focal plane shutter 102 are controlled by a clock signal supplied by the timing generator 112 that receives synchronization control of the system controller 114.

  The subject image converted into the digital signal is sequentially processed as image data. The image data converted into a digital signal by the A / D converter 105 is delivered to the image processing unit 107 and processed. Specifically, the image data converted into a digital signal by the A / D converter 105 is temporarily stored in the internal memory 108 under the control of the memory control unit 106. The internal memory 108 is a random access memory that can be read and written at high speed, and for example, a DRAM, an SRAM, or the like is used. The internal memory 108 serves as a buffer memory that waits for the order of image processing when image data is continuously generated at high speed in continuous shooting and moving image shooting. In addition, the image processing unit 107 also serves as a work memory in the image processing and compression processing. Therefore, the internal memory 108 has a sufficient memory capacity corresponding to these roles. The memory control unit 106 controls how much memory capacity is allocated to what work.

  The image processing unit 107 converts the image data into image data of a standardized image format according to the set shooting mode and an instruction from the user. For example, when having a still image shooting mode and a moving image shooting mode as part of the shooting mode of the camera 100 and generating a JPEG file as a still image in the still image shooting mode, the image processing unit 107 performs color conversion processing, After performing image processing such as gamma processing and white balance correction, compression processing is performed by performing adaptive discrete cosine transform or the like. For example, when generating an MPEG file as a moving image in the moving image shooting mode, the image processing unit 107 performs intra-frame coding, inter-frame coding, quantum coding on the generated continuous frame image as a still image. Compression processing is performed by performing encoding and entropy coding.

  The converted image data is stored in the internal memory 108 again. Still image data and moving image data processed by the image processing unit 107 are recorded on the recording medium 122 which is a nonvolatile memory from the internal memory 108 via the recording medium IF 111 under the control of the system control unit 114. The recording medium 122 is configured by a flash memory or the like. The recording medium 122 may be a non-volatile memory that can be attached to and detached from the main body of the camera 100.

  The image processing unit 107 generates display image data in parallel with the image data processed for recording. The generated display image data is displayed on the display unit 110 configured by a liquid crystal panel or the like under the control of the display control unit 109. In addition, the display control unit 109 can display various menu items related to various settings of the camera 100 on the display unit 110 with or without displaying an image.

  The camera 100 is directly or indirectly controlled by the system control unit 114 including each element in the image processing described above. The system control unit 114 includes a system memory that is an electrically erasable / recordable nonvolatile memory. The system memory is configured by EEPROM (registered trademark) or the like. The system memory included in the system control unit 114 records constants, variables, programs, and the like necessary when the camera 100 is operating so that they are not lost even when the camera 100 is not operating. The system control unit 114 expands constants, variables, programs, and the like as appropriate in the internal memory 108 and uses them for controlling the camera 100. The variable necessary for the operation of the camera 100 may include a variable indicating a blinking frequency of illumination light described later.

  The camera 100 includes an operation member 120 that receives an operation from a user. The system control unit 114 detects that the operation member 120 has been operated. The system control unit 114 performs an operation according to the detected operation. For example, when it is detected that the release switch as the operation member 120 is operated, the system control unit 114 performs a series of shooting operations for photoelectrically converting the subject image to generate image data. Further, as described later, as a part of the shooting mode of the camera 100, at least a flicker reduction mode that is a shooting mode for reducing color covering can be included. In the present embodiment, the flicker reduction mode is a shooting mode that reduces luminance unevenness and color covering. The system control unit 114 can accept an operation for instructing ON / OFF of the flicker reduction mode from the user through the operation member 120.

  Further, the system control unit 114 may receive information regarding the blinking frequency of the illumination light from the user through the operation member 120. For example, the system control unit 114 receives an operation for instructing the use area of the camera 100 through the operation member 120 as information on the blinking frequency of the illumination light. The system control unit 114 may determine the blinking frequency of the illumination light based on the frequency of the AC power source used in the instructed usage region. For example, the system control unit 114 sets a value indicating 100 Hz to the variable indicating the blinking frequency of the illumination light when the use area where the AC power supply of 50 Hz is used is instructed. On the other hand, the system control unit 114 sets a value indicating 120 Hz to the variable indicating the blinking frequency of the illumination light when the use region where the 60 Hz AC power supply is used is indicated. The blinking cycle can be calculated by the reciprocal of the blinking frequency. In addition to the operation from the user, the system control unit 114 may detect the current position of the camera 100 based on the GPS information and specify the blinking frequency of the illumination light based on the current position. In this case, the camera 100 may include a GPS module that receives a signal from a GPS satellite.

  The optical system 101 is controlled by the lens control unit 113. For example, the lens control unit 113 drives the zoom lens in accordance with a user instruction to change the angle of view of the subject image. In addition, the lens control unit 113 drives the focus lens based on the AF information, and focuses the subject image on the light receiving surface of the image sensor 103. The lens control unit 113 may perform autofocus control on a subject designated by the user through the operation member 120 or the like. The lens control unit 113 also calculates the exposure value by analyzing the image data processed by the image processing unit 107 or the output of the photometric sensor. The lens control unit 113 outputs a control signal for the diaphragm of the optical system 101 according to the calculated exposure value.

  FIG. 2 schematically shows an example of processing in the still image shooting mode in the camera 100 together with a blinking pattern of illumination light. Here, a blinking pattern 200 generated by blinking of a fluorescent lamp driven by an AC power source is illustrated as a blinking pattern of illumination light. The fluorescent lamp repeatedly blinks at a frequency twice the power frequency. For example, a fluorescent lamp driven by a 50 Hz AC power supply repeats blinking at 100 Hz.

  When the release switch is operated and still image shooting is instructed, the system control unit 114 controls each unit of the camera 100 so as to perform exposure for the exposure time T1 twice. Specifically, the focal plane shutter 102 of the camera 100 exposes twice with the image sensor 103 by running the shutter curtain twice over the front surface of the image sensor 103 over a time length T1. In this example, the focal plane shutter 102 travels in front of the image sensor 103 from time t1 to time t2 and performs the first exposure, and then travels in front of the image sensor 103 from time t3 to time t4. Perform the exposure. Image data 210 read from the image sensor 103 after the first exposure is stored in the internal memory 108. The image data 220 read from the image sensor 103 after the second exposure is also stored in the internal memory 108.

  At the time of still image shooting, the plurality of pixel lines of the image sensor 103 are sequentially exposed by the focal plane shutter 102, and thus luminance unevenness caused by flickering of illumination light appears in the image data 210 and the image data 220. In the image data 210 and the image data 220 in this figure, only the luminance component due to the illumination light is shown in order to easily explain the luminance unevenness. The luminance unevenness occurs in the traveling direction of the focal plane shutter 102. In this example, it is assumed that the traveling direction of the focal plane shutter 102 corresponds to the Y direction on the image, and each pixel line extending in the X direction is exposed substantially simultaneously. Therefore, in the image data 210 and the image data 220, luminance unevenness appears between a plurality of bands extending in the X direction.

  As in this example, when the exposure time T1 is shorter than the blinking cycle and longer than the half cycle of the blinking cycle, at least one of a strip-shaped bright portion and a strip-shaped dark portion appears on the image data 210. When exposure is started from time t1 when the luminance of illumination light is minimized as in this example, the dark band appears at the top in the Y direction of the image data 210. A bright band appears at a position in the Y direction corresponding to the time when the luminance of the illumination light reaches a maximum.

  In this example, the first exposure and the second exposure are performed with a deviation of 2.5 blinking cycles. That is, the second exposure is started from time t3 after 2.5 blinking cycles from time t1. Since the second exposure is started from time t3 when the luminance of the illumination light reaches a maximum, the luminance distribution in the Y direction of the image data 220 is reversed in phase with the luminance distribution in the Y direction of the image data 210. Specifically, the position where the dark band appears in the image data 220 matches the position where the bright part appears in the image data 210. The position where the bright portion band appears in the image data 220 matches the position where the dark portion appears in the image data 210.

  The image processing unit 107 generates superimposed image data 230 by superimposing and averaging the image data 210 and the image data 220. Since the superimposed image data 230 is generated by superimposing the image data whose luminance changes are opposite to each other, the luminance unevenness appearing in the image data 230 is compared with the luminance unevenness appearing in the image data 210 and the image data 220. , Significantly reduced.

  FIG. 3 schematically shows the image data 210 for each color component. A fluorescent lamp for white illumination uses a plurality of fluorescent materials having different emission spectra in order to emit illumination light close to white. When these phosphors have different light emission characteristics, the spectrum of the illumination light also changes with time. For example, according to a phosphor normally used for a fluorescent lamp, the luminance of the B component attenuates faster than the R component and the G component. In addition, the luminance of the R component may attenuate more slowly than the G component. For this reason, the image data 210 and the image data 220 are each covered with color.

  Since the luminance of the B component attenuates relatively quickly under fluorescent lamp illumination, the B component image data 210c is compared with the R component image data 210a and the G component image data 210b, as shown in the figure. A wide dark part appears. Further, a narrower dark portion appears in the R component image data 210a than in the G component image data 210b. Therefore, the dark area on the image data 210 is a band-like area with a strong yellow component corresponding to a strong drop in the brightness of the B component.

  The image data 220 is the same as the image data 210 except that the luminance distribution in the Y direction of each color component is in reverse phase. When such image data 210 and image data 220 are superimposed, color covering may remain on the image data 230. For example, a band having a color closer to yellow appears in the image data 230 by superimposing a substantially white bright part of the image data 210 and a yellow dark part of the image data 220.

  On the other hand, for example, at the time shifted from the time t1 by a quarter of the blinking cycle, if the B component of the illumination light is not significantly reduced compared to the other color components, the band of the corresponding image data 210 is It becomes almost white. Similarly, the band on the image data 220 corresponding to the time shifted by ¼ period from the time t3 is also substantially white. Accordingly, the corresponding band on the superimposed image data 230 is substantially white. In this case, the superimposed image data 230 includes at least a band having a yellowish color and a substantially white band. As described above, a plurality of band-like regions having different color temperatures appear in the image data 230 according to changes in the color temperature of the illumination light.

  FIG. 4 shows an example of a superimposed image 400 that has been subjected to white balance correction. The image processing unit 107 sets a plurality of band areas 410-1 to 6-6 having different positions in the Y direction in the image data obtained by superimposing the image data 210 and the image data 220. Then, the image processing unit 107 performs white balance correction for each of the band regions 410-1 to 410-6. Specifically, the image processing unit 107 determines the white balance gain for each of the band regions 410 based on the color difference signals (RY, BY) in each of the band regions 410, and The color signal is processed based on the gain determined for each. Further, the image processing unit 107 may perform brightness correction for each of the band regions 410-1 to 410-6. Compared with the case where the image processing unit 107 performs white balance correction for each of the band regions 410-1 to 6 and white balance correction is performed on the entire region of the image data 230, the color coverage included in the image data 230. Can be significantly reduced.

  Here, the width in the Y direction of each band region 410 may be a predetermined fixed value. Further, the number of band regions 410 to be divided may be a predetermined fixed value. On the other hand, the image processing unit 107 may variably control at least one of the width of the band region 410 and the number of band regions 410 to be divided.

  For example, the image processing unit 107 may set a band region 410 having a different width for each image region. As described above, the B component strongly falls at the timing when the blinking of the illumination light is turned off. Therefore, the color temperature changes greatly in the area corresponding to the dark part in the image data 230. For this reason, the image processing unit 107 may set the width of the band region 410 to be smaller than the other regions in the region corresponding to the dark portion in the image data 230.

  Here, the image processing unit 107 may set an area where the width of the band area 410 should be reduced based on at least one of the image data 210 and the image data 220. Specifically, the image processing unit 107 specifies a band region that is a dark part based on the luminance distribution in the Y direction of at least one of the image data 210 and the image data 220, and determines the region that is the specified dark part as a band region. The width of 410 may be set as an area to be reduced.

  In addition, the image processing unit 107 may specify an area where the width of the band area 410 should be reduced based on the comparison result between the image data 210 and the image data 220. For example, the image processing unit 107 may set an area where the difference in luminance value between the image data 210 and the image data 220 is larger than a predetermined value as an area where the width of the band area 410 should be reduced. . As illustrated in FIG. 2, when the exposure timing is shifted by a half cycle of the blinking cycle, a region where the difference in luminance value between images is maximum corresponds to a dark region. For this reason, there may be a case where a region where the color temperature greatly changes can be appropriately identified based on a difference in luminance value between image data. If the difference in exposure timing does not match the half cycle of the blinking cycle, or if the illuminance distribution of the illumination light is non-uniform, the area where the brightness value difference is the maximum and the dark area do not match completely There is a case. However, since a region with a large difference in luminance value can be said to be a region that is greatly affected by flickering of illumination light, there is a high probability that the color temperature is greatly changed in the region. Therefore, by reducing the width of the band region 410 in the region, the color temperature can be finely corrected for a region having a high probability that the color temperature changes greatly.

  Further, the image processing unit 107 may perform white balance correction by dividing into more band regions 410 as the shutter curtain speed of the focal plane shutter 102 is lower. As described with reference to FIG. 2, when exposure is performed in the exposure period T1, the image data 210 and the image data 220 include at least one of a bright part and a dark part. When the shutter curtain speed is lowered and the exposure period is made longer than the blinking cycle, at least one of a bright part and a dark part appears. That is, the lower the shutter curtain speed, the greater the rate of change in luminance on the image data. Therefore, the change rate of the color temperature of the illumination light is also increased. If the width of the band area 410 is a fixed value, the change rate of the color temperature in one band area 410 becomes large, and thus the color temperature may not be corrected correctly by white balance correction. The image processing unit 107 divides into more band regions 410 and performs white balance correction as the shutter curtain speed is slower, so that the color temperature of the illumination light can be finely corrected.

  FIG. 5 schematically shows an example of processing in a case where overlay and white balance correction are performed in a specific partial region. Image data 510 and image data 520 are image data including subject images corresponding to image data 210 and image data 520, respectively.

  The influence of the flashing of the fluorescent lamp appears greatly on the subject near the fluorescent lamp. In addition, the subject to which the fluorescent light is more intense than the ambient light is more affected by the flicker of the illumination light. In the illustrated image data 510 and image data 520, the influence of flicker appears greatly in the partial area 515 and the partial area 525. On the other hand, since the subject to which the ambient light is mainly reflected is reflected in the other image areas, the influence of flicker is relatively small.

  The image processing unit 107 superimposes the image of the partial area 525 of the image data 520 on the partial area 515 of the image data 510. At this time, it is preferable that the image processing unit 107 corrects a blur between the image data 510 and the image data 520 and performs an overlay process. For example, the image processing unit 107 calculates a movement vector between the image data 510 and the image data 520 and shifts and overlaps the partial areas 525 based on the calculated movement vector.

  The image processing unit 107 sets one or more band regions 540 for the region corresponding to the partial region 515 in the obtained image data 530. In this example, a plurality of band areas 540-1 and band areas 540-2 are set. Then, the image processing unit 107 performs white balance correction on each band region 540. For this reason, it is possible to reduce color coverage due to flickering of illumination light. In addition, the image processing unit 107 performs white balance correction on an area other than the partial area 515 in the image data 530.

  Since areas other than the partial area 515 in the image data 530 are not greatly affected by flicker, the color coverage is substantially only due to ambient light. For this reason, the color temperature of the ambient light can be properly corrected by performing white balance correction on the entire area other than the partial area 515. Further, according to the image processing of this example, since the image data 520 is not superimposed in an area other than the partial area 515, it is possible to provide a blur-free image in that area.

  The image processing unit 107 may specify the partial area 515 based on the image data 210. For example, the image processing unit 107 may specify a partial region whose luminance changes in a band shape as a partial region where the influence of flicker occurs. For example, the image processing unit 107 may specify a partial region where the luminance changes in a band shape based on the average value of luminance calculated for each band-like region. In addition, the image processing unit 107 is affected by flicker in a partial area in which a luminance change component larger than a predetermined value is detected based on, for example, an average luminance value calculated for each band-shaped area. It may be specified as a partial area. Further, the image processing unit 107 may identify the partial region 515 based on the comparison result between the image data 210 and the image data 220. For example, the image processing unit 107 may specify a partial area where the difference in luminance value between the image data 210 and the image data 220 is larger than a predetermined value as a partial area where flicker influence occurs. .

  FIG. 6 shows a process flow diagram during still image shooting. The processing flow is started when it is detected that the release switch for still image shooting is operated. In this flowchart, the system control unit 114 operates mainly unless otherwise specified.

  When the processing flow is started, a first still image that is the first still image is captured in step S602. Specifically, the first still image is taken by running and exposing the focal plane shutter 102. The memory control unit 106 stores the image data of the first still image in the internal memory 108. The timing generator 112 is set so as to generate a clock for driving the focal plane shutter 102 at the shooting timing at which the second still image should be shot. Specifically, the timing generator 112 is (N + 1/2) times after the blinking period of the illumination light from the start timing of the front curtain of the focal plane shutter 102 when the first still image is captured. It is set to generate the clock at the timing. Here, N is an integer of 0 or more.

  In step S604, it is determined whether or not the flicker reduction mode is set to ON. If the flicker reduction mode is set to ON, the image processing unit 107 determines whether or not the flicker has occurred based on the image data of the first still image (step S606). If the flicker effect has occurred, the image processing unit 107 detects a flicker area that is an area in which the flicker effect has occurred in the first still image (step S608). The area where the flicker influence occurs can be identified by the method described in relation to FIG.

  In step S610, a second still image, which is the second still image, is photographed with a phase shifted by a half cycle of the blinking cycle. At this time, the front curtain of the focal plane shutter 102 starts running in synchronization with the clock supplied from the timing generator 112. The memory control unit 106 stores the image data of the second still image in the internal memory 108. In step S612, the image processing unit 107 compares the image data of the first still image and the image data of the second still image, and detects the blur amount.

  In step S614, the image processing unit 107 adds the image of the flicker area of the second still image to the flicker area of the first still image. At this time, the image processing unit 107 performs blur correction on the image in the flicker area of the second still image based on the blur amount detected in step S612, and converts the blur-corrected image into the first still image. Add to the flicker area.

  In step S616, the image processing unit 107 sets a band-shaped image area in the flicker area, and performs white balance correction for each band-shaped image area. At this time, it is desirable to perform white balance correction for each band-shaped image region so that variation in color temperature for each image region is reduced. For example, it is desirable to perform white balance correction for each band-shaped image region using a region corresponding to the maximum luminance among a plurality of image regions as a reference for white balance. In step S618, the image processing unit 107 performs white balance correction on an image area other than the flicker area. Output still image data that has been subjected to white balance correction by the image processing unit 107 is stored in the internal memory 108. The output still image data stored in the internal memory 108 may be displayed on the display unit 110 under the control of the display control unit 109. When recording still image data for output on the recording medium 122, the process proceeds to step S620 described later.

  If the flicker reduction mode is not set to ON in step S604, the process proceeds to step S630. In step S606, if there is no flicker effect, the process proceeds to step S630. In step S630, the image processing unit 107 performs white balance correction on the entire image data of the first still image. The output still image data that has been subjected to white balance correction by the image processing unit 107 is stored in the internal memory 108. The output still image data stored in the internal memory 108 may be displayed on the display unit 110 under the control of the display control unit 109. When the still image data for output is recorded on the recording medium 122, the process proceeds to step S620.

  In step S620, the image processing unit 107 encodes still image data for output. The encoded still image data for output is recorded on the recording medium 122 through the recording medium IF111. When the recording on the recording medium 122 in step S620 is completed, the series of still image shooting operations is terminated.

  FIG. 7 schematically illustrates an example of processing in the moving image shooting mode in the camera 100 together with a blinking pattern 700 of illumination light.

  The system control unit 114 controls each unit of the camera 100 so as to perform rolling reading of the image sensor 103 as a CMOS sensor when the operation member 120 is operated to instruct moving image shooting. Specifically, the system control unit 114 opens the focal plane shutter 102 and performs drive operations such as a reset operation, a photocharge accumulation operation, and a read operation on the pixel lines of the image sensor 103 for a plurality of pixel lines. In sequence with a certain time difference. Specifically, the timing generator 112 generates a vertical synchronization signal at a time interval T <b> 2 and supplies it to the image sensor 103. Therefore, photographing is performed at every time interval T2. Here, it is assumed that the time interval T2 is set to ½ of the blinking cycle of the illumination light.

  Similar to the still image shooting described with reference to FIG. 2 and the like, since the plurality of pixel lines are sequentially exposed by the rolling readout of this example, there is no simultaneous charge accumulation in the plurality of pixel lines. For this reason, each image data has a luminance unevenness similar to the luminance unevenness described in the still image shooting. Also in this drawing, in order to easily explain the luminance unevenness, only the luminance component due to the illumination light is shown as the image data 710 and the image data 720. The luminance unevenness occurs in a direction perpendicular to the pixel line exposed at substantially the same time. When one pixel line is formed in the row direction, luminance unevenness occurs in the column direction.

  When the vertical synchronization signal is generated at time t10 when the luminance of the illumination light is minimized as in the present example, the dark band appears at the top of the image data 710 in the Y direction. Then, corresponding to the time t11 at which the luminance of the illumination light is maximized, a bright zone appears at the bottom in the Y direction. In addition, in the image data 720 captured in synchronization with the vertical synchronization signal generated at time t11, the bright band appears at the top in the Y direction, corresponding to time t12 when the luminance of the illumination light is minimized. A dark band appears at the bottom in the Y direction.

  The luminance distribution in the Y direction of the image data 720 is reversed in phase from the luminance distribution in the Y direction of the image data 710. The image processing unit 107 generates superimposed image data 730 as a frame constituting a moving image by superimposing the image data 710 and the image data 720 and averaging them. Since the superimposed image data 730 is generated by superimposing the image data whose luminance changes are opposite to each other, the luminance unevenness appearing in the image data 730 is compared with the luminance unevenness appearing in the image data 710 and the image data 720. , Significantly reduced.

  Similarly, for t12 to t14, which is the next cycle of flickering of illumination light, one frame is generated by superimposing two image data. The system control unit 114 controls each unit of the camera 100 so that this operation is repeated, thereby generating a plurality of frames constituting the moving image.

  In this example, for the purpose of explaining the processing at the time of moving image shooting in an easy-to-understand manner, luminance unevenness is described, and color covering is not particularly mentioned. However, as in still image shooting, there may be a case where color covering remains in the image data obtained by superimposing. As described above in connection with still image shooting, the image processing unit 107 performs white balance correction for each of the plurality of band regions on each superimposed image as a moving image frame. Thereby, the color covering of each moving image frame can be reduced.

  FIG. 8 shows a processing flow diagram during moving image shooting. The processing flow is started when it is detected that a moving image shooting instruction operation has been performed on the operation member 120. In this flowchart, the system control unit 114 operates mainly unless otherwise specified.

  When the processing flow is started, it is determined in step S802 whether the flicker reduction mode is set to ON. If the flicker reduction mode is set to ON, the frame rate is set according to the flicker (step S804). For example, based on a variable indicating the blinking frequency of the illumination light developed in the internal memory 108, the timing generator 112 generates the vertical synchronization signal at a time interval (N + 1/2) times the blinking period of the illumination light. Is set. Here, N is an integer of 0 or more. The timing generator 112 is set to generate a clock that indicates the timing for driving each pixel line.

  In step S806, rolling readout for the image sensor 103 is started. In step S808, shooting and frame generation processing is executed. A specific sequence will be described later. When the photographing and frame generation processing is completed, rolling readout is stopped in step S810. When recording a plurality of generated frames on the recording medium 122, the plurality of frames generated in step S808 are encoded as moving images by the image processing unit 107 and recorded as moving images on the recording medium 122 through the recording medium IF111. (Step S812). When the recording on the recording medium 122 in step S812 is completed, the series of moving image shooting operations is terminated.

  In step S804 of this flow, it has been described that the frame rate is set according to the flicker. For example, when shooting at the frame rate illustrated in FIG. 7 to cope with blinking at 100 Hz, a moving image of 100 fps is obtained as a moving image for output. On the other hand, when it is requested to generate a 30 fps moving image, it is necessary to convert the 100 fps moving image into a 30 fps moving image. For example, in step S812, the image processing unit 107 may convert a 100 fps moving image into a 30 fps moving image and record it on a recording medium. Although there is an intervening process for converting the frame rate, the image quality is not substantially deteriorated because it is a conversion process to a low frame rate moving image.

  Incidentally, the frame rate illustrated in FIG. 7 corresponds to the case of N = 0 described in relation to step S804. If N = 1, the frame rate of the moving image for output is 33.3 fps, which is higher than the requested 30 fps. Therefore, even if the frame rate of the moving image is converted to the required 30 fps, the image quality does not deteriorate significantly. However, when N = 2, the frame rate of the moving image for output is 20 fps, which is lower than the requested 30 fps. For this reason, if this moving image is converted to the required 30 fps, the degradation of image quality is greater than when N = 0 or N = 1. Therefore, in step S804, it is desirable to set the shooting frame rate based on the blinking cycle and the requested frame rate so that the frame rate of the moving image for output is higher than the requested frame rate.

  Specifically, assuming that the blinking cycle is T, in step S804, N that satisfies the condition that 2 × (N + ½) × T is smaller than the inverse of the requested frame rate is calculated. Then, it is desirable to set the shooting frame rate (N + 1/2) × T using the calculated N. At this time, the shooting frame rate may be set using the largest N that satisfies the condition.

  FIG. 9 is a flowchart for executing the photographing and frame generation processing. That is, it is a detailed flowchart of step S808 in FIG.

  When the reading of the first frame is completed in step S902, the memory control unit 106 stores the image data of the first frame in the internal memory 108 in step S904. When the reading of the second frame is completed in step S906, the memory control unit 106 stores the image data of the second frame in the internal memory 108 in step S906.

  In step S910, it is determined whether the flicker reduction mode is set to ON. When the flicker reduction mode is set to ON, the image processing unit 107 determines whether or not the flicker has occurred based on at least one of the first frame and the first frame (step S912). When the flicker influence is generated, the image processing unit 107 detects a flicker area that is an area where the flicker influence occurs in the first frame (step S914). The area where the flicker is generated can be detected by applying a method similar to the method described with reference to FIG. In step S916, the image processing unit 107 compares the first frame and the second frame to detect the shake amount.

  In step S918, the image processing unit 107 adds the image of the flicker area of the second frame to the flicker area of the first frame. At this time, the image processing unit 107 performs blur correction on the image in the flicker area of the second frame based on the blur amount detected in step S916, and converts the blur-corrected image into the flicker area of the first frame. Add to.

  In step S920, the image processing unit 107 sets a band-shaped image area in the flicker area, and performs white balance correction for each band-shaped image area. In step S922, white balance correction is performed on the image area other than the flicker area. The output frame subjected to the white balance correction by the image processing unit 107 is stored in the internal memory 108. The output frame stored in the internal memory 108 may be displayed on the display unit 110 under the control of the display control unit 109. After this step, the process proceeds to step S924.

  If the flicker reduction mode is not set to ON in step S910, the process proceeds to step S930. In step S912, if there is no flicker effect, the process proceeds to step S930. In step S930, the image processing unit 107 performs white balance correction on the entire image data of the first frame. The image processing unit 107 performs white balance correction on the entire image data of the second frame. Two frames generated by the image processing unit 107 performing white balance correction are stored in the internal memory 108 as output frames. The output frame stored in the internal memory 108 may be displayed on the display unit 110 under the control of the display control unit 109. After this step, the process proceeds to step S924.

  In step S924, it is determined whether or not the end of the moving image shooting is instructed through the operation member 120. If the end of moving image shooting is not instructed, the process proceeds to step S902, and output frames continue to be generated until the end of moving image shooting is instructed. When the end of moving image shooting is instructed, a series of processing relating to shooting and frame generation processing ends, and the flow returns to the flow of FIG.

  As described in the present embodiment, by driving the focal plane shutter 102 in the still photographing mode, a plurality of pixel lines included in the image sensor 103 are sequentially exposed. In addition, in the moving image shooting mode, a plurality of pixel lines included in the image sensor 103 are sequentially exposed by performing rolling readout of the image sensor 103 as a CMOS sensor.

  In the above description, it has been described that two images are superimposed to generate a superimposed image. However, the image processing unit 107 may generate a superimposed image by superimposing a plurality of three or more images. That is, the image processing unit 107 generates a superimposed image by superimposing a plurality of images generated by performing multiple exposures with the image sensor 103. Then, the image processing unit 107 performs white balance correction on the superimposed image for each of a plurality of strip-shaped image regions parallel to the plurality of pixel lines. By superimposing a plurality of images, a still image and a moving image frame with reduced luminance unevenness can be generated. In addition, by performing white balance correction for each band-shaped image region, it is possible to generate still images and moving image frames with reduced color coverage. That is, according to the camera 100, it is possible to generate still images and moving image frames in which the influence of flickering of illumination light is reduced.

  In the above description, it has been described that white balance correction is performed after a plurality of images are superimposed. However, the image processing unit 107 may generate a superimposed image on which white balance correction has been performed by performing white balance correction on each of the plurality of images after performing white balance correction on each band-shaped image region. Good. That is, regardless of the order in which the overlay process and the white balance process are applied, the image processing unit 107 generates a superimposed image in which white balance correction is performed for each of a plurality of strip-shaped image regions parallel to the plurality of pixel lines. be able to. Further, the image processing unit 107 may perform superimposition processing on at least a part of a region when superimposing a plurality of images. The image processing unit 107 may perform white balance correction in at least a part of the area. As described above, the image processing unit 107 may perform white balance correction for each of a plurality of band-shaped image regions on a partial region in which an effect of flicker in an image generated by the image sensor 103 is generated. In this manner, the image processing unit 107 may perform white balance correction for each of the plurality of strip-shaped image regions by superimposing a plurality of images on the partial region where the flicker is affected.

  In the above description, in the still image shooting mode, it has been described that exposure is performed with an exposure time T1 that is shorter than the blinking cycle of the illumination light and longer than the half cycle of the blinking cycle. In the moving image shooting mode, it has been described that a frame is shot at a time interval T2 that is half the blinking cycle. That is, it has been described that the image processing unit 107 superimposes two images generated by exposure by the image sensor 103 with a phase shifted by a half cycle with respect to the flicker cycle. However, the exposure time in still image shooting and the frame interval in moving image shooting are not limited to the above. The above overlay and white balance correction processes can be applied in any case as long as the exposure time and frame interval are affected by flicker. As the exposure time and frame interval affected by flicker, a case where the exposure time and frame interval are approximately the same as the blinking cycle can be exemplified. On the other hand, when the time length for which one pixel line is exposed is equal to or longer than the blinking cycle, it can be regarded as not affected by flicker. In still image shooting, when the shutter curtain speed is extremely high, for example, when the exposure time is sufficiently short as compared with the blinking cycle, it can be regarded as not affected by flicker.

  Therefore, when the exposure time or frame interval is affected by flicker, the above overlay and white balance correction processing is applied. When the exposure time or frame interval is not affected by flicker, the above overlay and white interval are applied. The balance correction process may not be applied. In the above description, it has been described that the process is switched by determining whether the flicker reduction mode is ON / OFF. However, ON / OFF determination of the flicker reduction mode can be replaced with determination of whether the exposure time or frame interval is affected by flicker.

  In the above description, it has been mainly described that the next exposure is started with a phase shifted by a half cycle of the blinking cycle with respect to the blinking of the illumination light. However, it is not necessary for the next exposure to be exactly half a cycle away from the blinking cycle. Most preferably, the next exposure is started with a phase that is shifted by a half cycle of the blink cycle, but if the time length until the start of the next exposure does not match an integer multiple of the blink cycle, the next exposure starts. The time length of may be any value. Unless the time length until the start of the next exposure coincides with an integral multiple of the blinking cycle, a certain effect of reducing the influence of flicker can be expected.

  In the above, the illumination light by a fluorescent lamp was illustrated as illumination light. However, the illumination light can be exemplified by illumination light from another light source whose color temperature changes with time, such as a mercury lamp. When shooting in an environment where a light source that is driven by a commercial AC power supply and blinks according to the power supply frequency is used as a light source of illumination light, the above processing, for example, the above overlay and white balance correction processing is performed. Can be applied. For example, a light source driven by a 50 Hz system power supply is expected to blink at, for example, 100 Hz. If the light source is driven by a 60 Hz system power supply, for example, it is expected to blink at 120 Hz. Therefore, it may be determined whether or not it is affected by the flicker based on a frequency according to the system power supply, for example, a blinking cycle of 100 Hz to 120 Hz. Further, as described above, the time length until the start of the next exposure may be set based on, for example, a blinking frequency of 100 Hz to 120 Hz. For example, the time length until the start of the next exposure may be set to a value that is not an integer multiple of 1/100 and not an integer multiple of 1/120. Further, the time length until the next exposure is started may be set based on the blink frequency so that the next exposure is started with a phase shifted by a half cycle of the blink cycle. For example, the time length until the next exposure starts may be set based on the frequency of the system power supply used in the area where the camera 100 is currently located. The current position of the camera 100 may be specified based on GPS information or the like.

  As described above, the image processing unit 107 performs white balance correction for each of a plurality of strip-shaped image regions when the image generated by the image sensor 103 is affected by flicker. If there is no flicker effect, the image processing unit 107 may perform white balance correction on the entire image. Further, as described above, the image processing unit 107 can determine that the influence of flicker has occurred when a band-like luminance distribution is generated. Specifically, the image processing unit 107 determines that the effect of flicker has occurred when the image generated by the image sensor 103 has luminance periodicity in a direction perpendicular to the plurality of pixel lines. In addition, a sensor that detects ambient light may be provided in the camera 100, and based on the sensor output, it may be determined whether an image generated by the image sensor 103 is affected by flicker. Examples of sensor output include brightness time change information, ambient light spectrum information, and the like. For example, when the brightness changes periodically, it may be determined that the influence of flicker occurs. Also, flicker effects occur when the ambient light spectrum matches a predetermined spectrum such as an emission spectrum from a fluorescent lamp or an emission spectrum from a mercury lamp with a degree of coincidence exceeding a predetermined value. You may judge as. In addition, as described above, when the exposure time or frame interval is affected by flicker, it may be determined that the image generated by the image sensor 103 is affected by flicker. In this manner, when shooting is performed under shooting conditions that may be affected by flicker without determining whether flicker is actually affected, the effect of flicker is exerted on the image generated by the image sensor 103. It may be determined that this occurs.

  In the above description, for example, as described with reference to FIG. 5, the image processing unit 107 has been described as correcting the blur and superimposing the partial areas when overlapping the partial areas. However, the image processing unit 107 may also superimpose the image by correcting the blur when superimposing the entire image. That is, the image processing unit 107 may correct and overlap blur between a plurality of images regardless of whether the partial areas are overlapped or the entire image is overlapped. According to this processing, it is possible to provide an image in which the influence of camera shake or the like is reduced. Further, according to the camera 100 described above, since image processing is performed after being stored as image data in the internal memory 108, various information such as information on flicker and the amount of camera shake can be obtained based on the image data. Can do.

  In the above description, it is assumed that the camera 100 includes the focal plane shutter 102, images are taken using the focal plane shutter 102 in the still image shooting mode, and rolling reading is performed in the moving image shooting mode. The camera 100 may not include the focal plane shutter 102 and may perform rolling reading even in the still image shooting mode. In the moving image shooting mode, the focal plane shutter 102 may be driven to shoot a moving image. The shutter function of the camera 100 may be implemented by a combination of an electronic front curtain shutter and a mechanical rear curtain shutter, and the image sensor 103 sequentially exposes pixel lines by the electronic front curtain shutter and the mechanical rear curtain shutter. May be.

  The camera 100 described in the above embodiment can be applied to a mobile phone with a camera function as well as a lens interchangeable single-lens reflex camera, a compact digital camera, a mirrorless single-lens camera, and a video camera. The image processing in the image processing unit 107 can be executed by an electronic information processing apparatus such as a personal computer. The electronic information processing apparatus may execute the image processing by loading a control program that controls execution of the image processing and operating according to the read control program. The electronic information processing apparatus can load the control program by reading a computer-readable recording medium storing the control program.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 100 Camera, 101 Optical system, 102 Focal plane shutter, 103 Image pick-up element, 104 Amplifier part, 105 A / D converter, 106 Memory control part, 107 Image processing part, 108 Internal memory, 109 Display control part, 110 Display part, 111 recording medium IF, 112 timing generation unit, 113 lens control unit, 114 system control unit, 120 operation member, 122 recording medium, 200 blinking pattern, 210, 220, 230, 510, 520, 530, 710, 720, 730 image Data, 400 Superimposed image, 410, 540, 540 Band area, 515, 525 Partial area

Claims (8)

An image sensor having a plurality of pixel lines exposed sequentially;
A plurality of strip-like images parallel to the plurality of pixel lines are obtained by superimposing a plurality of images generated by the image sensor being exposed a plurality of times on a partial region where the influence is caused when flicker is generated. An image processing unit for generating a superimposed image by performing white balance correction for each region ;
An imaging apparatus comprising: The imaging device according to claim 1, wherein the partial region is a region other than a region that substantially receives only ambient light. Wherein the image processing unit, an imaging apparatus according to claim 1 or 2 superposing two images generated by exposing by the imaging device in a half cycle out of phase with respect to the period of the flicker. Wherein the image processing unit, an image which the imaging device is generated, if it has a periodicity of the brightness in a direction perpendicular to said plurality of pixel lines, claims 1 to 3 determine that the influence of the flicker has occurred The imaging device according to any one of the above. Wherein the image processing unit, an imaging apparatus according to any one of claims 1 to 4, superimposed to correct the blur between the plurality of images. Imaging device according to claim 1, further comprising a reading unit for rolling read in any one of 5 with respect to the image sensor. The imaging apparatus according to any one of claims 1 to 5, further comprising a focal plane shutter for sequentially exposing the plurality of pixel lines. An image processing control program for a computer
Superimposing a plurality of images generated by exposure multiple times with an imaging device having a plurality of pixel lines that are sequentially exposed to a partial region where the effect occurs when flicker effects occur ;
Generating a superimposed image by performing white balance correction for each of a plurality of strip-shaped image areas parallel to the plurality of pixel lines;
An image processing control program for executing

Images