JP2012119788A - Imaging device, image processing device, imaging method and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable imaging at an arbitrary shutter speed to the possible extent while preventing an LED area in an imaged image from being imaged in a turn-off state when imaging a subject including periodically blinking LED light.SOLUTION: An imaging method, from an image imaged by imaging means, includes the steps of: detecting the LED area including the periodically blinking LED light (step S10); imaging a first image by the imaging means according to an imaging instruction (step S18); determining whether the LED area in the first imaged image is imaged in a turn-on state or a turn-off state (step S24); on determination that the LED area in the first imaged image is imaged in the turn-off state, imaging a second image by the imaging means with an exposure time longer than one period of the LED blink period (step S26); and synthesizing the LED area in the second imaged image with the first imaged image (steps S28, S30).

Description

  The present invention relates to a technique capable of preventing an LED area in a captured image from remaining in a light extinguished state when photographing a subject including LED light that blinks periodically.

  Patent Document 1 discloses a configuration in which an exposure time (charge accumulation time) of a solid-state image sensor is controlled to an integral multiple of 50 milliseconds in a photographing apparatus including a solid-state image sensor having a photoelectric conversion function.

  In Patent Document 2, the input video signal is divided into a plurality of fields in synchronization with the period of the vertical synchronization signal, and the gain of the gain control circuit is automatically adjusted according to the size of the video signal for each field. A configuration for reducing flicker of an image under fluorescent lamp illumination is disclosed.

Japanese Patent Publication No. 61-56951 Japanese Patent Publication No. 6-20276

  An LED light source having an inexpensive circuit configuration that does not incorporate an inverter causes periodic blinking (so-called “flicker”) according to the frequency of the AC power supply. Therefore, for example, when shooting an LED traffic light or gorgeous LED illumination under clear sky, the exposure time may be shorter than the blinking cycle of the LED, and it may be photographed as if it is not lit at all. Then, an image that does not match the photographer's intention is captured.

  In the configuration described in Patent Document 1, it is possible to prevent image fluctuations at different imaging timings by setting the exposure time of the imaging device to be equal to or longer than the cycle of the AC power supply. It is indispensable to set as described above, and photography cannot be performed at an arbitrary shutter speed. That is, there are cases where it is not possible to take pictures according to the photographing environment or the photographer's intention.

  In addition, the configuration described in Patent Document 2 is intended to reduce flicker under fluorescent lamp illumination, and even if similar gain control is performed on a captured image of a subject including LED light such as an LED traffic light or LED illumination, By adjusting the gain for each field divided in synchronization with the period of the vertical synchronization signal, only the gain for the entire image is increased, and the captured image (especially the LED when the LED area is in the extinguished state). Area) is not properly corrected.

  The present invention has been made in view of such circumstances, and when photographing a subject including LED light that periodically blinks, it is possible to prevent the LED area in the photographed image from remaining in the extinguished state. An object is to enable photographing at an arbitrary shutter speed as much as possible.

  In order to achieve the above object, the present invention periodically blinks from an imaging unit capable of continuously capturing an image of an object, an instruction input unit for receiving an input of an imaging instruction, and an image captured by the imaging unit. LED detection means for detecting an LED area including LED light; determination means for determining whether the LED area is imaged in a lit state or an extinguished state; and at least one by the imaging means in response to the imaging instruction Control means for taking a picture, when the LED area is detected by the LED detection means, and it is determined that the LED area of the first photographed image has been imaged in the extinguished state by the determination means, Control means for taking a second image by the imaging means with an exposure time of one cycle or more of the blinking cycle of the LED; and an LED area of the second photographed image. Providing an imaging apparatus characterized by comprising synthesizing means for synthesizing the image.

  That is, only when it is determined that the periodically blinking LED area of the first photographed image photographed in accordance with the photographing instruction is captured in the extinguished state, the exposure time is one or more of the LED blinking periods. Since the second shot is taken and the LED area of the second shot image is combined with the first shot image, the shot image can be taken even when shooting a subject including periodically flashing LED light. It is possible to prevent the inside LED region from being viewed with the light extinguished, and it is possible to photograph at an arbitrary shutter speed. Further, only the LED area is combined with the first captured image of the second captured image, and the other areas are captured at the release timing (timing instruction timing). Since the image remains as it is, an image suitable for the release timing can be obtained. In addition, since the number of times of photographing after the photographing instruction ends at the maximum, it does not give the photographer a sense of incongruity.

  In addition, the present invention provides an LED area that includes an imaging unit capable of continuously capturing an image of an object, an instruction input unit that receives an input of an imaging instruction, and LED light that periodically flashes from an image captured by the imaging unit. LED detection means for detecting the image, determination means for determining whether the LED area is imaged in a lit state or an extinguished state, and control for capturing at least one image by the imaging means in response to the imaging instruction And when the LED area is detected by the LED detection means and the LED area of the first photographed image is determined to have been taken out by the determination means, the LED area is in a lighting state. And a control unit that repeats the second and subsequent shots by the imaging unit until the image is taken.

  That is, only when it is determined that the periodically blinking LED area of the first photographed image photographed in accordance with the photographing instruction is captured in the extinguished state, it is determined that the LED area is captured in the lit state. Since the second and subsequent shots are repeated until the image is taken, it is possible to prevent the LED area in the shot image from being viewed with the light extinguished even when shooting a subject including LED light that periodically flashes. It is possible to shoot at an arbitrary shutter speed (exposure time) as much as possible. It is also possible to align the shutter speed between the first photographed image and the second photographed image.

  In one embodiment of the present invention, the image processing apparatus includes a synthesis unit capable of synthesizing a partial image of the second and subsequent captured images with the first captured image, and the control unit includes the second and subsequent captured images. The partial image of the LED region is separated from the captured image captured with the LED region turned on, and the separated partial image is combined with the first captured image by the combining means. And That is, only the LED area is combined with the first photographed image of the second and subsequent photographed images, and the other areas are the first photographed at the release timing (timing instruction timing). Since the captured image remains as it is, an image suitable for the release timing can be obtained. In the present embodiment, it is possible to combine the second and subsequent photographed images photographed at the same shutter speed as the first photographed image.

  In one embodiment of the present invention, the image pickup device includes combining means capable of combining the first shot image with a partial image of the second and subsequent shot images, and the image pickup device includes a plurality of photoelectric conversion elements arranged. An arbitrary partial area from the maximum readable area of the light-receiving pixel surface, and the control means is configured to capture the image sensor in the second and subsequent photographing after the photographing instruction. The partial area of the LED area is controlled by performing partial readout for reading out only the partial area corresponding to the LED area from among the maximum areas of the first area. Is combined with the first photographed image. That is, only the LED area is combined with the first photographed image of the second and subsequent photographed images, and the other areas are the first photographed at the release timing (timing instruction timing). Since the captured image remains as it is, an image suitable for the release timing can be obtained. In the present embodiment, it is possible to combine the second and subsequent photographed images photographed at the same shutter speed as the first photographed image. In re-photographing (second and subsequent shooting), only a partial area corresponding to the LED area is partially read out, so that the re-photographing interval can be shortened, and a captured image more suitable for the release timing can be obtained. it can.

  In addition, the present invention provides an LED area that includes an imaging unit capable of continuously capturing an image of an object, an instruction input unit that receives an input of an imaging instruction, and LED light that periodically flashes from an image captured by the imaging unit. LED detection means for detecting the image, determination means for determining whether the LED area is imaged in a lit state or an extinguished state, and control for capturing at least one image by the imaging means in response to the imaging instruction And when the LED region is detected by the LED detection unit and the LED region of the first photographed image is determined to be captured in the extinguished state by the determination unit, the There is provided a photographing device comprising: control means for controlling at least the brightness of a partial image of an LED region.

  That is, when it is determined that the periodically blinking LED area of the photographed image photographed in accordance with the photographing instruction is photographed in the extinguished state, the luminance of the partial image in the LED area is controlled. Even when shooting a subject including LED light that blinks, it is possible to prevent the LED area in the captured image from being viewed in the extinguished state, and it is possible to capture at an arbitrary shutter speed. In addition, an image suitable for the release timing (timing of shooting instruction) can be obtained.

  In one embodiment of the present invention, a blur detecting unit that detects a blur at the time of imaging, and a synthesizing unit capable of synthesizing a partial image of the second and subsequent shot images with the first shot image after the shooting instruction. The control means controls the luminance of the partial image of the LED area only when it is determined that the LED area of the photographed image has been captured in the extinguished state and a blur at the time of imaging is detected, and the photographing Even if it is determined that the LED area of the image is captured in the extinguished state, if the blur at the time of imaging is not detected, the LED area of the recaptured image is combined with the first captured image by the combining means. It is characterized by. That is, when there is a blur, it is possible to prevent the blurred LED region from being synthesized. In addition, when there is no blur, only the LED area is combined with the first photographed image among the re-captured images, and other areas are captured at the release timing (timing instruction timing). Since it is the same as the first photographed image, an image suitable for the release timing can be obtained.

  In one embodiment of the present invention, the control means determines the brightness of the LED area of the captured image when it is determined that the LED area of the captured image has been captured in the extinguished state and a blur at the time of capturing is detected. When the brightness is lower than the threshold, the brightness of the partial image in the LED area is controlled by replacing the partial image in the LED area of the captured image with an image according to the spectral characteristics of the LED. It is characterized by. That is, by replacing the image in accordance with the spectral characteristics of the LED, the LED region that has been photographed extremely dark can be corrected with good color development.

  In one embodiment of the present invention, the control means sets the same exposure time as the blinking cycle of the LED at the time of re-shooting after the shooting instruction. That is, it is possible to achieve both re-shooting only once (second shot) and shortening the re-shooting time.

  In one embodiment of the present invention, the control means sets the exposure time, the aperture amount, and the sensitivity to the same values as those at the time of the first photographing at the time of re-imaging after the photographing instruction. That is, since the same quality image is obtained by the first photographed image and the re-photographed image, it is possible to prevent the observer from feeling uncomfortable.

  In one embodiment of the present invention, the display unit displays a moving image captured by the imaging unit as a live view image before inputting the shooting instruction, and the LED detection unit is displayed on the display unit. The LED region is detected from a plurality of frames of the live view image. In other words, since the LED area can be easily detected before the shooting instruction by using the live view image, it is possible to prevent the shooting time from extending much.

  In addition, the present invention is different from an imaging unit that can continuously image a subject, an instruction input unit that receives an input of a shooting instruction, and a blinking cycle of an LED that blinks periodically according to the input of the shooting instruction. A control unit that continuously shoots by the imaging unit in a cycle; an LED detection unit that detects an LED region that includes the periodically flashing LED light from a plurality of captured images obtained by continuous shooting of the imaging unit; A determination unit that determines whether the LED area is imaged in a lit state or an extinguished light state, and selects a captured image that is determined to be captured in the lit state from the plurality of captured images And a selection means for providing a photographing apparatus.

  That is, since the LED area is detected from a plurality of captured images obtained by continuous shooting, and the captured image determined to have been imaged in the lighting state is selected from the plurality of captured images. Even when photographing a subject including LED light that periodically blinks, it is possible to prevent the LED region in the photographed image from being viewed in the extinguished state and to photograph at an arbitrary shutter speed.

  In one embodiment of the present invention, the LED detection unit divides an image continuously captured by the imaging unit into a plurality of divided regions, and detects periodic blinking of the LED for each of the divided regions. In addition, the LED region is detected by detecting a spectral characteristic of the divided area where the blinking is detected and determining whether the spectral characteristic is a spectral characteristic of the LED.

  In one embodiment of the present invention, the determination unit is a threshold value indicating a lower limit value of the luminance of the LED region when the luminance of the LED region detected by the LED detection unit is imaged in a lighting state. By comparing with a threshold value calculated based on the exposure time of the image pickup means, it is determined whether the LED area is picked up in a lighted state or a light extinguished state.

  The present invention is an image acquisition means for acquiring a plurality of captured images continuously shot at a period different from the blinking period of the periodically flashing LEDs, and a plurality of captured images obtained by the image acquisition means, LED detection means for detecting an LED area including the periodically flashing LED light; determination means for determining whether the LED area is imaged in a lit state or an extinguished state; and the plurality of captured images An image processing apparatus comprising: a selection unit configured to select a captured image determined to be captured when the LED region is in a lighting state.

  According to the present invention, when photographing a subject including LED light that periodically blinks, it is possible to prevent the LED region in the photographed image from remaining in the extinguished state and to photograph at an arbitrary shutter speed.

1 is an overall configuration diagram of an example of a photographing apparatus according to a first embodiment. The flowchart which shows the principal part of an example of the imaging | photography process in 1st Embodiment. Detailed flowchart of LED detection processing Explanatory drawing showing an example of spectral characteristics of single color LED The flowchart which shows the principal part of an example of the imaging | photography process in 2nd Embodiment. Overall configuration diagram of an example of an imaging apparatus according to the third embodiment The flowchart which shows the principal part of an example of the imaging | photography process in 3rd Embodiment. The flowchart which shows the principal part of an example of the imaging | photography process in 4th Embodiment. The flowchart which shows the principal part of an example of the imaging | photography process in 5th Embodiment. Overall configuration diagram of an example of an imaging apparatus according to the sixth embodiment Block diagram showing an example of an LED detection circuit Overall configuration diagram of an example of an imaging apparatus according to the seventh embodiment Block diagram showing an example of a luminance correction circuit

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is an overall configuration diagram of an example of an imaging apparatus according to the first embodiment.

  In FIG. 1, the photographing apparatus 10 of this example includes a photographing optical system 12, an image sensor 14, an AFE 16 (analog signal processing unit), a DSP 18 (digital signal processing unit), an operation unit 20, an LCD 22, a RAM 24, a flash ROM 26, and a microphone 28. A photographing optical system driver 46 and the like.

  The photographing optical system 12 includes a zoom lens 41, a focus lens 42, an iris 43, a mechanical shutter 44, a shake correction unit 45, and the like. The photographing optical system driver 46 includes a zoom lens 41, a focus lens 42, an iris 43, a circuit that drives the mechanical shutter 44, a motor, and the like. The zoom lens 41 and the focus lens 42 are provided so as to be movable along the optical axis L1. The photographing optical system driver 46 changes the photographing magnification by moving the zoom lens 41. Further, focusing is performed by moving the focus lens 42. Further, the amount of light received by the image sensor 14 is adjusted by changing the opening amount (aperture amount) of the iris 43. The optical path on the optical axis L1 is opened and closed by the mechanical shutter 44. The blur correction unit 45 includes a lens that is movably provided in a direction orthogonal to the optical axis L1. The shake detection unit 47 detects a shake of the photographing apparatus 10. The shake correction control unit 48 moves the shake correction unit 45 based on the detection result of the shake detection unit 47. Specifically, the blur correction unit 45 is moved in a direction perpendicular to the optical axis L1 according to the blur frequency of the photographing apparatus 10. The shake correction unit 45, the shake detection unit 47, and the shake correction control unit 48 may be known ones and will not be described in detail. The cause of the device shake to be detected is a camera shake of the photographer.

  The image sensor 14 is composed of a CMOS image sensor, a CCD image sensor, or the like. The image sensor 14 photoelectrically converts subject light imaged by the imaging optical system 12 on a light receiving surface on which photoelectric conversion elements (imaging pixels) are arranged, and outputs an analog image signal (captured image). The image sensor 14 has a so-called electronic shutter function. When a shutter pulse signal is input from a TG (timing generator) 49 of the AFE 16, the image pickup element 14 sweeps out and erases the charges accumulated so far. That is, the charge accumulation time (also referred to as “exposure time” or “shutter speed”) in the image sensor 14 is adjusted by the TG 49. The exposure amount in the image sensor 14 is determined by the shutter speed of the electronic shutter and the iris amount of the iris 43. The analog image signal output from the image sensor 14 is amplified with a gain corresponding to the ISO sensitivity by the gain control circuit 50 of the AFE 16, and converted into a digital image signal by the A / D converter 51.

  The imaging unit 11 that captures an image of a subject is configured including the imaging optical system 12, the imaging device 14, the AFE 16, and the imaging optical system driver 46. The imaging means 11 of this example can image a subject continuously in time.

  The DSP 18 (digital signal processing unit) temporarily stores an image signal (captured image) output from the A / D converter 51 of the AFE 16 as data, and a Y / Y for converting RGB signals into YC signals. A C processing unit 53; an AE / AF circuit 54 that detects the brightness and focus evaluation value of the subject; an AWB circuit 55 that calculates a white balance evaluation value; and a compression / decompression circuit 56 that compresses and decompresses the captured image. , A recording medium interface 57 for recording various data such as a captured image on the recording medium 30 and reading from the recording medium 30, a timer 58 for timing, and a face detection circuit 59 for detecting a face image from the captured image The CPU 60 is configured to include an encoder 61 that converts the captured image for display on the LCD 22 (display unit).

  A CPU (Central Processing Unit) 60 controls each unit of the photographing apparatus 10 by executing a predetermined program.

  The operation unit 20 is an instruction input device for a user to input various instructions to the photographing apparatus 10. In this example, a mode switch, a zoom lever, and a shutter switch are included. The shutter switch is a two-stage type. When the first switch (S1) is turned on, a shooting preparation instruction is accepted, and when the second switch (S2) is turned on, a photographing instruction is accepted. The mode switch accepts mode settings such as a shooting mode for shooting a subject and a playback mode for playing back an image.

  The LCD (liquid crystal display device) 22 displays various data such as a captured image encoded by the encoder 61 by driving the LCD driver 21. The LCD 22 is used for live view display that continuously displays in real time live view images (through images) taken continuously in time by the image pickup means 11 as an electronic viewfinder in the shooting mode. Further, the LCD 22 displays the captured image recorded on the recording medium 30 in the playback mode. A display device other than the LCD may be used.

  The RAM 24 stores information necessary for the operation of the program. The flash ROM 26 stores programs, setting information, and the like. The microphone 28 inputs sound by driving a microphone driver 29.

  FIG. 2 is a flowchart showing a main part of an example of the photographing process in the first embodiment. This process is executed by the CPU 60 according to a program.

  When the shooting mode is set by the operation unit 20, the CPU 60 starts the shooting process of FIG. 2 together with the start of the live view display. In the live view display, images taken continuously in time by the imaging means 11 are continuously displayed on the LCD 22.

  In step S <b> 10, the CPU 60 detects an LED region including an LED image that periodically blinks from captured images that are continuously captured in time. The detailed flow of this LED detection process is shown in FIG.

  In FIG. 3, the CPU 60 obtains a plurality of time-sequential captured images (in this example, a plurality of frames of a live view image) from the image buffer 52 (step S <b> 32) and arranges the captured images in two dimensions vertically and horizontally. Dividing into a plurality (n) of divided areas (step S34), loop processing (steps S36 to S54) is started with the division number n. In the loop processing, the CPU 60 detects flicker (LED blinking) for each divided region based on the luminance of the divided regions corresponding to each other over a plurality of captured images (step S38), and each detection result indicates The presence / absence of flicker in the divided area is determined (step S40). When there is flicker, the CPU 60 performs setting of the LED flicker frequency fn (step S42), setting of the luminance peak dn (step S44), and detection of the spectral characteristics of the LED (step S46).

  The flicker frequency fn of the LED is the light emission cycle of the LED that periodically blinks. For example, fn may be detected based on a frame rate from an image (divided image) for each divided region, or may be acquired from the flash ROM 26. Further, fn is the same as the frequency of the AC power supply (for example, 50 Hz or 60 Hz), and may be determined based on the current position.

  The imaging cycle in the live view image is ½ or less of the cycle of the AC power supply.

  The luminance peak dn is a measured value of the luminance of the divided region (LED region) where flicker is detected, and is an integrated value over the flicker cycle.

  The spectral (spectral) characteristics of the LED indicate the relationship between the color (wavelength) of light emitted from the LED and the intensity. For example, a monochromatic LED has spectral characteristics as shown in FIG. 4. Such spectral characteristics are stored in advance in the flash ROM 26, and a divided area (when flicker is detected and taken as an image in a lighting state) Compare with spectral characteristics of LED region). 4A shows the spectral characteristics of the purple LED, FIG. 4B shows the spectral characteristics of the blue LED and the green LED, and FIG. 4C shows the spectral characteristics of the red LED. FIG. 4D shows the spectral characteristics of the infrared LED. By comparing the spectral characteristics of the single color LEDs of each color with the spectral characteristics of the LED area in the lit state, it is possible to determine which color of the LED light is included in the LED area. In this example, for the sake of convenience of explanation, a case where an LED region including light of a single color LED (LED having a maximum spectral characteristic corresponding to one wavelength) is detected will be described as an example. The present invention can be similarly applied even to an LED that has been manufactured.

  The CPU 60 determines whether or not the LED has spectral characteristics (in this example, single-color LED spectral characteristics) (step S48). If the LED spectral characteristics, the CPU 60 turns on the single-color LED region flag in the RAM 24 (step S50). If the LED spectral characteristic is not satisfied, the single color LED region flag is turned off (step S52). When the loop is repeated n times, the loop process is terminated (step S54).

  In FIG. 2, the CPU 60 determines whether or not the S1 switch of the operation unit 20 has been pressed (step S12), and repeats step S10 until the switch is pressed. When the S1 switch is pressed, photographing preparation processing such as automatic exposure (AE) and automatic focusing (AF) is performed using the AE / AF circuit 54 (step S14).

  Next, the CPU 60 determines whether or not the S2 switch of the operation unit 20 has been pressed (step S16) and waits until it is pressed.

  If the S2 switch is pressed (when a shooting instruction is input), the first shot is taken in step S18. That is, imaging is performed by the imaging unit 11 and the captured image is temporarily stored in the RAM 24.

  In step S20, the shutter speed SS (exposure time) is compared with one cycle of flicker (1 / fn). If it is less than 1 / fn, the process proceeds to step S22, and if it is 1 / fn or more, the process proceeds to step S32. .

  In step S22, it is determined whether or not the monochrome LED region flag is on (that is, an LED region that periodically blinks and has LED spectral characteristics is detected). If it is on, the process proceeds to step S24. If it is off, the process proceeds to step S32.

  In step S24, the LED brightness (LED area brightness) is compared with the expected value (dn × SS × fn). If it is less than the expected value, the process proceeds to step S26, and if it is greater than the expected value, the process proceeds to step S32.

  That is, whether the divided area of the first captured image corresponding to the LED area detected in the live view image (the LED area of the first captured image) was captured in the extinguished state or in the lit state If it is determined that the image is captured in the extinguished state, the process proceeds to step S26.

  For example, if the LED blinks at the frequency of the AC power supply (for example, 50 Hz, 1 cycle 20 ms) and the shutter speed (exposure time) is 10 ms or less, the exposure time is less than half of the LED blinking cycle, so the exposure period is LED There are a case where the lighting period falls within the lighting period, a case where the lighting period falls within the lighting period, and a case where the lighting period spans the lighting period and the lighting period. In this specification, that the LED area is imaged in the “lighted state” includes not only the case where the exposure period is entirely within the lighting period, but also the case where the LED area spans the extinguishing period. Further, the fact that the LED area is imaged in the “light extinguished state” includes not only the case where the exposure period falls within the extinguished period but also the case where the LED area spans the lighting period. In this example, the luminance value (luminance peak dn) corresponding to the flicker cycle is multiplied by the ratio (SS × fn) between the shutter speed SS and the flicker cycle (1 / fn) to obtain the expected value (dn × SS × fn) is obtained and this expected value is used as a threshold value. Therefore, even when the exposure period spans the lighting state and the extinguished state, it is possible to appropriately determine the on / off state.

  In step S26, the shutter speed SS (exposure time) is set to one flicker cycle (1 / fn), and automatic shooting is performed. SS may be 1 / fn or more, but is preferably set to 1 / fn.

  In step S28, an LED region is extracted from the second captured image. That is, a divided region (LED region of the second photographed image) corresponding to the LED region of the live view image is separated from the second photographed image.

  In step S30, the partial image of the LED area of the second photographed image is synthesized with the first photographed image. That is, the LED area of the first captured image is replaced with the LED area of the second captured image.

  In step S32, this process ends.

  In the photographing apparatus 10 of the present embodiment, an LED detection unit that detects an LED area including LED light that periodically blinks from a live view image, and determines whether the LED area is imaged in a lit state or in a extinguished state. When the first imaging is performed by the imaging unit 11 according to the imaging instruction and the LED area of the first captured image is determined to be captured in the extinguished state, the blinking cycle of the LED is determined. The CPU 60 comprises a control means for taking a second image by the imaging means 11 with an exposure time of one cycle or more, and a combining means for combining the LED area of the second photographed image with the first photographed image. is doing.

Second Embodiment
Next, the imaging device in 2nd Embodiment is demonstrated. In addition, the point demonstrated in 1st Embodiment is abbreviate | omitted and a different point from 1st Embodiment is demonstrated.

  When the CPU 60 of the second embodiment determines that the LED area of the first photographed image is captured in the extinguished state, the imaging unit 11 captures the second and subsequent images until the LED area is captured in the lit state. The image capturing is repeated, and the captured image captured with the LED area turned on is recorded on the recording medium 30.

  FIG. 5 is a flowchart illustrating a main part of an example of the photographing process according to the second embodiment. This process is executed by the CPU 60 according to a program.

  Steps S10 to S24 are the same as in the first embodiment.

  In step S24, the CPU 60 determines whether the LED area of the first photographed image has been imaged in the extinguished state or in the lit state. If it is determined that the LED region has been imaged in the extinguished state, the process proceeds to step S60. If it is determined that the image is captured in the lighting state, the process proceeds to step S32.

  In step S60, the CPU 60 automatically performs re-shooting with the same settings as the first shot image. That is, the imaging unit 11 captures an image of the subject again under the same imaging conditions (exposure time, aperture value, sensitivity, etc.). Here, the timing of re-photographing (the time interval between the imaging timing of the first image and the imaging timing of the second image) is set not to be the flicker cycle (1 / fn).

  In step S62, it is determined whether the LED area of the re-captured image has been imaged in the extinguished state or in the lit state. If it is determined that the LED area has been imaged in the lit state, the process proceeds to step S64. If it is determined that the image has been captured, the process returns to step S60 to repeat the re-imaging.

  In step S64, the LED region is extracted from the re-captured image captured with the LED region being lit. That is, the divided area (LED area of the re-captured image) corresponding to the LED area of the live view image is separated from the second and subsequent captured images.

  In step S66, the partial image in the LED area of the recaptured image is synthesized with the first captured image. That is, the LED area of the first captured image is replaced with the LED area captured in the lighting state in the re-captured image.

  In step S32, the photographed image is recorded on the recording medium 30, and the process is terminated.

  In the photographing apparatus 10 of the present embodiment, an LED detection unit that detects an LED area including LED light that periodically blinks from a live view image, and determines whether the LED area is imaged in a lit state or in a extinguished state. When the first imaging is performed by the imaging unit 11 in response to the imaging instruction and the LED area of the first captured image is determined to be captured in the extinguished state, the LED area is in the lighting state. A control unit that repeats the second and subsequent shots by the imaging unit 11 until the image is picked up by the combining unit that combines the LED area captured in the lighting state of the retaken image with the first shot image; The CPU 60 is configured.

  In addition, although the case where the LED area is extracted from the recaptured image and combined in step S64 has been described as an example, the recaptured image (however, the LED area is in a lit state) is recorded on the recording medium 30 as it is. May be. When only the LED area is combined as in the present embodiment, the other area is an image immediately after the shooting instruction, which is more preferable.

<Third Embodiment>
Next, only the differences from the second embodiment will be described for the photographing apparatus according to the third embodiment.

  FIG. 6 is an overall configuration diagram illustrating an example of an imaging device according to the third embodiment.

  The image sensor 14 of this embodiment is configured by a CMOS image sensor. The imaging element 14 of this example is arbitrarily selected from the maximum readable area on the light receiving pixel surface in which photoelectric conversion elements (imaging pixels) that perform photoelectric conversion of a subject image are arranged in a predetermined arrangement shape (for example, honeycomb arrangement, Bayer arrangement). Only the partial area (also referred to as “capture area”) can be partially read out. By partial reading, only the focused partial area can be taken into the image buffer 52 at high speed.

  The capture area control unit 71 controls partial reading of the image sensor 14 by controlling the TG 49 in accordance with an instruction from the CPU 60. Note that a known technique may be used for partial reading, and detailed description thereof is omitted.

  FIG. 7 is a flowchart illustrating a main part of an example of the photographing process according to the third embodiment. This process is executed by the CPU 60 according to a program.

  Steps S10 to S24 are the same as in the first embodiment.

  In step S24, the CPU 60 determines whether the LED area of the first photographed image is imaged in the extinguished state or in the lit state. If it is determined that the LED region is imaged in the extinguished state, the process proceeds to step S70. If it is determined that the image is captured in the lighting state, the process proceeds to step S32.

  Step S70 is the same as step S60 in the second embodiment. That is, the imaging unit 11 captures an image of the subject again under the same imaging conditions (shutter speed, aperture value, gain, etc.).

  In step S72, the CPU 60 performs partial readout of only the LED region in the light receiving pixel surface of the image sensor 14 based on the LED region information obtained in step S10.

  In step S74, the process is the same as step S60 in the second embodiment. If it is determined that the image is captured in the lighting state, the process proceeds to step S76. If it is determined that the image is captured in the extinguished state, the process returns to step S70. .

  In step S76, a partial image of the LED area is acquired from the image buffer 52.

  Step S78 is similar to step S66 of the second embodiment, and the partial image of the LED area of the re-captured image is synthesized with the first captured image.

<Fourth embodiment>
Next, a photographing apparatus according to the fourth embodiment will be described. In the following, the same points as in the first embodiment will be omitted and different points will be described.

  In the fourth embodiment, the CPU 60 in FIG. 1 detects the LED area from the live view image captured by the imaging unit 11 before inputting the shooting instruction, and performs shooting by the imaging unit 11 in accordance with the shooting instruction. When an area (LED area) corresponding to the LED area of the live view image in the image is in the extinguished state, control is performed to adjust at least the luminance of the partial image in the LED area of the captured image. Note that the CPU 60 in this example performs control to correct the luminance of the LED area of the first captured image and record it on the recording medium 30 only when blurring is detected during the capture of the first captured image, and to record the first captured image. If no blur is detected at the time of shooting, the imaging unit 11 performs re-shooting, and the LED area of the re-shot image is combined with the first shot image and recorded on the recording medium 30. The CPU 60 constitutes LED detection means, determination means, control means, and composition means in the present invention.

  FIG. 8 is a flowchart illustrating a main part of an example of the photographing process in the fourth embodiment. This process is executed by the CPU 60 according to a program.

  Steps S10 to S26 are the same as those in the first embodiment.

  In step S80, based on the blur detection result of the blur detection unit 47, it is determined whether or not there has been a blur of the photographing apparatus 10 due to a camera shake or the like when the first photographed image is captured. A known technique may be used for blur detection. For example, blur detection may be performed using the output of a gyro sensor for camera shake correction, or blur detection may be performed by detecting a motion vector from a plurality of images (for example, live view images).

  If there is a blur at the time of imaging, the luminance (LED luminance) of the LED area in the first photographed image is compared with a threshold value (dn × SS × fn / 10 in this example) in step S82. If it is equal to or greater than the threshold value, it is determined that the brightness of the LED region is not extremely dark and the process proceeds to step S84. If it is less than the threshold value, it is determined that the brightness of the LED region is extremely dark and the process proceeds to step S86.

  In step S84, luminance correction is performed to increase the luminance of the LED area in the first photographed image.

  In step S86, the brightness of the LED region is corrected by replacing the brightness of the LED region with an image corresponding to the LED spectral characteristic (in this example, the spectral characteristic of a single color LED). This is because if the LED area is extremely dark, simply increasing the brightness will change the hue. For example, referring to the spectral characteristic detected in step S10 (step S46 in FIG. 3), the LED area of the first photographed image is replaced with an image corresponding to the spectral characteristic of the LED. Images corresponding to the LEDs of the respective colors may be stored in advance in the flash ROM 26 and replaced with images read from the flash ROM 26.

  If there is no blurring at the time of imaging in step S80, an LED area is extracted from the second captured image (step S88), and the LED area of the second captured image is extracted from the first captured image. Combining (step S90). Steps S88 to S90 and S32 are the same as steps S28 to S32 of the first embodiment shown in FIG. 2, and detailed description thereof is omitted.

  In addition, although the case where the threshold value for determining whether or not the LED region is extremely dark is 1/10 of the expected value of the LED brightness (dn × SS × fn) is shown as an example, May be used as a threshold value. The threshold is ½ or more of the expected value, and preferably １ ／ or more of the expected value.

<Fifth Embodiment>
Next, a photographing apparatus according to the fifth embodiment will be described. In addition, the point demonstrated in 1st Embodiment is abbreviate | omitted and a different point from 1st Embodiment is demonstrated.

  The CPU 60 of the fifth embodiment performs continuous shooting by the imaging unit 11 at a cycle different from the flicker cycle (flashing cycle) of the LED, and the LED image is in a lighting state among a plurality of photographed images acquired by the continuous shooting. A captured image is selected and recorded on the recording medium 30.

  FIG. 9 is a flowchart illustrating a main part of an example of the photographing process in the fifth embodiment. This process is executed by the CPU 60 according to a program.

  In FIG. 9, the CPU 60 determines whether or not the S1 switch of the operation unit 20 has been pressed (step S102). If the S1 switch has been pressed, the AE / AF circuit 54 is used to shoot AE, AF, or the like. Preparation processing is performed (step S104). Next, the CPU 60 determines whether or not the S2 switch of the operation unit 20 has been pressed (step S106) and waits until it is pressed.

  When the S2 switch is pressed (when a shooting instruction is input), a plurality of (m in this example) continuous shooting is performed by the imaging means 11 at a cycle different from the LED flicker cycle fn (step S108). The m captured images are temporarily stored in the image buffer 52 or the RAM 24.

  Next, the CPU 60 detects an LED area from the continuously shot captured images (step S110). That is, the LED area is detected from the live view image in the first embodiment (step S10 in FIG. 2), but the present embodiment is different in that the LED area is detected from the continuous shot image, but the other points are the first implementation. It is almost the same as the form.

  The CPU 60 starts a loop process with the number m of continuous shots (step S112), compares the shutter speed SS with one flicker period (1 / fn) (step S114), and determines whether or not the monochromatic LED area flag is on. (Step S116) and the comparison between the brightness of the LED area and the expected value (dn × SS × fn) (step S118). Here, if SS> 1 / fn, the monochrome LED area flag is on, and the LED brightness <expected value, the CPU 60 selects the captured image (step S120). When the loop of the continuous shooting number m ends (step S122), the CPU 60 records the selected photographed image on the recording medium 30 (step S124).

<Sixth Embodiment>
Next, the imaging device 10 according to the sixth embodiment will be described. In addition, the point demonstrated in 1st Embodiment is abbreviate | omitted and a different point from 1st Embodiment is demonstrated.

  FIG. 10 is an overall configuration diagram illustrating an example of an imaging apparatus according to the sixth embodiment. The DSP 18 of this embodiment includes an LED detection circuit 72 that detects an LED area from a captured image, and an LED area synthesis circuit 74 that combines the LED area of the second captured image with the first captured image.

  Details of the LED detection circuit 72 are shown in FIG. The LED detection circuit 72 includes an image dividing circuit 81, a flicker detection circuit 82, and a spectral characteristic detection circuit 83. The image dividing circuit 81 acquires a plurality of captured images (for example, live view images) that are temporally continuous from the image buffer 52, and divides each captured image into a plurality of divided regions arranged in two dimensions vertically and horizontally. , Output a divided image. The flicker detection circuit 82 detects flicker (flashing of LED) for each divided region based on the luminance of the divided regions (divided images) corresponding to each other over a plurality of captured images, and flicker frequency fn and luminance peak dn. Is output. The spectral characteristic detection circuit 83 detects whether or not the spectral characteristic of the divided area (LED area) captured in the lighting state is the spectral characteristic of the LED. The CPU 60 of the present embodiment performs the LED detection process shown in FIG. 3 using the LED detection circuit 72 of FIG. Further, the CPU 60 of the present embodiment performs image composition using the LED area composition circuit 74.

<Seventh embodiment>
Next, the imaging device 10 according to the seventh embodiment will be described. Below, a different point from 4th Embodiment is demonstrated.

  FIG. 12 is an overall configuration diagram illustrating an example of an imaging device according to the seventh embodiment. The DSP 18 of this embodiment includes an LED detection circuit 72, an LED area synthesis circuit 74, and a luminance correction circuit 76. Note that the LED detection circuit 72 and the LED region synthesis circuit 74 are the same as those in the sixth embodiment, and a description thereof will be omitted.

  Details of the luminance correction circuit 76 are shown in FIG. The luminance correction circuit 76 includes an LED luminance determination unit 84, a spectral characteristic replacement unit 85, and an LED area luminance correction unit 86. The LED luminance determination unit 84 acquires a partial image (LED region image) of the LED region from the image buffer 52, and compares the luminance (LED luminance) of the LED region with a threshold (for example, dn × SS × fn / 10). If it is less than the threshold, the spectral replacement flag is set to ON, and if it is greater than or equal to the threshold, the spectral replacement flag is set to OFF. The spectral characteristic replacement unit 85 corrects the luminance of the LED region image based on the LED spectral characteristics when the spectral replacement flag is on. The LED area brightness correction unit 86 performs brightness correction to increase the brightness of the LED area when the spectral replacement flag is off. The CPU 60 of the present embodiment performs the photographing process shown in FIG.

  In addition, although it divided and demonstrated to 7th Embodiment from 1st Embodiment, embodiment of these combinations may be implemented. For example, in the photographing apparatus of the fourth embodiment shown in FIG. 6, the LED detection circuit 72 and the LED area synthesis circuit 74 shown in FIG. 10 are provided to perform LED detection and LED area synthesis similar to those of the sixth embodiment. Alternatively, the brightness correction circuit 76 shown in FIG. 12 may be provided to perform brightness correction similar to that in the seventh embodiment.

  Further, in the first to seventh embodiments, the case where the LED that may exist as an object in the shooting scene is a single color LED (for example, blue, green, and red LEDs) has been described as an example. Is not particularly limited to such a case. For example, when there is a possibility that an LED in which a plurality of colors (a plurality of colors having different wavelengths) are mixed, such as a white LED, may exist as an object in the shooting scene, the LED region is detected based on the spectral characteristics of the white LED. Should be done.

  In addition, the present invention is not particularly limited when implemented with a photographing apparatus such as a digital camera. For example, the present invention may be implemented by using a computer apparatus such as a personal computer as the image processing apparatus of the present invention.

  The present invention is not limited to the examples described in the present specification and the examples illustrated in the drawings, and various design changes and improvements may be made without departing from the spirit of the present invention.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Imaging means, 12 ... Imaging optical system, 14 ... Imaging element, 16 ... AFE (analog signal processing part), 18 ... DSP (digital signal processing part), 20 ... Operation part, 22 ... LCD, DESCRIPTION OF SYMBOLS 30 ... Recording medium, 43 ... Iris (aperture), 46 ... Lens driver, 47 ... Blur detection part, 52 ... Image buffer, 60 ... CPU

Claims (24)

Imaging means capable of continuously imaging a subject;
Instruction input means for accepting input of shooting instructions;
LED detection means for detecting an LED region including LED light that periodically blinks from an image captured by the imaging means;
Determination means for determining whether the LED area is imaged in a lighting state or in a light extinction state;
Control means for photographing at least one sheet by the imaging means in response to the photographing instruction, wherein the LED area is detected by the LED detection means, and the LED area of the first photographed image is obtained by the determination means. When it is determined that the image is taken in the extinguished state, a control unit that takes a second image with the imaging unit with an exposure time of one cycle or more of the blinking cycle of the LED;
Combining means for combining the LED area of the second photographed image with the first photographed image;
An imaging apparatus comprising: Imaging means capable of continuously imaging a subject;
Instruction input means for accepting input of shooting instructions;
LED detection means for detecting an LED region including LED light that periodically blinks from an image captured by the imaging means;
Determination means for determining whether the LED area is imaged in a lighting state or in a light extinction state;
Control means for photographing at least one sheet by the imaging means in response to the photographing instruction, wherein the LED area is detected by the LED detection means, and the LED area of the first photographed image is detected by the determination means. When it is determined that the image is taken in the extinguished state, the control unit repeats the second and subsequent photographing by the imaging unit until the LED area is imaged in the lighting state;
An imaging apparatus comprising: A compositing means capable of compositing a partial image of the second and subsequent captured images with the first captured image;
The control means separates the partial image of the LED area from the second and subsequent photographed images taken with the LED area lit, and the synthesized partial image is separated by the synthesizing means. The photographing apparatus according to claim 2, wherein the photographing device is combined with the first photographed image. A compositing means capable of compositing a partial image of the second and subsequent captured images with the first captured image;
The imaging element has a light receiving pixel surface on which a plurality of photoelectric conversion elements are arranged and can partially read an arbitrary partial region from the maximum readable region of the light receiving pixel surface;
The control means controls partial reading of reading only a partial area corresponding to the LED area in the maximum area of the imaging element in the second and subsequent imaging after the imaging instruction. The photographing apparatus according to claim 2, wherein a partial image of the LED region that is a captured image and is partially read out after being lit is combined with the first captured image. Imaging means capable of continuously imaging a subject;
Instruction input means for accepting input of shooting instructions;
LED detection means for detecting an LED region including LED light that periodically blinks from an image captured by the imaging means;
Determination means for determining whether the LED area is imaged in a lighting state or in a light extinction state;
Control means for photographing at least one sheet by the imaging means in response to the photographing instruction, wherein the LED area is detected by the LED detection means, and the LED area of the first photographed image is detected by the determination means. Control means for performing control to correct at least the luminance of the partial image of the LED region of the captured image when it is determined that the image is captured in the extinguished state;
An imaging apparatus comprising: Blur detection means for detecting blur at the time of imaging;
Combining means capable of synthesizing a partial image of the second and subsequent photographed images with the first photographed image after the photographing instruction;
The control means controls the luminance of the partial image of the LED area only when it is determined that the LED area of the photographed image is captured in the extinguished state and a blur at the time of imaging is detected, and the LED area of the photographed image Even if it is determined that the image is captured in the extinguished state, if the blur at the time of imaging is not detected, the LED area of the recaptured image is combined with the first captured image by the combining means. The imaging device according to claim 5.   The control means, when it is determined that the LED area of the photographed image is captured in the extinguished state and a blur at the time of photographing is detected, compares the brightness of the LED area of the photographed image with a threshold value, and more than the threshold value The brightness of the partial image of the LED region is controlled by replacing the partial image of the LED region of the photographed image with an image corresponding to the spectral characteristic of the LED when the luminance is low. Shooting device.   8. The photographing apparatus according to claim 1, wherein the control unit sets the same exposure time as the blinking cycle of the LED at the time of re-photographing after the photographing instruction. 9.   5. The control unit according to claim 2, wherein the re-shooting after the shooting instruction sets the exposure time, the aperture amount, and the sensitivity to the same values as in the first shooting. The imaging device described in 1. Display means for displaying a moving image captured by the imaging means as a live view image before inputting the shooting instruction;
10. The photographing apparatus according to claim 1, wherein the LED detection unit detects the LED region from a plurality of frames of the live view image displayed on the display unit. 11. Imaging means capable of continuously imaging a subject;
Instruction input means for accepting input of shooting instructions;
In response to the input of the shooting instruction, control means for performing continuous shooting by the imaging means at a cycle different from the blinking cycle of the LED that blinks periodically;
LED detection means for detecting an LED region including the periodically flashing LED light from a plurality of captured images obtained by continuous shooting of the imaging means;
Determination means for determining whether the LED area is imaged in a lighting state or in a light extinction state;
Selecting means for selecting a photographed image determined to have been imaged while the LED region is lit from the plurality of photographed images;
An imaging apparatus comprising:   The LED detection unit divides an image continuously captured by the imaging unit into a plurality of divided regions, detects periodic blinking of the LED for each divided region, and the division in which the blinking is detected. 12. The LED area is detected by detecting spectral characteristics of the area and determining whether or not the spectral characteristics are spectral characteristics of the LED. The imaging device according to item.   The determination means is a threshold value indicating a lower limit value of the luminance of the LED area when the LED area is detected in a lighting state, based on the exposure time of the imaging means. The photographing according to any one of claims 1 to 12, wherein by comparing with the calculated threshold value, it is determined whether the LED region is imaged in a lighting state or in a light extinction state. apparatus. Image acquisition means for acquiring a plurality of photographed images continuously shot at a different period from the blinking period of the periodically blinking LED;
LED detection means for detecting an LED region including the periodically flashing LED light from a plurality of photographed images obtained by the image acquisition means;
Determination means for determining whether the LED area is imaged in a lighting state or in a light extinction state;
Selecting means for selecting a photographed image determined to have been imaged while the LED region is lit from the plurality of photographed images;
An image processing apparatus comprising:   The LED detection unit divides a plurality of captured images continuously shot by the imaging unit into a plurality of divided regions to detect periodic blinking of the LED for each divided region, and the blinking is detected. The image processing according to claim 14, wherein the LED region is detected by detecting a spectral characteristic of the divided area and determining whether the spectral characteristic is a spectral characteristic of the LED. apparatus. An imaging method that uses an imaging unit capable of continuously imaging a subject and an instruction input unit that receives an input of an imaging instruction,
Detecting an LED region including periodically flashing LED light from an image captured by the imaging means;
Determining whether the LED area of the first photographed image is imaged in the lit state or in the extinguished state by performing the first photographing by the imaging unit in response to the photographing instruction;
When it is determined that the LED area of the first photographed image is imaged in the extinguished state, the second image is photographed by the imaging means with an exposure time of one cycle or more of the blinking cycle of the LED;
Combining the LED area of the second photographed image with the first photographed image;
A photographing method characterized by comprising: An imaging method that uses an imaging unit capable of continuously imaging a subject and an instruction input unit that receives an input of an imaging instruction,
Detecting an LED region including periodically flashing LED light from an image captured by the imaging means;
Determining whether the LED area of the first photographed image is imaged in the lit state or in the extinguished state by performing the first photographing by the imaging unit in response to the photographing instruction;
When it is determined that the LED area of the first photographed image has been captured in a light-off state, the second and subsequent imaging steps are repeated by the imaging unit until the LED area is imaged in a lighting state;
A photographing method characterized by comprising:   The partial image of the LED area is separated from the second and subsequent captured images captured with the LED area turned on, and the separated partial image is combined with the first captured image. The imaging method according to claim 17, wherein: The imaging element has a light receiving pixel surface on which a plurality of photoelectric conversion elements are arranged and can partially read an arbitrary partial region from the maximum readable region of the light receiving pixel surface;
In the second and subsequent images after the imaging instruction, partial readout for reading out only a partial region corresponding to the LED region in the maximum region of the image sensor is controlled, and the second and subsequent images are captured. The photographing method according to claim 17, wherein a partial image of the LED region that is captured in a lighting state and partially read out is combined with the first photographed image. An imaging method that uses an imaging unit capable of continuously imaging a subject and an instruction input unit that receives an input of an imaging instruction,
Detecting an LED region including periodically flashing LED light from an image captured by the imaging means;
Determining whether the LED area of the first photographed image is imaged in the lit state or in the extinguished state by performing the first photographing by the imaging unit in response to the photographing instruction;
When it is determined that the LED area of the first photographed image is captured in a light extinguished state, at least the luminance of the partial image in the LED area of the photographed image is corrected. Using blur detection means to detect blur during imaging,
Only when it is determined that the LED area of the photographed image has been captured in the extinguished state and a blur at the time of imaging is detected, the brightness of the partial image in the LED area is controlled, and the LED area of the captured image is in the extinguished state. 21. The method according to claim 20, wherein, even if it is determined that the image has been picked up, if the blur at the time of image pick-up is not detected, the combining unit combines the LED region of the re-taken image with the first picked-up image. Shooting method.   When it is determined that the LED area of the photographed image is captured in the extinguished state and a blur at the time of imaging is detected, the brightness of the LED area of the photographed image is compared with a threshold value, and the brightness is smaller than the threshold value 23. The method according to claim 21, wherein a partial image of the LED region of the photographed image is replaced with an image corresponding to the spectral characteristic of the LED, and the luminance of the LED region of the photographed image is increased when the image is equal to or greater than the threshold value. The shooting method described. An imaging method that uses an imaging unit capable of continuously imaging a subject and an instruction input unit that receives an input of an imaging instruction,
In response to the input of the shooting instruction, the step of continuously shooting by the imaging means at a cycle different from the flashing cycle of the LED that flashes periodically;
Detecting a LED region including the periodically flashing LED light from a plurality of captured images obtained by continuous shooting of the imaging means;
Determining whether the LED region was imaged in a lit state or imaged in a extinguished state;
Selecting a captured image determined to have been captured in a lighting state from the plurality of captured images;
A photographing method characterized by comprising: Using image acquisition means for acquiring a plurality of captured images continuously shot at a period different from the blinking period of the periodically blinking LED,
Detecting a LED region including the periodically flashing LED light from a plurality of photographed images obtained by the image acquisition means;
Determining whether the LED region was imaged in a lit state or imaged in a extinguished state;
Selecting a photographed image determined to have been imaged while the LED region is lit from the plurality of photographed images;
An image processing method comprising:

Images