JP2018148500A - Imaging system, image determination program, image imaging device, image data structure - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging system, image determination program, image imaging device, and image data structure that can easily select a necessary image from a plurality of images.SOLUTION: An imaging system includes: position changing means for changing relative positions between a subject image entering an imaging element via an imaging optical system and the imaging element in a plane parallel to a light-receiving face of the imaging element; image recording means for obtaining two or more prescribed images by taking an image of the subject image with the imaging element and recording an image group including the prescribed images; positional information acquisition means for acquiring information on the relative positions between the subject image and the imaging element when the prescribed images were taken; and positional information addition means for adding the information on the relative positions to the prescribed images.SELECTED DRAWING: Figure 11

Description

  The present invention relates to an imaging system that captures an image, an image determination program that determines an image, an image capturing apparatus that captures image data, and an image data structure.

  It is known that when a subject including a high frequency component equal to or higher than the pixel pitch of the image sensor is picked up, moire such as false color (color moire) is generated and the captured image is deteriorated. Therefore, various techniques for removing this type of false color have been proposed. For example, Patent Document 1 describes a specific configuration of a photographing apparatus capable of detecting a false color generated in a photographed image. Patent Document 2 describes a specific configuration of an imaging apparatus that can reduce false colors.

  The imaging device described in Patent Document 1 captures an in-focus subject, and then captures an out-of-focus subject. In the out-of-focus photographed image, the contrast of the subject is reduced and high-frequency components are reduced, so that the false color generated in the in-focus photographed image is also reduced. Therefore, the imaging device described in Patent Document 1 divides a captured image into a plurality of blocks, and calculates a difference value between the color difference signal of the focused captured image and the color difference signal of the unfocused captured image for each block. Then, a block having a large difference value is detected as a block in which a false color is generated.

  The imaging apparatus described in Patent Document 2 captures a subject in a state in which a part of the imaging lens (anti-vibration lens) is vibrated in a plane perpendicular to the optical axis after imaging the subject once. Although there is no image blur in the captured image obtained in the first shooting, there is a possibility that a false color is generated. On the other hand, image blurring has occurred in a photographed image taken while vibrating the anti-vibration lens, but false colors are reduced. In the photographing apparatus described in Patent Document 2, a photographed image in which both image blur and false color are suppressed is obtained by combining the two photographed images.

JP 2011-109496 A JP 2008-182486 A

  As described above, in the photographing apparatuses described in Patent Document 1 and Patent Document 2, false colors are detected using a plurality of photographed images with different photographing states (in-focus state and presence / absence of vibration of the vibration-proof lens), or Reduces false colors. However, image processing such as false color detection processing and false color reduction processing is not always performed by the photographing apparatus immediately after photographing a plurality of photographed images. For example, it is assumed that only a plurality of captured images are captured and then image processing is performed at a timing desired by the user, or an external device other than the capturing device performs image processing. In this case, the user has to search for and select a captured image to be used for image processing from various captured images.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an imaging system, an image determination program, and an image imaging system that can easily select a required image from a plurality of images. To provide a device and data structure.

  An imaging system according to an embodiment of the present invention changes a relative position of a subject image incident on an imaging device via an imaging optical system and an imaging device in a plane parallel to a light receiving surface of the imaging device. A position changing means for capturing, an image recording means for capturing a subject image using an image sensor to acquire two or more predetermined images, and recording an image group including the predetermined images, and the predetermined images are captured Position information acquisition means for acquiring information regarding the relative position between the subject image and the image sensor, and position information addition means for adding information regarding the relative position to a predetermined image.

  According to an embodiment of the present invention, a necessary image can be easily selected from a plurality of images.

It is a block diagram which shows the structure of the imaging device of embodiment of this invention. 1 is a diagram schematically illustrating a configuration of an image shake correction apparatus provided in an imaging apparatus according to an embodiment of the present invention. 1 is a diagram schematically illustrating a configuration of an image shake correction apparatus provided in an imaging apparatus according to an embodiment of the present invention. It is a figure which shows the data structure of the image data produced | generated (recorded) with the imaging device of embodiment of this invention. It is a figure which assists description of the LPF drive in embodiment of this invention. It is a figure which shows the false color detection flow by the system controller in embodiment of this invention. It is a figure which shows the processing flow of the imaging | photography process by the system controller in embodiment of invention. It is a figure which shows the processing flow of the detection process by the system controller in embodiment of invention. It is a figure regarding the to-be-photographed object image | photographed in embodiment of this invention. It is a figure which shows the data structure of the image data produced | generated (recorded) by imaging | photography process in embodiment of this invention. It is a figure regarding the to-be-photographed object in embodiment of this invention. It is a figure which shows each color difference signal plotted in the color space of embodiment of this invention.

  Hereinafter, an imaging system, an image determination program, an image imaging device, and an image data structure according to an embodiment of the present invention will be described with reference to the drawings. In the following, a digital single lens reflex camera will be described as an embodiment of the imaging system and the imaging apparatus of the present invention. The imaging system and the imaging device of the present invention are not limited to a digital single lens reflex camera, but include, for example, a mirrorless single lens camera, a compact digital camera, a video camera, a camcorder, a tablet terminal, a PHS (Personal Handy phone System), a smartphone, It may be replaced with another form of device having an imaging function, such as a feature phone or a portable game machine. The imaging system may be a combination of an image imaging device having an imaging (imaging) function and an image processing device that processes image data generated (recorded) by the image imaging device. As the image processing device, for example, a device having an information processing function such as a personal computer, a server, a tablet terminal, or a smartphone is used. An image determination program according to an embodiment of the present invention is executed by an imaging system, an image imaging device, and an image processing device. The image data having the image data structure according to the embodiment of the present invention is generated (recorded) or processed by an imaging system, an image imaging device, or an image processing device.

  FIG. 1 is a block diagram showing a configuration of a digital single-lens reflex camera (hereinafter referred to as “imaging device”) 1 and an image processing device 3 of the present embodiment. As shown in FIG. 1, the photographing apparatus 1 includes a system controller 100, an operation unit 102, a drive circuit 104, a photographing lens 106, a diaphragm 108, a shutter 110, an image shake correction device 112, a signal processing circuit 114, and an image processing engine 116. , Buffer memory 118, card interface 120, LCD (Liquid Crystal Display) control circuit 122, LCD 124, ROM (Read Only Memory) 126, gyro sensor 128, acceleration sensor 130, geomagnetic sensor 132, GPS (Global Positioning System) sensor 134 And a communication interface 136. Although the photographing lens 106 has a plurality of lenses, it is shown as a single lens for convenience in FIG. Further, the same direction as the optical axis AX of the photographing lens 106 is defined as the Z-axis direction, and the two axial directions orthogonal to the Z-axis direction and orthogonal to each other are the X-axis direction (horizontal direction) and the Y-axis direction (vertical direction), respectively. It is defined as In the following, the positive X-axis direction is called right, the negative X-axis direction is left, the positive Y-axis direction is up, and the positive Y-axis direction is down.

  As illustrated in FIG. 1, the image processing apparatus 3 includes a controller 300, a memory 301, an LCD 302, a card interface 303, and a communication interface 304.

  The operation unit 102 includes various switches necessary for the user to operate the photographing apparatus 1, such as a power switch, a release switch, and a photographing mode switch. When the LCD 124 is a touch panel that accepts user operations, the operation unit 102 includes the LCD 124. When the user operates the power switch, power is supplied from the battery (not shown) to the various circuits of the photographing apparatus 1 through the power line.

  The system controller 100 includes a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). After supplying power, the system controller 100 accesses the ROM 126, reads out a control program, loads it into a work area (not shown), and executes the loaded control program to control the entire photographing apparatus 1.

  When the release switch is operated, the system controller 100 detects, for example, a photometric value calculated based on an image captured by a solid-state imaging device 112a (see FIG. 2 described later) or an exposure meter built in the photographing apparatus 1. The diaphragm 108 and the shutter 110 are driven and controlled via the drive circuit 104 so that proper exposure is obtained based on the photometric value measured in (not shown). More specifically, drive control of the aperture 108 and the shutter 110 is performed based on an AE function designated by a shooting mode switch such as a program AE (Automatic Exposure), shutter priority AE, aperture priority AE, and the like. Further, the system controller 100 performs AF (Autofocus) control together with AE control. For AF control, an active method, a phase difference detection method, an image plane phase difference detection method, a contrast detection method, or the like is applied. In addition, the AF mode includes a central single-point ranging mode using a single central ranging area, a multi-point ranging mode using a plurality of ranging areas, a full-screen ranging mode based on full-screen distance information, and the like. is there. The system controller 100 controls driving of the photographing lens 106 via the driving circuit 104 based on the AF result, and adjusts the focus of the photographing lens 106. Since the configuration and control of this type of AE and AF are well known, detailed description thereof is omitted here.

  2 and 3 are diagrams schematically showing a configuration of the image blur correction device 112. FIG. As shown in FIGS. 2 and 3, the image blur correction device 112 includes a solid-state image sensor 112a. The light beam from the subject passes through the photographing lens 106, the diaphragm 108, and the shutter 110, and is received by the light receiving surface 112aa of the solid-state imaging device 112a. The light receiving surface 112aa of the solid-state image sensor 112a is parallel to the XY plane including the X axis and the Y axis. The solid-state imaging device 112a is a single-plate color CCD (Charge Coupled Device) image sensor having a color filter with a Bayer pixel arrangement as a transmission wavelength selection device. The solid-state imaging device 112a accumulates an optical image formed by each pixel on the light receiving surface 112aa as a charge corresponding to the amount of light, and generates R (Red), G (Green), and B (Blue) image signals. Output. Note that the solid-state imaging device 112a is not limited to a CCD image sensor, and may be replaced with a complementary metal oxide semiconductor (CMOS) image sensor or other types of imaging devices. The solid-state imaging device 112a may be a filter having a complementary color filter, or a filter having a periodic color arrangement such as a Bayer arrangement if any color filter is arranged for each pixel. There is no need.

  The signal processing circuit 114 performs predetermined signal processing such as clamping and demosaicing (color interpolation) on the image signal input from the solid-state imaging device 112 a and outputs the processed signal to the image processing engine 116. The image processing engine 116 performs predetermined signal processing such as matrix operation, Y / C separation, and white balance on the image signal input from the signal processing circuit 114 to generate a luminance signal Y and color difference signals Cb and Cr. The image data is generated (recorded) after being compressed in a predetermined format such as JPEG (Joint Photographic Experts Group). The buffer memory 118 is used as a temporary storage location for processing data when the image processing engine 116 executes processing. Further, the format of the image data is not limited to the JPEG format, and may be a RAW format in which only minimal image processing (for example, clamping) is performed.

  In addition, metadata such as information for identifying a shooting condition and a shot image is added to the image data. FIG. 4 is a diagram illustrating a data structure of image data. The image data includes a photographed subject image and metadata related to the image. The metadata includes, for example, the shooting date and time, the file name of the image data, the resolution of the shot image, the exposure time, the aperture value, the white balance setting value, and the like. The metadata complies with, for example, an Exif (Exchangeable image file format) format.

  A memory card 200 is detachably inserted into a card slot of the card interface 120.

  The image processing engine 116 can communicate with the memory card 200 via the card interface 120. The image processing engine 116 stores the generated (recorded) image data in the memory card 200 as an image file. Further, the image data may be stored in a built-in memory (not shown) provided in the photographing apparatus 1. Alternatively, the memory card 200 storing the image data may be attached to the card interface 303 of the external image processing apparatus 3. Further, the image data stored in the memory card 200 may be stored in the memory 301. Alternatively, when the image capturing apparatus 1 and the image processing apparatus 3 are connected to each other via the communication interfaces 136 and 304, the image data is transferred from the image capturing apparatus 1 to the image processing apparatus 3 via the communication interfaces 136 and 304. May be sent. The communication interface 136 and the communication interface 304 may be connected by a cable or may be connected by wireless communication.

  Further, the image processing engine 116 buffers the generated luminance signal Y and color difference signals Cb and Cr in a frame memory (not shown) in units of frames. The image processing engine 116 sweeps the buffered signal from each frame memory at a predetermined timing, converts it into a video signal of a predetermined format, and outputs it to the LCD control circuit 122. The LCD control circuit 122 modulates and controls the liquid crystal based on the image signal input from the image processing engine 116. Thereby, the photographed image of the subject is displayed on the display screen of the LCD 124. The user can view through a display screen of the LCD 124 a real-time through image (live view) captured with appropriate brightness and focus based on AE control and AF control.

  When the user performs a playback operation of the captured image, the image processing engine 116 reads the image data designated by the operation from the memory card 200 or the built-in memory, converts the image data into an image signal of a predetermined format, and sends it to the LCD control circuit 122. Output. The LCD control circuit 122 performs modulation control on the liquid crystal based on the image signal input from the image processing engine 116, so that a captured image of the subject is displayed on the display screen of the LCD 124.

(Explanation regarding the drive of the shake correction member)
The image shake correction device 112 drives a shake correction member. In the present embodiment, the shake correction member is the solid-state image sensor 112a. Note that the shake correction member is not limited to the solid-state image sensor 112a, but is partially moved with respect to the optical axis AX, such as a part of lenses included in the photographing lens 106, so that the light-receiving surface 112aa of the solid-state image sensor 112a. Another configuration capable of shifting the incident position of the subject image above may be used, or a configuration in which two or more members of these and the solid-state imaging device 112a are combined may be used.

  The image blur correction device 112 only drives (vibrates) the shake correction member minutely in a plane orthogonal to the optical axis AX (that is, in the XY plane) in order to correct image blur (cancel camera shake and camera shake). In addition, the shake correction member is orthogonal to the optical axis AX so that an optical low pass filter (LPF) effect (reduction of moiré such as false color) is obtained by blurring the subject image by the pixel pitch. Drive minutely (micro-rotation) in the plane. Hereinafter, for convenience of description, driving the shake correction member with image shake correction is referred to as “image shake correction drive”, and driving the shake correction member so as to obtain the same effect as an optical LPF is referred to as “LPF”. "Drive".

(Explanation on image blur correction drive)
The gyro sensor 128 is a sensor that detects information for controlling image blur correction. Specifically, the gyro sensor 128 detects angular velocities around the two axes (around the X axis and around the Y axis) applied to the imaging apparatus 1, and the detected angular velocities around the two axes are within the XY plane (in other words, a solid state This is output to the system controller 100 as a shake detection signal indicating the shake of the light receiving surface 112aa of the image sensor 112a.

  As shown in FIGS. 2 and 3, the image blur correction device 112 includes a fixed support substrate 112 b fixed to a structure such as a chassis included in the photographing device 1. The fixed support substrate 112b slidably supports the movable stage 112c on which the solid-state imaging device 112a is mounted.

Magnets M _YR , M _YL , M _XD , and M _XU are attached on the surface of the fixed support substrate 112 b facing the movable stage 112 c. In addition, yokes Y _YR , Y _YL , Y _XD , and Y _{XU that} are magnetic bodies are attached to the fixed support substrate 112b. The yokes Y _YR , Y _YL , Y _XD , and Y _XU each have a shape that extends from the fixed support substrate 112 b to the position facing the magnets M _YR , M _YL , M _XD , and M _XU around the movable stage 112 c, A magnetic circuit is formed between the magnets M _YR , M _YL , M _XD , and M _XU . In addition, driving coils C _YR , C _YL , C _XD , and C _XU are attached to the movable stage 112c. When the driving coils C _YR , C _YL , C _XD , and C _XU receive a current in the magnetic field of the magnetic circuit, a driving force is generated. The movable stage 112c (solid-state imaging device 112a) is minutely driven in the XY plane with respect to the fixed support substrate 112b by the generated driving force.

Corresponding magnets, yokes and driving coils constitute a voice coil motor. Hereinafter, for the sake of convenience, a reference numeral VCM _YR is given to a voice coil motor made up of a magnet M _YR , a yoke Y _YR and a driving coil C _YR , and a voice coil motor made up of a magnet M _YL , a yoke Y _YL and a driving coil C _YL is called up. given the VCM _YL, a voice coil motor reference numeral _{VCM XD} magnet _{M XD,} the voice coil motor consisting of a yoke _{Y XD} and the driving coil _{C XD,} consisting of the magnet _{M XU,} yoke _{Y XU} and the driving coil _{C XU} The symbol VCM _XU is appended.

Each voice coil motor VCM _YR , VCM _YL , VCM _XD , VCM _XU (driving coils C _YR , C _YL , C _XD , C _XU ) is PWM (Pulse Width Modulation) driven under the control of the system controller 100. The voice coil motors VCM _YR and VCM _YL are arranged below the solid-state image sensor 112a and arranged side by side with a predetermined interval in the horizontal direction (X-axis direction). The voice coil motors VCM _XD and VCM _XU are On the side of the solid-state image sensor 112a, they are arranged side by side with a predetermined interval in the vertical direction (Y-axis direction).

Hall elements H _YR , H _YL , H _XD , and H _XU are attached to the positions near the driving coils C _YR , C _YL , C _XD , and C _XU on the fixed support substrate 112b. The Hall elements H _YR , H _YL , H _XD , and H _XU detect the magnetic _forces of the magnets M _YR , M _YL , M _XD , and M _XU , respectively, and the position of the movable stage 112c (solid-state imaging device 112a) in the XY plane. Is output to the system controller 100. Specifically, the Y-axis direction position and tilt (rotation) of the movable stage 112c (solid-state imaging device 112a) are detected by the Hall elements H _YR and H _YL , and the movable stage 112c (solid-state imaging) is detected by the Hall elements H _XD and H _XU. The X-axis direction position and inclination (rotation) of the element 112a) are detected.

The system controller 100 includes a driver IC for a voice coil motor. The system controller 100 determines the rated power (allowable power) of the driver IC based on the shake detection signal output from the gyro sensor 128 and the position detection signal output from the Hall elements H _YR , H _YL , H _XD , and H _XU. Calculate the duty ratio so as not to disturb the balance of the current flowing through each voice coil motor VCM _YR , VCM _YL , VCM _XD , VCM _XU (drive coil C _YR , C _YL , C _XD , C _XU ) within the range not exceeding To do. The system controller 100 sends a drive current to each of the voice coil motors VCM _YR , VCM _YL , VCM _XD , and VCM _XU with the calculated duty ratio to drive the solid-state imaging device 112 a with image blur correction. As a result, the image blur on the light receiving surface 112aa of the solid-state image sensor 112a is corrected while the solid-state image sensor 112a is held at a predetermined position against gravity, disturbance, or the like (in other words, on the light receiving surface 112aa). The position of the solid-state imaging device 112a is adjusted so that the incident position of the subject image does not fluctuate.

(Explanation regarding LPF drive)
Next, the LPF driving will be described. In the present embodiment, the image blur correction device 112 causes a predetermined drive current to flow through the voice coil motors VCM _YR , VCM _YL , VCM _XD , and VCM _XU , so that the image shake correction device 112 is predetermined in the XY plane for one exposure period. The movable stage 112c (solid-state image sensor 112a) is driven so as to draw a trajectory, and the subject image is incident on a plurality of pixels having different detection colors (R, G, or B) of the solid-state image sensor 112a. Thereby, the same effect as an optical LPF can be obtained.

  FIG. 5A and FIG. 5B are diagrams for assisting the description of the LPF drive. As shown in the figure, on the light receiving surface 112aa of the solid-state imaging device 112a, a plurality of pixels PIX are arranged in a matrix at a predetermined pixel pitch P. For convenience of explanation, each pixel PIX in the figure is given a reference (any one of R, G, and B) corresponding to the filter color arranged on the front surface.

  FIG. 5A shows an example in which the solid-state imaging device 112a is driven so as to draw a square locus centered on the optical axis AX. For example, the square locus can be a square closed path with the pixel pitch P of the solid-state imaging element 112a as one side. In the example of FIG. 5A, the solid-state imaging device 112a is driven so as to alternately form a square path in units of one pixel pitch P in the X-axis direction and the Y-axis direction.

FIG. 5B shows an example in which the solid-state imaging device 112a is driven so as to draw a rotationally symmetric circular locus centered on the optical axis AX. This circular locus can be, for example, a circular closed path having a radius r of (P / 2) × ^2½ .

  Note that information on the driving locus including the pixel pitch P is stored in advance in the internal memory of the system controller 100 or the ROM 126.

  As illustrated in FIG. 5A (or FIG. 5B), during the exposure period, the solid-state imaging device 112a is driven to draw a predetermined square locus (or circular locus) based on the information of the drive locus. Then, the subject image is uniformly incident on the four color filters R, G, B, and G (four (two rows and two columns) pixels PIX). Thereby, an effect equivalent to that of an optical LPF can be obtained. In other words, since the subject image incident on any color filter (pixel PIX) is necessarily incident on the surrounding color filter (pixel PIX), the same effect as when the subject image passes through the optical LPF. (Reduction of moiré such as false color) is obtained.

  Note that the user can switch on / off of each setting of image blur correction driving and LPF driving by operating the operation unit 102.

  Further, the user operates the operation unit 102 so that the photographing apparatus 1 continuously performs photographing with the LPF driving turned on and photographing with the LPF driving turned off (LPF driving auto bracket photographing). Can be set. In the auto bracket shooting of LPF drive, both the image data shot with the LPF drive turned on and the image data shot with the LPF drive turned off can be obtained by operating the release switch once. Therefore, the user does not need to determine whether or not the LPF drive setting is turned on every time shooting is performed, and the time and effort during shooting can be reduced.

(Explanation regarding false color detection)
Next, a method for detecting a false color generated in a captured image in the present embodiment will be described. The user can switch on / off the setting of the false color detection process by operating the operation unit 102.

  FIG. 6 shows a false color detection flow executed by the system controller 100. The false color detection flow includes an image data photographing process step S1 and a false color detection process step S2 using the image data. FIG. 7 and FIG. 8 show the processing flow of the imaging processing step S1 and the processing flow of the detection processing step S2, respectively. In the present embodiment, a case will be described in which both the imaging processing step S1 and the detection processing step S2 are executed by the system controller 100 of the imaging device 1. The false color detection flow shown in FIG. 6 is started, for example, when the release switch is pressed while the false color detection process is set to ON. In the false color detection flow, the detection processing step S2 may be executed by the controller 300 of the image processing device 3 outside the imaging device 1.

[S1 in FIG. 6 (shooting process)]
The photographing process step S1 shown in FIG. 6 will be described with reference to FIG.

[S11 in FIG. 7 (determination of state)]
In this processing step S11, it is determined whether or not the photographing apparatus 1 is in a stationary state. Illustratively, when the amplitude of a signal component having a frequency equal to or higher than a certain frequency among the vibration detection signals input from the gyro sensor 128 is within a certain threshold continuously for a certain period, it is determined that the camera is stationary. A typical example of the stationary state of the photographing apparatus 1 is a state in which the photographing apparatus 1 is fixed to a tripod. Note that the posture of the photographing apparatus 1 including the stationary state may be detected using information output from other sensors such as the acceleration sensor 130, the geomagnetic sensor 132, and the GPS sensor 134 instead of the gyro sensor 128. In order to improve the detection accuracy, for example, sensor fusion technology may be applied so that information output from these sensors may be used in combination.

  When the photographing apparatus 1 is not in a stationary state, such as a hand-held photographing state with a low shutter speed setting, a false color is unlikely to occur in the first place due to blurring of the subject due to camera shake (or subject shake). Therefore, in this embodiment, only when it is determined that the photographing apparatus 1 is in a stationary state (that is, in a state in which false color is likely to occur) (S11: YES), the false color in the photographed image is detected. Therefore, processing step S12 (photographing of the first image) and subsequent steps are executed. When the photographing apparatus 1 is not in a stationary state, the processing load of the system controller 100 is reduced because the processing step S12 (first image photographing) and subsequent steps are not executed. The photographer may be notified, for example, via the display screen of the LCD 124 that the processing step S12 (photographing the first image) and subsequent steps are not executed. Note that, when it is important to detect the false color in the photographed image, the processing step S12 (photographing the first image) and subsequent steps may be executed regardless of whether or not the photographing apparatus 1 is in a stationary state. In this case, the photographer may be notified through the display screen of the LCD 124 that the processing step S12 (photographing the first image) and subsequent steps are executed. Further, the determination threshold value for the still state may be changed according to the shutter speed set in the photographing apparatus 1.

[S12 in FIG. 7 (shooting the first image)]
FIGS. 9A and 10A are enlarged views showing different parts of the subject to be photographed. The subject shown in FIG. 9A is an oblique stripe pattern in which light and dark appear alternately at the same pitch as the pixel pitch P of the pixel PIX of the solid-state image sensor 112a, and the subject shown in FIG. It is a vertical stripe pattern in which light and dark appear alternately at the same pitch as P. For convenience of explanation, a 6 × 6 square subject surrounded by a thick solid line among the subjects shown in FIG. 9A is referred to as “obliquely striped subject 9a”, and a thick solid line among the subjects shown in FIG. The subject of 6 × 6 square surrounded by is denoted as “vertical stripe subject 10a”.

  In this processing step S12, a subject image (first image) is photographed (one exposure period is completed) with appropriate brightness and focus based on AE control and AF control. Here, it is assumed that both the diagonally striped subject 9a and the vertically striped subject 10a are in focus.

  FIGS. 9B and 10B are diagrams schematically showing the diagonally striped subject 9a and the vertically striped subject 10a captured by each pixel PIX of the solid-state image sensor 112a, and the light receiving surface 112aa of the solid-state image sensor 112a. Is a front view from the subject side. In FIG. 9B and FIG. 10B, as in FIG. 5, each pixel PIX is assigned a code (any one of R, G, and B) corresponding to the filter color arranged on the front surface. . In each of FIGS. 9B and 10B, the black pixel PIX indicates that the dark portion of the striped pattern has been captured, and the white pixel PIX has captured the bright portion of the striped pattern. It shows that. For convenience of explanation, pixel addresses (numerals 1 to 8, symbols 1 to 1) are attached to the diagrams of FIGS. 9B and 10B. Strictly speaking, the subject is imaged on the solid-state imaging device 112a by the imaging action of the photographic lens 106 in a state where the subject is turned upside down. As an example, the upper left corner portion of the “obliquely striped subject 9a” forms an image as the lower right corner portion on the solid-state imaging device 112a. However, in this embodiment, in order to avoid complication of explanation, it is assumed that the upper left corner portion of the “obliquely striped subject 9a” corresponds to the upper left corner portion of FIG. 9B.

[S13 in FIG. 7 (Shift of Solid-State Image Sensor 112a)]
In this processing step S13, the movable stage 112c is driven, and the position of the solid-state imaging device 112a is shifted (changed) in a plane parallel to the light receiving surface 112aa (in the XY plane). The shift direction and the shift amount of the solid-state imaging device 112a are controlled using outputs from the Hall elements H _YR , H _YL , H _XD , and H _XU . In the present embodiment, in this processing step S13, the solid-state imaging device 112a is shifted rightward by a distance of one pixel.

[S14 in FIG. 7 (Taking Second Image)]
Also in this processing step S14, the subject image (second image) is photographed (one exposure period is completed) based on the AE control and AF control at the time of photographing the first image.

  FIGS. 9C and 10C are diagrams similar to FIGS. 9B and 10B, respectively, and an obliquely striped subject 9a and a vertically striped subject captured by each pixel PIX of the solid-state imaging device 112a. 10a is schematically shown. As shown in FIGS. 9B and 10B, the incident position of the subject image on the light receiving surface 112aa of the solid-state image sensor 112a is the shift amount of the solid-state image sensor 112a in the processing step S13. According to (shifted by a distance of one pixel in the left direction on the light receiving surface 112aa).

  In addition, the imaging conditions in this processing step S14 are the same except that the solid-state imaging device 112a is shifted rightward by a distance of one pixel with respect to the imaging in processing step S12. For this reason, the second image is substantially a photograph of a range shifted to the right by one pixel with respect to the first image. As can be seen from FIGS. 9B and 10B, in the second image, the subject is shifted to the left by a distance of one pixel as a whole with respect to the first image. It is reflected.

[S15 in FIG. 7 (metadata addition processing)]
In this processing step S15, image data of the first image photographed in processing step S12 and image data of the second image photographed in processing step S14 are generated (recorded), and metadata is added to each image data. Is done. The metadata of the image data generated (recorded) in the shooting process step S1 (image data shot with the false color detection process set to ON) includes image data generated (recorded) in the normal shooting process. Additional information is included in addition to the metadata of (image data captured with the false color detection process set to OFF). The additional information is written, for example, in a maker note area of metadata that complies with the format of the Exif format.

  FIG. 11A and FIG. 11B show the data structure of the image data of the first image and the data structure of the image data of the second image. Items of the metadata other than the additional information are common when the false color determination process is set to on and off (see FIG. 4). The additional information includes information indicating that the image is captured with the false color detection process set to ON, and the image data generated (recorded) in the imaging process step S1 (processing steps S12 and S14) (that is, the same image). Information indicating that the image data included in the group is related to each other is included. Below, the information written in each item of additional information is demonstrated.

  In the “false color detection flag” item, information indicating whether the setting of the false color detection process is on or off is written. In this item, for example, “1” is written when the setting of the false color detection process is on, and “0” is written when the setting of the false color detection process is off. Note that when the setting of the false color detection process is off, information does not have to be written in this item.

  In the “file ID” item, information for identifying each image data is written. As the information described in the “file ID” item, the same information as the “file name” item of the metadata may be written. However, once information is written in the “file ID” item, the information in the “file name” item and the information in the “file ID” item are independent from each other. For example, even if the information of the “file name” item is changed after the metadata is added to the image data, the value of the “file ID” item of the additional information is maintained without being changed.

  In the “group ID” item, information indicating a group including each image data is written. In the present embodiment, a plurality of image data generated (recorded) by one photographing process step S1 is included in the same image group (image group). In the example shown in FIGS. 11A and 11B, the image data of the first image and the image data of the second image are both included in the same image group, and the ID of the group is “3”. is there. The group ID is changed every time the photographing process step S1 is executed.

  In the “number of shots” item, the number of shots (number of shot images, number of images belonging to the same group) in the shooting process step S1 is written. In the present embodiment, since a total of two shootings are performed in the processing step S12 and the processing step S14, “2” is written in the “number of shots” item. Note that, as will be described later, in another embodiment of the present invention, three or more shootings are performed in the shooting process S1. In this case, the value written in the “number of shots” item changes according to the number of shots.

  In the “shooting order” item, a shooting order among a plurality of images included in the same group is written. In the example shown in FIGS. 11A and 11B, in the group “3”, the shooting order of the first image is “1”, and the shooting order of the second image is “2”.

  In the “shooting time” item, information indicating the time when each image (first image, second image) was shot is written. The time written in the “shooting time” item is written in a unit (for example, in milliseconds) smaller than the “shooting date” item in the metadata so that the time when each image was shot can be distinguished. If the metadata conforms to the format of the Exif format, the “shooting time” item includes the value of the “DateTime” item in which the date is written in units of seconds or more in the metadata, and the second. The time information may be written with reference to the value of the “SubSecTime” item in which the time of a smaller unit is written.

In the “coordinate” item, information indicating the position of the solid-state imaging device 112a when each image is captured is written. The position of the solid-state imaging device 112a is indicated by coordinates (X, Y) in the XY plane orthogonal to the optical axis AX, for example, with the optical axis AX as a reference (coordinates (0, 0)). The unit of position is, for example, a pixel (pixel). In the present embodiment, when the first image is captured, the solid-state imaging device 112a is located at the coordinates (0, 0). Further, when the second image is captured, the solid-state imaging device 112a is shifted by one pixel in the right direction, and thus the coordinates are (1, 0). The value written in the “coordinate” item may be determined based on, for example, outputs from the hall elements H _YR , H _YL , H _XD , and H _XU in the processing step S13. Alternatively, when the shift amount of the solid-state imaging device 112a in the processing step S13 is determined in advance and stored in a predetermined storage area such as the ROM 126, the value written in the “coordinate” item is read from the predetermined storage area. The written value may be written.

  In the “file ID of previous image” item, the file ID of the image captured immediately before among the images included in the same group is written. In the example shown in FIGS. 11A and 11B, the first image is the first image taken in the group “3”. “Null” (or “null value” which is a value indicating “nothing”) indicating that there is no previously captured image is written. Further, since the second image is an image taken immediately after the first image in the group “3”, the “file ID of the previous image” item includes the file ID of the image data of the first image. A certain “Image011” is written.

  In the “file ID of subsequent image” item, the file ID of the image captured immediately after among the images included in the same group is written. In the example shown in FIG. 11A and FIG. 11B, the image taken immediately after the first image in the group “3” is the second image. In the “image file ID” item, “Image012” that is the file ID of the image data of the second image is written. Also, in group “3”, there is no image taken after the second image, and therefore there is no image taken immediately after in the “file ID of the subsequent image” item of the image data of the second image. “Null” (or a null value) is written.

  In the “file ID of the previous image” item and the “file ID of the subsequent image” item, the value of the “file ID” item, not the value of the “file name” item in the image data of the images taken before and after, respectively. Is written. For this reason, even if the value of the “file name” item of the image data is changed, the values of the “file ID of the previous image” item and the “file ID of the subsequent image” item are not affected.

  Image data to which metadata including additional information is added is stored in the memory card 200. Note that the image data may be stored not in the memory card 200 but in the internal memory of the photographing apparatus 1 or the memory 301 of the image processing apparatus 3 outside the photographing apparatus 1.

  In the above-described processing, the additional information is added to the image data as part of the metadata, but the metadata addition processing in the present embodiment is not limited to this. For example, the image data and the additional information may be stored separately in a memory card or the like, and information that associates the image data and the additional information corresponding to the image data may be added to at least one of the image data and the additional information. Further, the additional information may not include all the items described above. In short, if at least “information indicating the position of the solid-state imaging device 112a when each image is photographed” is recorded in association with each image data, the image capturing apparatus 1 or an external device is recorded later. This is effective when the image processing apparatus 3 automatically or manually detects the presence or absence of false colors. “Information indicating the position of the solid-state image sensor 112a when each image is photographed” is information written in the “coordinate” item in a straight line, but the position of the solid-state image sensor 112a is temporarily fixed by “imaging order”. If it is, “the information indicating the position of the solid-state imaging device 112a when each image is photographed” is the value of the “shooting order” item or the value of the “shooting time” item instead of the value of the “coordinate” item. Can be obtained from The information of each other item greatly assists the user who performs the process, but without it, “false color detection cannot be performed” is not possible.

  In addition, the image data of the first image and the image data of the second image generated (recorded) in the processing step S12 and the processing step S14 described above are all processed by the signal processing (clamping, demosaicing, matrix calculation, Y / C separation, white balance, etc.) are performed, converted into a luminance signal Y and color difference signals Cb, Cr, and stored in a predetermined format. Hereinafter, for convenience of explanation, the color difference signals (Cb, Cr) of the first image are referred to as “first color difference signals”, and the color difference signals (Cb, Cr) of the second image are referred to as “second color difference signals”. The “target pixel” refers to a pixel of each image after at least demosaic processing.

  In the present embodiment, although described in detail later, the occurrence of false colors is detected based on the signal difference value or signal addition value between the first color difference signal and the second color difference signal. However, at the edge portion with high contrast, the signal difference value between the first color difference signal and the second color difference signal may be large even if a false color is not generated. In this case, there is a risk of false detection that a false color has occurred in the edge portion. As will be described in detail later, the first color difference signal and the second color difference signal used for the calculation of the signal difference value and the signal addition value are the color difference signals of the pixels that capture the same subject image, but in processing step S13. Due to the shift of the solid-state imaging device 112a, the demosaic processing is performed using pixels having different addresses. For this reason, the first color difference signal and the second color difference signal may be slightly different in color information even though they are color difference signals of pixels that capture the same subject image. This appears as a false color on the captured image.

[S2 in FIG. 6 (detection process)]
The detection processing step S2 shown in FIG. 6 will be described with reference to FIG. In the detection processing step S2, false color detection processing is performed using the image data generated (recorded) and stored in the photographing processing step S1. The detection processing step S2 may be automatically started after the photographing processing step S1 is completed, or may be started in response to an operation on the operation unit 102. Further, when the image data is stored in the memory 301 of the external image processing device 3, the detection processing step S <b> 2 may be executed by the image processing device 3. Hereinafter, processing when image data is stored in the memory card 200 will be described.

[S20 in FIG. 8 (Metadata Acquisition Processing)]
In this processing step S20, metadata is acquired from each image data stored in the memory card 200. In the memory card 200, image data generated (recorded) by a normal shooting process (image data shot with the false color detection process set to OFF) and a shooting process step for use in false color detection are stored. The image data generated (recorded) in S1 (image data captured with the false color detection process set to ON) is stored. In this processing step S20, metadata is acquired from both the image data generated (recorded) in the normal shooting process and the image data generated (recorded) in the shooting process step S1.

[S21 in FIG. 8 (Image Data Extraction Processing)]
In this processing step S21, image data used for false color detection is extracted from the plurality of image data stored in the memory card 200 using the metadata acquired in the processing step S20. Illustratively, in this processing step S21, it is determined whether or not additional information is included in the metadata acquired in processing step S20. Next, the system controller 100 extracts image data to which additional information is added from among a plurality of image data. For example, the value of the “false color detection flag” item is used to determine whether additional information is included in the metadata. When the image data including the additional information is extracted, only the extracted image data is displayed on the LCD 124 from the image data stored in the memory card 200. The LCD 124 may display a thumbnail image of the selected image data, or may display a part of the metadata (for example, a file name) of the selected image data.

  Note that it is not necessary for the LCD 124 to display all the extracted image data. For example, among the image data whose “false color detection flag” item value is “1”, image data (for example, “shooting order” item value of the additional information is “1” from each group specified by the group ID). 1 ”) may be selected and displayed.

  In this processing step S21, the LCD 124 displays both extracted image data (image data including additional information) and image data not extracted (image data not including additional information). May be. In this case, the image data including the additional information is displayed so as to be distinguishable from the image data not including the additional information. For example, only image data including additional information is displayed with emphasis.

  In this processing step S21, values of items other than the “false color detection flag” item may be used for the determination processing of whether additional information is included in the metadata and the image data extraction processing. . For example, “information indicating the position of the solid-state imaging device 112a when each image is captured” may be used for these processes. Specifically, the “information indicating the position of the solid-state imaging device 112a when each image is photographed” is information of “coordinate” item, information of “shooting order” item, or “shooting time” item. Information etc. Alternatively, in the present processing step S21, “information indicating that the image data generated (recorded) in the photographing processing step S1 (that is, image data included in the same image group) is related to each other” is used. Good. Specifically, the “information indicating that the image data generated (recorded) in the shooting process step S1 are related to each other” includes information on the “group ID” item, information on the “number of shots” item, “ Information of “shooting order” item, information of “file ID of previous image” item, information of “file ID of subsequent image” item, and the like. Furthermore, in this process step S21, the combination of the information of the some item contained in additional information may be used.

[S22 in FIG. 8 (Process for Specifying Image Data)]
In this processing step S22, designation (selection) of image data for performing false color detection is received from the image data extracted in processing step S21 and displayed on the LCD 124. For example, the system controller 100 accepts one designation among a plurality of image data displayed on the LCD 124. This designation is performed by an operation on the operation unit 102 by the user. The system controller 100 reads the group ID of the image data designated by the user, and designates image data having the same group ID as the read group ID as image data used for false color detection. The number of image data specified in this processing step S22 is the same as the value written in the “number of shots” item described in the additional information of the image data specified by the user. If all the image data extracted in the processing step S21 has the same group ID, this processing step S22 may be omitted.

  In the following description, it is assumed that the image data of the first image and the image data of the second image are designated in this processing step S22.

[S23 in FIG. 5 (address conversion)]
In the present embodiment, a false color generation area in which a false color is generated in a captured image is detected on a pixel basis. In this case, in order to improve the false color detection accuracy, it is desirable to calculate the signal difference value and the signal addition value between the target pixels on which the same subject image is incident. Therefore, in this processing step S23, the addresses of the respective pixels forming the second image are processed by the solid-state imaging device 112a in the processing step S13 so that the same subject image is processed as being incident on the pixels having the same address. Conversion is performed according to the shift direction and the shift amount (in other words, the shift direction and shift amount of the incident position of the subject image on the light receiving surface 112aa of the solid-state imaging device 112a). This address conversion is performed by adding additional information added to each image data, for example, the value of the “coordinate” item (coordinate information) or “information indicating the position of the solid-state imaging device 112a when each image is captured. ”.

  First, based on the value of the “coordinate” item in the additional information added to each image data, the position of the solid-state imaging device 112a at the time of shooting each image is determined. In the example shown in FIGS. 11A and 11B, the first image is taken when the solid-state image sensor 112a is at the coordinates (0, 0), and then the solid-state image sensor 112a is at the coordinates (1,0). ) And the second image is taken. In this case, the shift direction of the solid-state imaging device 112a is rightward, and the shift amount is one pixel. In other words, the second image is taken with the subject image on the light receiving surface 112aa moved leftward by one pixel with respect to the first image. Therefore, in this processing step S23, the address of each pixel of the second image is converted so as to move by one pixel in the right direction opposite to the moving direction of the subject image. As an example, the address (c, 2) of the pixel PIXb in FIG. 9C is the same as the pixel that is located when the pixel PIXb is shifted by one pixel in the right direction according to the shift amount of the incident position of the subject image, that is, the pixel PIXb. Are converted into the same address (d, 2) as the pixel PIXa (see FIG. 9B) for capturing the subject image.

  In this processing step S23, instead of the value of the “coordinate” item, “shooting” is performed instead of the value of the “coordinate” item, for example, when the position of the solid-state imaging device 112a at the time of shooting each image is fixed according to the shooting order The position of the solid-state imaging device 112a at the time of shooting each image may be determined using the value of the “order” item or the information of the “shooting time” item. Alternatively, the shooting order of each image may be determined using information of the “file ID of the previous image” and the “file ID of the subsequent image”.

[S24 in FIG. 8 (Calculation of Difference Value of Color Difference Signal)]
The oblique stripe subject 9a and the vertical stripe subject 10a include a high-frequency component having the same pitch as the pixel pitch P of the pixel PIX of the solid-state image sensor 112a. Therefore, when the image signals of the diagonally striped subject 9a and the vertically striped subject 10a are demosaiced in the processing step S12 and the processing step S14, a false color is generated.

  Specifically, in the example of FIG. 9B, the luminance of the G component pixel PIX is high, and the luminance of the R and B component pixels PIX is low. Therefore, the G component is dominant in the color information of each pixel PIX after the demosaic process, and a green false color is generated in the diagonally striped subject 9a. On the other hand, in the example of FIG. 9C, the luminance of the R and B component pixels PIX is high and the luminance of the G component pixel PIX is low, contrary to the example of FIG. 9B. Therefore, R and B components are dominant in the color information of each pixel PIX after demosaic processing, and a purple false color is generated in the diagonally striped subject 9a.

  In the example of FIG. 10B, the luminance of the B component pixel PIX is high and the luminance of the R component pixel PIX is low. Therefore, the color information of each pixel PIX after the demosaic processing becomes a mixed color component of B and G, and a false color of an intermediate color between blue and green (for example, a color near the cyan color center) is generated. On the other hand, in the example of FIG. 10C, the luminance of the R component pixel PIX is high and the luminance of the B component pixel PIX is low, contrary to the example of FIG. Therefore, the color information of each pixel PIX after the demosaic processing becomes a mixed color component of R and G, and a false color of an intermediate color between red and green (a color near the orange center) is generated.

FIG. 12 shows a color space defined by two axes of Cb and Cr. In FIG. 12, reference numeral 9b is a plot corresponding to the green false color generated at the target pixel in the example of FIG. 9B, and reference numeral 9c is a purple color generated at the target pixel in the example of FIG. 9C. 10b is a plot corresponding to a false color of an intermediate color between blue and green generated in the target pixel in the example of FIG. 10B, and reference numeral 10c is a plot corresponding to FIG. It is a plot corresponding to the false color of the intermediate color of red and green generated in the target pixel in the example of c). The following shows the coordinate information of each plot. The origin O is the coordinates (0, 0).
Plot 9b: (Cb, Cr) = (− M, −N)
Plot 9c: (Cb, Cr) = (M, N)
Plot 10b: (Cb, Cr) = (M ′, −N ′)
Plot 10c: (Cb, Cr) = (− M ′ + α, N ′ + β)
However, all of M, N, M ′, N ′, α, and β are positive numbers.

  In the present embodiment, the false color itself generated when a subject image having a high frequency component equivalent to the pixel pitch P of the pixel PIX of the solid-state image sensor 112a is captured is the incident position of the subject image on the light receiving surface 112aa. A portion (a pixel of interest) where a false color is generated is detected by utilizing the change caused by shifting. More specifically, in a pixel of interest in which a high-frequency component subject image is captured, a portion in which the color information of the first color difference signal and the second color difference signal has a complementary color relationship is determined to be a portion where a false color is generated. It is detected.

  In the present embodiment, in order to obtain the above complementary color relationship, the incident position of the subject image on the light receiving surface 112aa is shifted by one pixel in the left direction (horizontal pixel arrangement direction) (the solid-state imaging device 112a is shifted from the subject image). However, the present invention is not limited to this. For example, the shift direction may be the right direction (horizontal pixel arrangement direction), or the upper direction (vertical pixel arrangement direction), the lower direction (vertical pixel arrangement direction), or the upper right direction. , Lower right, upper left, lower left diagonal directions (directions forming 45 degrees with respect to horizontal and vertical arrangement directions), or other directions according to the pixel arrangement may be used together. . In addition, the shift distance may be, for example, a distance corresponding to another odd pixel such as 3 pixels or 5 pixels, or a half pixel or a half pixel plus an odd pixel (for example, 1.. 5 pixels, 2.5 pixels, etc.) (ie, n pixels or (m + 0.5) pixels (where n = odd natural number, m = 0 or odd natural number) However, it can be selected according to the accuracy of the mechanism for shift driving. Further, the shift distance may be a distance other than the even number of pixels and the vicinity thereof (for example, 1.9 to 2.1 pixels) depending on the subject to be photographed and the photographing conditions. Even when the incident position of the subject image on the light receiving surface 112aa is shifted by these amounts (direction and distance), the first color difference signal is obtained at the target pixel in which the subject image of the high frequency component is captured (false color is generated). Since the color information of the second color difference signal and the second color difference signal are complementary to each other, the occurrence of false color can be detected.

In this processing step S24, after the address conversion in the processing step S23 (hereinafter the same), the difference value (Cb _sub , Cr _sub ) between the first color difference signal and the second color difference signal is obtained for each target pixel having the same address. Calculated. Specifically, when the Cb and Cr of the first color difference signal are defined as Cb1 and Cr1, respectively, and the Cb and Cr of the second color difference signal of the same address are defined as Cb2 and Cr2, respectively, the difference value (Cb _sub , Cr _sub ) is calculated by the following equation.
Cb _sub = Cb1-Cb2
Cr _sub = Cr1-Cr2

[S25 in FIG. 8 (Calculation of First Distance Information)]
In the process step S25, address for each identical target pixel, a distance in the color space of the first color difference signal and a second color difference signal (the first distance information Saturation_ _sub) is calculated by the following equation.
^{_{Saturation_ sub = (Cb sub 2 +}} Cr sub 2) 1/2

The first distance information Saturation_ _sub are each an example of each plot pair of FIG. 12 (plot 6b and plot 6c, plots 7b and plot ^{7c), 2 (M 2 +} N 2) 1/2, {(2M'-α ) ² + (2N ′ + β) ² } ^1/2 .

As understood from the positional relationship of each plot pair of FIG. 12, as the color information with the first color difference signal and a second color difference signal is strong complementary relationship first distance information Saturation_ _sub increases, first color difference signal and the color information having the second color difference signal is not (for example similar hue) in complementary relationship as the first distance information Saturation_ _sub decreases. That is, the first distance information Saturation_ _sub ideally if false color occurrence region is zero, the larger the false color occurrence region in which false color is generated strongly.

[S26 in FIG. 8 (Calculation of Addition Value of Color Difference Signal)]
In this processing step S26, an added value (Cb ′ _add , Cr ′ _add ) of the first color difference signal and the second color difference signal is calculated for each target pixel having the same address.

Specifically, in this processing step S26, provisional addition values (Cb _add , Cr _add ) are calculated by the following equations.
Cb _add = Cb1 + Cb2
Cr _add = Cr1 + Cr2

Next, the average value (Cb _mean , Cr _mean ) of the provisional addition values is calculated by the following equation.
Cb _mean = Cb _add / 2
Cr _mean = Cr _add / 2

Next, the added value (Cb ′ _add , Cr ′ _add ) is calculated by the following equation.
Cb ′ _add = Cb _add −Cb _mean
Cr ′ _add = Cr _add −Cr _mean

[S27 in FIG. 8 (Calculation of Second Distance Information)]
In the process step S27, address for each identical target pixel, the addition value _{(Cb 'add, Cr' add} ) the second distance information Saturation_ _{the add} in a color space based on the calculation by the following equation.
^{_{Saturation_ add = (Cb 'add 2}} + Cr' add 2) 1/2

Second distance information Saturation_ _{the add} each example of each plot pair of FIG. 12 (plot 9b and plot 9c, plots 10b and plot 10c), zero, and ^{(α 2 + β 2) 1/2} .

Here, the first and second color difference signals change under the influence of the light source, the exposure condition, the white balance and the like at the time of image capturing. However, first, since the second respective color difference signals varies in the same way, mutual relative distance in the color space (i.e., the first distance information Saturation_ _sub) changes little.

On the other hand, the second distance information Saturation_ _{the add,} the first, second light source for the respective color difference signal imaging, exposure conditions, the changes relative to the position of the origin O of the color space under the influence of such as white balance , Change a lot. Therefore, in the process step S26, the second distance information Saturation_ _{the add} provisional sum value (Cb _{add, Cr add)} without operation immediately using, to offset or reduce the change in position with respect to the origin O by the impact The added values (Cb ′ _add , Cr ′ _add ) are calculated (in order to bring the midpoint between the first color difference signal and the second color difference signal away from the origin O closer to the origin O due to the above influence).

As can be understood from the positional relationship between each pair of plots in FIG. 12, when the color information of the first color difference signal and the second color difference signal has a complementary color relationship, the signs are opposite to each other, so that the added value (Cb _'add, Cr' _add) is decreased, the second distance information Saturation_ _{the add} decreases. The second distance information Saturation_ to _add becomes smaller as is strong complementary relationship, the zero ideally. On the other hand, when the color information of the first color difference signal and the second color difference signal is not in a complementary color relationship (for example, the same hue), the signs are the same, so the added values (Cb ′ _add , Cr ′) _add) and increases, the second distance information Saturation_ _{the add} increases. That is, the second distance information Saturation_ _{the add} is larger if false color occurrence region decreases if false color occurrence region.

[S28 in FIG. 8 (Calculation of Difference Value of Luminance Signal)]
For convenience of explanation, the luminance signal Y of the first image is denoted as “first luminance signal”, and the luminance signal Y of the second image is denoted as “second luminance signal”. In this processing step S28, a difference value Y _diff between the first luminance signal and the second luminance signal is calculated for each target pixel having the same address. Specifically, in this processing step S28, when the first luminance signal is defined as Y1 and the second luminance signal is defined as Y2, the difference value Y _diff is calculated by the following equation.
Y _diff = | Y1-Y2 |

[S29 in FIG. 8 (Determination of False Color Generation Area)]
In this processing step S29, it is determined whether or not each pixel of interest having the same address is a false color generation region. Specifically, when all of the following conditions (1) to (3) are satisfied, it is determined that the pixel is a false color generation region.

・ Condition (1)
As described above, since the color information having the first color difference signal the larger the first distance information Saturation_ _sub and the second color difference signal is strong complementary relationship, likely the target pixel is false color occurrence region is high. Therefore, condition (1) is defined as follows.
Condition (1): Saturation_ _sub ≧ threshold T1

・ Condition (2)
As described above, since the color information having the second distance information Saturation_ _{the add} is small enough first color difference signal and a second color difference signal is strong complementary relationship, likely the target pixel is false color occurrence region is high. Therefore, condition (2) is defined as follows.
Condition (2): Saturation_ _add ≦ threshold T2

Consider a case where a large image blur occurs in only one of the first image photographed in the processing step S12 and the second image photographed in the processing step S14. In this case, the difference between the first color difference signal and the second color difference signal becomes large, and there is a possibility that a false color is erroneously detected. Therefore, the condition (3) is defined as follows.
Condition (3): Y _diff ≦ threshold value T3

That is, when the difference value Y _diff is larger than the threshold value T3, there is a risk of the erroneous detection described above, so that false color detection for the target pixel is not performed (assuming that the target pixel is not a false color generation region). It is processed.).

  Condition (1) and condition (2) are conditions for directly determining whether or not the target pixel is a false color generation region. Therefore, in another embodiment, when at least one of the condition (1) and the condition (2) is satisfied, the target pixel may be determined to be a false color generation region.

  Further, the user can change the false color detection sensitivity by operating the operation unit 102 to change the settings of the threshold values T1 to T3.

[S30 in FIG. 8 (detection of false color)]
In this processing step S30, the presence or absence of a false color is detected. Illustratively, when the number of target pixels determined as a false color generation region in processing step S29 (or the ratio of the target pixels determined as a false color generation region out of the total number of effective pixels) is equal to or greater than a predetermined threshold value. When the number of pixels of interest (or the ratio) is less than a predetermined threshold, the detection result is that there is no false color.

  The detection result of the false color by the detection process S2 is used for the photographing process of the subject and the notification to the user through the display device such as the LCD 124.

  Illustratively, if the detection process S2 is executed by the system controller 100 of the photographing apparatus 1 and it is determined that there is a false color, the subject may be photographed again under LPF driving. Thereby, an image in which false colors are suppressed can be obtained. It should be noted that when the photographing process S1 has already been performed under LPF driving and a false color is detected in that state, a solid optical so that a stronger optical LPF effect (reduction of moire such as false color) can be obtained. The drive cycle (rotation cycle) and drive amplitude (rotation radius) of the image sensor 112a are adjusted. That is, it is changed to a more advantageous shooting condition for reducing false colors. When the shooting condition is changed, the subject is shot again.

  For example, when it is determined that there is a false color in the detection process S <b> 2, an image (either one or both of the first image and the second image) is displayed on the LCD 124. Further, on the LCD 124, information indicating the location where the false color is generated is displayed superimposed on the photographed image. Thereby, the user can grasp | ascertain the location in which the false color in the image has generate | occur | produced. In addition, when the photographing apparatus 1 or the external image processing apparatus 3 includes an image processing unit that performs image processing on an image, false colors are reduced based on information indicating a false color occurrence location in the image. Such image processing (for example, blur processing) may be executed.

  The above is the description of the exemplary embodiments of the present invention. Embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the technical idea of the present invention. For example, the embodiment of the present application also includes an embodiment that is exemplarily specified in the specification or a combination of obvious embodiments and the like as appropriate.

  For example, in the above embodiment, the incident position of the subject image on the light receiving surface 112aa is shifted by shifting the solid-state imaging device 112a, but the embodiment of the present invention is not limited to this configuration. For example, the relative position between the light receiving surface 112aa and the subject image can be shifted by moving the relative position in the XY plane between the solid-state imaging device 112a and the optical member through which the light flux from the subject passes. Specifically, the incident position of the subject image on the light receiving surface 112aa is shifted by driving the image sensor 112a and another shake correction member (such as a part of the lenses included in the photographing lens 106) eccentrically. In addition, a parallel plate arranged in the optical path of the photographing lens 106 at a slight inclination with respect to the optical axis AX may be rotated around the optical axis AX, or the vertex angle variable prism and the solid-state image sensor 112a may be covered. The incident position of the subject image on the light receiving surface 112aa may be shifted by driving a glass (having a vibration function for removing dust or the like attached to the cover glass). As described above, when the imaging process S1 is executed by shifting an optical member other than the solid-state image sensor 112a, the additional information of the image data includes information indicating the position of the optical member, or the solid-state image sensor 112a and the optical member. Information indicating the relative position between and is written.

  Moreover, in said embodiment, although threshold value T1-T3 is unchanged in all the determinations with respect to each pixel in process step S29, this invention is not limited to this.

  For example, by performing AF control by the system controller 100, a range that can be regarded as being in focus in the image (a focus area, illustratively a range that falls within the depth of field) is obtained. Since the subject in the focus area (in-focus state) has high contrast and tends to include high-frequency components, false colors are likely to occur. On the other hand, a subject outside the in-focus area (out-of-focus state) has a low contrast and is unlikely to contain a high-frequency component, so that a false color is unlikely to occur. Therefore, in the processing step S29, the threshold values T1 to T3 may be changed to different values depending on whether the determination is made for the pixel in the in-focus area obtained by the calculation or the determination for the pixel outside the in-focus area. . Illustratively, since there is a high possibility that a false color is generated (conspicuous) in a pixel in the in-focus area, a threshold setting that increases detection sensitivity (for example, the threshold T1 is set to a low value) is performed. Since the possibility of false color is low (not conspicuous) in other pixels, threshold setting (for example, setting the threshold value T1 to a high value) that lowers the detection sensitivity or false color detection is performed. The process itself is omitted. As a result, the false color detection accuracy is further improved, and the detection speed is improved.

  In the above embodiment, false color detection is performed using a pair of images (a first image taken in processing step S12 and a second image taken in processing step S14). In another embodiment, false color detection may be performed using more than two images.

  For example, for the vertical stripe subject 10a, by shifting the solid-state image sensor 112a in the bright and dark direction (left direction) of the stripe in the processing step S13 (shift of the solid-state image sensor 112a) as in the above embodiment, It is easy to obtain a false color image having a complementary color relationship. However, when the solid-state image sensor 112a is shifted in the up, down, or diagonal directions with respect to the vertical stripe subject 10a, the color component with high luminance is less likely to change before and after the shift of the solid-state image sensor 112a, resulting in a complementary color relationship. It becomes difficult to obtain a false color image. Therefore, in order to further improve the accuracy of false color detection (in another aspect, to prevent detection omission), three or more images may be taken while changing the shift direction of the solid-state imaging device 112a.

  For example, in the imaging process S1, the first to fourth images are moved while the coordinates of the solid-state imaging element 112a are moved to (0, 0), (1, 0), (1, 1), and (0, 1), respectively. It may be taken. In this case, in the detection process S2, false color detection using a combination of the first image and the second image, false color detection using a combination of the first image and the third image, and the first image and the fourth image are detected. False color detection using the combination is performed. In each combination, the shift direction of the solid-state imaging device 112a when the other image is captured with respect to one image is different. Therefore, by determining the false color using a combination of a plurality of images, the false color can be accurately detected regardless of the appearance direction of the high-frequency component.

  In the above-described embodiment, information indicating the position of the solid-state image sensor 112a is written in the “coordinate” item of the additional information, but the information indicating the position of the solid-state image sensor 112a in the present embodiment is not limited to the coordinates. The additional information only needs to include information regarding the position of the solid-state imaging device 112a when each image is captured, and various expressions representing the position can be used for this information. For example, the additional information may have a “shift direction” item in which information indicating the shift direction of the solid-state imaging device 112a is written instead of the “coordinate” item. For example, when the shift amount of the solid-state image sensor 112a is determined in advance in the photographing process S1, the position of the solid-state image sensor 112a in the XY plane is determined by writing the shift direction of the solid-state image sensor 112a in the additional information. be able to. Further, the additional information may further include information indicating the shift amount.

  For example, in the “shift direction” item of the additional information, the declination angle in the circular coordinate system (rθ coordinate system) with the declination angle θ in the right direction (X-axis positive direction) set to zero with the coordinate (0, 0) as a reference. θ may be written. When shooting the first to fourth images while moving the coordinates of the solid-state imaging device 112a to (0, 0), (1, 0), (1, 1), (0, 1), shooting the first image At that time, since the solid-state image sensor 112a has not moved from the reference position, the “shift direction” item of the image data of the first image indicates “null” (or movement) indicating that the solid-state image sensor 112a has not moved. ("Null value"), which is a value indicating that this is not done, is written. Deflection angles θ “0 degrees”, “45 degrees”, and “90 degrees” of the position of the solid-state imaging device 112a with respect to the reference position are written in the “shift direction” items of the image data of the second to fourth images.

  Further, the additional information may not include all the items shown in FIGS. 11A and 11B. The additional information only needs to include at least information necessary for selecting the images included in the same group and the position of the solid-state imaging device 112a when the selected image is captured.

  For example, when the “group ID” item is included in the additional information, one or both of the “file ID of the previous image” item and the “file ID of the subsequent image” may not be included. If there is a “group ID” item, it is possible to select image data included in the same group as the image designated (selected) by the user in processing step S22.

  Also, in the shooting processing step S1, the correspondence between the shooting order of each shot image and the movement position of the solid-state imaging device 112a at the time of shooting each shot image is determined, and the corresponding relationship is a predetermined storage area such as the ROM 126. Is recorded in the additional information, the “coordinate” item may not be included. In this case, the system controller 100 determines the solid-state imaging device at the time of shooting each image based on the value of the “shooting order” item of the additional information and the correspondence relationship between the shooting order recorded in the predetermined storage area and the movement position. Information about the position of 112a can be acquired.

DESCRIPTION OF SYMBOLS 1 Image pick-up device 100 System controller 102 Operation part 104 Drive circuit 106 Shooting lens 108 Aperture 110 Shutter 112 Image blur correction device 112a Solid-state image sensor 112aa Light-receiving surface 112b (of solid-state image sensor) Fixed support substrate 112c Movable stage 114 Signal processing circuit 116 Image Processing engine 118 Buffer memory 120 Card interface 122 LCD control circuit 124 LCD
126 ROM
128 Gyro sensor 130 Acceleration sensor 132 Geomagnetic sensor 134 GPS sensor 136 Communication interface 200 Memory card 3 Image processing device 300 Controller 301 Memory 302 LCD
303 Card Interface 304 Communication Interface C _YR , C _YL , C _XD , C _XU Drive Coil H _YR , H _YL , H _XD , H _XU Hall Element M _YR , M _YL , M _XD , M _XU Magnet PIX Pixel VCM _YR , VCM _YL , VCM _XD , VCM _XU voice coil motor Y _YR , Y _YL , Y _XD , Y _XU yoke

Claims (16)

A subject image incident on the image sensor via the imaging optical system, and a position changing means for changing a relative position of the image sensor in a plane parallel to the light receiving surface of the image sensor;
Image recording means for capturing two or more predetermined images by capturing the subject image using the image sensor and recording an image group including the predetermined images;
Position information acquisition means for acquiring information related to the relative position between the subject image and the image sensor when the predetermined image is captured;
Position information adding means for adding information on the relative position to the predetermined image;
Comprising
Imaging system. The information on the relative position is obtained by capturing other images of the two or more predetermined images with respect to the relative position between the subject image and the image sensor when the first image is captured. Including information indicating a change in relative position between the subject image and the image sensor at the time of
The imaging system according to claim 1. The position changing means changes the position of the image sensor in a plane parallel to the light receiving surface,
The information on the relative position includes position coordinates with respect to a predetermined origin in a plane parallel to the light receiving surface of the imaging element.
The imaging system according to claim 1 or 2. The image recording means captures the subject images in a state where the relative positions are different from each other, acquires the two or more predetermined images, and records an image group including the predetermined images;
The imaging system according to any one of claims 1 to 3. The position information adding unit is configured to calculate the relative information based on imaging order information indicating an imaging order in an image included in the image group for each of the two or more predetermined images included in the image group. Add location information,
The imaging system according to any one of claims 1 to 4. Related information adding means for adding, to each of the two or more predetermined images included in the image group, related information indicating that the two or more predetermined images are related to each other;
Related image extraction means for extracting an image to which the related information is added from a plurality of images;
Further comprising
The imaging system according to any one of claims 1 to 5. Image identification information adding means for adding image identification information for identifying each of the two or more predetermined images to each of the two or more predetermined images included in the image group;
The related information adding means adds the image identification information of other images included in the image group to each of the two or more predetermined images.
The imaging system according to claim 6. The related information adding means captures each of the two or more predetermined images included in the image group in front of each of the two or more predetermined images among the images included in the image group. Adding at least one of the image identification information of the captured image and the image identification information of the image captured after each of the two or more predetermined images;
The imaging system according to claim 7. The related information includes image number information indicating the number of images included in the image group.
The imaging system according to any one of claims 6 to 8. The related information includes image group identification information for identifying the image group,
The imaging system according to any one of claims 6 to 9. An information acquisition step of acquiring information about the image added to the image from the image;
In the information acquired in the information acquisition step, relative to a subject image incident on the imaging device by the imaging device via the imaging optical system and the imaging device in a plane parallel to the light receiving surface of the imaging device. An information determination step for determining whether information relating to the relative position when the subject image is captured while changing the position and used for predetermined image processing on the image is included;
An image determination program including The information regarding the relative position is obtained when another image is captured with respect to the relative position between the subject image and the image sensor when the first image is captured among the two or more images. Including information indicating a change in the relative position of the subject image and the image sensor,
The image determination program according to claim 11. The information acquisition step acquires information on the image from each of the plurality of images,
The plurality of images include two or more predetermined images included in the same image group,
Each of the two or more predetermined images has associated information indicating that they are associated with each other;
A related image extracting step of extracting an image having the related information from the plurality of images;
The image determination program according to claim 11 or 12. The predetermined image processing is detection processing of a false color included in the predetermined image.
The image determination program according to any one of claims 11 to 13. A subject image incident on the image sensor via the imaging optical system, and a position changing means for changing a relative position of the image sensor in a plane parallel to the light receiving surface of the image sensor;
Image recording means for capturing two or more predetermined images by capturing the subject image using the image sensor and recording an image group including the predetermined images;
Position information acquisition means for acquiring information related to the relative position between the subject image and the image sensor when the predetermined image is captured;
Position information adding means for adding information on the relative position to the predetermined image;
Comprising
Imaging device. Each time the imaging device changes the relative position of the subject image incident on the imaging device via the imaging optical system and the imaging device in a plane parallel to the light receiving surface of the imaging device, the subject image Information about the relative position of the subject image and the image sensor when each image is recorded
An image data structure that enables predetermined image processing to be performed on two or more images in an imaging system.

Images