JP6407176B2 - Imaging apparatus, failure detection method, program, and recording medium - Google Patents

Description

  The present invention relates to an imaging device and a failure detection method for detecting a failure of the imaging device. The present invention further relates to a program for causing a computer to execute processing in the failure detection method, and a computer-readable recording medium for causing the computer to execute the program.

  There is known a method for detecting a failure of an imaging apparatus based on captured image data output from an imaging apparatus that continuously captures surrounding conditions. For example, in the vehicle camera exposure control device disclosed in Patent Document 1, when the vehicle is in a predetermined driving state, for example, the vehicle side brake is on, the gear position is neutral or parking, or the vehicle speed is a predetermined value. In the following cases, normal control is interrupted and the camera is diagnosed. In the diagnosis, the exposure amount of imaging is increased or decreased, and a failure is determined based on whether or not the brightness of the screen has increased or decreased according to the increase or decrease of the exposure amount.

  In the above apparatus, diagnosis is performed when normal imaging is not required, and normal imaging is not performed during diagnosis.

Japanese Patent Laying-Open No. 2005-73296 (page 10, FIG. 19)

  However, in an imaging device that continuously performs shooting, such as a monitoring camera or an in-vehicle camera, it is desirable to maintain a shooting state and a state in which a predetermined process is performed on image data obtained by shooting. . Therefore, it is not preferable to provide a time zone during which normal shooting is not performed for failure determination or failure detection. For this reason, it may not be appropriate to apply the failure determination method disclosed in Patent Document 1.

The imaging apparatus of the present invention
An imaging unit that receives light from the subject and performs imaging every imaging frame period;
An imaging control unit that controls an exposure condition for each imaging frame period in the imaging unit;
A signal processing unit that performs signal processing in a corresponding signal processing frame period for a captured image obtained by imaging in each imaging frame period in the imaging unit, and generates a processed image;
A low frequency processing unit that performs processing using the processed image at a frequency lower than the frequency of imaging in the imaging unit;
At least one imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low-frequency processing unit is obtained by imaging in the imaging unit, or at least one signal corresponding to the imaging frame period using the processing frame period, it possesses a failure detection processing unit for performing at least one of the failure detection processing of the image pickup unit and the signal processing unit,
The signal processing unit includes at least one signal processing frame period corresponding to an imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low frequency processing unit is obtained by imaging in the imaging unit In addition to receiving the image for failure detection instead of the captured image obtained by imaging in the imaging unit, the supplied image is subjected to signal processing under the failure detection processing conditions,
The failure detection processing unit determines whether or not a processed image generated as a result of performing signal processing on the failure detection image is as expected .

  According to the present invention, it is possible to detect a failure of an imaging apparatus while maintaining a normal shooting state.

1 is a functional block diagram schematically showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention. 2 is a functional block diagram schematically showing a configuration of an imaging unit according to Embodiment 1. FIG. 3 is a diagram illustrating a storage area in a memory used in the imaging apparatus according to Embodiment 1. FIG. It is a figure which shows roughly the order of imaging of a low exposure image and a high exposure image. The relationship between the imaging in the imaging part in Embodiment 1 and the process in each part in an image processing part is shown. It is a figure which shows an example of the brightness | luminance histogram of a low exposure image. It is a figure which shows an example of the brightness | luminance histogram of a high exposure image. It is a figure which shows the example of the combination pattern of the exposure conditions used. It is a figure which shows an example of the pattern table which prescribes | regulates the relationship between the feature-value obtained from a brightness | luminance histogram, and the transition between combination patterns. Operation in each part of the imaging unit and the image processing unit when only the default image use inspection is performed within one failure detection processing cycle is shown. The operations in the imaging unit and the image processing unit when only the inter-frame correlation test is performed within one failure detection processing cycle will be described. Operation in the imaging unit and the image processing unit when both the default image use inspection and the inter-frame correlation inspection are performed within one failure detection processing cycle will be described. It is a functional block diagram which shows roughly the structure of the imaging device of Embodiment 2 of this invention. 6 is a diagram illustrating a storage area in a memory used in the imaging device according to Embodiment 2. FIG. The relationship between the imaging in the imaging part 10 in Embodiment 2 and the process in each part in an image process part is shown. In the second embodiment, operations in the imaging unit and the image processing unit when both the default image use inspection and the inter-frame correlation inspection are performed in one failure detection processing cycle will be described. And FIG. 13 is a block diagram illustrating an example of a computer that executes processing of the imaging apparatus according to the first or second embodiment.

Embodiment 1 FIG.
FIG. 1 schematically shows a configuration of an imaging apparatus 1 according to Embodiment 1 of the present invention. The imaging device 1 includes an imaging unit 10, an image processing unit 20, and a memory 30.

  The imaging unit 10 receives light from the subject, performs imaging for each predetermined frame period, and outputs a time series of still images of consecutive frames as a moving image.

FIG. 2 schematically illustrates the configuration of the imaging unit 10.
As illustrated in FIG. 2, the imaging unit 10 includes an imaging optical system 12, a solid-state imaging device 13, a front end unit 14, and a drive circuit 15.

  The solid-state imaging device 13 is configured by a single plate type image sensor having a color filter array such as a Bayer array.

  The imaging optical system 12 forms an optical image by forming the incident light 11 on the imaging surface of the solid-state imaging device 13.

  The solid-state imaging device 13 photoelectrically converts the optical image formed by the imaging optical system 12 to generate an analog imaging signal Ra, and supplies the analog imaging signal Ra to the front end unit 14.

The front end unit 14 generates an analog signal by executing correlated double sampling (CDS) processing, programmable gain amplification (PGA), and the like on the imaging signal Ra.
The CDS process is a process for removing unnecessary components such as noise from the analog imaging signal Ra output from the solid-state imaging device 13.

  The front end unit 14 performs A / D conversion on the image signal subjected to the CDS process, and generates a digital image signal F in RAW format.

  As shown in FIG. 3, the memory 30 includes a frame buffer area 301, an exposure condition table storage area 302, a pattern table storage area 303, a default image storage area 304, a signal processing condition storage area 305, and an exposure condition storage. An area 306, an inspection data storage area 307, and a histogram storage area 309 are provided. The frame buffer area 301 includes a low-exposure image frame buffer area 301s and a high-exposure image frame buffer area 301t.

As illustrated in FIG. 1, the image processing unit 20 includes a signal processing unit 21, a histogram generation unit 22, an imaging control unit 23, an image composition unit 24, and a failure detection processing unit 25.
These components 21 to 25 are connected to each other via a bus (signal transmission path) 26 and also to the memory 30.

  The signal processing unit 21 performs signal processing on the digital image signal F output from the imaging unit 10. The signal processing performed by the signal processing unit 21 includes color synchronization processing, noise reduction processing, contour correction processing, white balance adjustment processing, signal amplitude adjustment processing, color correction processing, color space conversion, and the like. The signal processing unit 21 performs one or more of these processes. As a result of the processing, the signal processing unit 21 outputs a color image signal Re composed of a luminance component and a color difference component.

  The RAW format digital image signal F has only one color component of a plurality of color components for each pixel. The color synchronization process is performed in order to construct a color image signal in which all pixels have all color components, and is a process of interpolating color components that are lacking in each pixel. For example, when the color filter array of the solid-state imaging device 13 is a primary color Bayer array, the RAW digital image signal has only one color component of red, green, and blue for each pixel. In this case, the two missing color components for each pixel are interpolated from the color components of the surrounding pixels.

  The color image signal Re output from the signal processing unit 21 is transferred to the frame buffer area 301 of the memory 30 via the bus 26. The frame buffer area 301 stores the color image signal Re of each frame transferred by the bus 26 and holds (buffers) temporarily, that is, over a plurality of frame periods. As a result, the frame buffer area 301 maintains a state in which color image signals of a plurality of frames are stored.

Hereinafter, images represented by the image signals F and Re are represented by the same symbols F and Re. The same applies to other signals. Using the expression “image” instead of “image signal”, the signal processing unit 21 performs signal processing on the image F obtained by imaging by the imaging unit 10 to generate an image Re. I can say that there is. The same applies to other components.
For distinction, the image Re output from the signal processing unit 21 may be referred to as a “processed image”, and the image F output from the imaging unit 10 may be referred to as a “captured image”.

The histogram generation unit 22 receives the processed image Re of each frame generated by the signal processing unit 21 and held in the frame buffer area 301, and generates a luminance histogram of the luminance component of the processed image Re.
The generated luminance histogram is stored in the histogram storage area 309 of the memory 30 and is supplied to the imaging control unit 23.

The imaging control unit 23 controls the exposure conditions for imaging in the imaging unit 10 for each frame period.
The imaging control unit 23 controls the exposure conditions of the imaging unit 10 using the luminance histogram generated by the histogram generation unit 22 and the exposure condition table Tec stored in advance in the exposure condition table storage area 302 of the memory 30. To do.
The imaging control unit 23 also determines whether or not a condition for ending the failure detection process is satisfied during the failure detection process described later, and if it is determined that the above condition is satisfied, the failure detection process is performed. Terminate.

  The imaging control unit 23 individually controls the aperture of the imaging optical system 12, the exposure time (charge accumulation time) of the solid-state imaging device 13, and the gain (amplification factor) of the amplifier in the front end unit 14.

  The imaging control unit 23 supplies a control signal group EC to the imaging unit 10 for control of the imaging unit 10. As shown in FIG. 2, the control signal group EC includes a control signal ECa, a control signal ECs, and a control signal ECg.

The control signal ECa is for controlling the diaphragm and is supplied to the imaging optical system 12.
The control signal ECs is for controlling the exposure time (charge accumulation time) of the solid-state imaging device 13, and is supplied to the drive circuit 15, and the drive circuit 15 controls the exposure time (charge accumulation time). In order to control the exposure time, the drive circuit 15 generates a drive signal for driving the solid-state image sensor 13 according to the control signal ECs and supplies the drive signal to the solid-state image sensor 13.
The control signal ECg is for controlling the gain (amplification factor) of the amplifier in the front end unit 14 and is supplied to the front end unit 14.

  The imaging control unit 23 has a function of causing the imaging unit 10 to alternately perform imaging with the first exposure amount and imaging with the second exposure amount. That is, the imaging control unit 23 causes the imaging unit 10 to perform imaging with the first exposure amount in the first frame period, and output a captured image obtained by imaging with the first exposure amount. In the second frame period, imaging is performed with the second exposure amount, and a captured image obtained by imaging with the second exposure amount is output.

In the first imaging mode, that is, HDR (High Dynamic Range) mode, the imaging control unit 23 sets the exposure condition so that the first exposure amount and the second exposure amount are different from each other.
When the second exposure amount is larger than the first exposure amount, imaging with the first exposure amount is referred to as low-exposure imaging, and a captured image obtained by the imaging is referred to as a low-exposure image, and second exposure is performed. Imaging with a quantity is referred to as high exposure imaging, and a captured image obtained by the imaging is referred to as a high exposure image.
For distinction, a low-exposure image may be represented by a symbol “Fs” and a high-exposure image may be represented by a symbol “Ft”. Furthermore, in order to indicate which frame the image is, a number “(i)” (i is an integer) in parentheses may be attached. The low-exposure image Fs and the high-exposure image Ft are both images (captured images) output from the imaging unit 10, and the symbol “F” is used when they are not distinguished from each other or when they are common to both.
As a result of alternately performing low-exposure imaging and high-exposure imaging, low-exposure images and high-exposure images are obtained alternately.

  FIG. 4 shows low-exposure images Fs (1), Fs (3),..., And high-exposure images Ft (2), Ft (4) of each frame obtained by imaging in the HDR mode. , ... are shown schematically.

  In the second imaging mode, that is, the non-HDR mode, the imaging control unit 23 sets the exposure condition so that the first exposure amount and the second exposure amount have the same value.

  The image composition unit 24 includes a first processed image Re generated by performing signal processing in the signal processing unit 21 on the captured image Fs obtained by imaging in the first frame period by the imaging unit 10, and imaging. A synthesized image Rh is generated based on the second processed image Re generated by performing signal processing on the captured image Ft obtained by imaging in the second frame period by the signal processing unit 21. Output.

  In the HDR mode, the image composition unit 24 uses the processed image Re generated as a result of signal processing by the signal processing unit 21 for the captured image Fs obtained by imaging with the first exposure amount, and the second exposure amount. The synthesized image Rh with an expanded dynamic range is generated by HDR synthesizing the processed image Re generated as a result of signal processing by the signal processing unit 21 with respect to the captured image Ft obtained by imaging at, and output To do.

  In the non-HDR mode, the image synthesizing unit 24 processes the processed image Re generated as a result of signal processing by the signal processing unit 21 on the captured image Fs obtained by imaging with the first exposure amount, and the second exposure. Either one of the processed image Re generated as a result of signal processing by the signal processing unit 21 for the captured image Ft obtained by imaging with the amount is directly output as the composite image Rh.

In order to specify an image to be used for image composition, data indicating a frame number may be added to the processed image Re. In this case, the image composition unit 24 can specify an image to be used for image composition based on the data indicating the frame number.
The “image to be used for image composition” here may refer to a pair of images that are to be HDR-composed, or may be an image that is selected and used as it is as the composite image Rh.

  FIG. 5 shows a relationship between imaging in the imaging unit 10 and processing in each unit in the image processing unit 20 when the imaging apparatus is operating in the HDR mode. In FIG. 5, the 8 frame period is defined as one cycle, and the order of frames within each cycle is illustrated as a frame number in the uppermost row of FIG. The frame with frame number i in each cycle is referred to as the i-th frame in the cycle.

In FIG. 5, the start and end times of the frame periods with the same number are not necessarily the same in the imaging unit 10 and the processing units of the image processing unit 20, and there may be a time difference.
That is, the captured image obtained by imaging in the i-th frame period by the imaging unit 10 is processed by the signal processing unit 21 with a slight delay, but the period during which this processing is performed is the i-th period in the imaging unit 10. It can be seen that this is the i-th frame period related to the processing in the signal processing unit 21.
For the sake of distinction, a frame period related to imaging in the imaging unit 10 may be referred to as an “imaging frame period”, and a frame period related to signal processing in the signal processing unit 21 may be referred to as a “signal processing frame period”.

The processed image Re generated as a result of the signal processing in the i-th signal processing frame period by the signal processing unit 21 is once written in the memory 30 and then read and sent to the histogram generating unit 22. The luminance histogram is generated, and the period in which the luminance histogram is generated is a period corresponding to the i-th imaging frame period and the i-th signal processing frame period. It can be seen that it is the i-th frame period.
In the image composition unit 24, the processed image generated as a result of the processing in the i-th signal processing frame period by the signal processing unit 21 and the processed image generated as a result of the processing in the signal processing frame period immediately before that are processed. Synthesized. The period during which this composition is performed can be regarded as the i-th frame period related to image composition.
The i-th frame period regarding the exposure control of the imaging control unit 23 means a period during which exposure control for imaging in the i-th imaging frame period is performed.

  As shown in FIG. 5, low-exposure imaging is performed in the odd-numbered imaging frame period, that is, the first, third, fifth, and seventh imaging frame periods, and the low-exposure images Fs (1), Fs. (3), Fs (5) and Fs (7) are obtained, and high-exposure imaging is performed in the even-numbered imaging frame period, that is, the second, fourth, sixth and eighth imaging frame periods. High exposure images Ft (2), Ft (4), Ft (6) and Ft (8) are obtained. This is shown in the row of “imaging unit 10” in FIG.

The low-exposure image Fs and the high-exposure image Ft output from the imaging unit 10 are temporarily stored in the frame buffer area 301 of the memory 30 after being subjected to signal processing by the signal processing unit 21.
In the row of “signal processing unit 21” in FIG. 5, the signal processing unit 21 performs processing on the low-exposure image Fs (Fs (1), Fs (3), Fs (5), Fs (7)), Processing on the exposure image Ft (Ft (2), Ft (4), Ft (6), Ft (8)) is alternately performed, and processed images Re (1) to Re (8) are output. It is shown. For example, “Fs (1) → Re (1)” in the column of the first frame period indicates that a processed image Re (1) is generated as a result of signal processing on the low-exposure image Fs (1). The same applies to other frame periods.

  The histogram generation unit 22 reads out the processed image Re of each frame from the frame buffer area 301 of the memory 30 in the order of imaging, generates a luminance histogram of the processed image of each frame, and data (histograms) representing the generated luminance histograms Hs and Ht. Data) is output to the imaging control unit 23. Symbol Hs represents a luminance histogram of the low exposure image, and symbol Ht represents a luminance histogram of the high exposure image.

  In FIG. 5, the histogram generation unit 22 performs luminance histogram Hs (Hs (1)) of the processed image Re (Re (1), Re (3), Re (5), Re (7)) in the odd-numbered frame period. , Hs (3), Hs (5), Hs (7)) are generated, and processed images Re (Re (2), Re (4), Re (6), Re (8) are generated in the even-numbered frame period. )) Luminance histograms Ht (Ht (2), Ht (4), Ht (6), Ht (8)) are generated.

  Hereinafter, a captured image obtained by imaging in each (or i-th) imaging frame period is referred to as “a captured image of each (or i-th) frame” and a signal in each (or i-th) signal processing frame period. The processed image generated by the processing is referred to as “processed image of each (or i-th) frame”, and the luminance histogram of “processed image of each (or i-th) frame” is referred to as “each (or i-th) frame. May be referred to as a luminance histogram. In addition, when it can be distinguished by adding “Ft”, “Fs”, “Re”, etc., “the captured image of each (or i-th) frame” is simply referred to as “of each (or i-th) frame. “Image Ft” and “image Fs of each (or i-th) frame”, and “processed image of each (or i-th) frame” are simply referred to as “image Re of each (or i-th) frame”. Sometimes.

6 and 7 show examples of luminance histograms generated when the luminance value is represented by 8 bits, that is, when the luminance value is represented by a gradation value in the range of 0 to 255. FIG. FIG. 6 shows an example of the brightness histogram Hs of the low exposure image, and FIG. 7 shows an example of the brightness histogram Ht of the high exposure image.
6 and 7, the horizontal axis indicates the gradation value, and the vertical axis indicates the appearance frequency of the pixel having each gradation value.

  As described above, in the example shown in FIGS. 6 and 7, a luminance histogram is generated that represents the frequency of appearance of pixels having each gradation value. However, the present invention is not limited to this. For example, the present invention can also be applied to a case where a luminance histogram representing the frequency of appearance of a pixel having a gradation value in the class is generated for each of a plurality of classes each having a plurality of gradation values. The luminance histogram representing the frequency of appearance of pixels having each gradation value corresponds to the case where the number of gradation values belonging to each class is 1.

The imaging control unit 23 refers to the exposure condition table Tec stored in the exposure condition table storage area 302 of the memory 30 on the basis of the luminance histograms Hs and Ht generated by the histogram generation unit 22, and An exposure condition for imaging is determined for each frame period.
The exposure condition at the time of capturing the low-exposure image of each frame is determined based on the luminance histogram Hs generated from the low-exposure image obtained by the most recent image capturing, that is, two frames before.
On the other hand, the exposure conditions for capturing the high-exposure image of each frame are determined based on the luminance histogram Ht generated from the high-exposure image obtained by the most recent image capturing, that is, two frames before.

The determination of the exposure condition includes determination of the values of the control signals ECa, ECs, and ECg.
The imaging control unit 23 supplies the control signals ECa, ECs, and ECg having the determined values to the imaging unit 10 to control the imaging unit 10.

  For example, “Hs (1) → EB (3)” in the column of the third frame period in the “exposure control” row of the imaging control unit 23 in FIG. 5 is the luminance histogram Hs (1) of the first frame. , The exposure condition EB (3) for the third frame period is determined, and exposure control is performed using the determined exposure condition. Similarly, “Hs (8) → EB (2)” in the column of the second frame period indicates the second frame period based on the luminance histogram Ht (8) of the eighth frame of the previous cycle. The exposure condition EB (2) is determined and exposure control is performed using the determined exposure condition. The same applies to other frame periods.

  In the exposure condition table Tec, the correspondence relationship between the data representing the characteristics of the luminance histogram and the exposure conditions is stored in advance. The exposure conditions stored in the exposure condition table Tec include an exposure condition for low exposure imaging and an exposure condition for high exposure imaging.

As one of the exposure condition items, the exposure time is determined.
As the exposure time, for example, as shown in FIG. 8, five combination patterns P1 to P5 are prepared as combinations of the exposure time for high exposure imaging and the exposure time for low exposure imaging, and any of these combination patterns is prepared. Is selected.

In the example of FIG. 8, the exposure time for long exposure imaging (long exposure time) is fixed to 32 milliseconds in all of the combination patterns P1 to P5.
On the other hand, as shown in FIG. 8, the exposure time for short exposure imaging (short exposure time) varies depending on the combination pattern, and is 2 milliseconds for the combination pattern P1, 4 milliseconds for the combination pattern P2, and 8 for the combination pattern P3. It is 16 milliseconds for the combination pattern P4, and 32 milliseconds for the combination pattern P5.

  FIG. 9 shows an example of a pattern table Tps indicating the relationship between data (parameters) representing the characteristics of the luminance histogram and the combination pattern to be selected. The pattern table Tps is stored in the pattern table storage area 303 of the memory 30.

  The pattern table Tps of FIG. 9 is a luminance histogram of the processed image corresponding to the combination patterns P1 to P5 (transition source pattern) selected during the latest low-exposure imaging and the captured image (low-exposure image) obtained by the imaging. Based on the values of the five parameters Yo200, Yu100, Yu50, Yu25 and Yu12 representing the characteristics of the combination, P1 to P5 (transition destination patterns) to be selected at present (that is, next) are determined. is there.

  Here, Yo200 is a parameter indicating a ratio (unit: percent) of pixels having a luminance value of 200 or more to the total number of effective pixels. Yu100 is a parameter indicating the ratio (unit: percent) of the pixels having luminance values in the range of 0 to 100 to the total number of effective pixels. Yu50 is a parameter indicating a ratio (unit: percent) of pixels having luminance values in the range of 0 to 50 to the total number of effective pixels. Yu25 is a parameter indicating a ratio (unit: percent) of pixels having a luminance value in the range of 0 to 25 to the total number of effective pixels. Yu12 is a parameter indicating the ratio (unit: percent) of the pixels having luminance values in the range of 0 to 12 to the total number of effective pixels.

  For example, in a state where the combination pattern P1 is selected, if the parameter Yu100 is 90% or more, the combination pattern P2 is selected next. If the parameter Yo200 is 50% or more in the state where the combination pattern P2 is selected, the combination pattern P1 is selected next.

  The imaging control unit 23 calculates the parameters Yo200, Yu100, Yu50, Yu25, and Yu12 from the luminance histogram Hs of the low-exposure image obtained by imaging immediately before, that is, two frames before, and these parameters Yo200, Yu100, Yu50, One of the combination patterns P1 to P5 is selected based on the values of Yu25 and Yu12 and the most recently selected combination pattern.

  In the example of FIGS. 8 and 9, the exposure time at the time of high exposure imaging is fixed to a fixed 32 milliseconds, but the present invention is not limited to this. For example, in the case of high-exposure imaging as well as low-exposure imaging, the combination pattern selected during the most recent high-exposure imaging and the brightness of the processed image corresponding to the captured image (high-exposure image) obtained by the high-exposure imaging Even if a pattern table for predetermining a combination pattern to be selected next from data (parameter) values representing the characteristics of the histogram is prepared in advance, the exposure conditions for high exposure imaging can be set using this pattern table. Good.

  Further, not only the exposure time but also the aperture of the imaging optical system 12 and the gain of the amplifier in the front end unit 14 can be set as the exposure conditions. The larger the aperture, the smaller the exposure amount, the longer the exposure time, the larger the exposure amount, and the larger the gain, the larger the exposure amount.

  As described above, the image composition unit 24 generates a composite image based on the high-exposure image obtained by imaging in each imaging frame period and the low-exposure image obtained by imaging in the immediately preceding or immediately following imaging frame period. To do.

  In FIG. 5, in the second frame period, the processed image Re (2) of the second frame and the immediately preceding frame, that is, the processed image Re (1) of the first frame are used for synthesis. This is indicated by “Re (1) + Re (2) → Rh (2)”, and the same is shown for the other even-numbered frame periods.

  Image composition in the image composition unit 24 is performed once every two frame periods. That is, the processing in the image composition unit 24 is performed at a lower frequency than the image capturing in the image capturing unit 10. Therefore, it can be said that the image composition unit 24 is an aspect of a low-frequency processing unit that performs processing at a lower frequency than the image capturing in the image capturing unit 10.

  When the combination pattern P5 is selected, the exposure condition for high exposure imaging is the same as the exposure condition for low exposure imaging. In this state, the imaging apparatus according to the present embodiment operates in the non-HDR mode. In the non-HDR mode, the image composition unit 24 outputs one of the high exposure image and the low exposure image as a composite image as it is.

  In the imaging in the non-HDR mode, the high exposure image and the low exposure image are obtained by imaging under the same exposure conditions. For convenience, the “high exposure image” and the “low exposure image” used in the HDR mode are used. "Is also used for a captured image obtained by imaging in the non-HDR mode. That is, among the captured images obtained by the non-HDR mode imaging, the captured image obtained by the imaging at the same timing (in the same imaging frame period) as the high exposure imaging in the HDR mode is referred to as the high exposure image. In other words, among the captured images obtained by the non-HDR mode imaging, the captured image obtained by the imaging at the same timing (in the same imaging frame period) as the low exposure imaging in the HDR mode is referred to as the low exposure image. say.

As described above, while the imaging apparatus is operating in the non-HDR mode, a captured image F other than the captured image F corresponding to the processed image Re used for image synthesis is obtained by imaging among the high-exposure image and the low-exposure image. In the signal processing frame period corresponding to the imaging frame period to be performed, the signal processing unit 21 can perform processing different from processing for generating an image used in image synthesis. For example, when a composite image is generated only from a high-exposure image, a process for generating an image used for image composition is performed in a signal processing frame period corresponding to an imaging frame period in which a low-exposure image is obtained by imaging. Different processing can be performed. Therefore, in one aspect of the present invention, failure detection is performed using one or more signal processing frame periods of such signal processing frame periods. Failure detection is performed by the failure detection processing unit 25.
“Signal processing frame period corresponding to an imaging frame period in which a low-exposure image is obtained by imaging” is “a signal processed by the signal processing unit 21 if the low-exposure image output from the imaging unit 10 is in the HDR mode. “Processing frame period”.
“The signal processing frame period corresponding to the imaging frame period in which the captured image F other than the captured image F corresponding to the processed image Re used for image synthesis is obtained by imaging” is “the processed image Re used for image synthesis is signal processed. In other words, a signal processing frame period other than the signal processing frame period processed by the unit 21 ”.

Further, while the imaging apparatus is operating in the non-HDR mode, the imaging apparatus F is used for imaging the captured image F other than the captured image F corresponding to the processed image Re used for image synthesis of the high exposure image and the low exposure image. The exposure condition in the imaging frame period can be freely determined separately from the exposure condition used for imaging the captured image corresponding to the processed image used for image synthesis. For example, when a composite image is generated only from a high-exposure image, the exposure conditions during the imaging frame period during which the low-exposure image is captured can be freely determined. Therefore, in another aspect of the present invention, failure detection is performed using one or more of the imaging frame periods. The exposure condition is controlled by the imaging control unit 23, and the failure detection is performed by the failure detection processing unit 25.
“An imaging frame period for imaging the captured image F other than the captured image F corresponding to the processed image Re used for image synthesis” is “for imaging the captured image F corresponding to the processed image Re used for image synthesis”. In other words, the imaging frame period other than the imaging frame period.

In short, the failure detection processing unit 25 is one or more of the imaging frame periods in which the captured image F other than the captured image F corresponding to the processed image Re used in the image synthesis in the image synthesis unit 24 is obtained by imaging. The failure detection process of at least one of the imaging unit 10 and the signal processing unit 21 is performed in the imaging frame period or one or more signal processing frame periods among the signal processing frame periods corresponding to the imaging frame period.
The “captured image F corresponding to the processed image Re used for image composition” may be simply referred to as “captured image F used for image composition” or “image F used for image composition”. This is because the image F is also used in image composition after undergoing signal processing.

Hereinafter, the failure detection process will be described in more detail.
When the imaging apparatus is in a state of operating in the non-HDR mode, the imaging control unit 23 instructs the failure detection processing unit 25 to execute failure detection processing.
In FIG. 5, “Hs (1)”, “Hs (3)”, “Hs (5)”, and “Hs (7)” are entered in the “failure detection process start” row of the imaging control unit 23. Thus, the imaging control unit 23 is based on the parameters Yo200, Yu100, Yu50, Yu12 obtained from the luminance histograms Hs (1), Hs (3), Hs (5) or Hs (7) (more specifically, Specifically, it is shown that it is determined whether or not the failure detection process should be started (based on whether or not the combination pattern P5 is selected).
As the failure detection process, at least one of a default image use inspection and an inter-frame correlation inspection is performed.

In the default image use inspection, at least one signal processing frame period corresponding to an imaging frame period in which a low exposure image is obtained by imaging (that is, at least one signal processing frame period in which signal processing for low exposure imaging is performed in the HDR mode). ), The signal processing unit 21 processes the default image for failure detection (prepared or defined for failure detection) instead of the low-exposure image under the processing conditions for failure detection, and outputs a signal as a result of the processing. The failure detection processing unit 25 determines whether or not the processed image output from the processing unit 21 is as expected, that is, whether or not it matches the expected image stored in advance.
As the default image, an image stored in the default image storage area 304 of the memory 30 is used. As processing conditions for failure detection, those stored in the signal processing condition storage area 305 of the memory 30 are used.

In the inter-frame correlation test, imaging is performed using the exposure conditions for failure detection in at least one imaging frame period in which low-exposure imaging is performed by the imaging control unit 23. Whether or not the change between the image obtained by imaging in the imaging frame period corresponds to the change between the exposure condition for failure detection and the exposure condition in the other imaging frame period, That is, the failure detection processing unit 25 determines whether or not there is a correlation between the change in the exposure condition and the change in the image.
The other imaging frame period may be an imaging frame period in which high exposure imaging is performed, or may be an imaging frame period in which low exposure imaging is performed.

  When the other imaging frame period is an imaging frame period in which high-exposure imaging is performed, the exposure condition in the imaging frame period is determined to obtain an image used for image synthesis, and is used for failure detection. It is necessary to determine the exposure conditions based on the exposure conditions in the other imaging frame period. As described above, when the exposure condition is determined based on the exposure condition in the other imaging frame period, for example, the exposure condition in the other imaging frame period has a known relationship (for example, the exposure amount is known). The exposure condition may be determined so that the ratio becomes a ratio.

  When the other imaging frame period is an imaging frame period in which low-exposure imaging is performed, even in the imaging frame period, a further failure detection exposure condition that is different from the above-described failure detection exposure condition is used. Therefore, the exposure conditions for failure detection in the two imaging frame periods may be determined so that a known change occurs between them.

As a change between exposure conditions, a ratio between exposure amounts determined by the exposure conditions can be used, and as a change between images, a ratio between image brightnesses can be used.
An image for which a change between them, for example, a ratio is required, may be a processed image or a captured image.

  While performing the failure detection process, the imaging control unit 23 checks a change in the luminance histogram generated by the histogram generation unit 22, and if a change in the luminance histogram exceeds a predetermined threshold occurs, the failure detection processing unit 25 is instructed to end the failure detection process.

FIG. 10 is a diagram similar to FIG. 5 showing the operation of each part of the imaging unit 10 and the image processing unit 20 when only the default image use inspection is performed within one failure detection processing cycle.
In the example shown in FIG. 10, eight frame periods are set as one failure detection processing cycle, and inspection is performed once in each failure detection processing cycle. “Frame number” in the uppermost row in FIG. 10 indicates the order of each frame in each failure detection processing cycle.

  In the example shown in FIG. 10, as in the case of FIG. 5, high-exposure imaging is performed in the even-numbered imaging frame period, that is, the second, fourth, sixth, and eighth imaging frame periods. The low-exposure imaging is performed in the imaging frame period, that is, the first, third, fifth, and seventh imaging frame periods. However, in the non-HDR mode, the exposure condition for low exposure imaging is the same as the exposure condition for high exposure imaging, and the low exposure image is not used for image synthesis.

  Even if a low exposure image is output from the imaging unit 10, the signal processing unit 21 does not need to perform signal processing on the low exposure image, and as a result, the processed image generated as a result is not used for image synthesis. Therefore, at least one signal processing frame period corresponding to an imaging frame period in which a low-exposure image is obtained by imaging is selected, and a default image usage inspection is performed using the selected signal processing frame period.

For this inspection, the imaging control unit 23 instructs the failure detection processing unit 25 to perform a default image use inspection. This instruction includes designation of a timing at which a default image use inspection is to be performed, designation of a default image to be used in the examination, and designation of signal processing conditions for the default image.
As the above timing, at least one signal processing frame period corresponding to an imaging frame period in which a low-exposure image is obtained by imaging, which is not used for image synthesis, is specified. In the following, it is assumed that the third signal processing frame period is designated as the above timing.
Further, it is assumed that the default image Fk for failure detection stored in advance in the default image storage area 304 of the memory 30 is designated as the default image.
Furthermore, it is assumed that the signal processing conditions stored in advance in the signal processing condition storage area 305 of the memory 30 are designated as the signal processing conditions.
In accordance with such an instruction, the failure detection processing unit 25 performs a default image use inspection.

  In this inspection, the failure detection processing unit 25 reads the default image Fk for failure detection from the default image storage area 304 of the memory 30 according to the designation by the imaging control unit 23, and is obtained by imaging in the third imaging frame period. Instead of the captured image F (3), the signal is supplied to the signal processing unit 21. At the same time, the failure detection processing unit 25 reads the signal processing conditions from the signal processing condition storage area 305 of the memory 30 in accordance with the designation by the imaging control unit 23, and sets the read signal processing conditions in the signal processing unit 21. .

  As a result, the signal processing unit 21 performs signal processing on the predetermined image Fk under the set signal processing conditions, and outputs a processed image Re (3) as a result of the processing. This is indicated by “Fk → Re (3)” in the column of the third frame period in the row of the “signal processing unit 21” in FIG.

The signal processing unit 21 operates in the same manner as in FIG. 5 in the signal processing frame period other than the third signal processing frame period. However, the result of signal processing for an odd-numbered frame image (low-exposure image) other than the third frame is not used for image synthesis or failure detection. Therefore, signal processing for images of these frames may be omitted. In order to show that, no character is written in the odd-numbered frame period column other than the third frame period in the row of “signal processing unit 21” in FIG.
The same applies to the histogram generation unit 22 described later.
In the following description, it is assumed that signal processing is not performed on an odd-numbered frame image other than the third frame, and a processed image is not generated. The signal processing for the even-numbered frame image is performed in the same manner as in FIG. This is indicated by “Ft (2) → Re (2)” in the row of “signal processing unit 21” in FIG.

  The signal processing unit 21 stores the processed images Re (2), Re (4), Re (6) and Re (8), and the processed image Re (3) generated as a result of the signal processing in the memory 30. Among these, the processed images Re (2), Re (4), Re (6), and Re (8) are stored in the frame buffer area 301, and the processed image Re (3) is stored in the inspection data storage area 307. .

  When the writing of the processed image Re (3) to the inspection data storage area 307 is completed, the signal processing unit 21 notifies the failure detection processing unit 25 of that, that is, the processing completion.

The histogram generator 22 reads the processed images Re (2), Re (4), Re (6), and Re (8) from the memory 30, and sequentially displays the luminance histograms Ht (2), Ht (4), Ht (6). ) And Ht (8), the processed image Re (3) is also read from the memory 30, and its luminance histogram Hk (3) is generated.
The generation of these luminance histograms is indicated by “Re (2) → Ht (2)”, “Re (3) → Hk (3)”, etc. in the row of “histogram generation unit 22” in FIG.

The luminance histogram generated by the histogram generator 22 is stored in the memory 30.
Among these, the luminance histograms Ht (2), Ht (4), Ht (6), and Ht (8) are stored in the histogram storage area 309, and the luminance histogram Hk (3) is stored in the inspection data storage area 307.
When the writing of the luminance histogram Hk (3) to the memory 30 is completed, the histogram generation unit 22 notifies the failure detection processing unit 25 of the completion of the processing.

  Supplying the default image Fk read from the memory 30 to the signal processing unit 21, writing the processed image Re output from the signal processing unit 21 to the memory 30, and generating a histogram of the processed image Re read from the memory 30 The supply to the unit 22 and the writing of the luminance histogram generated by the histogram generation unit 22 to the memory 30 are performed via the bus 26.

  The failure detection processing unit 25 stores data stored in the inspection data storage area 307 of the memory 30 (that is, the processed image Re (3) output from the signal processing unit 21 and the luminance histogram Hk ( 3)) is read, and it is determined whether or not the read data is as expected. The determination as to whether it is as expected is performed by checking the correlation (similarity) with each expected value stored in advance in the expected value storage area 308 of the memory 30.

The determination as to whether or not there is a correlation with respect to the image is, for example, a value obtained by performing a predetermined process on the processed image Re (3) (processed image corresponding to the default image Fk), This is performed by comparing the correlation with the stored expected value with a predetermined threshold value.
A value obtained by performing a predetermined process is, for example, a checksum value. In this case, the expected value of the checksum value is stored in the expected value storage area 308 of the memory 30. The checksum value generation rule is also stored in the memory 30, and the failure detection processing unit 25 generates a checksum value from the processed image Re (3) according to the generation rule stored in the memory 30. .
Even when a value obtained by performing a predetermined process other than the checksum is used, information representing the predetermined process is stored in the memory 30.

  The determination as to whether or not there is a correlation with respect to the luminance histogram is performed, for example, by comparing the frequency of appearance of each gradation value and integrating the comparison results for all gradation values.

  The failure detection processing unit 25 performs the failure detection based on the processed image Re (3) and the luminance histogram Hk (3) in the third frame period. “Re (3), Hk (3)” is shown in the third frame period column of the “inspection” row.

As described above, it is determined that the data (that is, the processed image Re (3) and the luminance histogram Hk (3)) stored in the inspection data storage area 307 of the memory 30 are as expected. At this time, it is determined that the imaging unit 10, the signal processing unit 21, the low-exposure image frame buffer area 301s of the memory 30, and the bus 26 are operating normally.
When it is determined that it is not as expected, it is determined that there is a failure in any of the imaging unit 10, the signal processing unit 21, the low-exposure image frame buffer area 301s of the memory 30, and the bus 26.

  Image composition in the image composition unit 24 is performed based only on the processed image corresponding to the high-exposure image Ft. For example, “Re (2) → Rh (2)” in the column of the second frame period in the row of the “image composition unit 24” in FIG. 10 is composed only from the processed image Re (2) of the second frame. This means that an image Rh (2) is generated. The same applies to the other even-numbered frame periods.

  As described above, while performing the failure detection process, the imaging control unit 23 checks a change in the luminance histogram of the high-exposure image generated by the histogram generation unit 22. The change of the brightness histogram of the high exposure image is performed by comparing the brightness histogram of the high exposure image of each frame with the brightness histogram of the high exposure image immediately before, that is, two frames before.

  For example, “Ht (8), Ht (2)” in the column of the second frame period in the row of “Fault detection processing end” of the imaging control unit 23 in FIG. 10 is the luminance histogram Ht (2) of the second frame. And the luminance histogram Ht (8) of the eighth frame of the previous failure detection processing cycle is used for the determination. The same applies to other frame periods.

As a change in the luminance histogram, a change in the appearance frequency of each gradation value may be comprehensively determined. As a simpler method, a change in the average value, median value, difference between the minimum value and the maximum value (tone distribution width) may be used.
As the change, a change between frames, that is, a difference between a value in one frame and a value in another frame is used.
“Change has occurred” means that the amount of change between frames is equal to or greater than a threshold value.

  When a change occurs in the brightness histogram of the high-exposure image, the imaging control unit 23 instructs the failure detection processing unit 25 to end the failure detection processing. If there is no change in the brightness histogram of the high exposure image, that is, if the amount of change is less than the threshold, the failure detection process is continued.

  FIG. 11 is a diagram similar to FIG. 10 showing operations in the imaging unit 10 and the image processing unit 20 when only inter-frame correlation inspection is performed within one failure detection processing cycle.

  In the example shown in FIG. 11, as in the case of FIG. 10, high-exposure imaging is performed in the even-numbered imaging frame period, that is, the second, fourth, sixth, and eighth imaging frame periods. The low-exposure imaging is performed in the imaging frame period, that is, the first, third, fifth, and seventh imaging frame periods. However, in the non-HDR mode, the exposure condition for low exposure imaging is the same as the exposure condition for high exposure imaging, and the low exposure image is not used for image synthesis.

  Therefore, at least one imaging frame period in which a low-exposure image is obtained by imaging is selected, and an inter-frame correlation test is performed using the selected imaging frame period. In this inspection, the exposure condition in the selected imaging frame period is changed from the exposure condition in the immediately preceding or immediately following imaging frame period (the imaging frame period in which high exposure imaging is performed), and the above The failure detection processing unit 25 determines whether an image corresponding to the change of the exposure condition is obtained, in other words, whether there is a correlation between the change of the exposure condition and the change of the image obtained by imaging. Judgment. The image obtained by imaging may be a captured image or a processed image. Hereinafter, a case where a processed image (a processed image corresponding to a captured image obtained by imaging) is used as the “image obtained by imaging” will be described.

For this inspection, the imaging control unit 23 instructs the failure detection processing unit 25 to perform an interframe correlation inspection. This instruction includes information indicating the timing for performing the inter-frame correlation inspection and the exposure condition for failure detection to be used for imaging for the inspection.
As the above timing, at least one imaging frame period in which a low exposure image that is not used for image synthesis is obtained by imaging is designated. In the following, it is assumed that the seventh imaging frame period is designated as the above timing.
In addition, as the exposure condition for detecting the failure, the imaging control unit 23 specifies the exposure condition EB (a) determined based on the exposure condition EB (6) in the sixth imaging frame period. . Specifically, it is assumed that an exposure condition that gives an exposure amount having a predetermined ratio is specified with respect to the exposure condition EB (6) in the sixth imaging frame period. The imaging control unit 23 determines the exposure condition EB (6) for failure detection as described above, and notifies the failure detection processing unit 25 of the determined exposure condition.
In accordance with this instruction, the failure detection processing unit 25 performs an inter-frame correlation test.

  The imaging control unit 23 performs exposure control in an imaging frame period other than the seventh imaging frame period in the same manner as in FIG. 5, and performs the seventh imaging under the exposure condition EB (a) determined as described above. Exposure control is performed in the frame period. For this purpose, the imaging control unit 23 supplies the imaging unit 10 with a control signal group EC corresponding to the determined exposure condition EB (a).

  In FIG. 11, “EB (a)” is written in the column of the seventh frame period in the “exposure control” row of the imaging control unit 23, thereby exposing in the seventh imaging frame period. It is shown that condition EB (a) is used.

  The exposure conditions for imaging in the seventh imaging frame period are determined as described above, and the captured image F (a) obtained by imaging in the seventh imaging frame period and the imaging in the sixth imaging frame period are obtained. The obtained high exposure image Ft (6) is used for inspection. Specifically, the ratio between the exposure amount determined by the exposure condition EB (6) in the sixth imaging frame period and the exposure amount determined by the exposure condition EB (a) in the seventh imaging frame period, and the sixth imaging frame. The brightness of the processed image Re (6) corresponding to the captured image Ft (6) obtained by imaging during the period and the processed image Re (corresponding to the captured image F (a) obtained by imaging during the seventh imaging frame period It is determined whether or not there is a correlation between the ratio to the brightness of 7).

For such determination, the signal processing unit 21 performs normal operation (frame) on the captured images other than the captured image F (a) of the seventh frame among the captured images output from the imaging unit 10. Signal processing is performed in the same manner as when the inter-correlation test is not performed), and signal processing is performed on the captured image F (a) of the seventh frame in the same manner as signal processing on these captured images.
When the signal processing conditions in the signal processing unit 21 change for each frame, the signal processing conditions for the captured image F (a) of the seventh frame are set for the captured image Ft (6) of the sixth frame. Same as signal processing conditions.

The result of signal processing for an odd-numbered frame image (low-exposure image) other than the seventh frame is not used for image synthesis or failure detection. Therefore, signal processing for images of these frames may be omitted. In order to show this, no characters are written in the odd-numbered frame period column other than the seventh frame period in the row of “signal processing unit 21” in FIG.
The same applies to the histogram generation unit 22 described later.
In the following description, it is assumed that signal processing is not performed on an odd-numbered frame image other than the seventh frame, and a processed image is not generated.

  The signal processing unit 21 stores the processed images Re (2), Re (4), Re (6) and Re (8), and Re (7) generated as a result of the signal processing in the memory 30. Among these, the processed images Re (2), Re (4), Re (6), and Re (8) are stored in the frame buffer area 301, and the processed image Re (7) is stored in the inspection data storage area 307. .

  When the writing of the processed images Re (6) and Re (7) to the memory 30 is completed, the signal processing unit 21 notifies the failure detection processing unit 25 of the completion of the processing.

  In FIG. 11, “F (a)” indicates that the captured image F (a) is obtained by imaging in the seventh imaging frame period, and for the captured image F (a) in the seventh signal processing frame period. It is indicated by “F (a) → Re (7)” that signal processing is performed to generate a processed image Re (7).

  The histogram generator 22 reads the processed images Re (2), Re (4), Re (6), and Re (8) from the memory 30, and sequentially displays the luminance histograms Ht (2), Ht (4), Ht (6). ) And Ht (8), the processed image Re (7) is also read out from the memory 30, and its luminance histogram Ha (7) is generated.

  The generation of these luminance histograms is indicated by “Re (2) → Ht (2)”, “Re (7) → Ha (7)”, etc. in the row of “histogram generation unit 22” in FIG.

The luminance histogram generated by the histogram generator 22 is stored in the memory 30.
Among these, the luminance histograms Ht (2), Ht (4), Ht (6) and Ht (8) are stored in the histogram storage area 309, and the luminance histogram Ha (7) is stored in the inspection data storage area 307.
When the writing of the luminance histograms Ht (6) and Ha (7) to the memory 30 is completed, the histogram generation unit 22 notifies the failure detection processing unit 25 of the completion of the processing.

  Writing the processed image Re output from the signal processing unit 21 into the memory 30, supplying the processed image Re read from the memory 30 to the histogram generating unit 22, and the memory of the luminance histogram generated by the histogram generating unit 22 Writing to 30 is performed via the bus 26.

  The failure detection processing unit 25 reads the data stored in the memory 30 (that is, the processed images Re (6) and Re (7) and the luminance histograms Ht (6) and Ha (7)), and the read data; Using the exposure conditions EB (6) and data representing EB (a) used for imaging to obtain the captured image F corresponding to the processed images Re (6) and Re (7), the presence / absence of a failure is determined. Do. For this process, the exposure conditions EB (6) and EB (a) determined by the imaging control unit 23 are notified to the failure detection processing unit 25 as described above.

  The failure detection processing unit 25 determines that the change between the processed image Re (6) and the processed image Re (7) is an exposure condition EB (6) and a captured image at the time of imaging for obtaining the captured image Ft (6). It is determined whether or not there is a correlation with the change with the exposure condition EB (a) at the time of imaging for obtaining F (a).

  The failure detection processing unit 25 also changes the exposure between the luminance histogram Ht (6) and the luminance histogram Ha (7) according to the exposure condition EB (6) and the imaging for obtaining the captured image Ft (6). It is determined whether or not there is a correlation with the change with the exposure condition EB (a) at the time of imaging for obtaining the image F (a).

  For example, if the exposure time is changed to ½ as one of the exposure condition items and the other exposure condition items (aperture, gain) are kept the same (assuming that the subject itself has not changed) The brightness of the image should be about ½.

The failure detection processing unit 25 determines whether or not there is a correlation between the change in the exposure condition and the change in the luminance histogram based on such a known relationship.
For example, the failure detection processing unit 25 determines that the ratio between the brightness of the processed images Re (6) and Re (7) corresponding to the captured image obtained by imaging is the exposure condition EB ( It is determined whether or not there is a correlation with the ratio between the exposure amounts determined by 6) and EB (a).
The failure detection processing unit 25 also determines whether the brightness of the images indicated by the luminance histograms Ht (6) and Ha (7) of the processed images Re (6) and Re (7) corresponding to the captured image obtained by imaging. It is determined whether or not there is a correlation with the ratio between the exposure amounts determined by the exposure conditions EB (6) and EB (a) used in the imaging.

For example, the exposure time in the seventh imaging frame period is set to ½ of the exposure time in the sixth imaging frame period (other exposure conditions are the same), and the processed image Re (7) of the seventh frame is set. The ratio of the brightness of the image indicated by the brightness or brightness histogram Ha (7) to the brightness of the processed image Re (6) of the sixth frame or the brightness indicated by the brightness histogram Ht (6) is , 1/2 or a value close thereto, it is determined that there is a correlation.
Whether or not the brightness ratio is close to 1/2 can be determined by whether or not the difference between the calculated brightness ratio and 1/2 is equal to or less than a predetermined threshold. .

  The failure detection processing unit 25 performs failure detection based on the processed images Re (6) and Re (7), the luminance histograms Ht (6) and Ha (7), and the exposure conditions EB (6) and EB (a). 11, “Re (6), Re (7), Ht (6), Ha (7)” in the column of the seventh frame period in the “interframe correlation check” row of the failure detection processing unit 25 in FIG. , EB (6), EB (a) ".

As described above, when it is determined that there is a correlation between the change in the exposure condition and the change in the processed image corresponding to the captured image obtained by imaging, the imaging unit 10, the signal processing unit 21, and the memory It is determined that the 30 low-exposure image frame buffer areas 301 s and the bus 26 are operating normally.
When it is determined that there is no correlation, it is determined that there is a failure in any of the imaging unit 10, the signal processing unit 21, the low-exposure image frame buffer region 301 s of the memory 30, and the bus 26.

  Similar to the case of FIG. 10, the image composition in the image composition unit 24 is performed based only on the processed image corresponding to the high exposure image Ft. “Re (2) → Rh (2)” in the column of the second frame period, for example, in the row of the “image composition unit 24” in FIG. 11 is composed only from the processed image Re (2) of the second frame. This means that an image Rh (2) is generated. The same applies to the other even-numbered frame periods.

  As described above, while performing the failure detection process, the imaging control unit 23 checks a change in the luminance histogram of the high-exposure image generated by the histogram generation unit 22. When a change occurs in the brightness histogram of the high exposure image, the imaging control unit 23 instructs the failure detection processing unit 25 to end the failure detection processing.

  The default image use inspection and the inter-frame correlation inspection have been described above with reference to FIGS. 10 and 11. These two inspections may be performed one after the other. For example, a predetermined image use inspection may be performed in a certain failure detection cycle, an interframe correlation inspection may be performed in the next failure detection processing cycle, and such an operation may be repeated.

  In addition, after performing one of the default image use inspection and inter-frame correlation inspection repeatedly over a plurality of failure detection cycles, the other inspection is repeated over a plurality of failure detection processing cycles, or only once (one failure detection It may be performed only in the processing cycle.

  Furthermore, both the default image use inspection and the inter-frame correlation inspection may be performed within one failure detection processing cycle, and such an operation may be repeated over a plurality of failure detection processing cycles.

Hereinafter, a case where both the default image use inspection and the interframe correlation inspection are performed in one failure detection processing cycle will be described in more detail.
When the imaging apparatus enters a state of operating in the non-HDR mode, the imaging control unit 23 instructs the failure detection processing unit 25 to perform the default image use inspection and the inter-frame correlation inspection.

This instruction includes designation of the timing at which the default image utilization inspection is to be performed and designation of the timing at which the interframe correlation inspection is to be performed. As the timing at which the default image utilization inspection should be performed, at least one signal processing frame period corresponding to an imaging frame period in which an image other than the frame image used in image synthesis is obtained by imaging is designated. As the timing at which the inter-frame correlation test is to be performed, at least one imaging frame period in which an image other than the frame image used in image synthesis is obtained by imaging is designated.
In the following, it is instructed that the default image use inspection should be performed using the third signal processing frame period, and the interframe correlation inspection should be performed using the sixth and seventh imaging frame periods. And
In accordance with this instruction, the failure detection processing unit 25 performs a default image use inspection and an interframe correlation inspection.

FIG. 12 shows the operations in the imaging unit 10 and the image processing unit 20 when both the default image use inspection and the inter-frame correlation inspection are performed in one failure detection processing cycle. FIG.
In the example shown in FIG. 12, the default image use inspection and the inter-frame correlation inspection default image failure detection are performed once in each failure detection processing cycle, with 8 frame periods as one failure detection processing cycle.

  In the example shown in FIG. 12, as in the case of FIGS. 10 and 11, high-exposure imaging is performed in the even-numbered imaging frame period, that is, the second, fourth, sixth, and eighth imaging frame periods. The low-exposure imaging is performed in the odd-numbered imaging frame period, that is, in the first, third, fifth, and seventh imaging frame periods. However, in the non-HDR mode, the exposure condition for low exposure imaging is the same as the exposure condition for high exposure imaging, and the low exposure image is not used for image synthesis.

In the example shown in FIG. 12, as in the case of FIG. 10, the default image use inspection is performed using the third signal processing frame period.
Similarly to the case of FIG. 11, a failure detection exposure condition is set as the exposure condition in the seventh imaging frame period, and the processed image Re (()) corresponding to the captured image F (a) obtained by the imaging is set. 7) and the processed image Re (6) corresponding to the high-exposure image Ft (6) obtained by imaging in the immediately preceding imaging frame period, that is, the sixth imaging frame period, are used for inter-frame correlation inspection. Is done.

  The details of the default image utilization inspection are the same as those described with reference to FIG. The details of the inter-frame correlation check are the same as described with reference to FIG.

  If a change occurs in the brightness histogram of the high-exposure image generated by the histogram generation unit 22 during the failure detection processing, the imaging control unit 23 causes the failure detection processing unit 25 to end the failure detection processing. Instruct.

  By performing both the default image use inspection and the inter-frame correlation inspection, and combining the results of these detection processes, it may be possible to identify a site with a failure. For example, in the inter-frame correlation test, it is determined that there is a failure in any of the imaging unit 10, the signal processing unit 21, the histogram generation unit 22, and the memory 30, while the signal processing unit 21 is determined from the result of the default image usage test. When it is determined that the histogram generation unit 22 and the memory 30 are normal, it is possible to determine that a failure has occurred in the imaging unit 10 by combining the results of these two tests.

  In the present embodiment, at least one imaging frame period for capturing a low-exposure image or the imaging frame period when only a high-exposure image is used for image synthesis when the combination pattern P5 of exposure conditions is selected. Since the failure detection process is performed using at least one signal processing frame period corresponding to, it is possible to detect the failure while maintaining the photographing state.

In the example described with reference to FIGS. 10, 11, and 12, at least one of the default image use inspection and the inter-frame correlation inspection is performed once in a failure detection cycle having a certain length.
However, the present invention is not limited to this. For example, when each inspection (default image use inspection or inter-frame correlation inspection) is completed, the failure detection processing unit 25 notifies the imaging control unit 23 accordingly, and the imaging control unit immediately or after a certain period of time accordingly. 23 may give an instruction to start the next inspection (default image use inspection or interframe correlation inspection).

In this case, in order to perform the above-described two inspections (default image use inspection and inter-frame correlation inspection) in succession, the imaging control unit 23 determines the order in which the two inspections are performed, and the failure detection processing unit 25 Give instructions to start each test.
Hereinafter, the case where the default image use inspection is performed first will be described in detail.

The imaging control unit 23 instructs the failure detection processing unit 25 to start the default image use inspection. This instruction includes specification of the timing at which the default image use inspection should be performed. The failure detection processing unit 25 executes a default image use inspection according to the instruction. When the default image use inspection ends, the failure detection processing unit 25 notifies the imaging control unit 23 of the end.
Upon receiving this notification, the imaging control unit 23 instructs the failure detection processing unit 25 to start an interframe correlation test. This instruction includes designation of the timing at which inter-frame correlation determination should be performed. The failure detection processing unit 25 performs an inter-frame correlation test according to this instruction. When the inter-frame correlation test is completed, the failure detection processing unit 25 notifies the imaging control unit 23 of the completion.

Further, as described above, at least one of the two inspections may be repeated a plurality of times instead of performing the default image use inspection and the interframe correlation inspection once.
For example, after one of the default image use inspection and the interframe correlation inspection is repeatedly performed a plurality of times, the other inspection may be performed once or a plurality of times.

  Failure detection processing, for example, each of the default image use inspection and inter-frame correlation inspection is performed at appropriate intervals determined based on the respective failure occurrence rates of the imaging unit 10, the image processing unit 20, and the memory 30. As good as

For example, when at least one of the default image use inspection and the inter-frame correlation inspection is completed, the failure detection processing unit 25 notifies the imaging control unit 23 to that effect, and a predetermined time has elapsed accordingly. Later, the imaging control unit 23 may give an instruction to start the next inspection.
When both the default image use inspection and the interframe correlation inspection are performed, the default image use inspection and the interframe correlation inspection may be performed at different time intervals.

  If the failure detection processing is performed every appropriate period, it is not necessary to perform excessive failure detection processing, which leads to a reduction in the processing amount of the entire imaging apparatus.

In the above example, the signal processing conditions for the failure detection default image Fk in the signal processing unit 21 are set in advance, but this is not necessarily limited to the signal processing conditions used. Does not mean
It is also possible to detect a failure in a specific part inside the image processing by changing the above signal processing conditions. If the failed part can be specified, control can be performed so as to avoid the failed part thereafter.

  Here, the “specific part” means, for example, any one of various functions in the signal processing unit 21. For example, the signal processing unit 21 performs “color synchronization processing, noise reduction processing, contour correction processing, white balance adjustment processing, signal amplitude adjustment processing, color correction processing, and color space conversion” as described above. Means one of the following processes. In this case, it is determined whether each process has a failure, and the process determined to have a failure is not executed during normal operation.

  For example, for each of the cases where the contour correction process of the signal processing unit 21 is performed and the case where the contour correction process is not performed, the resulting processed image is compared with the respective expected values, so that the fault correction in the contour correction process is performed. If the presence / absence is detected and it is detected that there is a failure in the contour correction processing, the contour correction processing is not executed during normal operation.

  In addition, each of the signal processing unit 21, the histogram generation unit 22, and the bus 26 is set as a specific part, and it is determined which of the parts has failed, and the part that has been determined to have failed is operated during normal operation. It is good not to let it. In this case, the backup signal processing unit 21, the histogram generation unit 22, and the bus 26 may be provided, and a part prepared for backup may be used as a part where a failure is detected. In a system that requires high reliability, it is desirable to have a configuration including a backup circuit as described above.

In the above embodiment, the default image Fk used in the default image utilization inspection is stored in advance in a predetermined area (default image storage area 304) of the memory 30, but the present invention is not limited to this. . For example, the image generation rule is stored in advance in the failure detection processing unit 25, the failure detection processing unit 25 transmits the image generation rule to the signal processing unit 21 at the start of the failure detection processing, and the signal processing unit 21 fails. A default image may be generated in accordance with the image generation rule transmitted from the detection processing unit 25, the generated default image may be written in the low-exposure image frame buffer area 301s of the memory 30, and the written data may be read out. An example of the default image generated according to the image generation rule is a gradation pattern (a pattern in which gradation values gradually increase).
With such a configuration, a special memory area (304) for holding the default image (Fk) is not necessary, and the low-exposure image frame buffer area 301s of the memory 30 used in normal processing is not necessary. A failure can be detected. In this case, information indicating a rule for generating a default image or information indicating a part in which the rule is stored, for example, an address in the memory 30, may be given as information for specifying the default image.

In the above embodiment, in order to determine whether or not there is a correlation between the processed image Re obtained by performing signal processing on the default image and the expected value in the default image utilization inspection, the checksum value and its expected value Are comparing.
The present invention is not limited to this.
For example, the image may be divided into a plurality of regions, and an average value or an integrated value of luminance values for each divided region may be compared with an expected value obtained in advance, and the comparison result may be comprehensively determined. In this case, the average value or integrated value of the luminance values for each of the divided regions corresponds to “a value obtained by performing a predetermined process” on the image. In short, it is only necessary to check the correlation between the value obtained by applying a predetermined process to the image and the expected value. More generally, it may be determined whether or not there is a correlation between the processed image obtained as a result of processing the default image and the expected image.

  In the above example, the exposure condition in one imaging frame period (for example, the sixth imaging frame period) in which a captured image corresponding to the processed image used for image composition in the non-HDR mode is obtained by imaging, and the image in the non-HDR mode. Exposure conditions in one imaging frame period (for example, a seventh imaging frame period) in which an image that is not used for synthesis is obtained by imaging, and captured images (Ft (6), F) obtained by imaging in those imaging frame periods (A)) (specifically, processed images (Re (6), Re (7)) obtained by signal processing them) and their luminance histograms (Ht (6), Ha (7)) And inter-frame correlation test.

  Instead, in the non-HDR mode, exposure conditions for two imaging frame periods (for example, the fifth and seventh imaging frame periods) in which an image that is not used for image synthesis is obtained by imaging are separately determined for failure detection. Using the exposure conditions EB (b) and EB (c), these exposure conditions EB ((b) and EB (c)) and the captured images (F (b), F (c)) (specifically, processed images (Re (5), Re (7)) obtained by signal processing them) and their luminance histograms (Hc (5), Ha (7)). In such a configuration, exposure conditions (EB (b), EB (c)) at the time of imaging can be freely determined (for image synthesis). Imaging corresponding to the processed image used Therefore, the exposure conditions (EB (b), EB (c)) can be determined to be optimal for failure detection, and the exposure conditions (EB). (B), EB (c)) can be determined in advance and stored in, for example, the memory 30. In this case, as information for specifying the exposure condition, information necessary for obtaining the exposure condition, for example, exposure What is necessary is just to give the information which shows the site | part where the conditions are memorize | stored, for example, the address in the memory 30.

  In the above embodiment, a composite image is generated using only a high-exposure image, and at least one imaging frame period for capturing a low-exposure image or at least one signal processing frame period corresponding to the imaging frame period is used. Thus, failure detection processing (imaging in inter-frame correlation inspection or signal processing in default image utilization inspection) is performed. This may be reversed. That is, a fault detection process (at least one imaging frame period in which a composite image is generated using only a low-exposure image and at least one signal processing frame period corresponding to the imaging frame period is captured ( Imaging in the inter-frame correlation inspection or signal processing in the default image utilization inspection) may be performed.

  Also, the above two processes may be performed alternately. By doing so, for example, the signal processing unit 21, the histogram generation unit 22, the bus 26, and the memory 30 are divided into a part related to processing or storage of a high exposure image and a part related to processing or storage of a low exposure image. If it is separable, it can be determined which part has failed.

Embodiment 2. FIG.
FIG. 13 schematically shows a configuration of an imaging apparatus 1a according to the second embodiment of the present invention. The imaging apparatus 1a is generally the same as the imaging apparatus 1 of FIG. 1, but includes an image processing unit 20a instead of the image processing unit 20. The image processing unit 20 a is generally the same as the image processing unit 20, but an average luminance value calculation unit 27 is provided instead of the histogram generation unit 22, and an image recognition unit 28 is provided instead of the image synthesis unit 24. It is different in point.

  The imaging apparatus 1 according to the first embodiment operates in either the HDR mode or the non-HDR mode. In the HDR mode, the exposure condition is switched for each frame. In the imaging apparatus 1a according to the second embodiment, the exposure condition is switched. Such switching is not performed.

  In FIG. 13, the same reference numerals as those in FIG. 1 denote the same components. However, the memory 30 in FIG. 13 includes a frame buffer area 301, an exposure condition table storage area 302, a default image storage area 304, a signal processing condition storage area 305, and an exposure condition storage area 306, as shown in FIG. A test data storage area 307, an expected value storage area 308, and an average luminance value storage area 310.

  The average luminance value calculation unit 27 calculates the average luminance value Ya of the processed image Re of each frame read from the frame buffer area 301 of the memory 30. The processed image Re is a color image, and the average luminance value Ya is an average value of the luminance component of the processed image Re over the entire screen. The calculated average luminance value Ya is stored in the average luminance value storage area 310 of the memory 30.

  The imaging control unit 23 determines an exposure condition using the average brightness value Ya calculated by the average brightness value calculation unit 27 and the exposure condition table Tec stored in advance in the exposure condition table storage area 302 of the memory 30. . The determination of the exposure condition includes determination of the values of the control signals ECa, ECs, and ECg. The imaging control unit 23 supplies control signals ECa, ECs, and ECg having determined values to the imaging unit 10 to control the exposure conditions of the imaging unit 10.

  The imaging control unit 23 of the imaging apparatus 1a according to the present embodiment controls the operation of the imaging unit 10 so that the variation of the average luminance value Ya is reduced, and performs imaging.

  Similarly to the first embodiment, the signal processing unit 21 performs signal processing on the captured image F of each frame output from the imaging unit 10, and generates a processed image Re of each frame as a result of the processing. To do. The processed image Re is stored in the frame buffer area 301 of the memory 30.

The image recognition unit 28 performs image recognition on the processed image Re of some frames among the processed images Re of a plurality of frames. Specifically, out of the processed images of a plurality of frames stored in the frame buffer area 301, a processed image corresponding to a captured image of a frame obtained by imaging at a fixed time interval in the imaging unit 10 is extracted. Then, image recognition is performed.
For example, an image for each predetermined number of frames is extracted. Below, the case where the image of 1 frame of 8 frames is extracted and image recognition is performed about the extracted image is demonstrated.

  FIG. 15 is a diagram similar to FIG. 5 showing the relationship between the imaging in the imaging unit 10 and the processing in each unit in the image processing unit 20a.

The captured image F output from the imaging unit 10 is subjected to signal processing by the signal processing unit 21, and the processed image Re generated by this signal processing is temporarily held in the frame buffer area 301 of the memory 30.
In FIG. 15, the signal processing unit 21 performs processing on the captured image F (F (1) to F (8)) in each signal processing frame period, so that the processed image Re (Re (1) to Re (8) is processed. )) Is output. For example, “F (1) → Re (1)” in the column of the first frame period indicates that a process is performed on the captured image F (1), and as a result, a processed image Re (1) is generated. Show. The same applies to other frame periods.

The average luminance value calculation unit 27 reads out the processed image Re of each frame from the frame buffer area 301 of the memory 30 in the order of imaging, calculates the average luminance value Ya of the processed image Re of each frame, and represents the calculated average luminance value Ya. Data is output to the imaging control unit 23.
In FIG. 15, the average luminance value calculation unit 27 calculates the average luminance values Ya (1) to Ya (8) of the processed images Re (1) to Re (8) in each frame period. It is shown.

The imaging control unit 23 refers to the exposure condition table Tec stored in the memory 30 based on the average luminance value Ya calculated by the average luminance value calculating unit 27, and the exposure condition at the time of imaging in the imaging unit 10 To decide.
The exposure condition EB (i) at the time of imaging in each imaging frame period is the average of the processed images Re obtained by performing signal processing on the captured image F obtained by imaging in the latest, that is, the previous frame period. It is determined based on the luminance value Ya (i-1). Conversely, the exposure used in the next imaging frame period based on the average luminance value Ya (i) of the processed image Re obtained by performing signal processing on the captured image F obtained by imaging in each imaging frame period. Condition EB (i + 1) is determined.
“Ya (1) → EB (2)” in the column of the second frame period in the “exposure control” row of the imaging control unit 23 in FIG. 15 is the average luminance value Ya (1) of the first frame. Based on this, the exposure condition EB (2) in the second imaging frame period is determined, and exposure control in the second imaging frame period is performed using the determined exposure condition. The same applies to other frame periods.

Of the processed images Re stored in the memory 30, processed images Re of a predetermined number of frames are input to the image recognizing unit 28, and the image recognizing unit 28 receives the input processed images Re. Then, image recognition is performed, and a result Rr of image recognition is output.
In order to be able to specify the processed image Re to be used for image recognition, data indicating a frame number may be added to the processed image Re. In this case, the image recognition unit 28 can identify the processed image Re to be used for image recognition based on the data indicating the frame number.
In FIG. 15, image recognition is performed on the processed image Re (3) corresponding to the captured image F (3) obtained by imaging in the third imaging frame period, and the recognition result Rr (3) is output. Is indicated by “Re (3) → Rr (3)” in the column of the third frame period in the row of the “image recognition unit 28”.

  The image recognition mentioned here includes, for example, determination of the type of a subject in the image, for example, determination of whether a person is included or a vehicle, and determination of the size of a subject determined to be included.

As described above, the image recognition in the image recognition unit 28 is performed on the processed images Re of some frames among the processed images Re corresponding to the captured images F of a plurality of frames output from the imaging unit 10. .
For example, in the illustrated example, the frame rate of the image used for image recognition is 1/8 of the frame rate of the image output from the imaging unit 10.
Therefore, it can be said that the image recognizing unit 28 is an aspect of a low-frequency processing unit that performs processing at a lower frequency than the image capturing in the imaging unit 10.

Note that the image recognition is not necessarily performed regularly, and the image recognition may be performed irregularly, for example, only when some condition is satisfied.
For example, the image recognition may be performed when the brightness of the image changes significantly, or when a scene change is detected, or the image recognition may be performed only when an instruction is given from the outside. .
In such a case, it is possible to specify an image to be used for image recognition, and to add information indicating that the brightness of the image has changed significantly or that a scene change has been detected. Also good. In this case, the image recognition unit 28 can specify an image to be used for image recognition based on the additional information.

  In the present embodiment, at least one imaging frame period in which a captured image F other than the captured image F corresponding to the processed image Re of the frame used for image recognition in the image recognition unit 28 is obtained by imaging, or at least corresponding to this. Fault detection is performed using one signal processing frame period.

Specifically, in the signal processing frame period corresponding to the imaging frame period in which the captured image other than the captured image corresponding to the processed image of the frame used for image recognition is obtained by imaging, the signal processing unit 21 performs image recognition. A process different from the process for generating the image to be used can be performed. Therefore, in one aspect of the present invention, failure detection is performed using one or more signal processing frame periods of such signal processing frame periods. Failure detection is performed by the failure detection processing unit 25.
“Signal processing frame period corresponding to imaging frame period in which captured image other than captured image corresponding to processed image of frame used for image recognition is obtained by imaging” is “captured image of frame not used in image recognition” If it is output from the unit 10 and is in a normal operation state (a state in which failure detection is not performed), it means a signal processing frame period processed by the signal processing unit 21, and “the processed image of the frame used for image recognition is In other words, a signal processing frame period other than the signal processing frame period processed by the signal processing unit 21 can be used.

In addition, in an imaging frame period in which a captured image other than the captured image corresponding to the processed image used for image recognition is obtained by imaging, the exposure condition is used for capturing a captured image corresponding to the processed image used for image recognition. It can be freely determined separately from the exposure conditions. Therefore, in another aspect of the present invention, an image obtained by performing imaging using an exposure condition for failure detection using one or more imaging frame periods of such imaging frame periods, and an image obtained by this imaging Fault detection will be performed using. The exposure condition is controlled by the imaging control unit 23, and the failure detection is performed by the failure detection processing unit 25.
“An imaging frame period in which a captured image other than the captured image corresponding to the processed image used for image recognition is obtained by imaging” is “other than an imaging frame period in which the captured image corresponding to the processed image used for image recognition is obtained by imaging. It can be paraphrased as “an imaging frame period”.

As the failure detection process, as in the first embodiment, at least one of the default image use inspection and the inter-frame correlation inspection is performed.
In the following, in the same manner as described with reference to FIG. 12 with respect to the first embodiment, the 8 frame period is defined as one failure detection processing cycle, and within each failure detection processing cycle, the default image utilization inspection and inter-frame correlation are performed. A case where both inspections are performed will be described.
FIG. 16 is a diagram similar to FIG. 15 showing operations in the imaging unit 10 and the image processing unit 20a when performing the failure detection processing in this way.
The “frame number” in the top row of FIG. 16 indicates the order of each frame in each failure detection processing cycle.

  In the example shown in FIG. 16, as in the case of FIG. 15, image recognition is performed on the processed image Re (3) of the third frame. Processed images Re of other frames are not used for image recognition.

  In the example shown in FIG. 16, the default image use inspection is performed using the fifth signal processing frame period. In addition, the interframe correlation test is performed using the sixth and seventh imaging frame periods.

The imaging control unit 23 instructs the failure detection processing unit 25 to perform the default image use inspection and the inter-frame correlation inspection.
This instruction includes the specification of the timing at which the default image use inspection should be performed, the specification of the default image to be used in the inspection, the specification of the signal processing conditions for the default image, and the specification of the timing at which the interframe correlation inspection should be performed. Information indicating exposure conditions for failure detection to be used in imaging for inspection is included.
As a timing at which the default image utilization inspection should be performed, at least one signal processing frame period corresponding to an imaging frame period in which an image other than the frame image used in image recognition is obtained by imaging is designated.
As the timing at which the inter-frame correlation inspection is to be performed, at least one imaging frame period in which an image other than the frame image used in image recognition is obtained by imaging is designated.
In the following, it is instructed that the default image use inspection should be performed using the fifth signal processing frame period, and the interframe correlation inspection should be performed using the sixth and seventh imaging frame periods. And
In accordance with this instruction, the failure detection processing unit 25 performs a default image use inspection and an interframe correlation inspection.

  For the inspection using the default image, the failure detection processing unit 25 reads the default image Fk for failure detection stored in advance in the default image storage area 304 of the memory 30 according to the designation from the imaging control unit 23, and performs the fifth imaging. Instead of the captured image F (5) obtained by imaging in the frame period, it is supplied to the signal processing unit 21. At the same time, the failure detection processing unit 25 reads a predetermined signal processing condition stored in the signal processing condition storage area 305 of the memory 30 in accordance with the designation from the imaging control unit 23, and the read signal processing condition Is set in the signal processing unit 21.

As a result, the signal processing unit 21 performs signal processing on the default image Fk under the set signal processing conditions, and outputs a processed image Re (5) as a result of the processing.
When the signal processing unit 21 performs processing on the default image Fk and outputs the processed image Re (5) in the fifth signal processing frame period, the line “signal processing unit 21” in FIG. “Fk → Re (5)” is shown in the column of 5 frame periods.

For the inter-frame correlation inspection, the imaging control unit 23 preliminarily sets exposure conditions EB (c) and EB (d) for failure detection, which are stored in the exposure condition storage area 306 of the memory 30. It shall be specified to be used.
The imaging control unit 23 reads out the exposure conditions EB (d) and EB (e) stored in the exposure condition storage area 306 of the memory 30, and uses the read exposure conditions EB (d) and EB (e) as the sixth. And imaging in the seventh imaging frame period. Therefore, the imaging control unit 23 generates a control signal group EC corresponding to the exposure conditions EB (d) and EB (e) and supplies the control signal group EC to the imaging unit 10.
The imaging control unit 23 further notifies the failure detection processing unit 25 that the exposure conditions EB (d) and EB (e) are used for inspection.

  In FIG. 16, “EB (d)” and “EB (e)” are entered in the columns of the sixth and seventh frame periods in the “exposure control” row of the imaging control unit 23. Shows that the exposure conditions EB (d) and EB (e) are used in the frame periods of the sixth and seventh imaging.

  The exposure conditions EB (d) and EB (e) are different from each other. For example, the exposure time EB (e) is 1/2 the exposure time with respect to the exposure condition EB (d), and the other items (aperture, gain) of the exposure condition are the same.

The captured images F (d) and F (e) obtained by imaging using the exposure conditions EB (d) and EB (e) are also subjected to signal processing by the signal processing unit 21, and processed images Re (6), Re (7) is output.
The signal processing conditions for the captured image F (d) and the signal processing conditions for the captured image F (e) in the signal processing unit 21 are the same.

In FIG. 16, the captured images F (d) and F (e) are obtained by imaging in the sixth and seventh imaging frame periods, and the sixth and seventh frame periods in the row of “imaging unit 10”. Are indicated by “F (d)” and “F (e)” in the column.
Also in FIG. 16, signal processing is performed on the captured images F (d) and F (e) in the sixth and seventh signal processing frame periods to generate processed images Re (6) and Re (7). Is indicated by “F (d) → Re (6)” and “F (e) → Re (7)” in the columns of the sixth and seventh frame periods in the row of the “signal processing unit 21”. Has been.

In an imaging frame period other than the sixth and seventh imaging frame periods, imaging is performed as in the case of FIG.
In signal processing frame periods other than the fifth to seventh signal processing frame periods, signal processing is performed in the same manner as in FIG.

  The processed images Re (1) to Re (8) output from the signal processing unit 21 are stored in the memory. Among them, the processed images Re (5) to Re (7) are stored in the inspection data storage area 307, and the processed images Re (1) to Re (4) and Re (8) are stored in the frame buffer area 301. .

The average luminance value calculation unit 27 reads the processed images Re (1) to Re (8) from the memory 30, and the average luminance value Ya (1) for the read processed images Re (1) to Re (8). -Ya (8) is calculated.
This is indicated by “Re (1) → Ya (1)” in the row of “average luminance value calculation unit 27” in FIG.

  The average luminance value calculating unit 27 outputs data representing the average luminance values Ya (1) to Ya (4) and Ya (8) among the calculated average luminance values Ya (1) to Ya (8). The data representing the average luminance values Ya (5), Ya (6), and Ya (7) are stored in the inspection data storage area 307 of the memory 30.

  Supply the default image Fk read from the memory 30 to the signal processing unit 21, write the processed images Re (1) to Re (8) output from the signal processing unit 21 to the memory 30, and read from the memory 30. Supplying the processed images Re (1) to Re (8) to the average luminance value calculating unit 27 and writing the luminance average value calculated by the average luminance value calculating unit 27 to the memory 30 via the bus 26. Done.

  For the determination in the default image utilization inspection, the failure detection processing unit 25 processes the processed image Re (5) (the processed image obtained as a result of the signal processing in the signal processing unit 21) stored in the inspection data storage area 307 of the memory 30. Re (5)) and the average luminance value Ya (5) calculated by the average luminance value calculation unit 27 are determined as expected. The determination as to whether it is as expected is performed by checking the correlation (similarity) with each expected value stored in advance in the expected value storage area 308 of the memory 30. If the correlation is equal to or greater than the threshold, it is determined that the signal processing unit 21, the average luminance value calculation unit 27, and the bus 26 are operating normally. If the correlation is less than the threshold, it is determined that any of the signal processing unit 21, the average luminance value calculation unit 27, and the bus 26 has a failure.

The determination as to whether or not there is a correlation with respect to the image is, for example, a value obtained by performing a predetermined process on the processed image Re (5) (processed image corresponding to the default image Fk), This is performed by comparing the correlation with the stored expected value with a predetermined threshold value.
A value obtained by performing a predetermined process is, for example, a checksum value. In this case, the expected value of the checksum value is stored in the expected value storage area 308 of the memory 30. The checksum value generation rule is also stored in the memory 30, and the failure detection processing unit 25 generates a checksum value from the processed image Re (5) according to the generation rule stored in the memory 30. .
Even when a value obtained by performing a predetermined process other than the checksum is used, information representing the predetermined process is stored in the memory 30.

  Whether or not there is a correlation with respect to the average luminance value is determined based on whether or not the difference between the average luminance value and the expected value is equal to or less than a threshold value, for example.

  The failure detection processing unit 25 performs the default image use inspection based on the processed image Re (5) and the average luminance value Ya (5) in the fifth frame period. “Re (5), Ya (5)” is shown in the fifth frame period column in the row of “default image use inspection”.

  For the determination in the inter-frame correlation test, the failure detection processing unit 25 reads out the data indicating the average luminance values Ya (6) and Ya (7) stored in the memory 30, and reads the read data and the imaging control unit 23. The presence / absence of a failure is determined based on the exposure conditions EB (d) and EB (e) that have been notified.

  The failure detection processing unit 25 in FIG. 16 performs the inter-frame correlation inspection based on the average luminance values Ya (6), Ya (7) and the exposure conditions EB (d), EB (e). This is indicated by “Ya (6), Ya (7), EB (d), EB (e)” in the column of the seventh frame period in the 25 “inter-frame correlation check” row.

  The failure detection processing unit 25 determines whether there is a correlation between the change between the exposure conditions EB (d) and EB (e) and the change between the average luminance values Ya (6) and Ya (7). Judgment is made.

  For example, the exposure time of the exposure condition EB (e) is ½ of the exposure condition of the exposure condition EB (d), and other items (aperture, gain) of the exposure conditions EB (d) and EB (e) are If they are the same, the average luminance value Ya (7) should be about ½ of the average luminance value Ya (6).

The failure detection processing unit 25 determines whether or not there is a correlation between a change between exposure conditions and a change between average luminance values based on such a known relationship.
For example, the failure detection processing unit 25 determines between the ratio between the exposure amounts determined by the exposure conditions EB (d) and EB (e) and the ratio between the average luminance values Ya (6) and Ya (7). It is determined whether or not there is a correlation.
The average luminance values Ya (6) and Ya (7) represent the brightness of the images Re (6) and Re (7), respectively. Therefore, the above determination is made based on the exposure conditions EB (d) and EB (e ) And the ratio between the exposure amounts determined by the average brightness values Ya (6) and Ya (7) and the ratio between the brightnesses of the images Re (6) and Re (7). In other words, it can be called determination of whether or not there is.

As described above, when it is determined that there is a correlation between the change in the exposure condition and the change in the average luminance value of the processed image corresponding to the captured image obtained by imaging, the imaging unit 10, the signal processing It is determined that the unit 21, the frame buffer area 301 of the memory 30, and the bus 26 are operating normally.
Conversely, when it is determined that there is no correlation, it is determined that there is a failure in any of the imaging unit 10, the signal processing unit 21, the frame buffer area 301 of the memory 30, and the bus 26.

  As described above, when imaging in the sixth and seventh imaging frame periods is performed under exposure conditions separately defined for failure detection, in the immediately following imaging frame period, that is, in the eighth imaging frame period, It is not appropriate to determine the exposure condition based on the average luminance value Ya of the processed image corresponding to the captured image obtained by imaging in the immediately preceding imaging frame period (seventh imaging frame period). Therefore, in the present embodiment, the same exposure condition EB (5) as that used in the imaging frame period before the sixth imaging frame period, that is, the fifth imaging frame period is also used in the eighth imaging frame period. . In FIG. 16, this is indicated by “EB (5) → EB (8)”.

  In the above example, correlation determination is performed using exposure conditions prepared separately for failure detection as exposure conditions in two imaging frame periods, for example, the sixth and seventh imaging frame periods. Instead, one of the two imaging frame periods is imaged under an exposure condition determined based on the brightness of the subject, as in normal imaging, and the other of the two imaging frame periods is Images are taken using the exposure conditions determined based on the exposure conditions, and the ratio between the exposure amounts determined by the exposure conditions between these two imaging frame periods and the imaging in these two imaging frame periods are obtained. It may be determined whether there is a correlation between the brightness of the processed image corresponding to the captured image, for example, the ratio between the average luminance values. The exposure condition determined based on the exposure condition determined based on the brightness of the subject has a known relationship with the exposure condition determined based on the brightness of the subject (for example, the exposure amount is known). The exposure conditions can be used.

  While the failure detection processing unit 25 performs the failure detection processing as described above, the imaging control unit 23 monitors whether or not a condition for ending the failure detection processing is satisfied. Specifically, the average luminance value Ya (1) of the processed image corresponding to the captured image (image of the first to fourth frames) obtained by imaging in the first to fourth frame periods of each failure detection processing cycle. ) To Ya (4), it is determined whether or not the average luminance value has changed.

  For example, the average luminance values Ya (1) and Ya (2) of the processed images of the first and second frames, and the average luminance values Ya (3) and Ya of the processed images of the third and fourth frames. If the difference between the average value of (4) and the difference is greater than or equal to the threshold value, it is determined that a change has occurred.

  When it is determined that the average luminance value has changed, the imaging control unit 23 instructs the failure detection processing unit 25 to end the failure detection processing, and the failure detection processing unit 25 follows the failure detection processing according to this. Exit.

  The determination as to whether or not the condition for ending the failure detection process is satisfied is based on the processed image used in the default image use inspection (the processed image Re (5) in the fifth frame in the above example), and the failure detection Images other than the processed images (in the above example, the processed images Re (6) and Re (7) of the sixth and seventh frames) obtained by imaging under the exposure conditions for In the example, it is necessary to use the average luminance value of the processed images Re (1), Re (2), Re (3), Re (4), and Re (8)).

  With reference to FIG. 16, the operations in the imaging unit and the image processing unit when both the default image use inspection and the interframe correlation inspection are performed in one failure detection processing cycle have been described. As described above, only one of the default image use inspection and the interframe correlation test may be performed, and the default image use inspection is performed in one failure detection cycle and the interframe correlation test is performed in another failure detection cycle. Alternatively, after one of the default image use inspection and the inter-frame correlation inspection is repeated a plurality of times, the other may be repeated a plurality of times.

In the second embodiment, a histogram generation unit may be provided as in the first embodiment, and a luminance histogram may be used in the default image use inspection and the interframe correlation inspection.
Modifications other than those described above with respect to the first embodiment can also be applied to the second embodiment.

  In the second embodiment, failure detection processing is performed in an imaging frame period in which a captured image corresponding to a processed image of a frame that is not used in image recognition is obtained by imaging, or in a signal processing frame period corresponding to such an imaging frame period. Therefore, it is possible to detect a failure of the imaging apparatus while continuing image recognition.

In Embodiment 1 described above, the processed image Re generated by performing signal processing on the captured image F obtained by imaging with the first exposure amount in the imaging unit 10 by the signal processing unit 21, and the imaging unit 10, the processed image Re generated by performing signal processing in the signal processing unit 21 on the captured image F obtained by imaging at the second exposure amount in 10 is synthesized by the image synthesis unit 24. A configuration for generating Rh will be described. In the second embodiment, a processed image of a plurality of frames obtained by performing signal processing on a captured image F of a plurality of frames obtained by imaging by the imaging unit 10 by the signal processing unit 21. The configuration in which the image recognition unit 28 performs image recognition on the processed image Re of some frames of Re has been described.
Both the image composition unit 24 and the image recognition unit 28 are common in that the processing using the processed image Re is performed at a frequency lower than the imaging frequency in the imaging unit 10, and each of them is an aspect of the low-frequency processing unit. Can be seen.
However, the present invention is not limited to the above configuration, and the processing using the processed image Re is performed at a frequency lower than the imaging frequency in the imaging unit 10 other than the image synthesis unit 24 and the image recognition unit 28. The present invention can be applied to a configuration in which a frequency processing unit is used.

  Although the present invention has been described above as an imaging apparatus, an image processing method implemented by the above-described imaging apparatus, particularly a failure detection method thereof, also forms part of the present invention.

In the first and second embodiments described above, each part (the part illustrated as a functional block) of the image processing units 20 and 20a is realized by a processing circuit. The processing circuit may be dedicated hardware or a CPU that executes a program stored in a memory.
For example, the functions of the respective parts in FIG. 1 or FIG. 13 may be realized by separate processing circuits, or the functions of a plurality of parts may be realized by a single processing circuit.

  When the processing circuit is a CPU, the function of each part of the image processing unit is realized by software, firmware, or a combination of software and firmware. Software or firmware is described as a program and stored in a memory. The processing circuit reads out and executes the program stored in the memory, thereby realizing the function of each unit. That is, the image processing unit includes a memory for storing a program in which the functions of the respective parts shown in FIG. 1 or FIG. 13 are executed as a result when executed by the processing circuit. Further, it can be said that these programs cause the computer to execute an image processing method executed by the image processing unit, in particular, a process in the failure detection method, or a procedure thereof.

In addition, a part of the functions of each part of the image processing unit may be realized by dedicated hardware, and a part may be realized by software or firmware.
As described above, the processing circuit can realize the above-described functions by hardware, software, firmware, or a combination thereof.

FIG. 17 shows an example of a configuration in which the above processing circuit is a CPU and all functions of the image processing unit and the memory are realized by a computer (indicated by reference numeral 50) including a single CPU, together with the imaging unit 10. Show. The computer 50 and the imaging unit 10 constitute an imaging device.
A computer 50 shown in FIG. 17 includes a CPU 51, a memory 52, a first interface 53, and a second interface 54, which are connected by a bus 55.
A signal representing the captured image F from the imaging unit 10 is input to the first interface 53. Further, the control signal group EC is supplied to the imaging unit 10 via the first interface 53.

  The CPU 51 operates in accordance with a program stored in the memory 52, performs processing of each unit of the image processing unit of the first or second embodiment on the video signal input via the first interface 53, and performs processing The output signal obtained as a result of is output from the second interface 54.

  The contents of the processing by the CPU 51 are the same as those described in the first or second embodiment. The memory 52 serves as the memory 30.

  An image processing method implemented by the image processing unit of the image pickup apparatus of the present invention, particularly a failure detection method thereof, processing of each part of the image processing unit, or each processing in the image processing method, particularly the failure detection method, is executed by a computer With respect to the program and a computer-readable recording medium on which the program is recorded, the same effect as described for the image processing unit can be obtained. Therefore, these programs and recording media also form part of the present invention.

  DESCRIPTION OF SYMBOLS 1, 1a imaging device, 10 imaging part, 12 imaging optical system, 13 solid-state image sensor, 14 front end part, 20, 20a image processing part, 21 signal processing part, 22 histogram generation part, 23 imaging control part, 24 image composition Unit, 25 failure detection processing unit, 26 bus, 27 average luminance value calculation unit, 28 image recognition unit, 30 memory.

Claims (15)

An imaging unit that receives light from the subject and performs imaging every imaging frame period;
An imaging control unit that controls an exposure condition for each imaging frame period in the imaging unit;
A signal processing unit that performs signal processing in a corresponding signal processing frame period for a captured image obtained by imaging in each imaging frame period in the imaging unit, and generates a processed image;
A low frequency processing unit that performs processing using the processed image at a frequency lower than the frequency of imaging in the imaging unit;
At least one imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low-frequency processing unit is obtained by imaging in the imaging unit, or at least one signal corresponding to the imaging frame period using the processing frame period, it possesses a failure detection processing unit for performing at least one of the failure detection processing of the image pickup unit and the signal processing unit,
The signal processing unit includes at least one signal processing frame period corresponding to an imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low frequency processing unit is obtained by imaging in the imaging unit In addition to receiving the image for failure detection instead of the captured image obtained by imaging in the imaging unit, the supplied image is subjected to signal processing under the failure detection processing conditions,
The failure detection processing unit determines whether or not a processed image generated as a result of performing signal processing on the failure detection image is as expected . A memory for storing the failure detection image;
The imaging apparatus according to claim 1 , wherein the failure detection processing unit reads the failure detection image from the memory and supplies the failure detection image to the signal processing unit as the failure detection image. The imaging apparatus according to claim 1 , wherein the signal processing unit generates the failure detection image according to a rule for generating a failure detection image. The failure detection processing unit is stored in advance with a value obtained by performing predetermined processing on a processed image generated as a result of performing signal processing on the failure detection image. correlation between the expected value there are, the imaging apparatus according to claim 2 or 3, characterized in that to determine the presence or absence of a failure based on whether it is a predetermined threshold or more. The imaging control unit is configured to detect failure in the imaging unit during an imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low-frequency processing unit is obtained by imaging with the imaging unit. Image with the exposure conditions of
The failure detection processing unit is configured such that a change between an image obtained by imaging under the exposure condition for failure detection and an image obtained by imaging during another imaging frame period is the exposure condition for failure detection. The imaging apparatus according to any one of claims 1 to 4 , wherein a determination is made as to whether or not the change corresponds to an exposure condition in the other imaging frame period. The imaging control unit, as the determination, the ratio between the brightness of the image obtained by imaging under the exposure condition for failure detection and the brightness of the image obtained by imaging in another imaging frame period, and the failure 6. The method according to claim 5 , wherein a determination is made as to whether or not there is a correlation between a ratio between an exposure amount determined by an exposure condition for detection and an exposure amount determined by an exposure condition in the other imaging frame period. The imaging device described. The other imaging frame period is an imaging frame period in which a captured image corresponding to a processed image used in processing in the low frequency processing unit is obtained by imaging,
The imaging control unit
It said failure detection for the exposure conditions, the image pickup apparatus according to claim 5 or 6, characterized in that determining, based on the exposure conditions in the other imaging frame period. The other imaging frame period is an imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low frequency processing unit is obtained by imaging,
The imaging control unit, as an exposure condition in the other imaging frame period, different from the exposure conditions for the fault detection, according to claim 5 or 6, comprising using the exposure conditions for further fault detection Imaging device. The imaging apparatus according to claim 6 , wherein the brightness of the image is brightness of an image indicated by a luminance histogram of the image. The imaging apparatus according to claim 6 , wherein the brightness of the image is brightness represented by an average luminance value of the image. The low frequency processing unit is
In a first imaging frame period, a first processed image generated by performing signal processing in the signal processing unit on a captured image obtained by imaging in the imaging unit, and a second imaging frame period An image composition unit that generates and outputs a composite image based on a second processed image generated by performing signal processing on the captured image obtained by imaging at the image capture unit. Yes,
In the first imaging mode,
The imaging control unit performs imaging by setting an exposure condition in the imaging unit such that an exposure amount in the first imaging frame period and an exposure amount in the second imaging frame period are different from each other. Let
The image synthesis unit synthesizes the first processed image and the second processed image, and outputs an image with an expanded dynamic range as the synthesized image,
In the second imaging mode,
The image synthesis unit outputs the first processed image as it is as the synthesized image,
The failure detection processing unit performs the failure detection process using the second imaging frame period or a signal processing frame period corresponding to the second imaging frame period. The imaging apparatus according to any one of 10 . The low frequency processing unit is
Image recognition is performed on processed images of some of the processed images of a plurality of frames obtained by performing signal processing on the captured images of the plurality of frames obtained by imaging by the imaging unit, respectively. An image recognition unit to perform,
The failure detection processing unit includes at least one imaged image obtained by imaging at the imaging unit other than the imaged image corresponding to the processed image of the frame used in the image recognition among the processed images of the plurality of frames. The imaging apparatus according to any one of claims 1 to 10 , wherein the failure detection process is performed using a frame period or at least one signal processing frame period corresponding to the imaging frame period. An imaging unit that receives light from the subject and performs imaging every imaging frame period;
An imaging control unit that controls an exposure condition for each imaging frame period in the imaging unit;
A signal processing unit that performs signal processing in a corresponding signal processing frame period for a captured image obtained by imaging in each imaging frame period in the imaging unit, and generates a processed image;
In an imaging apparatus having a low frequency processing unit that performs processing using the processed image at a frequency lower than the frequency of imaging in the imaging unit,
At least one imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low-frequency processing unit is obtained by imaging in the imaging unit, or at least one signal corresponding to the imaging frame period using the processing frame period, have lines at least one of the failure detection processing of the image pickup unit and the signal processing unit,
The signal processing unit includes at least one signal processing frame period corresponding to an imaging frame period in which a captured image other than the captured image corresponding to the processed image used in the processing in the low frequency processing unit is obtained by imaging in the imaging unit In addition to receiving the image for failure detection instead of the captured image obtained by imaging in the imaging unit, the supplied image is subjected to signal processing under the failure detection processing conditions,
The failure detection processing is performed by determining whether or not a processed image generated as a result of performing signal processing on the failure detection image is as expected. Method. A program for causing a computer to execute the process of the failure detection method according to claim 13 . The computer-readable recording medium which recorded the program of Claim 14 .

Images