JP7023120B2 - Endoscope device, operation method and program of the endoscope device - Google Patents

Description

The present invention relates to an endoscope device, an operation method and a program of the endoscope device.

Endoscopic devices are widely used not only in the medical field but also in the industrial field. The endoscope device is provided with an elongated insertion portion and is used for observing and inspecting damage, corrosion, etc. inside transportation equipment such as aircraft, other equipment and structures. Since there are various subjects in general, some endoscope devices are equipped with an automatic exposure (AE) control mechanism. The AE control is a process of adjusting the brightness of an image according to observation conditions, and includes feedback control for determining a brightness control parameter based on the difference between the brightness of the captured image and the target brightness.

For example, the endoscope device described in Patent Document 1 switches the exposure time and sensitivity according to the movement of the camera. More specifically, when the movement of the camera is detected, a plurality of images are read out until the image of one frame is displayed. As a result, the followability to a sudden change in brightness is improved, and a higher quality image is acquired.
In the endoscope device described in Patent Document 2, AE control is stopped when hunting is detected.

Japanese Unexamined Patent Publication No. 2009-188644 Japanese Unexamined Patent Publication No. 2002-325729

However, during the execution of the AE control, the subject to be observed operates periodically, so that the brightness of the image representing the subject may fluctuate periodically. Such a phenomenon may appear, for example, in an image taken while rotating a turbine blade of a gas turbine engine in a power plant using an industrial endoscope. In addition, when the insertion part is inserted into a cavity such as an engine space, the camera attached to the tip moves closer to or farther from the wall surface, causing repeated sudden fluctuations in the brightness of the captured image. There is. Even if the captured image includes a portion that is not originally accompanied by a change in brightness, the brightness of that portion also changes. Such fluctuations in brightness can be annoying and tired for the user.

Further, even if the followability of the AE control is improved as in the endoscope device described in Patent Document 1, each part in the image to be captured is under a situation where a sudden change in the amount of incident light to the camera occurs repeatedly. There is no change in the fact that sudden changes in the brightness of the camera occur repeatedly.
Further, in the endoscope device described in Patent Document 2, even if the luminance level is further changed after the AE control is stopped, the change in the luminance level cannot be followed.

The present invention has been made in view of the above problems, and the operation of an endoscope device and an endoscope device capable of more comfortably controlling the brightness even in a situation where changes in the amount of incident light occur repeatedly. It is intended to provide methods and programs.

One aspect of the present invention includes an image pickup element that generates an image of a subject, an insertion unit provided with the image pickup element, and a control unit, and the control unit evaluates the brightness of the image and the brightness is reduced. The brightness control parameter is set so as to approach a predetermined target value, the fluctuation status of both the brightness and the control parameter is detected, the control parameter is determined based on the fluctuation status, and the fluctuation status is determined. Includes the luminance, the fluctuation frequency of the control parameter, and the periodicity of the luminance and the control parameter. The control unit detects the fluctuation frequency of the control parameter, and the fluctuation frequency of the control parameter is predetermined. When it is higher than the fluctuation frequency, it is determined whether or not the fluctuation of the control parameter has periodicity, and when it is determined that the fluctuation of the control parameter has periodicity, the average value of the control parameter in the cycle up to the present time is determined. Is an endoscopic device characterized in that is defined as the control parameter at the present time .

According to the present invention, the brightness is controlled more comfortably even in a situation where changes in the amount of incident light occur repeatedly.

It is a block diagram which shows the structural example of the endoscope apparatus which concerns on this embodiment. It is a flowchart which shows the example of the image pickup control which concerns on this embodiment. It is a figure which shows an example of the luminance control by this embodiment. It is a figure which shows another example of the luminance control by this embodiment. It is a figure which shows the example of the image taken. It is a figure which shows the further example of the luminance control by this embodiment. It is a figure which shows the other example of the image taken. It is a figure which shows the control example of the followability by this embodiment. It is a figure which illustrates the gamma table.

Next, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration example of the endoscope device 1 according to the present embodiment.
The endoscope device 1 according to the present embodiment includes an insertion portion 10 and a main body portion 11. The insertion unit 10 is inserted inside the subject and at least acquires an image inside the subject. The type of subject is not particularly limited. The subject may be, for example, an industrial device such as an engine, a turbine, a boiler, or a living body. The insertion portion 10 is flexible and has an elongated tubular shape. The length of the insertion portion 10 is, for example, 1 to 30 m.

At the tip of the insertion portion 10, a CCD (Charge Coupled Device) image pickup element 104, an objective lens 105, an LED (Light Emitting Diode) 106, a thermistor 107, an acceleration sensor 108, and a wire fixing portion 109 are provided. , A wire connecting portion 124 and a detachable connector 134. Four bending wires 126 are inserted in the longitudinal direction of the insertion portion 10.

The CCD image sensor 104 photoelectrically converts the light from the subject. Light receiving elements are arranged for each pixel on the image pickup surface of the CCD image pickup device 104, and each light receiving element generates a voltage corresponding to the intensity of the incoming light. The CCD image pickup device 104 generates an image pickup signal indicating a signal value for each pixel from the voltage value obtained by photoelectric conversion based on the timing indicated by the drive signal input from the main body 11. The generated imaging signal indicates an image of the subject. The CCD image pickup device 104 outputs the generated image pickup signal to the main body 11.

The objective lens 105 converges the light incident on the tip of the insertion portion 10 and forms an image of the subject on the image pickup surface of the CCD image pickup device. The objective lens 105 is arranged so as to face the light receiving window (not shown) at the tip thereof.

The LED 106 is a light source that emits light based on the driving power supplied from the main body 11. The LED 106 irradiates the subject with the emitted light.

The thermistor 107 is a temperature sensor that detects the temperature at the tip of the insertion portion 10. The thermistor 107 generates a thermistor signal indicating the detected temperature, and outputs the generated thermistor signal to the main body 11.

The acceleration sensor 108 detects the acceleration at the tip of the insertion portion 10. The acceleration sensor 108 is, for example, a 3-axis acceleration sensor. The 3-axis accelerometer detects accelerations in each of the three directions orthogonal to each other in the three-dimensional space. The acceleration sensor 108 outputs an acceleration signal indicating the detected acceleration to the main body 11.

The wire fixing portion 109 is installed at the tip end portion of the insertion portion 10. The tips of the four bending wires 126 are connected to the wire fixing portion 109 so as to be alternately tiltable in the biaxial direction orthogonal to the longitudinal direction of the insertion portion 10. The directions of the two axes are orthogonal to each other. The two-axis directions include a UD (Up-Down) direction and an RL (Light-Left) direction. The UD direction and the RL direction correspond to the vertical direction and the horizontal direction of the pixels arranged on the imaging surface. Each of the two bending wires related to the bending in the UD and RL directions is referred to as a UD and RL wire, respectively.

The wire connecting portion 124 is installed at the base end portion of the insertion portion 10. The wire connecting portion 124 pulls each bending wire 126 in the longitudinal direction of the insertion portion 10 in response to the pulling by the wire connecting portion 125 at the time of connecting to the wire connecting portion 125 included in the main body portion 11.

The detachable connector 134 is installed at the base end portion of the insertion portion 10. The detachable connector 134 includes a detachable mechanism that can be attached to and detached from the detachable connector 135 included in the main body 11. The detachable connector 134 has a shape that fits each other with, for example, the detachable connector 135.

A connection cable for transmitting each of the image pickup control signal, the image pickup signal, the drive power, the thermistor signal, and the acceleration signal is inserted in the insertion unit 10. These connection cables are electrically connected to the respective source and destination components when the detachable connector 134 is attached to the detachable connector 135 of the main body 11. These connecting cables are, for example, composite coaxial lines. The detachable connectors 134 and 135 include contacts that electrically connect to the base end of each connection cable.

The main body portion 11 includes a wire connecting portion 125 and a detachable connector 135 at an end portion that may come into contact with the base end portion of the insertion portion 10. The main body 11 includes a system control unit 110, a user interface unit 111, a storage medium 112, a parameter storage unit 113, a timing generator 114, a CCD drive circuit 115, an LCD (Liquid Crystal Display) 118, an LED drive circuit 119, and a curve. It includes a control unit 121, a UD bending motor 122, an RL bending motor 123, a preamplifier 136, an AFE (Analog Front End) 137, and an image processing unit 140.

The system control unit 110 is a component that controls the entire operation of the endoscope device 1. The system control unit 110 is connected to the user interface unit 111, the storage medium 112, the parameter storage unit 113, the CCD drive circuit 115, the LCD 118, the LED drive circuit 119, the bending control unit 121, the contacts of the detachable connector 135, and the image processing unit 140. Has been done. The system control unit 110 controls the operation of each component of the endoscope device 1 based on an operation signal input from the user interface unit 111, various parameters stored in the parameter storage unit 113, and the like. The system control unit 110 includes, for example, lighting / extinguishing control of the LED 106, image pickup operation control of the CCD image pickup device 104, bending control of the bending control unit 121, temperature detection control, and the like. In the temperature detection control, the system control unit 110 causes the LCD 118 to display information indicating the current temperature indicated by the thermistor signal input from the thermistor 107 via the LCD controller 110b. Further, the system control unit 110 is based on the upper limit or the lower limit of the allowable temperature stored in advance in the parameter storage unit 113, when the current temperature is within a predetermined range from the upper limit or the lower limit, when it is higher than the upper limit or the lower limit. If it is lower than, the LCD 118 displays a predetermined warning message. The system control unit 110 includes an image recording unit 110a and an LCD controller 110b.

The image recording unit 110a stores the image signal input from the image processing unit 140 in the storage medium 112 based on the control signal input from the user interface unit 111. For example, the image recording unit 110a stores the image signal from the recording start time to the recording end time indicated by the control signal in the storage medium 112.

The LCD controller 110b outputs an image signal input from the image processing unit 140 via the image recording unit 110a to the LCD 118 based on the control signal input from the user interface unit 111. As a result, the LCD 118 can display various moving images and still images acquired by the image processing unit 140. When displaying a still image, the LCD controller 110b continuously outputs the image signal of the frame acquired at that time to the LCD 118. The image recording unit 110a may record an image signal acquired as a still image on the storage medium 112. The LCD controller 110b also outputs image signals indicating various display information to the LCD 118. The display information includes, for example, a guidance screen for information necessary for operation input and the like.

The user interface unit 111 includes an operation unit. The operation unit accepts operation input by the user. The operation unit is, for example, a member such as a joystick, an operation switch, and an operation button. The user interface unit 111 generates a control signal according to the received operation input, and outputs the generated control signal to the system control unit 110.

The storage medium 112 is a storage medium for storing image signals. The storage medium 112 stores, for example, an image signal indicating an image such as a moving image or a still image obtained by the image processing unit 140.

The parameter storage unit 113 is changed by the user's operation, such as operation input to the user interface unit 111, image processing parameters used for various image processing executed by the image processing unit 140, and bending control parameters related to bending control by the bending control unit 121. Stores various parameters to be processed. Further, the parameter storage unit 113 stores the relationship between the transmission path length of the non-imaging signal and the drive processing parameter of the CCD drive circuit 115. In addition, the parameter storage unit 113 may store various values and data obtained by the image processing unit 140.

The timing generator 114 generates a timing signal for driving the CCD image pickup device based on the image pickup control signal input from the image pickup control unit 147 (described later) of the system control unit 110. The image pickup control signal includes information such as an exposure time notified from the image processing unit 140. The exposure time is the time from the start of exposure to the end of exposure in each pixel, and corresponds to the reciprocal of the shutter speed. Therefore, the longer the exposure time, the higher the signal level obtained by one reading. The exposure time is used as a control parameter for controlling the brightness of the image to be captured. Here, the longer the exposure time, the higher the brightness, and the shorter the exposure time, the lower the brightness. The timing generator 114 outputs the generated timing signal to the CCD drive circuit 115. When the exposure time is not controlled by the image pickup control unit 147, the timing generator 114 may generate a timing signal using the exposure time set in advance in the own unit.

The CCD drive circuit 115 generates a drive signal based on the timing signal input from the timing generator 114 and the control signal from the system control unit 110. The control signal from the system control unit 110 is a signal indicating image pickup operation control. Imaging start, imaging stop, brightness adjustment, etc. are instructed as imaging operation control. The CCD drive circuit 115 outputs the generated drive signal to the CCD image pickup device 104. The drive signal indicates the frame rate, the order of pixels for acquiring the signal level in each frame, the start of exposure for each pixel, and the end of exposure.

The LCD 118 is an image display unit that displays an image based on an image signal input from the system control unit 110.

The LED drive circuit 119 controls lighting / extinguishing of the LED 106 based on a control signal input from the system control unit 110. The LED drive circuit 119 starts supplying drive power to the LED 106 when it is instructed to turn on, and stops supplying drive power when it is instructed to turn off.

The bending control unit 121 drives one or both of the UD bending motor 122 and the RL bending motor 123 based on the control signal input from the system control unit 110. The control signal indicates, for example, the bending amount for each bending direction corresponding to each of the operation amounts in the two directions detected by the user interface unit 111. The bending direction corresponds to the above-mentioned UD direction and RL direction. The bending control unit 121 instructs the UD bending motor 122 and the RL bending motor 123 of the driving amount corresponding to the bending amount in the UD direction and the driving amount corresponding to the bending amount in the RL direction indicated by the control signal.

The UD bending motor 122 and the RL bending motor 123 pull the UD wire and the RL wire by the driving amount instructed by the bending control unit 121 by their respective rotations.
The wire connecting portion 125 connects the UD wire and the RL wire connected to the wire connecting portion 124 of the inserting portion 10 when the detachable connector 135 is attached to the detachable connector 134 of the inserting portion 10. The UD wire and the RL wire are towed by the UD bending motor 122 and the RL bending motor 123. As a result, the tip of the insertion portion 10 is curved according to the user's operation.

The detachable connector 135 is installed at a position where it comes into contact with the base end portion of the insertion portion 10. The detachable connector 135 includes a detachable mechanism that can be attached to and detached from the detachable connector 134 of the insertion portion 10. The detachable connector 135 has a shape that fits each other with, for example, the detachable connector 134.

The preamplifier 136 amplifies the image pickup signal input from the CCD image sensor 104, and outputs the amplified image pickup signal to the AFE 137. This amplification is done to compensate for the signal level attenuated by the transmission.

The AFE137 performs CDS (Correlated Double Sampleting) processing, AGC (Automatic Gain Control, automatic gain control) processing, and AD (Analog-to-Digital, analog) processing for the analog imaging signal input from the preamplifier. / Digital) Performs conversion processing to generate a digital image pickup signal. An image pickup control signal indicating a gain may be input to the AFE137 from the image pickup control unit 147 (described later). In that case, the AFE137 adjusts the signal level by multiplying the analog image pickup signal by the gain indicated by the input image pickup control signal instead of the gain set in the own part in advance. The AFE137 performs AD conversion processing on an analog image pickup signal whose amplitude has been adjusted. The AFE137 outputs the generated digital image pickup signal to the image processing unit 140.

The image processing unit 140 includes a black correction unit 142, a pixel interpolation / luminance color difference conversion unit 143, a correction processing unit 144, a luminance evaluation unit 145, and an image pickup control unit 147.

The black correction unit 142 calculates the corrected signal level by subtracting the black level from the signal level for each pixel indicated by the input imaging signal. As the black level, the black correction unit 142 may use, for example, the signal level during the extinguishing period of the LED 106, the lowest value between pixels of each frame or the signal level within a predetermined period, and the like. The black correction unit 142 outputs an image pickup signal indicating the corrected signal level for each pixel to the pixel interpolation / luminance color difference conversion unit 143.

The pixel interpolation / luminance color difference conversion unit 143 performs pixel interpolation processing on the image pickup signal input from the black correction unit 142. The input imaging signal has a signal level of one of red (R), green (G), and blue (B) color components for each pixel. Pixels related to red, green, and blue are periodically arranged in a two-dimensional space. This sequence is, for example, a Bayer sequence. Hereinafter, the set of pixels related to the color component of each one cycle is referred to as a pixel cycle. In the pixel interpolation processing, the pixel interpolation / brightness color difference conversion unit 143 is arranged within a predetermined range from the target pixel for each target pixel, and interpolates the signal level of each pixel related to the same color component as the target pixel. Calculate the signal level of the target pixel. The target pixel is a pixel to be processed at that time. Each pixel in the frame is treated as a target pixel. In interpolation, for example, methods such as bilinear interpolation and bicubic interpolation can be used.

The pixel interpolation / luminance color difference conversion unit 143 performs color space conversion on the signal level of each pixel for each color component obtained by the interpolation, and performs color space conversion for each pixel cycle, the luminance level (Y) and the two color difference levels (Cr), ( Cb) is calculated. That is, the pixel interpolation / luminance color difference conversion unit 143 converts the RGB value represented by the RGB color system into the YCrCb value represented by the YCrCb color system in the color space conversion. In color space transformation, for example, ITU-R BT. The conversion formula specified in 601 can be used. The color difference levels (Cr) and (Cb) indicate the hue and saturation of the red and blue colors, respectively. The pixel interpolation / luminance color difference conversion unit 143 outputs a luminance signal indicating the luminance level for each pixel cycle to the correction processing unit 144 and the luminance evaluation unit 145. The pixel interpolation / luminance color difference conversion unit 143 outputs a color difference signal indicating two color difference levels (Cr) and (Cb) for each pixel period to the correction processing unit 144.

The correction processing unit 144 performs various correction processing for each of the luminance signal and the two luminance signals input from the pixel interpolation / luminance color difference conversion unit 143. The correction processing includes, for example, gamma correction, aberration correction, noise reduction processing, and the like. Gamma correction is a process of converting an input signal level (hereinafter referred to as “input level”) as a signal level (hereinafter referred to as “output level”) to output 1 / γ power. The gamma correction compensates for the characteristics of the output value according to the amount of input light in the CCD image sensor 104. γ indicates a gamma value. The gamma value indicates the ratio of the change in the output level to the change in the input level. Generally, the larger the gamma value is, the higher the contrast is obtained, and the smaller the gamma value is, the lower the contrast is obtained. When the gamma value is set by the image pickup control unit 147 (described later), the correction processing unit 144 replaces the set gamma value with the gamma value set in advance in its own unit and acts on the input level. Convert to output level. The correction processing unit 144 performs color space conversion on each of the luminance signal and the two color difference signals obtained by the correction processing to generate an image signal indicating the signal level for each pixel after correction. That is, in the color space conversion, the correction processing unit 144 converts the YCrCb value represented by the YCrCb color system into the RGB value represented by the RGB color system. This color space conversion process corresponds to the inverse conversion of the color space conversion in the pixel interpolation / brightness color difference conversion unit 143. The correction processing unit 144 outputs the generated image signal to the system control unit 110.

The luminance evaluation unit 145 evaluates the luminance of the entire image of each frame from the luminance signal input from the pixel interpolation / luminance color difference conversion unit 143. The luminance evaluation unit 145 calculates, for example, an average value between pixel periods in one frame of the luminance level as an evaluation value of the luminance level. The evaluation value of the brightness is not necessarily limited to the average value, and may be a value representing the brightness of the entire image of one frame. For example, the luminance evaluation unit 145 may calculate any of the center-weighted evaluation value, the median between pixel cycles, and the mode as the luminance evaluation value. The center-weighted evaluation value is an evaluation value calculated by giving priority to the luminance level in the central portion rather than the peripheral portion of the image, or ignoring or relatively neglecting the luminance level in the peripheral portion. The luminance evaluation unit 145 stores the evaluation value information indicating the calculated luminance evaluation value in the parameter storage unit 113.

The image pickup control unit 147 reads the latest evaluation value information from the parameter storage unit 113 for each frame, and the image pickup is performed so that the brightness indicated by the evaluation value indicated by the read evaluation value information approaches a predetermined brightness target value. Define control parameters for controlling the brightness of the image. Here, the image pickup control unit 147 detects, for example, the periodicity of one or both of them and the frequency of fluctuations (hereinafter referred to as “variation frequency”) as the fluctuation status of one or both of the evaluation value of the luminance and the control parameter. do. The image pickup control unit 147 determines the control parameters by a method different for each case of the fluctuation state. As the control parameter, either one or both of the exposure time and the gain can be used.

When the control parameter is the exposure time, the image pickup control unit 147 can determine the exposure time by performing, for example, the following processing.
(Case C01) In the imaging control unit 147, the fluctuation frequency of the exposure time within the predetermined period up to that time (current time) is higher than the threshold value of the fluctuation frequency of the predetermined exposure time, and the fluctuation of the exposure time has periodicity. When it is determined to have, the average value of the exposure time within the latest one cycle up to that point is calculated, and the calculated average value is determined as the control value of the exposure time. Examples of detection of fluctuation frequency and periodicity will be described later.
(Case C02) When the image pickup control unit 147 determines that the fluctuation frequency of the exposure time within a predetermined period up to that point is higher than the threshold value of the predetermined fluctuation frequency and the fluctuation of the exposure time has no periodicity. , The exposure time is determined by AE control. However, the image pickup control unit 147 performs AE control by lowering the followability when the evaluation value of the luminance approaches the target luminance to be lower than the followability of a predetermined standard. A control example of followability in AE control will be described later.

(Case C03) In the imaging control unit 147, the fluctuation frequency of the exposure time within the predetermined period up to that point is equal to or less than the threshold value of the predetermined fluctuation frequency, and the fluctuation frequency of the evaluation value of the luminance is the fluctuation of the predetermined evaluation value. When it is determined that the frequency is equal to or less than the threshold value, the exposure time is determined by AE control. However, in that case, the image pickup control unit 147 performs AE control with the followability when the evaluation value of the luminance approaches the target luminance as a predetermined standard followability.

(Case C04) In the imaging control unit 147, the fluctuation frequency of the exposure time within a predetermined period up to that point is equal to or less than the threshold value of the predetermined fluctuation frequency, and the fluctuation frequency of the luminance evaluation value is the fluctuation frequency of the predetermined evaluation value. When it is determined that the value is higher than the threshold value and the fluctuation of the evaluation value of the luminance has periodicity, the exposure time is determined as follows. The image pickup control unit 147 calculates the average value of the evaluation values of the latest brightness within one cycle up to that time, and determines the average value of the calculated evaluation values as the control value of the brightness. Then, the image pickup control unit 147 performs AE control so that the control value of the determined luminance approaches the target value of the predetermined luminance, and determines the exposure time.

(Case C05) In the imaging control unit 147, the fluctuation frequency of the exposure time within the predetermined period up to that point is equal to or less than the threshold value of the predetermined fluctuation frequency, and the fluctuation frequency of the luminance evaluation value is the fluctuation frequency of the predetermined evaluation value. When it is determined that the fluctuation is higher than the threshold value and the fluctuation of the evaluation value of the luminance has no periodicity, the exposure time is determined by AE control. In that case, the image pickup control unit 147 performs AE control by lowering the followability when the evaluation value of the luminance approaches the target luminance to be lower than the followability of a predetermined standard.

The image pickup control unit 147 outputs an image pickup control signal indicating an exposure time determined for each frame to the timing generator 114. As a result, the determined exposure time is set in the timing generator 114.

Further, the image pickup control unit 147 also determines the corresponding case of the gain based on the fluctuation frequency and the periodicity of the fluctuation, the fluctuation frequency of the evaluation value of the luminance and the periodicity of the fluctuation, by the same method as the exposure time. The gain can be determined by a method according to the determined case.
(Case C01) When the image pickup control unit 147 determines that the gain fluctuation frequency within a predetermined period up to the present time is higher than the threshold value of the predetermined gain fluctuation frequency and the gain fluctuation has periodicity, the gain fluctuation frequency is determined. The average value of the gain within the latest one cycle up to the time point is calculated, and the calculated average value is determined as the gain control value.
(Case C02) When the image pickup control unit 147 determines that the gain fluctuation frequency within a predetermined period up to that point is higher than the threshold value of the predetermined fluctuation frequency and the gain fluctuation has no periodicity, the AE. The exposure time is determined by control. In that case, the image pickup control unit 147 performs AE control by lowering the followability when the evaluation value of the luminance approaches the target luminance to be lower than the followability of a predetermined standard.

(Case C03) In the imaging control unit 147, the fluctuation frequency of the gain within a predetermined period up to that point is equal to or less than the threshold value of the predetermined fluctuation frequency, and the fluctuation frequency of the evaluation value of the luminance is the fluctuation frequency of the predetermined evaluation value. When it is determined that the value is equal to or less than the threshold value of, the gain is determined by AE control. In that case, the image pickup control unit 147 performs AE control with the followability when the evaluation value of the luminance is brought close to the target luminance as a predetermined standard followability.

(Case C04) In the imaging control unit 147, the fluctuation frequency of the gain within a predetermined period up to that point is equal to or less than the threshold value of the predetermined fluctuation frequency, and the fluctuation frequency of the luminance evaluation value is the threshold value of the fluctuation frequency of the predetermined evaluation value. When it is determined that the value is higher than the above and the fluctuation of the luminance evaluation value has periodicity, the average value of the luminance evaluation values within the latest one cycle up to that point is calculated, and the average value of the calculated evaluation values is calculated. Is defined as the brightness control value. Then, the image pickup control unit 147 determines the gain by AE control so that the control value of the determined luminance approaches the target value of the predetermined luminance.

(Case C05) In the imaging control unit 147, the gain fluctuation frequency within a predetermined period up to that point is equal to or lower than the predetermined fluctuation frequency threshold value, and the luminance evaluation value fluctuation frequency is the predetermined evaluation value fluctuation frequency threshold value. If it is higher than that and it is determined that the fluctuation of the evaluation value of the luminance does not have periodicity, the gain is determined by AE control. In that case, the image pickup control unit 147 performs AE control by lowering the followability when the evaluation value of the luminance approaches the target luminance to be lower than the followability of a predetermined standard.
The image pickup control unit 147 outputs an image pickup control signal indicating a gain determined for each frame to the AFE 137. As a result, the determined gain is set to AFE137.

(Example of fluctuation frequency)
Next, an example of the fluctuation frequency for determining the conditions for determining the control parameters will be described.
In the image pickup control unit 147, as the fluctuation frequency of the control parameters such as the exposure time and the gain, for example, the number of times the change tendency of the control parameters changes from an increase to a decrease within a predetermined period up to that point and the change tendency of the control parameters are determined. Count the total number of changes from decrease to increase. The predetermined time may be a period longer than the fluctuation cycle assumed as the fluctuation cycle of the control parameter.

More specifically, the image pickup control unit 147 performs the process of determining the above control parameters for each frame of the image, and stores the determined control parameters in the parameter storage unit 113. The image pickup control unit 147 compares the control parameter in the latest frame (hereinafter referred to as “current frame”) at that time with the control parameter in the previous frame, and changes the control parameter from the control parameter in the previous frame to the present. Determine if the control parameters in the frame have increased or decreased. Then, the image pickup control unit 147 can count the total number of events in which the change tendency changes from an increase to the next decrease and the event in which the change tendency changes from a decrease to the next increase within a predetermined period as the fluctuation frequency. can. In the determination of whether or not the control parameter has increased, the imaging control unit 147 determines that the control parameter has increased when the increase amount of the control parameter is larger than the threshold value of the predetermined increase amount, and determines whether or not the control parameter has decreased. In the present invention, it may be determined that the control parameter has decreased when the decrease amount of the control parameter is smaller than the threshold value of the predetermined decrease amount. As a result, the number of vertical fluctuations associated with significant fluctuations in the control parameters is counted as the fluctuation frequency, so that the influence of errors and noise added to the control parameters is reduced.

The image pickup control unit 147 determines the frequency of fluctuation of the representative value of the luminance, for example, an event in which the representative value of the luminance changes from a state in which the representative value of the luminance is larger than the target value of the predetermined luminance to a state in which the representative value of the luminance changes from a state larger than the target value of the predetermined luminance within a predetermined period up to that point, and the luminance. The total number of times with the event that the representative value of is changed from the state where the representative value is larger than the target value of the predetermined brightness to the state where the representative value is smaller than the target value is counted. The predetermined time may be a period longer than the fluctuation cycle assumed as the fluctuation cycle of the control parameter.
The image pickup control unit 147 may be preset with a second target value larger than the target value of the predetermined brightness and a third target value smaller than the target value of the predetermined brightness. Then, the image pickup control unit 147 is an event in which the representative value of the brightness changes from a state in which the representative value of the brightness is smaller than the third target value of the predetermined brightness to a state in which the representative value is larger than the second target value, and is larger than the second target value. The number of events that change from a state to a state smaller than the third target value may be counted as the fluctuation frequency. As a result, the number of vertical fluctuations associated with the significant fluctuation of the luminance from the predetermined target value is counted as the fluctuation frequency, so that the influence of the error and noise added to the luminance target value is reduced.

(Example of periodicity detection)
Next, an example of periodicity detection for determining the conditions for determining the control parameters will be described.
The image pickup control unit 147 performs a Fourier transform on the control parameters within a predetermined period up to the present time, and calculates the intensity for each frequency. The image pickup control unit 147 determines that there is periodicity in the time change of the control parameter when there is a frequency in which the calculated intensity becomes larger than the threshold value of the predetermined intensity, and determines the periodicity when such frequency does not exist. Judge not to have. When the image pickup control unit 147 determines that the frequency has periodicity, the reciprocal of the frequency can be calculated as the fluctuation period.
When detecting the periodicity, the image pickup control unit 147 may calculate the autocorrelation coefficient between the control parameters up to the present time and the time-shifted control parameters for each shift amount. Then, the image pickup control unit 147 determines that the time change of the control parameter has periodicity when there is a shift amount in which the autocorrelation coefficient is larger than the threshold value of the predetermined autocorrelation coefficient, and such a shift amount. If does not exist, it is determined that there is no periodicity. When the image pickup control unit 147 determines that the image has periodicity, the image pickup control unit 147 can determine the shift amount as the fluctuation period.

Further, when the image pickup control unit 147 determines that the change of the control parameter has periodicity, the imaging control unit 147 may further determine whether or not the amplitude of the control parameter has stationarity. In the determination of steadyness, the image pickup control unit 147 detects, for example, the maximum value and the minimum value of the control parameter for each cycle in the section determined to have periodicity. Then, the image pickup control unit 147 determines an absolute value of the difference between the maximum value in the latest cycle (hereinafter referred to as "current cycle") and the maximum value in the immediately preceding cycle (hereinafter referred to as "previous cycle"). When the absolute value of the difference between the minimum value in the current cycle and the minimum value in the previous cycle is less than the threshold value of the predetermined difference, it is determined to have steadiness. On the other hand, in the image pickup control unit 147, the absolute value of the difference between the maximum value in the current cycle and the maximum value in the previous cycle is equal to or more than the threshold value of the predetermined difference, or the difference between the minimum value in the current cycle and the minimum value in the previous cycle. When the absolute value of is equal to or greater than the threshold value of a predetermined difference, it is determined that the product does not have steadyness.

The image pickup control unit 147 may further impose as the determination condition of the above case C01 on the condition that the amplitude of the control parameter has stationarity with respect to the periodicity of the change of the control parameter. That is, when the amplitude of the control parameter does not have stationarity, the image pickup control unit 147 determines the control parameter according to the case C02. As a result, the case where the control parameter is set to the average value of one cycle is limited to the case where the change in the brightness of the image of the subject is periodic and the amplitude is steady, and the change in the brightness of the image of the subject is periodic. However, when the amplitude is unsteady, the ability to follow the target value of the brightness obtained by the control is not lost. Such a condition determination is suitable when the change in luminance accompanying the movement of the subject is periodic and the amplitude is constant, as in the case of an image of a turbine blade of a gas turbine engine in a power plant.

The image pickup control unit 147 also determines the presence or absence of periodicity in the evaluation value of luminance by the same method as the control parameter, and if it is determined to have periodicity, the period can be determined. The image pickup control unit 147 can determine whether or not the amplitude is stationary with respect to the evaluation value of the luminance by the same method as the control parameter. The image pickup control unit 147 may impose the determination condition of the above case C04 on the condition that the amplitude of the evaluation value of the luminance has stationarity with respect to the periodicity of the evaluation value of the luminance. That is, when the amplitude of the evaluation value of the luminance does not have stationarity, the image pickup control unit 147 determines the control parameter according to the case C05.

(Image control)
Next, an example of image pickup control according to this embodiment will be described. In the following description, a case where one or both of the exposure time and the gain are used as control parameters will be taken as an example.
FIG. 2 is a flowchart showing an example of image pickup control according to the present embodiment.
(Step S102) The luminance evaluation unit 145 calculates and calculates the luminance evaluation value indicating the luminance of the entire image of the latest one frame at that time based on the luminance signal input from the pixel interpolation / luminance color difference conversion unit 143. The evaluation value information, the exposure time, and the gain indicating the evaluated evaluation value are stored in the parameter storage unit 113. After that, the process proceeds to step S104.
(Step S104) The image pickup control unit 147 refers to the control parameter information stored in the parameter storage unit 113, and counts the change frequency of the exposure time or the gain within a predetermined period up to that point. When the image pickup control unit 147 determines that the change frequency is higher than the threshold value of the predetermined change frequency (YES in step S104), the process proceeds to the process of step S106. When the image pickup control unit 147 determines that the change frequency is equal to or less than the threshold value of the predetermined change frequency (step S104 NO), the process proceeds to step S120.

(Step S106) The image pickup control unit 147 determines whether or not the fluctuation of the exposure time or the gain has periodicity. When the image pickup control unit 147 determines that the exposure time has periodicity, the image pickup control unit 147 further determines whether or not the exposure time or the amplitude of the gain has stationarity. When the image pickup control unit 147 determines that the fluctuation of the exposure time or the gain is periodic and the amplitude is steady (step S106 YES), the image pickup control unit 147 proceeds to the process of step S108. When the image pickup control unit 147 determines that the fluctuation of the exposure time or the gain is not periodic, or even if the fluctuation is periodic, the amplitude is not steady (step S106 NO), the process proceeds to the process of step S130.
(Step S108) The image pickup control unit 147 refers to the control parameter information stored in the parameter storage unit 113, and calculates the average value of the exposure time or gain within the latest one cycle up to that point. The image pickup control unit 147 sets the average value of the calculated exposure time in the timing generator 114, or sets the average value of the calculated gain in the AFE137. After that, the process proceeds to step S110.
(Step S110) The image pickup control unit 147 erases (reset) the evaluation value information stored up to that point and the control parameter information indicating the exposure time or gain. After that, the process proceeds to step S112.

(Step S112) The luminance evaluation unit 145 calculates the latest luminance evaluation value of one frame at that time based on the luminance signal input from the pixel interpolation / luminance color difference conversion unit 143. The brightness evaluation unit 145 stores the evaluation value information indicating the calculated evaluation value in the parameter storage unit 113, and the imaging control unit 147 stores the control parameter information indicating the exposure time or gain set at that time in the parameter storage unit 113. Remember. After that, the process proceeds to step S114.
(Step S114) The image pickup control unit 147 determines whether or not the evaluation value information indicating the evaluation value and the control parameter information indicating the exposure time or the gain are stored for at least a predetermined time with reference to the parameter storage unit 113. do. When it is determined that the image pickup control unit 147 has been stored for at least a predetermined time (YES in step S114), the image pickup control unit 147 proceeds to the process of step S102 and changes the frame to be processed to the next frame. When the image pickup control unit 147 determines that the memory is not stored for a predetermined time (step S114 NO), the process proceeds to the process of step S112, and the frame to be processed is changed to the next frame.

(Step S120) The image pickup control unit 147 refers to the evaluation value information stored in the parameter storage unit 113, and counts the frequency of changing the evaluation value of the luminance within a predetermined period up to that point. The image pickup control unit 147 determines whether or not the change frequency is higher than the threshold value of the predetermined change frequency. When it is determined that the change frequency is higher than the threshold value of the predetermined change frequency (YES in step S120), the process proceeds to step S122. When the image pickup control unit 147 determines that the change frequency is equal to or less than the threshold value of the predetermined change frequency (step S120 NO), the process proceeds to step S140.
(Step S122) The image pickup control unit 147 determines whether or not the fluctuation of the evaluation value of the luminance has periodicity. When the image pickup control unit 147 determines that it has periodicity, it further determines whether or not the amplitude of the evaluation value has stationarity. When the image pickup control unit 147 determines that the fluctuation of the evaluation value is periodic and the amplitude is steady (step S122 YES), the imaging control unit 147 proceeds to the process of step S124. When the image pickup control unit 147 determines that the fluctuation of the evaluation value is not periodic, or even if the fluctuation is periodic, the amplitude is not steady (step S122 NO), the process proceeds to the process of step S130.

(Step S124) The image pickup control unit 147 refers to the evaluation value information stored in the parameter storage unit 113, and calculates the average value of the evaluation values of the latest luminance within one cycle up to that point. The image pickup control unit 147 determines the average value of the calculated evaluation values as the brightness control value, executes AE control based on the determined brightness control value and the predetermined brightness target value, and calculates the exposure time or gain. .. The image pickup control unit 147 sets the average value of the calculated exposure time in the timing generator 114, or sets the average value of the calculated gain in the AFE137. Further, the image pickup control unit 147 stores the control parameter information indicating the calculated exposure time or gain in the parameter storage unit 113. After that, the process proceeds to step S102, and the frame to be processed is changed to the next frame.

(Step S130) The image pickup control unit 147 sets, as an update parameter for updating the exposure time or gain, an update parameter for low speed for lowering the followability than the normal update parameter. After that, the process proceeds to step S150.
(Step S140) The image pickup control unit 147 sets a normal update parameter as an update parameter for updating the exposure time or the gain. After that, the process proceeds to step S150.

(Step S150) The image pickup control unit 147 determines the evaluation value in the latest frame at that time as the luminance control value, and executes AE control based on the predetermined luminance control value and the predetermined luminance target value for exposure. Calculate time or gain. The image pickup control unit 147 sets the average value of the calculated exposure time in the timing generator 114, or sets the average value of the calculated gain in the AFE137. Further, the image pickup control unit 147 stores the control parameter information indicating the calculated exposure time or gain in the parameter storage unit 113. After that, the process proceeds to step S102, and the frame to be processed is changed to the next frame.

(Example of brightness control)
Next, an example of luminance control according to the present embodiment will be described.
FIG. 3 is a diagram showing an example of luminance control according to the present embodiment.
In FIG. 3, I31, I32, and I33 show the amount of incident light on the objective lens 105 at each time, the reciprocal of the exposure time, and the evaluation value of the luminance, respectively. In this example, the incident light amount I31 fluctuates periodically as in the image of the turbine blade of the gas turbine engine in the initially rotating power plant as a subject. The image pickup control unit 147 performs AE control using normal update parameters to obtain an exposure time (FIG. 2, steps S140 and S150). Here, the exposure time is calculated so that the difference between the luminance evaluation value I33 and the luminance target value becomes small. Therefore, the fluctuation of the incident light amount I31 is followed by the reciprocal I32 of the exposure time, and the fluctuation of the evaluation value I33 of the luminance is relaxed more than the fluctuation of the incident light amount I31. Since the exposure time is also periodically and the amplitude is controlled to be constant according to the fluctuation of the incident light amount I31 of the luminance (FIG. 2, step S106), the image pickup control unit 147 exposes the exposure within the latest one cycle at the time _t31 . The average value of time is calculated, and the calculated average value of exposure time is set in the timing generator 114 (FIG. 2, step S108). Therefore, although the brightness of the moving subject fluctuates, the brightness of the subject whose brightness does not originally change, such as a non-moving subject, becomes constant, which reduces or eliminates the annoyance for the user.

In the example shown in FIG. 3, since the set exposure time is fixed to a constant value, the fluctuation of the incident light amount I31 and the fluctuation of the luminance evaluation value I33 are synchronized. Since the exposure time is constant from at least the time t ₃₁ to the time t ₃₂ after the lapse of a predetermined time, it is determined that the fluctuation frequency of the exposure time is equal to or less than the predetermined fluctuation frequency at the time immediately after that (FIG. 2, step). S104 NO). At that time, the fluctuation frequency of the evaluation value is higher than the predetermined fluctuation frequency (FIG. 2, step S120 YES), and it is determined that the fluctuation frequency of the evaluation value is periodic and the amplitude is steady (FIG. 2, step S122 YES). ). Therefore, the image pickup control unit 147 starts AE control using the average value of the latest evaluation values within one cycle up to that point as the control value of the luminance (FIG. 2, step S124). Therefore, the brightness is not controlled according to the periodic fluctuation of the incident light amount, but the exposure time is controlled according to the more macroscopic change tendency (macro trend) such as the fluctuation of the amplitude of the incident light amount or the evaluation value of the brightness. To. Further, since the periodic fluctuation of the brightness of the subject whose brightness does not originally change does not occur, the annoyance for the user is reduced.

At time t ₃₃ , it is determined that the fluctuation frequency of the exposure time is equal to or less than the predetermined fluctuation frequency (FIG. 2, step S104 NO), and the fluctuation frequency of the evaluation value is higher than the predetermined fluctuation frequency (FIG. 2, step S120 YES). It is determined that the fluctuation frequency of the evaluation value is periodic and the amplitude is not constant (FIG. 2, step S122 NO). Therefore, in the image pickup control unit 147, the image pickup control unit 147 starts the AE control using the update parameter for low speed and obtains the exposure time (FIG. 2, steps S130, S150).
At time t ₃₄ , it is determined that the fluctuation frequency of the exposure time is equal to or less than the predetermined fluctuation frequency (FIG. 2, step S108 NO), and the fluctuation frequency of the evaluation value is higher than the predetermined fluctuation frequency (FIG. 2, step S120 YES). It is determined that the fluctuation frequency of the evaluation value is periodic and the amplitude is constant (FIG. 2, step S122 YES). The AE control is restarted with the average value of the evaluation values in the latest one cycle up to that point as the control value of the brightness (FIG. 2, step S124).

According to the above-mentioned control, after the time t ₃₁ , the change tendency of the incident light amount and the change tendency of the evaluation value of the luminance are substantially synchronized. This means that the brightness of the part of the subject whose brightness originally fluctuates in the captured image fluctuates according to the fluctuation, and the brightness of the part of the subject whose brightness does not originally fluctuate does not fluctuate. show. Therefore, it is expected that the discomfort and fatigue of the image captured by the user will be reduced or eliminated.

FIG. 4 is a diagram showing another example of luminance control according to the present embodiment.
In FIG. 4, I41 and I42 indicate the amount of incident light on the objective lens 105 at each time and the reciprocal of the exposure time, respectively. In this example, it is assumed that the image pickup control unit 147 performs AE control using normal update parameters to obtain the exposure time (FIG. 2, steps S140 and S150). The incident light amount I41 fluctuates randomly and has no periodicity. Therefore, the reciprocal I42 of the exposure time quickly follows the incident light amount I41. However, as described above, when the incident light amount I41 and the fluctuation of the exposure time have periodicity, it causes annoyance to the user.

FIG. 5 is a diagram showing an example of a captured image. In FIG. 5, it is assumed that the exposure time is obtained by performing AE control using normal update parameters (FIG. 2, steps S140 and S150). The images Im51-Im55 are images taken when the insertion portion 10 is inserted inside the pipe. The times when the images Im51-Im55 are imaged are in that order. The image Im51 is an image taken when the insertion portion 10 is inserted inside the pipe. In this example, the tip of the insertion portion 10 floats in the center of the pipe. Therefore, the central part of the screen is darker than the surrounding area. The image Im52 is an image taken in a state where the lower right portion of the screen at the tip of the insertion portion 10 is in contact with the inner wall surface of the pipe. Therefore, halation occurs in the portion representing the inner wall surface. Halation is a phenomenon in which the signal value is limited to the maximum value because the luminance detected for each pixel exceeds a predetermined maximum value that can be expressed as a signal value. The image Im53 is an image taken with the lower right portion of the screen at the tip of the insertion portion 10 still in contact with the inner wall surface of the pipe. However, it becomes darker than the image Im52 due to AE control. In particular, the hollow part of the pipe, which is not expected to change in brightness, becomes dark. The image Im54 is an image taken again with the tip of the insertion portion 10 floating in the center of the pipe. The brightness of the image Im54 is expected to be the same as that of the initially captured image Im51, but it becomes darker than that of the image Im51 due to AE control. The image Im55 is an image taken in a state where the tip of the insertion portion 10 continues to float in the center of the pipe. The image Im55 becomes brighter than the image Im54 by AE control, and becomes equivalent to the brightness of the image Im51. As described above, when the tip of the insertion portion 10 hits or separates from the inner wall surface of the pipe, the AE control works when the amount of incident light changes, resulting in a troublesome image in which the change in brightness is large. In particular, when the change in the brightness of the entire image is periodic, the brightness of the portion where the brightness does not originally change changes, so that the annoyance becomes remarkable.

FIG. 6 is a diagram showing still another example of luminance control according to the present embodiment.
In FIG. 6, I61 and I62 indicate the amount of incident light on the objective lens 105 at each time and the reciprocal of the exposure time, respectively. In this example, it is assumed that the image pickup control unit 147 performs AE control using the update parameter for low speed to obtain the exposure time (FIG. 2, steps S130, S150). The incident light amount I51 fluctuates randomly and has no periodicity. In this case, the change in the brightness of the captured image (live image) is suppressed, but the deviation of the evaluation value of the brightness from the target value becomes large.

FIG. 7 is a diagram showing another example of the captured image. In FIG. 7, it is assumed that the exposure time is obtained by performing AE control using the update parameter for low speed (FIG. 2, steps S130 and S150). The imaging conditions of the images Im71-Im75 are the same as the imaging conditions of the images Im51-Im55 except for the update parameters. Therefore, the images Im71, Im72, and Im75 are the same as the images Im51, Im52, and Im55, respectively. However, since the followability in the AE control is lower than that in the example shown in FIG. 5, a portion where halation occurs in the image Im73 is left, but the change in brightness from the image Im72 as a whole image is alleviated. Further, the brightness of the image Im74 is closer to the brightness of the initial image Im71. Therefore, the annoyance to the user is alleviated or eliminated.

(Example of followability control)
Next, a control example of followability in AE control will be described.
When determining the exposure time as a control parameter, the image pickup control unit 147 performs, for example, proportional control (Proportional Control, P control). In P control, the image pickup control unit 147 multiplies the luminance difference, which is the difference between the evaluation value of the luminance immediately before the control and the target value, by a predetermined proportional gain, and obtains the amount of change in the exposure time, which is a control parameter, for each frame. Calculate to. The image pickup control unit 147 adds the calculated change amount to the value of the immediately preceding control parameter (hereinafter referred to as “control value”) to determine the control value at that time. Proportional gain is a parameter for controlling the control speed. The larger the proportional gain, the higher the control speed, and the smaller the proportional gain, the lower the control speed. Therefore, when the image pickup control unit 147 lowers the followability, the amount of fluctuation of the exposure time per frame is reduced by using a proportional gain smaller than the normal proportional gain as a predetermined proportional gain, so that the followability is improved. Can be lowered. For example, the curves I81 and I82 in FIG. 8 show the case where the control of the exposure time is started at the time _t81 . However, the curve I82 shows that the amount of change in the exposure time with respect to the passage of time, that is, the followability is lower than that of the curve I81 showing the standard followability.

Further, in the image pickup control unit 147, when the followability is lowered, instead of lowering the control speed, a first dead region and a second dead region wider than the first dead region are set. , Hysteresis may be provided. The first dead region is the range of luminance that stops AE control, while the second dead region is the range of luminance that does not start AE control, that is, the range of luminance that starts AE control outside that region. Is. Here, the lower the followability, the wider the range of the second dead region with respect to the first dead region. By providing a difference between the range of the brightness at which the AE control is started and the range of the brightness at which the AE control is stopped in this way, the difference between the target value of the brightness and the evaluation value becomes large, so that the start of the AE control is delayed. In the example shown in FIG. 8, E8 and E8'indicate the lower limit and the upper limit of the first dead region for stopping the AE control, respectively. However, the first dead region is constant regardless of the followability. A81 and A83 indicate a second dead region that does not start the AE control related to the standard followability, and a second dead region that stops the AE control when the followability is lowered, respectively. The median values of the second dead regions A81 and A83 are the target values E0 for luminance, respectively. E81 and E81'indicate the lower limit and the upper limit of the second dead region A81, respectively. E83 and E83'indicate the lower limit and the upper limit of the second dead region A83, respectively.

The curve C81 shows the evaluation value of the luminance at each time when the standard followability is set. Since the evaluation value of the luminance exceeds the upper limit E81'of the second dead region A81 from the beginning (time t ₈₁ ), the image pickup control unit 147 starts AE control, and the evaluation value is the upper limit E8 of the first dead region. When it reaches', AE control is stopped. On the other hand, the curve C83 shows the evaluation value of the luminance at each time when the followability is lowered. The image pickup control unit 147 starts AE control at time t ₈₃ when the evaluation value of the luminance exceeds the upper limit E83'of the second dead region A83, and when the evaluation value reaches the upper limit E8'of the first dead region. AE control is stopped.
The image pickup control unit 147 can control the luminance followability by controlling the tracking speed or the dead region for the gain as well as the exposure time.

(Brightness control example)
When the fluctuation amount of the evaluation value of the luminance is larger than the threshold value of the predetermined fluctuation amount, the image pickup control unit 147 may make the range of the output level in the correction processing unit 144 narrower than the predetermined standard range. .. When narrowing the range of the output level, the image pickup control unit 147 determines one gamma table according to the range. A gamma table corresponding to each of a plurality of output level ranges is set in advance in the image pickup control unit 147. Each gamma table is configured for each of the plurality of input levels in association with the output level corresponding to each input level. The correction processing unit 144 performs gamma correction and acquires an output level by referring to the gamma table determined by the image pickup control unit 147. The correction processing unit 144 specifies an output level corresponding to an input level in gamma correction. FIG. 9 is a diagram illustrating a gamma table. Curves I91 and I92 show standard gamma tables, respectively, and show gamma tables with a smaller range of output levels. By using the gamma table of I92, the output level range Dg92 is narrower than the standard output level range Dg91. Therefore, when the fluctuation of the brightness is remarkable, the fluctuation of the brightness of each one frame image is also suppressed by gamma correction using the gamma table of I92. Therefore, the annoyance due to the fluctuation of the brightness of the image is reduced. Further, when detailed observation is not always required, such as during insertion of the insertion portion 10 into the inspection object, such reduction of the output level range Dg91 can be allowed.

When the image pickup control unit 147 detects an operation for the operation of the insertion unit 10 such as a bending operation, the image pickup control unit 147 does not reduce the luminance followability in the AE control regardless of the fluctuation frequency or periodicity of the control parameter or the luminance evaluation value. In addition, it may be maintained in a predetermined normal followability. This is because it is expected that the brightness of the captured image immediately follows the target value with respect to the change in the amount of incident light due to the operation intended by the user. The image pickup control unit 147 may detect the presence or absence of an operation based on the control signal input from the user interface unit 111 via the system control unit 110. The image pickup control unit 147 determines that an operation for the operation is detected when a control signal instructing an operation such as bending of the insertion unit 10 is input, and operates when a control signal indicating the operation is not input. It may be determined that the operation for is not detected.

As described above, the endoscope device 1 according to the present embodiment includes a CCD image pickup device 104 that generates an image of a subject, an insertion section 10 provided with the CCD image pickup device 104, and an image processing section 140. Be prepared. The image processing unit 140 includes a brightness evaluation unit 145 that evaluates the brightness of an image, and an image pickup control unit 147 that sets brightness control parameters so that the evaluated brightness approaches a predetermined target value. The image pickup control unit 147 detects at least the fluctuation state of one of the evaluated brightness and the set control parameter, and determines the control parameter based on the fluctuation frequency of the fluctuation status.
With this configuration, the brightness of the image is controlled based on the periodicity of the fluctuation state of at least one of the brightness of the captured image and the controlled control parameter. Therefore, the discomfort such as annoyance to the user due to the fluctuation factor such as the periodic fluctuation of the brightness of the image is suppressed or eliminated, and the image is taken, as compared with the case where the brightness is uniformly controlled without considering the fluctuation frequency of the fluctuation situation. The brightness of the image is controlled more comfortably for the user.

Further, when the image pickup control unit 147 determines that the fluctuation of the control parameter has periodicity, the image pickup control unit 147 may determine the average value of the control parameter in the cycle up to the present time as the current control parameter.
With this configuration, when the fluctuation of the control parameter has periodicity, the control parameter is set to the average value in the cycle. Therefore, the periodic fluctuation of the brightness in the portion of the image obtained by the periodic fluctuation of the control parameter, which originally does not cause the change in the brightness, is eliminated. Therefore, the discomfort associated with the periodic fluctuation of the brightness of the image is eliminated, and more appropriate brightness control is performed.

Further, when the imaging control unit 147 determines that the fluctuation of the luminance has periodicity, the imaging control unit may determine the control parameter so that the average value in the period up to the present time of the luminance approaches the target value of the luminance. With this configuration, the followability to the periodic variation of the luminance is relaxed or eliminated without completely losing the followability to the global variation of the luminance. Therefore, the discomfort associated with the periodic fluctuation of the brightness of the image is eliminated, and the brightness of the image is controlled more comfortably.

Further, when the image pickup control unit 147 determines that the control parameter or the fluctuation of the luminance does not have periodicity, the imaging control unit 147 may determine the control parameter by lowering the followability of the luminance to the target value than the predetermined followability. .. Since this configuration alleviates the fluctuation of the brightness of the image due to the fluctuation of the control parameters, the discomfort caused by the aperiodic fluctuation of the brightness of the image is reduced, and the brightness is controlled more comfortably.

Further, the image pickup control unit 147 may reduce the amount of change in the control parameter when the followability of the luminance to the target value is made lower than the predetermined followability. With this configuration, it is possible to reduce the discomfort associated with the aperiodic variation in the brightness of the image without completely stopping the control of the brightness.

Further, when the image pickup control unit 147 makes the followability of the brightness to the target value lower than the predetermined followability, the image pickup control unit 147 may increase the range of the brightness that does not start the change of the control parameter. With this configuration, the difference between the brightness of the captured image and its target value is increased to reduce the followability, so that the brightness of the image is aperiodic without changing the process for updating the control parameters. The discomfort associated with fluctuations can be reduced.

Further, the image pickup control unit 147 detects the fluctuation frequency of the control parameter, and when the fluctuation frequency of the control parameter is higher than the fluctuation frequency of the predetermined control parameter, determines whether or not the fluctuation of the control parameter has periodicity. You may.
With this configuration, when the fluctuation of the control parameter is relatively remarkable, the control parameter is set according to the presence or absence of periodicity of the control parameter. Therefore, the discomfort caused by the fluctuation of the brightness of the image due to the fluctuation of the control parameter can be reduced by a method according to the presence or absence of the periodicity of the fluctuation.

Further, the image pickup control unit 147 detects the variation frequency of the luminance when the variation frequency of the control parameter is equal to or less than the variation frequency of the predetermined control parameter, and when the variation frequency of the luminance is higher than the predetermined variation frequency, the luminance is changed. It may be determined whether or not the fluctuation has periodicity.
With this configuration, even if the fluctuation of the control parameter is relatively gradual, when the fluctuation of the luminance is relatively remarkable, it is determined whether or not the fluctuation of the luminance has periodicity, and the control parameter according to the presence or absence of the periodicity. Can be set. Therefore, even when the fluctuation of the control parameter is relatively gentle, when the fluctuation of the brightness of the image is relatively remarkable, it is possible to reduce the discomfort depending on the presence or absence of the periodicity of the fluctuation.

Further, when the operation of the insertion unit 10 is detected, the image pickup control unit 147 maintains the followability to the target value of the luminance at a predetermined followability and determines the control parameter. With this configuration, when the operation of the insertion unit 10 is detected, therefore, in a situation where a change in the brightness of the image after control, which causes annoyance, is allowed, the followability to the target value is not reduced. The brightness of the captured image is controlled.

In the above embodiment, the case where the image pickup device provided at the tip of the insertion portion 10 is the CCD image pickup device 104 is taken as an example, but the present invention is not limited to this. The insertion unit 10 may include a CMOS (Complementary metal-axis-semiconductor) image pickup element instead of the CCD image pickup element 104. When the CMOS image sensor is provided, the timing generator 114, the CCD drive circuit 115, the preamplifier 136, and the AFE 137 may be omitted from the main body 11. Therefore, the image pickup control unit 147 generates an image pickup control signal indicating the drive timing and gain of each element according to the exposure time as a control parameter used for image pickup, and outputs the generated image pickup control signal to the CMOS image sensor. The CMOS image sensor generates an image pickup signal indicating a signal value for each pixel at a drive timing specified by the image pickup control signal, and amplifies the generated image pickup signal with a gain specified by the image pickup control signal. The CMOS image sensor outputs the amplified image pickup signal to the black correction unit 142 of the image processing unit 140.

Further, in the above case C01, the case where the image pickup control unit 147 calculates the average value of the latest control parameters within one cycle up to that point and uses the calculated average value for the brightness control is taken as an example. Not limited to this. The image pickup control unit 147 may calculate the average value of the control parameters in the past one cycle or a plurality of cycles up to that point, and use the calculated average value for the brightness control.
Further, the imaging control unit 147 takes as an example the case where the average value of the evaluation values of the brightness within the latest one cycle up to that point is calculated as the brightness control value, but within the past multiple cycles up to that point in time. The average value of the evaluation values of the brightness may be calculated as the control value of the brightness.

In the above embodiment, the case where P control is mainly adopted in AE control is taken as an example, but control variables such as exposure time are calculated so as to reduce the size of the difference between the controlled amount and the target amount. Any method may be used as long as it is a method. For example, PI (Proportional-Integral Control) control and PID (Proportional-Integral-Differential Control) control may be used.
The LED 106 may be provided in the main body 11. In that case, the insertion portion 10 may be provided with a light guide that transmits the light from the LED 106 to the objective lens 105 and emits the light from the tip portion of the insertion portion 10. Further, if the image signal from the system control unit 110 can be received and the image based on the received image signal can be displayed, the LCD 118 may be omitted in the main body unit 11.
In the above embodiment, the case where the exposure time and the gain are the control targets in the AE control is taken as an example, but as a means for controlling the amount of light received by the image sensor, for example, a parameter of a mechanical shutter (for example, The control target of the AE control may include exposure time, aperture, etc.) and filter parameters (for example, filter coefficient, etc.).

It should be noted that a part of the endoscope device 1 according to the above embodiment, for example, the system control unit 110 and the image processing unit 140, respectively or as a whole, has one or more processors, a memory, an input / output interface, and a computer read. It may be realized by a computer made of a possible recording medium or the like. In that case, it is realized by recording a program for realizing the functions of the system control unit 110 and the image processing unit 140 individually or as a whole on a recording medium, loading the recorded program into a computer system, and executing the program. May be good. The "computer system" is a computer system built in the endoscope device 1, and includes hardware such as an OS (Operating System) and peripheral devices. The processor may be any of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a PLD (Programmable Logic Device), and the like.
The "computer-readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD (Compact Disc) (registered trademark) -ROM, or a computer system. A storage device such as a hard disk. Further, a "computer-readable recording medium" is a medium that dynamically holds a program for a short time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In that case, a program may be held for a certain period of time, such as a volatile memory inside a computer system serving as a server or a client. Further, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Further, a part of the endoscope device 1 according to the above-described embodiment, for example, each or a combination of the system control unit 110 and the image processing unit 140, or all of them may be an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit). ) Etc. may be realized as an integrated circuit. Each functional block of the endoscope device 1 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of making an integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, when an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology, an integrated circuit based on this technology may be used.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and variations thereof. It is possible to add, omit, replace, and make other changes to the configuration without departing from the spirit of the present invention.
Further, the present invention is not limited by the above description, but only by the scope of the attached claims.

1 ... Endoscope device, 10 ... Insertion part, 11 ... Main body part, 104 ... CCD image sensor, 105 ... Objective lens, 106 ... LED, 107 ... Thermista, 108 ... Acceleration sensor, 109 ... Wire fixing part, 110 ... System Control unit, 110a ... Image recording unit, 110b ... LCD controller, 111 ... User interface unit, 112 ... Storage medium, 113 ... Parameter storage unit, 114 ... Timing generator, 115 ... CCD drive circuit, 118 ... LCD, 119 ... LED drive Circuit, 121 ... Curved control unit, 122 ... UD curved motor, 123 ... RL curved motor, 124, 125 ... Wire connecting part, 126 ... Curved wire, 134, 135 ... Detachable connector, 136 ... Preamp, 137 ... AFE, 140 ... Image processing unit (control unit), 142 ... Black correction unit, 143 ... Pixel interpolation / brightness color difference conversion unit, 144 ... Correction processing unit, 145 ... Brightness evaluation unit, 147 ... Image pickup control unit

Claims (14)

An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The control unit
The fluctuation frequency of the control parameter is detected, and when the fluctuation frequency of the control parameter is higher than a predetermined fluctuation frequency, it is determined whether or not the fluctuation of the control parameter has periodicity.
An endoscope device, characterized in that, when it is determined that the fluctuation of the control parameter has periodicity, the average value of the control parameter in the cycle up to the present time is determined as the control parameter at the present time. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The control unit
The frequency of variation in luminance is detected, and when the frequency of variation in luminance is higher than a predetermined frequency of variation, it is determined whether or not the variation in luminance has periodicity.
An endoscope device, characterized in that, when it is determined that the variation in luminance has periodicity, the control parameter is determined so that the average value of the luminance in the cycle up to the present time approaches the target value. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The control unit
The fluctuation frequency of the control parameter is detected, and when the fluctuation frequency of the control parameter is higher than a predetermined fluctuation frequency, it is determined whether or not the fluctuation of the control parameter has periodicity.
When it is determined that the fluctuation of the control parameter does not have periodicity, the endoscope device is characterized in that the followability of the luminance to the target value is made lower than the predetermined followability to determine the control parameter. .. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The control unit
The frequency of variation in luminance is detected, and when the frequency of variation in luminance is higher than a predetermined frequency of variation, it is determined whether or not the variation in luminance has periodicity.
An endoscope device characterized in that when it is determined that the variation in luminance does not have periodicity, the control parameter is determined by lowering the followability of the luminance to the target value below a predetermined followability. The control unit
The endoscope device according to claim 3 or 4, wherein when the followability is lowered, the amount of change in the control parameter is reduced. The control unit
The endoscope device according to claim 5, wherein when the followability is lowered, the range of luminance that does not start the change of the control parameter is increased. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The control unit
The fluctuation frequency of the control parameter is detected, and the fluctuation frequency is detected.
When the fluctuation frequency of the control parameter is equal to or less than the fluctuation frequency of the predetermined control parameter, the fluctuation frequency of the luminance is detected.
When the fluctuation frequency of the luminance is higher than the predetermined fluctuation frequency, it is determined whether or not the variation of the luminance has periodicity.
When it is determined that the variation in luminance has the periodicity, the control parameter is determined so that the average value of the luminance in the cycle up to the present time approaches the target value.
An endoscope device characterized in that when it is determined that the variation in luminance does not have the periodicity, the control parameter is determined by lowering the followability of the luminance to the target value below a predetermined followability. .. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The control unit
The fluctuation frequency of the control parameter is detected, and the fluctuation frequency is detected.
When the fluctuation frequency of the control parameter is equal to or less than the fluctuation frequency of the predetermined control parameter, the fluctuation frequency of the luminance is detected.
When the fluctuation frequency of the luminance is higher than the predetermined fluctuation frequency, it is determined whether or not the variation of the luminance has periodicity.
An endoscope device, characterized in that, when it is determined that the variation in luminance has the periodicity, the control parameter is determined so that the average value of the luminance in the cycle up to the present time approaches the target value. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The control unit
The fluctuation frequency of the control parameter is detected, and the fluctuation frequency is detected.
When the fluctuation frequency of the control parameter is equal to or less than the fluctuation frequency of the predetermined control parameter, the fluctuation frequency of the luminance is detected.
When the fluctuation frequency of the luminance is higher than the predetermined fluctuation frequency, it is determined whether or not the variation of the luminance has periodicity.
An endoscope device characterized in that when it is determined that the variation in luminance does not have the periodicity, the control parameter is determined by lowering the followability of the luminance to the target value below a predetermined followability. .. The control unit
When detecting the operation of the insertion portion,
The endoscope device according to any one of claims 1 to 9, wherein the control parameter is determined by maintaining the followability of the brightness to the target value at a predetermined followability. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
A method of operating an endoscope device including a control unit.
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
The fluctuation status of both the brightness and the control parameter is detected, and the fluctuation status is detected.
Based on the fluctuation situation, the control parameters are determined.
The fluctuation situation includes the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter.
The fluctuation frequency of the control parameter is detected, and the fluctuation frequency is detected.
When the fluctuation frequency of the control parameter is equal to or less than the fluctuation frequency of the predetermined control parameter, the fluctuation frequency of the luminance is detected.
When the fluctuation frequency of the luminance is higher than the predetermined fluctuation frequency, it is determined whether or not the variation of the luminance has periodicity.
When it is determined that the variation in luminance has the periodicity, the control parameter is determined so that the average value of the luminance in the cycle up to the present time approaches the target value.
An endoscope device characterized in that when it is determined that the variation in luminance does not have the periodicity, the control parameter is determined by lowering the followability of the luminance to the target value below a predetermined followability. How to operate. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
To the computer of the endoscope device equipped with
The procedure for evaluating the brightness of the image and
A procedure for setting the brightness control parameters so that the brightness approaches a predetermined target value, and
A procedure for detecting fluctuations in both the brightness and the control parameters, and
The control parameter is determined based on the fluctuation situation, and the fluctuation situation is a program for executing a procedure including the variation frequency of the luminance and the control parameter and the periodicity of the luminance and the control parameter. hand,
To the computer
The fluctuation frequency of the control parameter is detected, and the fluctuation frequency is detected.
When the fluctuation frequency of the control parameter is equal to or less than the fluctuation frequency of the predetermined control parameter, the fluctuation frequency of the luminance is detected.
When the fluctuation frequency of the luminance is higher than the predetermined fluctuation frequency, it is determined whether or not the variation of the luminance has periodicity.
When it is determined that the variation in luminance has the periodicity, the control parameter is determined so that the average value of the luminance in the cycle up to the present time approaches the target value.
A program for determining the control parameter by lowering the followability of the luminance to the target value from the predetermined followability when it is determined that the variation of the luminance does not have the periodicity. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
To the computer of the endoscope device equipped with
The procedure for evaluating the brightness of the image and
A procedure for setting the brightness control parameters so that the brightness approaches a predetermined target value, and
A procedure for detecting fluctuations in both the brightness and the control parameters, and
The control parameter is determined based on the fluctuation situation, and the fluctuation situation records a program for executing a procedure including the fluctuation frequency of the brightness and the control parameter, and the periodicity of the brightness and the control parameter. Computer-readable storage medium
The program is applied to the computer.
The fluctuation frequency of the control parameter is detected, and the fluctuation frequency is detected.
When the fluctuation frequency of the control parameter is equal to or less than the fluctuation frequency of the predetermined control parameter, the fluctuation frequency of the luminance is detected.
When the fluctuation frequency of the luminance is higher than the predetermined fluctuation frequency, it is determined whether or not the variation of the luminance has periodicity.
When it is determined that the variation in luminance has the periodicity, the control parameter is determined so that the average value of the luminance in the cycle up to the present time approaches the target value.
A storage medium that determines the control parameter by lowering the followability of the luminance to the target value from the predetermined followability when it is determined that the variation of the luminance does not have the periodicity. An image sensor that generates an image of the subject and
An insertion unit provided with the image sensor and
With a control unit,
The control unit
Evaluate the brightness of the image and
The brightness control parameters are set so that the brightness approaches a predetermined target value.
When at least the fluctuation frequency of one of the brightness and the control parameter is detected and the fluctuation frequency of at least one of the brightness and the control parameter is higher than the predetermined fluctuation frequency, at least one of the brightness and the control parameter. Detects the periodicity of fluctuations and
When it is determined that there is no periodicity in the fluctuation of at least one of the luminance and the control parameter, the control parameter is determined by lowering the followability of the luminance to the target value than the predetermined followability. A featured endoscope device.

Images