JP2010183206A - Method and device for processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce variation in luminance among a plurality of output images. <P>SOLUTION: A pixel value correction means 12 corrects the value of each pixel so that representative luminance is included in a prescribed range for respective acquired images for a plurality of continuously captured images in a time series. After that, a luminance change rate determination means 140 determines whether a rate of change in luminance of the representative luminance deviates from a prescribed range, and a periodical variation determination means 141 determines whether temporal variation in the representative luminance is periodical or irregular. A processing means 16 for irregular variation smoothes the acquired image by an already acquired image when the rate of change in luminance deviates from a prescribed range or when it is determined that temporal variation of the representative luminance is irregular. When it is determined that temporal variation of the representative luminance is periodical, a processing means 15 for periodical variation extracts one prescribed acquired image or a plurality of acquired images to generate output images in synchronization with the period of a processing term in the plurality of acquired images included in the processing term that is an integer multiple of the time variation period of the representative luminance. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing method and an image processing apparatus that acquire a plurality of images obtained by continuously capturing an imaging region in time series and perform image processing on the plurality of acquired images.

  Conventionally, as an image processing device that acquires a plurality of images obtained by continuously capturing a shooting area in time series and performs image processing on the plurality of acquired images, the pixel values of the acquired images are added in the time axis direction. Is known as an output image (see, for example, Patent Document 1). A conventional image processing apparatus smoothes variations in luminance between acquired images and reduces random noise.

  Such an image processing apparatus can be used in an intercom system capable of displaying an image, particularly an apartment intercom system.

JP 2007-36741 A

  By the way, when the above-described image processing apparatus is used in an intercom system for condominiums, an imaging unit connected to the image processing apparatus is attached to a lobby of the apartment. A discharge lamp such as a fluorescent lamp is generally used as a light source for a lobby to which an image pickup means is attached. Since the discharge lamp is lit at a cycle that is ½ that of the AC power supply, flicker occurs when an image is taken in the illumination environment of the discharge lamp. For example, in the case of the interlace method, as shown in FIG. 15, the discharge lamp fluctuation period is 10 milliseconds (= 1 / (50 Hz × 2)), and the rate of each field image is about 60 fields / second (field interval). 16.67 milliseconds), the brightness of the field image will fluctuate with a period of 50 milliseconds.

  However, in the conventional image processing apparatus, although random noise can be reduced by adding and smoothing the pixel value of the acquired image in the time axis direction, flicker generated by the fluctuation cycle of the discharge lamp can be reduced. I couldn't. For this reason, the conventional image processing apparatus has a problem that the luminance fluctuation due to flicker remains in the display image.

  The present invention has been made in view of the above points, and an object thereof is to provide an image processing method and an image processing apparatus capable of reducing luminance fluctuation between output images with respect to a plurality of output images. It is in.

  The image processing method according to the first aspect of the present invention acquires a plurality of images obtained by continuously capturing a shooting region in time series, performs image processing on the plurality of acquired images, and outputs an image after image processing as an output image. As an image processing method for outputting to the image display means, a luminance value calculated from a pixel value of a plurality of predetermined pixels for each acquired image for each of the plurality of acquired images is set as a representative luminance, and the representative luminance for each acquired image Correcting the pixel value of each pixel so as to be included in the preset range, and determining whether the luminance change rate indicating the corrected representative luminance with respect to the representative luminance before correction deviates from the specified range, It is determined whether or not the temporal variation in luminance of one or more pixels included in the acquired image is irregular, and when it is determined that the luminance change rate is out of the specified range, or one or more included in the acquired image Painting When it is determined that the luminance temporal variation is irregular, the latest acquired image is smoothed with the acquired image already acquired for each latest acquired image, and the plurality of sequentially smoothed for each of the latest acquired images. A first step of setting an image as a plurality of new acquired images, and determining whether the temporal variation of the representative luminance is periodic before or after the first step. When the time variation of the representative luminance is determined to be periodic, a predetermined period that is an integral multiple of the time variation period of the representative luminance is set as a processing period, and all the processing periods are included in the processing period. Among the plurality of acquired images, a predetermined one or a plurality of acquired images are extracted at a timing synchronized with the period of the processing period to generate an image, and the plurality of images generated in all the processing periods are replaced with the new plurality of images A second step of obtaining an obtained image, and outputting the plurality of acquired images after the first step and the second step as the output image to the image display means, To do.

  According to a second aspect of the present invention, in the first aspect of the present invention, when it is determined in the first aspect that the temporal variation of the representative luminance is periodic, the image display means displays the output image at the variation cycle of the representative luminance. Are sequentially output.

  The image processing method according to a third aspect of the present invention is the image processing method according to the first or second aspect, wherein the representative luminance is obtained from pixel values of a plurality of pixels included in a region where face detection or motion detection is performed in the acquired image. The average brightness is calculated.

  The invention of the image processing device according to claim 4 acquires a plurality of images obtained by continuously capturing a shooting region in time series, performs image processing on the plurality of acquired images, and outputs an image after image processing as an output image As an image processing device for outputting to the image display means, a luminance value calculated from pixel values of a plurality of predetermined pixels is set as a representative luminance for each acquired image for the plurality of acquired images, and the representative for each acquired image A pixel value correcting unit that corrects the pixel value of each pixel so that the luminance is included in a preset range, and a luminance change rate that indicates the representative luminance after correction with respect to the representative luminance before correction by the pixel value correcting unit are specified. A luminance change rate determination means for determining whether or not to deviate from a range; and an irregular fluctuation determination means for determining whether or not a temporal fluctuation in luminance of one or more pixels included in the acquired image is irregular; Brightness change When it is determined by the determining means that the rate of change in luminance deviates from the specified range, or the time variation of the luminance of one or more pixels included in the acquired image is determined by the irregular variation determining unit to be irregular. The latest acquired image is smoothed with the acquired image that has already been acquired for each latest acquired image, and a plurality of images that are sequentially smoothed for each of the latest acquired images are used as the new acquired images. A processing unit for variation, a periodic variation determination unit that determines whether or not the temporal variation of the representative luminance is periodic, and a determination that the temporal variation of the representative luminance is periodic by the periodic variation determination unit In this case, a predetermined period that is an integral multiple of the time variation period of the representative luminance is set as the processing period, and the processing period is set to be the same among the plurality of acquired images included in the processing period. For periodic variation, a predetermined one or a plurality of acquired images are extracted at a timing synchronized with the period of the physical period to generate an image, and the plurality of images generated in all the processing periods are newly acquired as the plurality of acquired images. And a processing means.

  According to the first aspect of the present invention, when the time variation of the luminance of the evaluation target pixel is irregular, it is possible to reduce the random noise by smoothing the acquired image in the time axis direction, and the time of the representative luminance. When the fluctuation is periodic, flicker can be reduced by generating an output image with the same brightness for each period that is an integral multiple of the time fluctuation period of the representative brightness. As a result, luminance fluctuations between output images can be reduced.

  According to the second aspect of the present invention, the output image can be displayed at the same speed as the acquired image by sequentially outputting the output image in accordance with the period of time variation of the representative luminance.

  According to invention of Claim 3, each process can be performed using the brightness | luminance of the important part in an acquired image.

  According to the invention of claim 4, when the time variation of the representative luminance is irregular, random noise can be reduced by smoothing the acquired image in the time axis direction, and the time variation of the representative luminance is periodic. If this is the case, flicker can be reduced by generating an output image with the same luminance for each period that is an integral multiple of the time variation period of the representative luminance, and as a result, output images for a plurality of output images. It is possible to reduce the luminance fluctuation between the two.

1 is a block diagram illustrating a configuration of a first embodiment. It is a figure which shows the usage example same as the above. It is a block diagram which shows the structure of the process means for a periodic fluctuation which concerns on the same as the above. It is a figure for demonstrating operation | movement of the process means for periodic fluctuations based on the same as the above. It is a block diagram which shows the structure of the processing means for irregular fluctuations based on the same as the above. (A) is an example of setting the measurement points of the motion vector of the same subject area, (b) is a measurement result of each motion vector measurement point of the same subject area, (c) is the movement of the face area. FIG. 4D is a setting example diagram of vector measurement points, and FIG. 4D is a measurement result diagram of each motion vector measurement point in the face area same as above. It is an equivalent circuit diagram of the infinite length impulse response IIR filter corresponding to the processing operation of the irregular variation processing means according to the above. It is explanatory drawing of the addition ratio of the preservation | save image of a synthesized image, and an acquisition image in the same as the above. It is principle explanatory drawing of the motion correction in the same as the above. It is a flowchart which shows operation | movement of the image processing apparatus which concerns on the same as the above. FIG. 10 is a diagram for explaining an operation of a periodic variation processing unit according to the second embodiment. It is a figure for demonstrating operation | movement of the process means for periodic fluctuations concerning Embodiment 3. FIG. It is a figure for demonstrating operation | movement of the process means for periodic fluctuations concerning Embodiment 4. FIG. FIG. 10 is a diagram for explaining an operation of a periodic variation processing unit according to the fifth embodiment. It is a figure which shows the relationship between a light source and a captured image.

(Embodiment 1)
First, the configuration of the image processing apparatus according to the first embodiment will be described with reference to FIG. The image processing apparatus 1 shown in FIG. 1 acquires a moving image obtained by continuously capturing a shooting area in time series from the image capturing unit 2, performs image processing on a plurality of acquired images constituting the moving image, and performs image processing. Is an image processing apparatus that outputs the image as an output image to the image display means 3. The image processing apparatus 1 includes a moving object detection unit 10 and a face detection unit 11 for setting a target subject area, a pixel value correction unit 12 that corrects a pixel value of each pixel of an acquired image, and a temporary acquired image. The image storage means 13 to preserve | save, the determination means 14 which performs determination using the below-mentioned representative brightness | luminance and the brightness | luminance of an evaluation object pixel, and the process means 19 which performs an image process with respect to each acquisition pixel are provided.

  The image pickup means 2 includes an image pickup device (not shown) such as a CCD (Charged Coupled Device), for example, and is attached to a ceiling C of a building (ceiling of a condominium lobby) C as shown in FIG. Then, a predetermined imaging region including the subject Mn is continuously imaged in time series. A discharge lamp (not shown) such as a fluorescent lamp is attached as a lighting fixture to the ceiling C to which the imaging means 2 is attached.

  Each acquired image captured by the imaging unit 2 is an interlaced odd field image or even field image. Therefore, as shown in FIG. 15, the rate of each field image is about 60 fields / second (field interval 16.67 milliseconds). The fluctuation cycle of the discharge lamp is 10 milliseconds (= 1 / (50 Hz × 2)). Note that the exposure amounts of the first field (first odd field) and the fourth field (second even field) shown in FIG. 15 are the same, and the second field (first even field) and The exposure amount for the fifth field (third odd field) is the same.

  The image display means 3 is, for example, an interphone master set installed in a house, has a display screen 30 such as a liquid crystal panel, and sequentially displays output images from the image processing apparatus 1 on the display screen 30.

  The moving object detection means 10 shown in FIG. 1 creates a difference image between, for example, an acquired image and a stored image to be synthesized, extracts a change area, binarizes the change area, and sets the change areas of the proximity distances to one. A circumscribed rectangle to be combined into one is generated to be an integrated region, and for each integrated region, density pattern matching is performed between the acquired image and the saved image, and an integrated region having a low similarity is extracted as a moving object candidate (FIG. 6A). (Refer to FIG. 5), which will be described later as subject area information.

  For example, the face detection unit 11 compares the color histogram generated from the image data of the acquired image with the template histogram of the face part created in advance to determine the face area using a method of determining the face part area. This face area information is given to the image motion calculation means 163.

  The pixel value correction unit 12 calculates the average luminance of the region used for face detection from the pixel value of each pixel of the region used for face detection for each of the plurality of acquired images from the imaging unit 2. Calculated as representative luminance, and for each acquired image, corrects the pixel value of each pixel (red, green, and blue pixel values in the case of a color image) so that the representative luminance is included in the preset range. To do. Specifically, the pixel value correcting unit 12 converts the pixel value of each pixel according to how much the representative luminance is out of the above range, using a conversion table stored in advance in the storage unit 120. The acquired image output from the pixel value correction unit 12 is temporarily stored in the image storage unit 13. Note that the representative luminance may be the average luminance of the region where motion detection is performed, instead of the average luminance of the region used for face detection. When the representative luminance is included in the above range from the beginning, the acquired image is temporarily stored in the image storage unit 13 without being corrected by the pixel value correcting unit 12.

  The determination unit 14 includes a luminance change rate determination unit 140 that determines whether or not the luminance change rate indicating the representative luminance after correction with respect to the representative luminance before correction by the pixel value correction unit 12 deviates from the specified range, and the representative luminance Periodic variation determination means 141 that determines whether or not temporal variation is periodic, and temporal variation in luminance of one or more pixels (evaluation target pixels) included in the region used for face detection are irregular. And irregular fluctuation determining means 142 for determining whether or not.

  The irregular fluctuation determination unit 142 obtains data obtained by collecting the luminance values of the evaluation target pixels in the time axis direction, and when the standard deviation of the data exceeds a preset value, the temporal fluctuation of the luminance of the evaluation target pixels Is determined to be irregular. On the other hand, when the standard deviation of the data is less than or equal to the set value, the irregular variation determination unit 142 determines that the temporal variation in luminance of the evaluation target pixel is not irregular. Note that the evaluation target pixel may not be one or more pixels included in the region used for face detection, but may be one or more pixels included in the region where motion detection is performed.

  The processing unit 19 includes a periodic variation processing unit 15 that is executed when the temporal variation of the representative luminance is periodic, and an irregular variation that is performed when the temporal variation of the luminance of the evaluation target pixel is irregular. Processing means 16.

  The periodic variation processing unit 15 is executed when the periodic variation determination unit 141 determines that the temporal variation of the representative luminance is periodic. As shown in FIG. An image composition / thinning unit 150 that performs composition processing and thinning processing, and an image storage unit 151 that temporarily stores an acquired image and an output image are provided.

  Next, the operation of the periodic variation processing means 15 will be described. First, the period of the representative luminance is 50 milliseconds (see FIG. 15). Therefore, as shown in FIG. 4, three field images (acquired images) are included in one period of time variation of the representative luminance.

  The image composition / thinning means 150 sets the first odd field image among the three field images included in one period of the temporal variation of the representative luminance as a single image. The image composition / thinning unit 150 generates a single image for each period of time variation of the representative luminance. The periodic fluctuation processing means 15 that generates a single image for each period of time variation of the representative luminance sequentially outputs the single image as an output image to the image display means 3 (see FIG. 1) with the fluctuation period of the representative luminance. . On the other hand, for the remaining field images (second field image and third field image), the image composition / thinning process 150 performs a thinning process. That is, the image composition / thinning means 150 outputs the field image as an output image by skipping two, and outputs the odd field image and the even field image alternately to the image display means 3 at 20 fps. In this way, the luminance between output images can be made substantially equal, and flicker does not occur.

  When the luminance change rate determining unit 140 determines that the luminance change rate is out of the specified range, the irregular variation processing unit 16 has an irregular time variation in the luminance of the evaluation target pixel. As shown in FIG. 5, an image storage unit 160 that stores an output image after image synthesis, which is described later, and a storage image stored in the image storage unit 160 and the latest image are stored. Reference is made to an image composition unit 161 that performs image composition with the acquired image, an image motion conversion unit 162 that converts a motion vector in the acquired image from the posture information and condition information of the imaging unit 2, and a target subject area on the acquired image. And a motion vector measuring point 163 for setting motion vector measurement points and calculating a motion vector between the acquired image and the saved image. The attitude information of the imaging unit 2 is acquired from the attitude control unit 17 (see FIG. 1) that controls the attitude of the imaging unit 2, and the condition information of the imaging unit 2 is an imaging condition control unit that controls the imaging conditions of the imaging unit 2. 18 (see FIG. 1).

  The irregular fluctuation processing means 16 smoothes the acquired image with the acquired image, updates the smoothed image as a new saved image, and sets it as an output image. Details will be described below.

  The image motion calculation unit 163 sets a plurality of motion vector measurement points for the moving object (target subject region) detected by the moving object detection unit 10 (see FIG. 1). Here, when the subject Mn (see FIG. 2) is an article, even if the subject Mn moves within the frame of the captured image, there is no deformation, but when the subject Mn is a person, partial movement or deformation of the limbs or the like. In order to avoid these effects and measure the overall movement of a person, it is necessary to measure the movement of the face and torso. Therefore, the image motion calculation means 163 has the motion vector measurement points (FIG. 6A) so as to be dense in the upper half of the region set as a moving body (the circumscribed rectangular region indicated by the broken line in FIG. 6A). In FIG. 6B, the motion vector at each motion vector measurement point is obtained as indicated by “→” in the block matching method.

  Further, the image motion calculation means 163 calculates a representative value from these calculated motion vectors. The representative value is obtained by, for example, setting a motion vector equal to more than a majority of the obtained motion vectors as the representative value motion vector M1. In this embodiment, since a plurality of motion vectors are handled as described later, the representative value is not limited to a unique value, and a plurality of motion vector measurement values may be output to the image synthesis unit 161.

  The image motion calculating unit 163 applies shadows such as eyes, ears, and nose to the face area (inside the rectangle indicated by the broken line in FIG. 6C) detected by the face detecting unit 11 (see FIG. 1). Correspondingly, motion vector measurement points (indicated by □ in FIG. 6C) are set as illustrated, and the motion vectors at the respective motion vector measurement points are indicated by “→” in FIG. 6D by the block matching method. Measure as follows. In this case, since the mouth periphery is deformed when words are generated, it is necessary to set the motion vector measurement points around the mouth densely, but the mouth periphery is not important when determining the displacement of the entire face. It is not necessary to set motion vector measurement points around the mouth. At each motion vector measurement point, an accurate position can be obtained by recognizing facial parts such as eyes, ears, and nose by a method of extracting a facial region. Depending on the required set position accuracy, the relative position of each face part with respect to the face contour may be obtained in advance from the statistical information of the face shape and geometrically applied to the rectangle serving as the face contour.

  FIG. 6D shows a motion vector distribution M2 (x, y) (hereinafter referred to as “motion vector M2”) obtained at each motion vector measurement point set in the target subject region (face region). Assuming that the face is a rigid body with no deformation except for the deformed portion, the movement of the face between the acquired image and the saved image can be approximated by a primary transformation. However, as shown in the drawing, processing such as data mapping is required to cope with a state in which the movement is not uniform around the mouth.

  The image pickup means 2 is controlled by a posture control means 17 (see FIG. 1) by a drive means (not shown) such as a motor, whereby the horizontal (pan) and vertical (tilt) postures are controlled. However, the image motion conversion means 162 shown in FIG. 5 takes in the posture information of the imaging means 2 from the posture control means 17 and measures the relative shake angle θ at the time of acquisition of each image used for image composition. Of course, when the posture control of the imaging means 2 is not performed, zero displacement is adopted as the posture information. The image motion conversion unit 162 converts the shake angle θ obtained as described above into a motion vector using condition information including the focal length l of the optical system of the imaging unit 2 obtained from the imaging condition control unit 18 (see FIG. 1). Convert. The amount of movement Δx of the image on the imaging plane α due to the difference in shooting time between the acquired image and the stored image using the shake angle θ and the focal length l is expressed as Δx = [(l2 + x2) / l] tan θ. The motion vector distribution M3 (x, y) (hereinafter referred to as “motion vector M3”) is set by adding the distortion information from the posture control means 17 as necessary.

  The image synthesizing unit 161 obtains a weighted average of m: n pixel values of the same coordinates on the acquired image and the saved image, outputs an image based on the averaged pixel value as a synthesized image, and outputs the synthesized image The image is stored in the image storage unit 160 and used as a new saved image when processed with a new image acquired from the imaging unit 2. This series of processing can be shown as an infinite length impulse response IIR filter shown in FIG. 7, and this filter acts as a smoothing filter to reduce noise.

  By the way, the addition ratio of the individual images acquired in time series in the latest synthesized image is as shown in FIG. 8, and the moving information is blurred because the past information remains as if it is tailed. Note that the bar at the left end in FIG. 8 shows the newly acquired image, and shows the bar of the image acquired in the form of going back in the past from the right to the right, and the bar length of this image is the latest composite. An addition ratio in an image, that is, a stored image is shown.

  Therefore, the image synthesizing unit 161 shown in FIG. 5 performs motion correction using the motion vector M1, the motion vector M2 from the image motion calculating unit 163, and the motion vector M3 from the image motion converting unit 162 as motion vector candidates. .

  In the stored image and the acquired image, the subject Mn captured over time has a movement. Here, as shown in FIG. 9 (d), a processing unit consisting of one or more pixels (□ shown in FIG. 9 (d)) is set on the acquired image, the subject Mn is a person, and the torso of the moving person In the processing unit corresponding to, since the similarity with the acquired image is high in the saved image in which the motion is corrected by the motion vector M1 obtained by moving object detection from the acquired image (FIG. 9D), the motion vector M1 is applied. (FIG. 9 (c)). On the other hand, in the processing unit corresponding to the face of a moving person, the saved image that has been motion-corrected by the motion vector M2 obtained by face detection has a high similarity to the acquired image, so the motion vector M2 is applied (FIG. 9 (b)). Furthermore, in the processing unit corresponding to the background, the saved image that has been motion-corrected by the motion vector M3 obtained by the displacement of the imaging means 2 has a high similarity to the acquired image, so motion correction by the motion vector M3 is applied. The

  Subsequently, the image synthesizing unit 161 performs a process of evaluating the similarity between the corrected image after motion correction and the acquired image as described above before synthesizing the image.

  In this case, the absolute value of the difference between the pixel values between the corrected image and the acquired image is integrated within the processing unit or in the vicinity of the processing unit to obtain a block matching error for evaluation, or a spatial difference image Similarly, a block matching error is obtained and evaluated. In this case, in the block matching error, the larger the value, the lower the similarity.

  Through the process of evaluating the similarity in this way, the image synthesis is performed by determining the addition ratio of the pixel values of the stored image and the acquired image at the time of synthesis based on the evaluation result. Here, the ratio of the added value is determined by, for example, using an evaluation value of similarity to a stored image that has been subjected to motion correction using the motion vector M1 and an evaluation value of similarity to a stored image that has been subjected to motion correction using the motion vector M2. In comparison, when the similarity on the motion vector M1 side is high, the acquired image and the saved image corrected by the motion vector M1 are weighted average at a ratio of 1: 3. When the similarity on the side of the motion vector M2 is high, the weighted average is taken at a ratio of 1: 3 between the acquired image and the stored image corrected for motion with the motion vector M2. If any evaluation value does not exceed a preset threshold value, the pixel value of the stored image is not used and the pixel value of the acquired image is applied as it is.

  Here, each unit configuring the image processing apparatus 1 may be configured by a function realized by executing a program in a microcomputer, or may be configured independently by a hardware configuration. . In addition, the image processing apparatus 1 is separate from the imaging unit 2 as shown in FIG. 2, but may be configured integrally with the imaging unit 2. Furthermore, the image display unit 3 may be integrated into the imaging device.

  Next, the operation of the image processing apparatus 1 according to the present embodiment will be described with reference to FIG. First, the image processing apparatus 1 acquires a new image from the imaging unit 2 (S1 in FIG. 10). The moving object detection means 10 detects a moving object from the acquired image, and the face detection means 11 detects a face portion from the acquired image (S2). For each acquired image, the area used for face detection is set as the luminance evaluation area (S3), and the average luminance of the luminance evaluation area is measured as the representative luminance (S4).

  Subsequently, the periodic variation determination unit 141 analyzes the temporal variation of the representative luminance (S5), and determines whether the temporal variation of the representative luminance is periodic (S6). If the temporal variation of the representative luminance is periodic, the periodic variation processing means 15 operates (intra-variation period synthesis processing on) (S7). That is, the periodic variation processing unit 15 sets the first odd field image among the three field images included in one period of the temporal variation of the representative luminance as a single image (output image), and the remaining field images. (2nd field image, 3rd field image) is thinned out.

  Thereafter, for each acquired image, the pixel value correction unit 12 corrects the pixel value of each pixel using the conversion table stored in the storage unit 120 so that the representative luminance is included in a preset range. (S8).

  Thereafter, the luminance change rate determination means 140 determines whether or not the luminance change rate deviates from the specified range (whether or not the luminance change amount is a predetermined value or less) (S9). When the luminance change rate deviates from the specified range (when the luminance change amount is a predetermined value or less), the luminance of the acquired image pixel is measured (S10), and the pixel luminance statistical processing is performed (S11).

  Thereafter, the irregular variation determination unit 142 determines whether or not the temporal variation in luminance of the evaluation target pixel is irregular (S12). That is, it is determined whether or not the standard deviation of the luminance value is equal to or less than a predetermined value. When the standard deviation of the luminance value is larger than the predetermined value, the irregular variation processing means 16 operates (S13). That is, the irregular fluctuation processing means 16 smoothes the acquired image with the acquired image (the previous output image integration process is on).

  As described above, according to the present embodiment, when the time variation of the luminance of the evaluation target pixel is irregular, the random noise can be reduced by smoothing the acquired image in the time axis direction, and the time of the representative luminance can be reduced. When the variation is periodic, flicker can be reduced by generating a single image with the same luminance for each period of time variation of the representative luminance, and as a result, between output images for a plurality of output images. The luminance fluctuation can be reduced.

  Further, according to the present embodiment, the output image can be displayed at the same speed as the acquired image by sequentially outputting the output image in accordance with the time variation period of the representative luminance.

  Furthermore, according to the present embodiment, each process can be performed using the luminance of the important part in the acquired image.

(Embodiment 2)
As shown in FIG. 11, the image processing apparatus 1 according to the second embodiment is different from the first embodiment in that the periodic variation processing unit 15 generates an output image by combining two field images (acquired images). This is different from the image processing apparatus 1. Note that the description of the same components as those in the first embodiment is omitted.

  The image synthesizing / decimating unit 150 (see FIG. 3) of the periodic variation processing unit 15 according to the present embodiment performs two consecutive field images (first field image, 2) for each period of time variation of the representative luminance. The second field image) is integrated into an output image. Specifically, the image composition / thinning unit 150 adds a pixel value between the second field image and the first field image stored in the image storage unit 151 to perform a bit shift operation (1/2 times). I do. Thereafter, the image composition / thinning means 150 outputs the composite image as an output image to the image display means 3 at 20 fps. On the other hand, for the third field image, the image composition / thinning means 150 performs a thinning process. The periodic variation processing unit 15 of the present embodiment is the same as the periodic variation processing unit 15 of the first embodiment except for the points described above.

  As described above, according to the present embodiment, the luminance between the output images can be made substantially equal as in the first embodiment, and further, the vertical movement of the output image can be prevented from occurring.

  Further, since the rate of each field image is precisely 59.94 fields / second, it is actually 10 seconds (= 0.05 seconds × 0.01 seconds / {(3 / 59.94 fields / second). )-(3/60)}) in some cases, the luminance fluctuation may occur. However, according to the present embodiment, the luminance fluctuation in the 10-second period can also be reduced.

(Embodiment 3)
As shown in FIG. 12, the image processing apparatus 1 according to the third embodiment is different from the first embodiment in that the periodic variation processing unit 15 generates an output image by combining three field images (acquired images). This is different from the image processing apparatus 1. Note that the description of the same components as those in the first embodiment is omitted.

  The image synthesizing / decimating unit 150 (see FIG. 3) of the periodic variation processing unit 15 according to the present embodiment performs three consecutive field images (all fields in one cycle) for each period of time variation of the representative luminance. Image) is integrated into an output image. Specifically, the image composition / thinning means 150 adds the pixel value and multiplies the constant by the third field image and the first field image and the second field image stored in the image storage means 151. Perform bit shift operations. At this time, the first field image (odd field image in the illustrated example): the second field image (even field image in the illustrated example): the weight of the third field image (odd field image in the illustrated example) is 1: 2. : 1. That is, the sum of the weights of the odd-numbered field images is made equal to the sum of the weights of the even-numbered field images. Thereafter, the image composition / thinning means 150 outputs the composite image as an output image to the image display means 3 at 20 fps. The periodic variation processing unit 15 of the present embodiment is the same as the periodic variation processing unit 15 of the first embodiment except for the points described above.

  As described above, according to the present embodiment, the luminance between output images is substantially equal, and flicker does not occur. Further, when the interlace output image is sequentially displayed on the image display means 3, it is possible to reduce the vertical movement of the output image, that is, the shaking of the display image.

  In the present embodiment, the weighting of the first field image: the second field image: the third field image is 1: 2: 1. However, as a modification of the present embodiment, the first field image: The weight of the second field image: third field image may be 1: 1: 1. That is, all the field images may have the same weight. In such a case, it is possible to maximize the effect of suppressing the luminance fluctuation in the 10-second cycle.

(Embodiment 4)
As shown in FIG. 13, the image processing apparatus 1 according to the fourth embodiment is an image according to the first embodiment in that the periodic variation processing unit 15 synthesizes odd field images and synthesizes even field images. It is different from the processing apparatus 1. Note that the description of the same components as those in the first embodiment is omitted.

  The image composition / thinning unit 150 (see FIG. 3) of the periodic variation processing unit 15 according to the present embodiment sets a processing period that is twice the time variation period of the representative luminance. The first odd field image and the second odd field image (third field image) are combined with a weight of 1: 1 (the pixel value multiplied by 1/2: 1/2) to generate an output image. On the other hand, the second even field image (fourth field image) and the third even field image (sixth field image) are 1: 1 (1/2: 1/2 as a value to be multiplied by the pixel value). ) To generate an output image. For the first even field image (second field image) and the third odd field image (fifth field image), the image composition / thinning means 150 performs a thinning process. In the present embodiment, as shown in FIG. 13, the processing sections of the odd-numbered field images and the processing sections of the even-numbered field images do not overlap and are alternated. That is, in this embodiment, the center (centroid) of the processing section of the odd field image is separated from the center (centroid) of the processing section of the even field image. The output image by the odd field image and the output image by the even field image are alternately output to the image display means 3.

  According to the present embodiment, in the interlace method, it is possible to keep the resolution while maintaining the independence of each field.

  Further, according to the present embodiment, since the processing interval of the odd field image and the processing interval of the even field image are separated, the processing timing can be shifted between the odd field image and the even field image. When a set of frame images composed of an output image obtained by synthesizing an image and an output image obtained by synthesizing two even field images is displayed on the image display means 3 as a moving image, the movement of the subject Mn can be made smooth.

(Embodiment 5)
As illustrated in FIG. 14, the image processing apparatus 1 according to the fifth embodiment is different from the fourth embodiment in that a processing section for combining odd field images and a processing section for combining even field images overlap. This is different from the image processing apparatus 1. Note that the description of the same components as those in the fourth embodiment is omitted.

  The image composition / thinning unit 150 (see FIG. 3) of the periodic variation processing unit 15 according to the present embodiment sets a processing period that is twice the time variation period of the representative luminance. The 1: 1 odd field image, the second odd field image (third field image), and the third odd field image (fifth field image) are 1: 1: 2 (the value multiplied by the pixel value is 1 / 4: 1/4: 1/2) is combined to generate an output image, while a first even field image (second field image) and a second even field image (fourth field image). And the third even field image (sixth field image) are combined at a weighting ratio of 2: 1: 1 (the pixel value multiplied by 1/2: 1/4: 1/4) to produce an output image. Generate. In the present embodiment, as shown in FIG. 14, the processing interval of the odd field image and the processing interval of the even field image overlap. That is, in this embodiment, the center (centroid) of the processing section of the odd field image and the center (centroid) of the processing section of the even field image are substantially the same. The output image by the odd field image and the output image by the even field image are alternately output to the image display means 3.

  According to the present embodiment, since the processing interval of the odd field image and the processing interval of the even field image overlap, the processing timing can be made closer to the odd field image and the even field image. When one frame image consisting of the synthesized output image and the output image obtained by synthesizing the three even field images is displayed on the image display means 3 as a still image, blurring as a still image can be reduced.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Moving body detection means 11 Face detection means 12 Pixel value correction means 140 Luminance change rate determination means 141 Periodic fluctuation determination means 142 Irregular fluctuation determination means 15 Periodic fluctuation processing means 16 Irregular fluctuation processing means 2 Imaging means 3 Image display means

Claims (4)

An image processing method for acquiring a plurality of images obtained by continuously capturing a shooting region in time series, performing image processing on the plurality of acquired images, and outputting the image processed image to an image display means as an output image. And
The luminance value calculated from the pixel values of a plurality of predetermined pixels for each acquired image for the plurality of acquired images is used as a representative luminance, and the representative luminance of each pixel is included in a preset range for each acquired image. The pixel value is corrected, it is determined whether the luminance change rate indicating the corrected representative luminance with respect to the representative luminance before correction deviates from the specified range, and the luminance time of one or more pixels included in the acquired image It is determined whether or not the variation is irregular, and it is determined that the luminance change rate deviates from the specified range or that the temporal variation in luminance of one or more pixels included in the acquired image is irregular In this case, for each latest acquired image, the latest acquired image is smoothed with the acquired image that has already been acquired, and a plurality of images that are sequentially smoothed for each of the latest acquired images are used as the new plurality of acquired images. And the step of,
When determining whether the temporal variation of the representative luminance is periodic before or after the first step, and determining that the temporal variation of the representative luminance is periodic, A predetermined period that is an integral multiple of the time variation period of the representative luminance is set as a processing period, and is determined at a timing synchronized with the period of the processing period among a plurality of acquired images included in the processing period in all the processing periods. A second step of extracting one or a plurality of acquired images and generating an image and using the plurality of images generated in all processing periods as the new plurality of acquired images,
An image processing method, comprising: outputting the plurality of acquired images after performing the first step and the second step to the image display unit as the output images.   2. The image processing method according to claim 1, wherein when the temporal variation of the representative luminance is determined to be periodic, the output image is sequentially output to the image display means at the variation cycle of the representative luminance.   The image processing according to claim 1, wherein the representative luminance is an average luminance calculated from pixel values of a plurality of pixels included in a region where face detection or motion detection is performed in the acquired image. Method. An image processing apparatus that acquires a plurality of images in which a shooting region is continuously captured in time series, performs image processing on the plurality of acquired images, and outputs the processed image as an output image to an image display unit. And
For each acquired image, the luminance value calculated from the pixel values of a plurality of predetermined pixels is used as the representative luminance for each of the plurality of acquired images, and the representative luminance is included in a preset range for each acquired image. Pixel value correcting means for correcting the pixel value of
A luminance change rate determination unit that determines whether or not a luminance change rate indicating the representative luminance after correction with respect to the representative luminance before correction by the pixel value correction unit deviates from a specified range;
Irregular fluctuation determining means for determining whether or not the temporal fluctuation of luminance of one or more pixels included in the acquired image is irregular;
When the luminance change rate is determined to depart from the specified range by the luminance change rate determination unit, or the time variation of luminance of one or more pixels included in the acquired image is irregular by the irregular variation determination unit. A plurality of images obtained by smoothing the latest acquired image for each latest acquired image with the acquired image already acquired and sequentially smoothing for each of the latest acquired images. Processing means for irregular fluctuations, and
Periodic variation determining means for determining whether or not the temporal variation of the representative luminance is periodic;
When the time variation of the representative luminance is determined to be periodic by the periodic variation determining means, a predetermined period that is an integral multiple of the time variation period of the representative luminance is set as a processing period, and in all the processing periods A plurality of acquired images included in the processing period are extracted at a timing synchronized with the period of the processing period to extract a predetermined one or a plurality of acquired images, and the plurality of images generated in all the processing periods An image processing apparatus comprising: periodic change processing means that sets a plurality of acquired images as new images.

Images