JP2020021983A - Information processing apparatus and program - Google Patents

Abstract

To provide an information processing apparatus capable of suppressing a flicker fast with simple constitution.SOLUTION: The present invention relates to an information processing apparatus comprising: an image pickup part 1 which picks up an image at a predetermined image pickup frequency and obtains picked-up images of respective points of time as video, and a processing part 3 which obtain, as a processed image formed by processing the picked-up images of the respective points of time, an image by averaging a predetermined number of picked-up images of corresponding time or past time at predetermined intervals from the corresponding time. The predetermined intervals correspond to the predetermined image pickup frequency and a frequency of the quantity of light of a light source in the environment in which the images are picked up, and minimizes variation in average value calculated by selecting the predetermined number of points based upon the predetermined image pickup frequency as a unit interval from on a predetermined light quantity variation function of the frequency of the quantity of light in the environment in which the images are picked up.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus and a program capable of suppressing flicker at high speed with a simple configuration.

  There is a problem that unintended flicker (flicker) occurs in a captured image (video). In other words, when the update timing of the imaging system is high or the exposure time is short, there is a problem that the captured image flickers when an image is captured under a light source whose light amount varies depending on the frequency of the power supply.

  For example, Patent Documents 2 to 4 disclose methods for reducing the flicker of a captured image. In Patent Document 2, a captured image is corrected by separating a light source component from the captured image. Patent Literature 3 uses the fact that the update timing of the imaging system and the frequency of the power supply cycle at a constant interval, and uses the exposure time after the next period so that the luminance level integrated for each update timing of the imaging system becomes a constant value. To adjust. Patent Literature 4 utilizes the fact that the update timing of the imaging system and the frequency of the power supply cycle at regular intervals, as in Patent Literature 3, so that the luminance level integrated for each update timing of the imaging system becomes a constant value. A correction coefficient is applied to captured images in the next cycle and thereafter.

Japanese Patent Application No. 2018-019335 JP 2018-032971 JP JP 2010-200133 A JP 2013-187603 A

  However, the above-described conventional techniques have the following problems with respect to suppressing flicker in a captured image at high speed with a simple configuration.

  Patent Document 2 uses a fast Fourier transform to separate a light source component, and therefore has a problem that a terminal with scarce computational resources takes a long processing time. In particular, there is a problem that as the update timing of the display system is higher, the processing cannot be completed in time. In Patent Literature 3, the above problem is solved because the exposure time is changed according to the integrated value. However, there is a problem that a mechanism capable of changing the exposure time in a unit of several milliseconds at every update timing of the imaging system is required, and there is a problem that the apparatus scale cannot be reduced.

  In Patent Document 4, the above problem is solved because a correction coefficient is applied to a captured image. However, Patent Literature 4 (and Patent Literature 3) also has the following four problems depending on the premise thereof.

  The first problem is that the time interval for calculating the integrated value is derived from the least common multiple of the update timing of the imaging system and the frequency of the light amount (twice the frequency of the power supply). There is a problem that the amount of memory for storing the integrated value becomes enormous.

  The second problem is that the exposure time is the update timing interval of the imaging system, but since the exposure time is actually shorter than the update timing of the imaging system, the time interval at which the integrated value should be calculated at the least common multiple is accurately derived. There is a problem that it cannot be done. For example, if the frequency of the light amount is 100 Hz and the update timing of the imaging system is 1/200 second, the least common multiple is 200 and the integration interval is 1/200 second. You can only get the kind. That is, in Patent Document 3, the exposure time is not adjusted, and in Patent Document 4, the correction coefficient is 1, and there is no problem with the captured image as it is. However, when the process is actually executed for the above-mentioned reason, flicker occurs in many terminals. .

  As a third problem, it is assumed that the update timing of the imaging system and the frequency of the power supply are completely accurate, but there is a problem that the assumption often does not hold in practice. For example, the frequency of a commercial power supply in eastern Japan (TEPCO jurisdiction) is 50 Hz, but an error of ± 0.2 Hz (otherwise, for example, ± 0.3 Hz in Hokkaido electric power jurisdiction) can occur, and within a certain accuracy , The power supply frequency is managed, and the premise may not hold. Further, depending on the terminal, a processing drop or a frame drop due to an overload may occur, and not only the update timing may change but also a timing at which the captured image itself does not exist. In particular, there is a problem that as the least common multiple mentioned in the first problem is large and the image reflecting the integrated value is further apart in time, the premise is more likely to be broken and both methods are broken.

  As a fourth problem, the integrated value is based on the assumption that only the fluctuation of the light source is reflected. However, when the imaging unit or the subject moves greatly as in Patent Document 1, a change in brightness other than the light source is added to the integrated value. Since the reflection is reflected, there is a problem that the light source may not be reduced, but may be amplified.

  In view of the above-described problems of the related art, an object of the present invention is to provide an information processing apparatus and a program that can suppress flicker at high speed with a simple configuration.

  In order to achieve the above object, the present invention provides an imaging unit that performs imaging at a predetermined imaging frequency and obtains a captured image at each time as a video, and a processed image obtained by processing the captured image at each time at a predetermined interval from the time. A processing unit that obtains an average of a plurality of images captured at a predetermined time or the past time in the predetermined interval, wherein the predetermined interval is the predetermined imaging frequency and the imaging is performed. And the frequency of the light amount of the light source in an environment. Further, it is a program for causing a computer to function as the information processing device.

  According to the present invention, it is possible to suppress flicker in a processed image obtained by a simple and high-speed calculation process of averaging.

It is a functional block diagram of an information processor concerning one embodiment. It is a figure which shows the schematic example of generation | occurrence | production of a flicker. It is a plot of a discrete value distribution in each combination at a power supply frequency of 50 Hz and an imaging frequency of 240 Hz. 9 is a plot of a discrete value distribution in each combination at a power supply frequency of 50 Hz and an imaging frequency of 220 Hz. It is a figure which shows the effect of the flicker reduction by this invention typically with a comparative example.

  FIG. 1 is a functional block diagram of an information processing apparatus according to one embodiment. The information processing device 10 includes an imaging unit 1, a selection unit 2, a processing unit 3, and a display unit 4. The processing contents of each unit are as follows.

  The imaging unit 1 outputs a captured image I (t (i)) obtained by imaging at time t (i) to the selection unit 2. Here, the imaging unit 1 performs imaging as a video at a predetermined imaging frame rate (imaging frequency or an imaging cycle as a reciprocal thereof), and captures an image I ((i) at each time t (i) specified by the integer index i. t (i)). As the hardware for realizing the imaging unit 1, a digital camera or the like which is often provided as a standard equipment in recent mobile terminals can be used.

  The selection unit 2 performs a selection process in accordance with the update timing of imaging in the imaging unit 1 and the frequency of the power source from the time-series captured images I (t (i)) captured by the imaging unit 1, The selection is performed at each display timing by the display unit 4 on the side, and the selection result is output to the processing unit 3. Here, the frequency of the power source for the selection process is the frequency of the power source of the light source in the environment in which the imaging unit 1 is performing imaging (for example, the fluorescence as a light source in the room when performing imaging in a room). Frequency of a power supply such as a lamp or LED). Details of the processing of the selection unit 2 will be described later.

  The processing unit 3 obtains a processed image as an average of the plurality of captured images I (t (i)) selected by the selection unit 2, and outputs the processed image to the display unit 4. The display unit 4 displays an image at a predetermined frame rate by displaying the obtained processed image at each predetermined display timing.

  Here, the display frame rate (display frequency) by the display unit 4 can be set as the same as the imaging frame rate by the imaging unit 1 according to the embodiment. It may be set as a value (low frame rate) smaller than the rate (high frame rate). In any of the embodiments, the present invention can reduce the flicker that is assumed to occur when the present invention is not applied. That is, in a configuration in which the selection unit 2 and the processing unit 3 are omitted from the information processing apparatus 10 without applying the present invention, the captured images I (t (i)) at the respective times obtained by the imaging unit 1 are left as they are. At a high frame rate, or at a low frame rate thinned out at regular intervals, a flicker may occur when displayed on the display unit 4, whereas the flicker is reduced by applying the present invention. It becomes possible to reduce.

  In the present invention, in particular, the processing unit 3 averages the plurality of captured images I (t (i)) (that is, averages the pixel values at the same pixel position (u, v) in the plurality of captured images) ), A processed image with reduced flicker can be obtained. Therefore, even when the computational resources and the like that can be used in the information processing apparatus 10 are limited, it is possible to perform the flicker reduction processing at high speed.

  Hereinafter, the details of the selection processing by the selection unit 2 will be described together with the significance of the processed image obtained as a result of the processing performed by the processing unit 3 on the captured image selected by the selection unit 2 and having reduced flicker.

First, the cause of the flicker will be described. Assuming that the frequency of the power supply is f _p , the amplitude is V, and the time is t, the voltage of the AC power supply can be represented as Vsin (2πf _p t). Light source that varies the amount of light by the frequency f _p of the time power supply voltage absolute value | sin (2πf _p t) | amount in proportion to the changes. (Thus, since the variation in absolute value of the sine function, the frequency of the light intensity variation becomes 2f _p is a multiple of the power frequency f _p.) Imaging unit 1 is subjected to image pickup under a light source for the variation and things, when the frequency to be imaged by the imaging unit 1 and the f _c, when the exposure time is short, so that the flicker and the power frequency f _p and the image frequency f _c do not match occurs.

FIG. 2 is a diagram illustrating a schematic example in which the flicker occurs. In Figure 2, the horizontal axis represents time, the vertical axis as a voltage (and the light quantity proportional thereto), the voltage absolute value by a solid line | sin (2πf _p t) | with showing a continuous graph of a series on the graph the discrete "●" (circles), shows a discrete graph of light intensity fluctuations (i.e. luminance value variation) in the captured image on the time-series captured by the image capturing frequency f _c. Note that the luminance value fluctuation is a fluctuation as an average value over the entire pixel range in one captured image, and the same applies to the following description.

Specific numerical examples in FIG. 2, the imaging frequency f _c = 240 Hz as the update timing of the imaging (and thus, as imaging cycle = 1/240 seconds shown) and, also, if a power supply frequency f _p = 50 Hz Is shown. Since the brightness of the light source is changed by the update timing of the imaging, if the captured image captured immediately before the timing of displaying as an image (1/30 second in FIG. 2) on the display unit 4 is displayed as it is, as shown in FIG. The nearest vertical line (the vertical line at 0, 1/30, 2/30, and 3/30 seconds indicated at every 1/30 second of the display timing) Is displayed, and the brightness of the image fluctuates greatly, and the image appears to flicker when reproduced as an image. Specifically, the brightness of the light source is minimum at t = 0 seconds and t = 3/30 seconds, but is maximum at t = 1/30 seconds and t = 2/30 seconds. Because these are close, if these are sequentially displayed as video on the display unit 4, the captured image as the displayed video will blink.

In the above example, when a display is performed on the display unit 4 at a display frequency (referred to as f _v ) f _v = 30 Hz at a low frame rate with respect to the imaging frequency f _c = 240 Hz as a high frame rate, flickering, that is, flicker However, as can be seen from the graph of FIG. 2, the display unit 4 performs display at a high frame rate setting that matches the display frequency f _v with the imaging frequency f _c = 240 Hz. As can be seen as a change in the value of ●, blinking occurs.

As schematically example of FIG. 2, power supply frequency f varying voltage at the _{p | sin (2πf p t)} | ( i.e. imaged) are sampled according to the imaging frequency f _c from above captured image generally involves blinking It will be. Therefore, from the captured image to be sampled in accordance with the imaging frequency f _c the selecting unit 2, so that the flicker is reduced by averaging by selecting a predetermined plurality, and selects.

  As an embodiment that can be generally applied, a case will be described in which the selection unit 2 selects a plurality of predetermined n captured images as targets for averaging by the processing unit 3 in the subsequent stage. In this case, the brightness of the average image of n images selected at each time t is represented by the following equation (1) under the influence of the brightness by the fluctuating light source. (That is, when the n images are selected, the processing unit 3 obtains a processed image having the luminance as in the following equation (1).)

In the above equation (1), h _j (0 ≦ j <n) is a non-negative integer, and each of the n captured images I (th _j / f _c ) selected as being selectable at the current time t is It is for representing. (I.e., if the current time t is due to the imaging unit 1 k-th imaging time t (k), the image I (th _j / f _c) is .h _j representable with image I (t (kh _j)) This means that only h _j previous images are selected from the current time image captured by the imaging unit 1. If h _j = 0, the image is at the current time.) The above equation (1) As a continuous function of time t, f (t, {h}) (where {h} represents a set of selected h _j (0 ≦ j <n), ie, {h} = {h _j | 0 ≦ j <n}, and the same applies to the following.), The correspondence between the imaging start timing and the brightness cycle of the light source is unknown, and the power supply frequency is within a certain error range. Considering that there is a possibility that f (t, {h}) as a continuous function of time t keeps a constant value as much as possible and exhibits such a behavior that f (t, {h}) does not fluctuate (time t In the selection unit 2 It is desirable to be-option.

Here, in practice, it is not always necessary to evaluate the fluctuation behavior of the continuous function f (t, {h}) at the time t for each selection {h}. In other words, in practice, it is sufficient if the variation is constant (the variation is as small as possible and constant) only at the update timing of the imaging, so that the variation behavior may be evaluated discretely. Specifically, in the “update timing of a plurality of periodic imagings” by the discrete imaging unit 1, “average of the brightness of the light source” (the average is a processed image of the processed image obtained by the processing unit 3). The combination {h} of h _j that minimizes the “dispersion” of the luminance (which coincides with the luminance) may be determined by the selection unit 2 and used as the selection result. Expressed as an expression, a combination {h _j } as a selection result that minimizes “variance” (where the argument of argmin is used as an object to find the minimum case) as in the following expression (2) Should be determined. Incidentally, L is the least common multiple of twice the frequency 2f _p of the imaging frequency f _c and the power supply frequency f _p, the range of "update timing of the periodic plurality of imaging" for evaluating the "dispersed" It is to determine. (As described above, the voltage change of the absolute value | a frequency of the double frequency _{2f p | sin (2πf p t} ).)

In the case both frequencies f _c, 2f _p is not an integer (if accompanied by decimal) is rounded to integer round english (us) (f _c), may be obtained with L as the least common multiple of the round english (us) (2f _p). (Here, round () is a function of the rounding process.) In Equations (2) and (3), 0 ≦ k <L / 2f _p −1 is a range that is a unit in which periodicity appears in order to evaluate variance. Is used, but the variance may be evaluated in a narrower range.

As described above, as a generally applicable embodiment, the selection unit 2 can determine the combination {h} so as to minimize the variance of the above equation (2). That is, the selection unit 2 can determine the fixed combination {h} in the following (procedure 1) to (procedure 3).
(Step 1) and the imaging frequency f _c as the known information about the camera imaging section 1, the image pickup unit 1 is obtained in advance and the power supply frequency f _p as known information, previously in the environment to be imaged.
(Procedure 2) An administrator or the like sets the number n of images to be selected by the selection unit 2 and averaged by the processing unit 3 as a predetermined one.
(Step 3) above f _c, as a predetermined combination determined according to f _p, n, to obtain a combination {h} that minimizes the variance of the formula (2) in the selection section 2.

Note that each h _j (0 ≦ j <n) as a non-negative integer constituting the combination {h} in (Procedure 3) may be obtained within a predetermined range. For example, the above-described equations (2) and (3) ) May be obtained in a range based on the least common multiple L for performing the variance evaluation. That is, each h _j may be obtained within the range of 0 ≦ h _j ≦ L / 2f _p −1. When the range based on L becomes large, from the viewpoint of delay reduction described below and from the viewpoint of preventing the search space from becoming enormous, each _hj is determined within a predetermined smaller range. You may.

Also, when there are a plurality of combinations {h} that minimize the variance in (Procedure 3), or when there are a plurality of combinations {h} that are variance values within the threshold range from the minimum value of the variance (that is, from the minimum value of the variance If those in closely there is more than one), from the viewpoint of delay reduction to be described later, it may be adopted a smaller value of the h _j constituting a {h}. For example, it may be adopted in which the sum of the values of each h _j is small, it may be adopted as the maximum value is smaller of the values of each h _j. In (Procedure 3), one of the combinations {h} is always set to 0 (for example, h ₀ = 0), that is, the captured image at the current time is always included in the combination {h}. May be imposed.

  Also, in Equations (2) and (3), the magnitude of the variation of the discrete value is evaluated by the variance, but instead of the variance, the magnitude of the variation is evaluated by any other statistical index. Is also good. For example, the evaluation may be performed based on a range (range) as a difference between the maximum value and the minimum value.

  In the following, matters that are desirably specifically considered when applying the general embodiment according to (procedure 1) to (procedure 3) to an actual situation, specific application examples, and the like will be described.

From the viewpoint of reducing the processing load, it is desirable to set the number n to be small.From the viewpoint of reducing the delay of the captured image displayed as an image on the display unit 4 and reducing the amount of memory for holding frames, the combination (h }, It is desirable to reduce each h _j as a non-negative integer. In one embodiment, the number n may be set as n = 2. As an embodiment that can be combined with this, the above-described constraint that 0 is always included as one of h _j (the current time t (or if the display timing and the imaging timing are different, The combination {h} of (Procedure 3) may be determined under the constraint that the image at the latest time is always selected.

Numerical examples 1 and 2 for obtaining a combination {h} under the constraint that n = 2 and that 0 is always included as one of h _j are introduced below. Here, since n = 2, {h} = {h ₀ , h ₁ }, and h ₀ = 0 may be assumed without loss of generality. Therefore, the numerical examples are given f _c, with respect to f _p, becomes to determine the h ₁ undetermined of {h} = {0, h 1}.

<Numerical example 1>
Power supply frequency f _p = 50 Hz, giving as an imaging frequency f _c = 240 Hz. In this case, the least common multiple L = 1200 Hz, the above-mentioned range 0 ≦ h ₁ ≦ L / 2f _p −1 becomes the range 0 ≦ h ₁ ≦ 11, and h ₀ = 0 has already been selected because it has already been selected. Satisfies the range 1 ≦ h ₁ ≦ 11. Within this range, h ₁ = 6 minimizes the variance of equation (2).

FIG. 3 is a plot of the distribution of discrete values for which the variance is evaluated by equation (2) for each of h ₁ = 1 to 11 separately from [1] to [11]. In each of [1] to [11] in FIG. 3, as in the example of FIG. 2, the voltage absolute value by a solid line | sin (2πf _p t) | with showing a continuous graph of a series on the graph A discrete graph of light amount fluctuation (ie, luminance value fluctuation) in a time-series captured image captured at an imaging frequency f _c = 240 Hz is drawn by a discrete “●” (black circle). Represents a discrete value distribution (distribution of the time variation of the average value by the equation (1)) in each case of h ₁ = 1 to 11. In FIG. 3, for reference, the distribution in the case where Equation (1) is a continuous function of time is also indicated by a broken line, and the distribution of discrete values due to “◇” substantially matches the distribution of continuous values due to the broken line. (That the continuous value distribution does not significantly deviate at locations other than the discrete values) can be seen.

In FIG. 3, as shown in [6], h ₁ = 6, that is, {h} = {0,6}, and the past at which the image at the current time t (or its latest time) is six times apart from the image at the current time t It can be seen that the average of the image and the average minimizes the variance of Expression (2), assuming that the fluctuation width of “Δ” in the vertical direction is the smallest.

<Numerical example 2>
Power supply frequency f _p = 50 Hz, as the imaging frequency f _c = 220 Hz, giving change only the imaging frequency f _c from the case of the numerical example 1. In this case, the least common multiple L = 1100 Hz, the above-mentioned range 0 ≦ h ₁ ≦ L / 2f _p −1 becomes the range 0 ≦ h ₁ ≦ 10, and h ₀ = 0 has already been selected, so it is excluded. Satisfies the range 1 ≦ h ₁ ≦ 10. Within this range, h ₁ = 1 or h ₁ = 10, that is, {h} = {0,1} or {0,10}, the image at the current time t (or its latest time), and the imaging timing 1 When averaging the previous image (the image at the immediately preceding time) for 10 or 10 times, the variance of Expression (2) is minimized. (Here, even if there is a difference in the variance between {h} = {0,1} and {0,10}, they can be determined to be within the error range, and both give the same minimum value. It is possible to adopt {h} = {0, 1}, which has a smaller value from the viewpoint of delay reduction or the like.

FIG. 4 is a plot of the distribution of discrete values for which the variance is evaluated by equation (2) for each of the cases of h ₁ = 1 to 10, separately from [1] to [10]. The method of expressing is the same as that described with reference to FIG. As in the case of FIG. 3, also in FIG. 4, it can be seen that the distribution when the equation (1) indicated by a broken line is a continuous function of time does not greatly deviate from the distribution of discrete values indicated by “◇”. Can be. In FIG. 4, as shown in [1], the fact that h ₁ = 1,10 minimizes the variance of Equation (2) indicates that the fluctuation width of “◇” in the vertical direction is the smallest. It can be seen as having become. Then, the smaller value of h ₁ = 1 can be adopted.

As described above, according to the present invention, flicker can be reduced by a simple process of averaging images of a predetermined combination determined according to given f _c , f _p , n. FIG. 5 is a diagram schematically showing the effect of reducing flicker according to the present invention, together with a comparative example. That is, in FIG. 5, when there is a movement as shown in [A], the resulting images obtained by each method are shown as [R1] to [R4]. In [A], it is assumed that the sphere rapidly moves rightward as indicated by an arrow in a time interval (1/30 second) of a low frame rate of 30 fps assuming display. In the video [R1] as a result of applying the present invention, the flicker is reduced and the motion blur due to the high-speed motion is small. [R2] as a comparative example is a case where imaging is also performed at a low frame rate of 30 fps, and although flicker cannot be perceived, motion blur occurs. [R3] as a comparative example is a case where imaging and display are performed as they are at a high frame rate of 240 fps, and flicker occurs instead of motion blur. [R4] as comparative example is the technique of Patent Document 3, suppress flicker by changing the exposure time according to the light source brightness of f _p = 50 Hz is the motion blur accidentally occur.

  In each of the results [R1] to [R4] in FIG. 5, the flickering (blinking) in the image of the sphere moving at high speed as in [A] is represented by shading. Also, the background portion other than the sphere is similarly expressed in shades. That is, [R2] has a long exposure time, so that the entire image including the background is brightly imaged. [R3] has a bright and dark image due to a change in the light source, but has a short exposure time, so that the entire image is darkly imaged. [R1] and [R4] are imaged intermediate between the two.

As can be understood from the above description, the result [R1] of the present invention is shown in the schematic example of FIG. 5 together with the comparative examples [R2] to [R4], and the present invention reduces the motion blur by utilizing the averaging process for the selected image. It is possible to reduce the flicker while suppressing it to an acceptable range and at the same time. Note that if the time interval of all captured images used for the averaging process is within about 1/30 second, the time interval is smaller than the standard 30 fps used in a general imaging device, so that even if averaging, Bokeh cannot be perceived. Rather, a single captured image a piece 1 / f motion blur for being imaged within the exposure time _c s is not generated, the motion blur can be suppressed as compared with Patent Document 3 to extend the exposure time The effect will be obtained.

Also, as can be seen from the behavior of Equation (1) as a continuous function in the examples of FIGS. 3 and 4, the blinking variation of the average image (processed image) by the processing unit 3 evaluated as the variance in Equation (2). behaviors, imaging frequency f _c and the power supply frequency f _p is also varied within a predetermined error, it does not vary greatly. That average, if f _c and f _p fluctuates error range, although the graph of FIG. 3 and FIG. 4 so that the slightly stretch or move the horizontal axis (time axis) direction, by the slight expansion or movement The blinking behavior of the image (processed image), that is, the behavior of the fluctuation in the vertical axis direction does not change significantly.

Therefore, also the actual value of the imaging frequency f _c and the power supply frequency f _p according to the present invention in a case with the error from the standard value, by selection by the selection section 2 on the assumption standard value, suppresses flicker It is possible. Similarly, when the update timing of the processing by the processing unit 3 varies (for example, by performing a separate process in the information processing device 10 in parallel, the calculation resources available in the processing unit 3 are assumed). Flickering can be suppressed even in the case where the number is smaller than that of

  Hereinafter, additional descriptions (1) to (6) regarding additional embodiments and the like in the present invention will be made.

(1) The selection unit 2 selects a predetermined combination {h} determined according to given f _c , f _p , n, and the selection result does not change according to time t. . That is, in the above description, for convenience of explanation that flicker reduction is realized, the selection unit 2 is provided and described, but the selection unit 2 is integrated with the processing unit 3 and the processing unit 3 In, directly selecting a plurality of images corresponding to a fixed selection result based on the combination {h} determined in advance by the same processing as the selection unit 2 described above, from the captured image obtained by the imaging unit 1 Thus, a processed image as an average image may be obtained. In other words, the combination {h} may be treated as a preset parameter, and by integrating the fixed selection process of the selection unit 2 with the processing process of the processing unit 3, (for each time t, each time t The selection unit 2 (as a functional unit that performs processing) may be omitted.

(2) When a part (or all) of a plurality of captured images selected by the selection unit 2 and averaged by the processing unit 3 is missing due to processing omission, frame drop, etc., and cannot be used for reference In this case, the selected images in the entire combination {h} may be shifted one by one to the past, and the combination available in the closest past may be used. Hereinafter, for the sake of explanation, the combination shifted m times in the past is denoted by {h} _[m] .

For example, in the example of FIG. 3, {h} = {0,6}, that is, the current time 0 and the past time 6 before 6 are to be selected. If at least one of them is unavailable, first, , It is confirmed whether all of {h} _[1] = {1,7} shifted one past in the past are available, and if available, this is set as a selection result instead of {h}. If {h} _[1] is not available, further check {h} _[2] = {2,8}, and if {h} _[2] is also unavailable, further {h} _[3] = {3 , 9}, and so on, the available result shifted m times in the past {h} _[m] = {m, 6 + m} to {h} It may be adopted as an alternative.

In the embodiment using the above result {h} _[m] , since the prerequisite {h} is fixed, the selection process of the selection unit 2 is performed by the processing unit as described in (1). 3 may be treated as integrated.

(3) A preferred embodiment of the predetermined combination {h} determined according to given f _c , f _p , n is as follows. Here, it is assumed that n = 2 and {h} = {0, h ₁ } are imposed as constraints so that an average image can be obtained at high speed as in the examples of FIGS.

For example, in Japan, the power supply frequency f _p differs from 50 Hz or 60 Hz depending on the region, and there are exceptions (near boundaries, railway networks, etc.), so the imaging update timing must correspond to multiple power supply frequencies f _p. (i.e. imaging frequency f _c) it is desirable to set the h ₁ in and {h} (h ₁ as solution that minimizes the variance of the formula (2)). For example, if the imaging frequency _{f c = 240Hz, f p =} 50Hz in solution h ₁ = 6, and the so becomes f _p = 60 Hz the solution h ₁ = 1, on the frequency of the power supply is determined whether it is there arises a need to set the appropriate h _1. However, when the imaging frequency f _c = 220, the solution h ₁ = 1 at both f _p = 50 Hz and f _p = 60 Hz, so that f _c = 220 Hz, h ₀ = 0, h ₁ = 1 By doing so, flicker can be suppressed without depending on the frequency of the power supply.

That is, in the constraint under high speed to the average image n = 2 and as capable of obtaining a {h} = {0, h 1}, if the candidate of the power supply frequency f _p is more, the imaging frequency f _c, solution regard candidate h ₁ of the plurality of power supply frequency f _p may be set as such that the common. Similarly, for example, under the constraints of n = 3 and {h} = {0, h ₁ , h ₂ }, if the imaging frequency f _c is set to 220 Hz, the candidates for the power supply frequency f _p are f _p = 50 Hz and f _{Even when p} = 60 Hz, there is a common solution of {h}, and it can be used as a setting for suppressing flicker. Specifically, when h ₁ = 3, h ₂ = 8, that is, the solution {h} = {0,3,8} has two different power supply frequencies f _p = under the imaging frequency f _c = 220 Hz. There will be a common solution that minimizes the variance at both 50 Hz and f _p = 60 Hz.

(4) As described with reference to FIG. 2, the present invention provides a case where the display frequency f _{v is set} to, for example, a low frame rate f _v = 30 Hz while the imaging frequency f _c = 240 Hz is set to a high frame rate, for example. and also for example if the same high frame rate f _v = 240 Hz and the imaging frequency f _c, be any of, when displaying on the display unit 4 by the display frequency f _v in the low frame rate or a high frame rate It is possible to reduce flickering and suppress motion blur.

Note that, in the embodiment described in the above (2), in which an available combination is searched for from the combination {h} _[m] shifted by m in the past, the display is performed at a low frame rate and the imaging is performed at a high frame rate. This is also a preferred embodiment. This is because m shifts can be performed at a high frame rate, and the range of search (a searchable range) is wider than that of display at a low frame rate.

Patent Document 1 cited above is a preferred example to be implemented in combination with the present invention (when the above display is performed at a low frame rate). Acquires a captured image by the imaging frequency f _c = 240 Hz as Patent Document 1, for example, a high frame rate, performs motion detection of an object in an image by analyzing the captured image of the high frame rate, the detected motion , Such as augmented reality display, is performed at a low frame rate f _v = 30 Hz, for example. Here, a motion at a low frame rate used for display is obtained by synthesizing motion detected at a high frame rate.

(5) In Expressions (2) and (3), the combination {h} that minimizes the variance and the like is determined as the selection result in the selection unit 2. However, when the search space of the combination {h} is enormous, etc. May be selected as an asymptotic or approximate solution that satisfies a predetermined criterion, instead of a combination as an exact minimum value. For example, a solution in which the variance or the like is equal to or smaller than a threshold value for determination may be used as a solution, or a solution that gives the minimum value among the results obtained by thinning out the search space may be used.

(6) The information processing device 10 can be realized as a computer having a general configuration. That is, a CPU (Central Processing Unit), a main storage device that provides a work area for the CPU, an auxiliary storage device that can be configured with a hard disk, an SSD, and the like, an input interface that receives input from a user such as a keyboard, a mouse, a touch panel, and a network. The information processing apparatus 10 can be configured by a general computer including a communication interface for performing communication by connecting to a computer, a display for performing display, a camera, and a bus for connecting these. Further, the processing of each unit of the information processing apparatus 10 can be realized by a CPU that reads and executes a program for executing the processing, but it is preferable that any part of the processing is realized by a separate dedicated circuit or the like. You may.

  10 information processing device, 1 imaging unit, 2 selection unit, 3 processing unit, 4 display unit

Claims (9)

An imaging unit that performs imaging at a predetermined imaging frequency and obtains a captured image at each time as a video,
A processing unit that obtains an average of the captured images at a predetermined plurality of times or past times at predetermined intervals from the time as a processed image obtained by processing the captured image at each time,
The information processing apparatus according to claim 1, wherein the predetermined interval is in accordance with the predetermined imaging frequency and a frequency of a light amount of a light source in an environment where the imaging is performed.   The predetermined interval is an average calculated by selecting the predetermined plurality of points with the predetermined imaging frequency as a unit interval from a predetermined light amount variation function at a frequency of the light amount of the light source in the environment where the imaging is performed. 2. The information processing apparatus according to claim 1, wherein the change in the value satisfies a predetermined criterion.   The information processing apparatus according to claim 2, wherein the predetermined criterion is to minimize the fluctuation. There are a plurality of candidates for the frequency of the light amount of the light source in the environment where the imaging is being performed,
The information processing apparatus according to claim 3, wherein the predetermined imaging frequency is given such that the predetermined interval for minimizing the fluctuation is common to each of the plurality of candidates. .   The information processing apparatus according to claim 2, wherein the fluctuation is evaluated as a variance.   In the processing unit, if there is a non-referenced among the captured images for averaging, by shifting the predetermined interval to the past side, and averaging the referenced captured images, The information processing apparatus according to claim 1, wherein a processed image is obtained.   The processing unit obtains, as a processed image obtained by processing the captured image at each time, an average of two of the captured image at the time and the captured image at one past time at a predetermined interval from the time. The information processing apparatus according to claim 1, wherein:   The information processing apparatus according to claim 1, further comprising a display unit that displays the processed image at a display frequency lower than the predetermined imaging frequency.   A program for causing a computer to function as the information processing apparatus according to claim 1.