JP2009181259A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform generation of a path-composition image with a high responsiveness and stability. <P>SOLUTION: A background pixel selecting unit 130 selects m (m≤n) pixel values centering one (image I<SB>t</SB>to be processed) of input images I<SB>1</SB>-I<SB>n</SB>held in an input image holding unit 110. A background estimating unit 140 estimates a background value of each pixel position of the image I<SB>t</SB>to be processed based on the selected pixel values. A background updating unit 210 updates a background update value held in a background update value holding unit 221 based on the estimated background value. A difference level generating unit 160 generates a difference level between the pixel value of each pixel position of the image I<SB>t</SB>to be processed and the background update value. A path composition unit 170 generates a composition ratio based on the difference level and composites the image I<SB>t</SB>to be processed at the time t and the path composition result at the time t-1 held in the path composition result holding unit 180 by the composition ratio to generate a path composition result at the time t. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that displays a trajectory in a captured image taken in time series, an image processing method thereof, and a program that causes a computer to execute the method.

  As a method of generating an image showing a moving subject's trajectory from a plurality of captured images (hereinafter referred to as frame images) captured in time series, for example, a method of simply superimposing a plurality of frame images and combining them can be considered. According to this simple method, there is a problem that as the number of frames increases, the contribution rate of one frame to the synthesis result decreases, and the locus of the moving subject becomes thin. For example, assuming five frame images as shown in FIG. 7, if the composition is simply performed, the contribution ratio of each pixel to the composition result is 20%, and the moving subject is located at the pixel position where the moving subject crosses. Since 20% and the background are combined at a ratio of 80%, the locus of the moving subject becomes thin as shown in FIG.

  On the other hand, as a method for improving simple synthesis, a method has been proposed in which a plurality of frame images are extracted from video data, and a single still image is synthesized with weights according to luminance values ( For example, see Patent Document 1.) In this prior art, in the pixel at the same position in each frame image, a pixel whose distance from the average value of the luminance value is more than the variance of the luminance value is given a large weight, and the distance from the average value of the luminance value is Pixels that are less than or equal to the variance of luminance values are combined with a small weight, and the result is output as a pixel at the same position in the output image. As a result, the weight of the moving subject in the synthesized image becomes larger than the weight of the background, and the locus of the moving subject is clarified as compared with a simple synthesis method.

  In addition, a method of synthesizing the locus of a moving subject by extracting and tracking the moving subject has been proposed (see, for example, Patent Document 2). In this conventional technique, when tracking of a moving subject fails, the accuracy of extraction and tracking of the moving subject is improved by interpolating the moving subject from the feature amount or position of the moving subject at the time when tracking was successful. .

  However, the method of combining the moving subject with a larger weight has the effect of clarifying the locus of the moving subject as compared with the simple combining method. The larger the number of other frame images, the lower the contribution ratio of one piece to the trajectory, and it was difficult to create a trajectory as clear as the original image. Further, in the method of determining the background based on the dispersion of luminance values, it may be difficult to clearly distinguish the background from the other depending on the state of the image. In addition, according to the conventional technique for tracking a moving subject, a trajectory can be displayed in an arbitrary manner while interpolating the moving subject. In this case, the moving subject's existing area, color, brightness, texture, etc. And the like are used, there is a problem that these changes and deformations cannot be handled. Further, when there are many moving subjects, there is a problem that it is difficult to track all the moving subjects due to the cost required for calculation and the like.

Therefore, a background image is obtained in advance, a difference between the pixel value of the background image and the pixel value of the input image is generated, and the pixel value of the input image is output as an output image according to the difference. There has been proposed an image processing apparatus that generates a trajectory still image or a trajectory moving image of a moving subject by reflecting it in a trajectory composite image (see, for example, Patent Document 3).
JP-A-10-290450 (FIG. 1) Japanese Patent Laying-Open No. 2005-123824 (FIG. 1) JP 2007-328755 A (FIG. 2)

  As described above, according to the prior art in which the pixel value of the input image is reflected in the locus synthesized image according to the degree of difference from the background image, the contribution rate to the locus synthesized image increases as the region different from the background becomes clearer. The trajectory of the moving subject can be obtained. However, in this case, since the background image is calculated in advance, there is a problem that a waiting time occurs before the display of the trajectory composite image is started. In addition, since all input images are required to generate a background image, there is a problem that locus synthesis cannot be performed until the end point of imaging is determined. Further, when the background image is generated in advance, it becomes impossible to follow the background fluctuation, and there is a possibility that a problem may occur in imaging for a certain period of time.

  The present invention has been made in view of such a situation, and an object thereof is to generate a trajectory composite image with high responsiveness and stability.

  The present invention has been made to solve the above-described problems, and a first aspect of the present invention is that a plurality of input images that are processing targets at respective pixel positions corresponding to a plurality of input images arranged in time series are centered. A pixel selection means for selecting a pixel value of the image, a background value holding means for holding a background value at the pixel position, and a background at the pixel position of the input image to be processed based on the selected pixel values. A background value estimating means for estimating a value, and a background updating means for updating the background value held in the background value holding means based on the estimated background value and holding the background value as a new background value in the background value holding means And a dissimilarity generation means for generating a dissimilarity between the pixel value of the input image to be processed and the updated background value, and the processing according to the dissimilarity for each of the pixel positions. The image processing apparatus characterized by comprising an output image generation means for generating a pixel value of an output image to reflect the pixel values of the input image which is the target, its method and program. Thus, the background value is updated at each of the corresponding pixel positions of the plurality of input images, and an output pixel is generated based on the updated background value.

  The first aspect further includes background reliability generation means for generating the reliability of the estimated background value based on the selected plurality of pixel values, and the difference generation means includes: The degree of difference may be generated according to the reliability. This brings about the effect that the degree of difference is generated according to the reliability of the background value. In this case, the dissimilarity generation means can generate the dissimilarity so that the dissimilarity increases as the reliability decreases.

  In the first aspect, the background update unit may update the background value held in the background value holding unit according to the reliability. This brings about the effect that the background value is updated according to the reliability of the background value. In this case, the background update unit can update the background value held in the background value holding unit so that the higher the reliability, the higher the specific gravity of the estimated background value.

  In the first aspect, the background value estimating means may estimate a mode value among the selected plurality of pixel values as a background value. This brings about the effect that the background value is updated with the mode value in the pixel value.

  In the first aspect, the output image generation unit generates an output image holding unit that holds a pixel value of the output image, and a combination ratio that generates a combination ratio according to the difference between the pixel positions. A new output image is generated by combining the pixel value of the input image to be processed with the generation unit and the pixel value of the output image held in the output image holding unit according to the synthesis ratio for each of the pixel positions. And a synthesized value calculating means for holding the output image holding means as the pixel value. Thereby, the pixel value of the input image to be processed and the pixel value of the output image are combined by the combination ratio generated based on the degree of difference and the background combination ratio.

  In the first aspect, the pixel selection unit may select the plurality of pixel values at regular intervals in time, and the closer the temporal distance to the input image to be processed is, The plurality of pixel values may be selected with a high frequency in time. By thinning out the selected pixel values, the number of pixel values is reduced without changing the selected time range, and the deterioration of the accuracy of moving subject probability generation is suppressed.

  In the first aspect, the background value update by the background update unit may be different from the frequency of the difference generation by the difference generation unit. This brings about the effect of effectively improving the throughput by controlling the background update frequency with respect to the generation of the degree of difference.

  The second aspect of the present invention is based on pixel selection means for selecting a plurality of pixel values at each of corresponding pixel positions of a plurality of input images arranged in time series, and on the basis of the selected plurality of pixel values. Background value estimating means for estimating a background value at the pixel position, background reliability generating means for generating the reliability of the estimated background value based on the selected plurality of pixel values, and an input to be processed A difference generation means for generating a difference between the pixel value of the image and the estimated background value according to the reliability, and an input image to be processed according to the difference for each of the pixel positions An image processing apparatus, a method and a program therefor, comprising output image generation means for generating pixel values of an output image by reflecting pixel values. This brings about the effect | action that an output pixel is produced | generated based on the estimated background value and its reliability.

  According to the present invention, it is possible to achieve an excellent effect that it is possible to generate a trajectory composite image with high responsiveness and stability.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of an imaging apparatus according to an embodiment of the present invention. This imaging apparatus is roughly composed of an optical system, a signal processing system, a recording system, a display system, and a control system.

  The optical system includes a lens 11 that collects a light image of a subject, a diaphragm 12 that adjusts the amount of light in the light image, and an image sensor 13 that photoelectrically converts the collected light image into an electrical signal. The The image sensor 13 is realized by, for example, a CCD (Charge Coupled Devices) image sensor.

  The signal processing system includes a sampling circuit 21 that samples an electrical signal from the image sensor 13, an A / D conversion circuit 22 that converts an analog signal supplied from the sampling circuit 21 into a digital signal, and an A / D conversion circuit 22. And an image processing circuit 23 that performs predetermined image processing on the input digital signal. The sampling circuit 21 is realized by, for example, a correlated double sampling circuit (CDS: Correlated Double Sampling). Thereby, noise generated in the image sensor 13 is reduced. Details of processing executed by the image processing circuit 23 will be described later.

  The recording system encodes the image signal processed by the image processing circuit 23 and the memory 32 that stores the image signal, and records the image signal in the memory 32, and also reads and decodes the image signal from the memory 32, and the image processing circuit 23. And an encoder / decoder 31 to be supplied.

  The display system includes a D / A conversion circuit 41 that converts the image signal processed by the image processing circuit 23 into an analog signal, and a video encoder that encodes the analog image signal into a video signal in a format suitable for the display unit 43 in the subsequent stage. 42 and a display unit 43 that displays an image corresponding to the input video signal. The display unit 43 is realized by, for example, an LCD (Liquid Crystal Display) or the like, and also has a function as a finder.

  The control system is a timing generator 51 that controls the operation timing of the image sensor 13, the sampling circuit 21, the A / D conversion circuit 22, and the image processing circuit 23; An operation input receiving unit 52, a driver 53 for connecting peripheral devices, and a control unit 54 for controlling the entire imaging apparatus are configured. The driver 53 is connected to a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like. The control unit 54 reads out the control programs stored therein via the driver 53, and performs control based on the read control program, a user command input from the operation input receiving unit 52, and the like.

  The image processing circuit 23, the encoder / decoder 31, the memory 32, the timing generator 51, the operation input receiving unit 52, and the control unit 54 are connected to each other via a bus 59.

  In this image pickup apparatus, an optical image (incident light) of a subject enters the image pickup device 13 through the lens 11 and the diaphragm 12, and is photoelectrically converted by the image pickup device 13 into an electric signal. A noise component is removed from the obtained electric signal by the sampling circuit 21, digitized by the A / D conversion circuit 22, and then temporarily stored in an image memory (not shown) built in the image processing circuit 23.

  In a normal state, the image signal built in the image processing circuit 23 is constantly overwritten at a fixed frame rate by the control of the signal processing system by the timing generator 51. The image signal in the image memory built in the image processing circuit 23 is converted into an analog signal by the D / A conversion circuit 41, converted into a video signal by the video encoder 42, and a corresponding image is displayed on the display unit 43.

  The display unit 43 also serves as a viewfinder of the imaging device. When the user presses the shutter button included in the operation input receiving unit 52, the control unit 54 causes the timing generator 51 to hold the image signal immediately after the shutter button is pressed, that is, image processing. The signal processing system is controlled so that the image signal is not overwritten in the image memory of the circuit 23. The image data held in the image memory of the image processing circuit 23 is encoded by the encoder / decoder 31 and recorded in the memory 32. With the operation of the imaging apparatus as described above, the acquisition of one piece of image data is completed.

  FIG. 2 is a diagram illustrating a first example of the image processing circuit 23 of the imaging device according to the embodiment of the present invention. The image processing circuit 23 according to this embodiment includes an input image holding unit 110, a trajectory pixel selection unit 120, a background pixel selection unit 130, a background estimation unit 140, a background reliability generation unit 150, and a background update unit 210. A background update value holding unit 221, a dissimilarity generation unit 160, a trajectory synthesis unit 170, a trajectory synthesis result holding unit 180, and a trajectory composite image display unit 190.

The input image holding unit 110 is a memory that holds _n (n is an integer of 2 or more) input images (I _{1 to} I _n ) captured in time series. Among the n input images, they are treated as processing target images in order from the oldest in time series. Since the input image holding unit 110 is limited, when a new input image is input, the oldest input image is invalidated and overwritten in order. Therefore, this input image holding unit 110 always holds an input image corresponding to a certain period.

The trajectory pixel selection unit 120 selects a pixel value at each pixel position using _{one of} the input images (I _{1 to} I _n ) held in the input image holding unit 110 as a processing target image. The selection order of the pixel positions may be, for example, the scan line order from the upper left. For example, when each of the input images is composed of p × q pixels having coordinates (1, 1) to (p, q) (p and q are integers of 1 or more), at time t (t is an integer of 2 or more). When the input image and the processing target image I _t, the locus pixel selection unit 120 selects the pixel value of the pixel position of the coordinate (1,1) of the processing target image I _t, then the coordinates of the target image I _t The pixel value at the pixel position (1, 2) is selected. Locus pixel selection unit 120 repeats the process until the pixel values of all the pixel positions is selected for the target image in this manner, selects a pixel value of p × q pieces min to the processing target image I _t. The pixel value selected by the trajectory pixel selection unit 120 is supplied to the dissimilarity generation unit 160 and the trajectory synthesis unit 170 via the signal line 129 as a trajectory pixel value.

The background pixel selection unit 130 uses _{one of} the input images (I _{1 to} I _n ) held in the input image holding unit 110 as a processing target image, and an input image in a certain range in time series centering on the processing target image. In FIG. 5, m (m ≦ n; m is an integer of 2 or more) pixel values are selected from the corresponding pixel positions. The selection order of the pixel positions may be, for example, the scan line order from the upper left. For example, when each of the input images is composed of p × q pixels having coordinates (1, 1) to (p, q) (p and q are integers of 1 or more), at time t (t is an integer of 2 or more). When the input image and the processing target image I _t, background pixel selection unit 130 firstly processed as the center image I _t from among the pixel values of the pixel positions of the coordinates of the n input images (1,1) Select m. Background pixel selection unit 130 then selects an m number around the target image I _t from among the pixel values of the pixel positions of the coordinates of the n input images (1, 2). In this manner, ultimately m pixel value is to be selected p × q of sets for the input image (I 1 _{~I n).} These selected pixel values are supplied to the background estimation unit 140.

Background estimation unit 140, based on the m number of pixel values selected in the background pixel selection unit 130, and estimates the background value of each pixel position of the processing target image I _t. For m pixel values supplied p × q of sets as described above, the background estimation unit 140 estimates the p × q pieces of background values with respect to one of the processing target image I _t. The estimated background value Be _t (hereinafter referred to as background estimation value) is supplied to the background update unit 210. In this background value estimation, as will be described later, statistical processing using the temporal constancy of the background is performed. With this process, when m pixel values are sampled as reference values, the number of pixel values close to each reference value is counted, and the reference value that maximizes the count number is estimated as the background value. The estimated count number is the maximum count number c. The maximum count number (c) and the total sampling number (m) are supplied to the background reliability generation unit 150.

The background reliability generation unit 150 generates the reliability T _t of the background estimation value estimated by the background estimation unit 140 based on the maximum count number (c) and the total sampling number (m) supplied from the background estimation unit 140. To do. Specifically, the reliability T _t of the background value can be obtained by the following equation. Incidentally, the reliability T _t of the background value generated by the background reliability determining unit 150, the processing is determined for each pixel position in the target image I _t, background update unit through the signal lines 229 210 and degree-of-difference generation unit 160.
T _t = c / m

The background update unit 210 updates the background value (background update value) held in the background update value holding unit 221 based on the background value estimated by the background estimation unit 140. The background update section 210, the background update value _{Bu t-1} Metropolitan held in the background value Be _t and background update value holding section 221 estimated by the background estimation unit 140, a background update value in the processing target image _{I t} Bu _t is generated and stored in the background update value storage unit 221. In this case, it is possible in accordance with the reliability T _t of the background value supplied from the background reliability determining unit 150, changing the ratio between the background value Be _t and background update value Bu _t-1. That is, the higher the reliability _{T t} of the background value Be _t with a ratio of the background value Be _t, a higher ratio of the background updated value _{Bu t-1} A low reliability _{T t} of the background value Be _t. For example, the reliability of the background value of the coordinate (x, y) at time t is T (x, y, t), the estimated background value of the coordinate (x, y) at time t is Be (x, y, t), Assuming that the background update value at the coordinates (x, y) at time t−1 is Bu (x, y, t−1), the background update value Bu (x, y, t) at the coordinates (x, y) at time t−1. Is calculated by the following equation.

However, k is an update parameter. If k is a filter coefficient, the above equation is equivalent to the equation of an IIR (Infinite Impulse Response) filter transfer function. That is, the background update value Bu _t at time t, with reference to the background update value _{Bu t-1} of the background value of the time t Be _t and time t-1, will have been updated by an IIR filter. Thereby, it is possible to cope with the case where the background fluctuates, and when the background reliability is low, the background update value is updated so that the weight of the background update value at the previous time is increased. The background value can be updated in consideration of the background reliability. Incidentally, the background update value Bu _t updated by the background update unit 210 via the signal line 228 is supplied to the background update value holding unit 221 and the degree-of-difference generation unit 160.

The background update value holding unit 221 holds the background update value updated by the background update unit 210. The background update value held by the background update value holding unit 221 is supplied to the background update unit 210 at the next time (t) as the previous background update value But _-1 .

Degree-of-difference generation unit 160, the dissimilarity d between the background update value Bu _t updated by the pixel value and the background update unit 210 of each pixel position of the processing target image I _t selected by the trajectory pixel selection unit 120 Is to be generated. For example, the degree of difference can be set so as to be proportional to the distance in the color space between the pixel value of the processing target image and the background update value. Thus, the greater the distance between the two, the greater the degree of difference.

Here, the distance between the two is defined as follows, for example. That is, when the pixel value is represented by an RGB value, the distance between the pixel value V1 (R1, G1, B1) of the processing target image and the background update value V2 (R2, G2, B2) is a distance in the Euclidean space, It can be expressed by the sum of the distances in each color space, the distance in the color space showing the maximum difference value, or the like. Although an example of the RGB color space is shown here, for example, a YCbCr color space or other color spaces may be used.

In addition, when the reliability T _t supplied from the background reliability generation unit 150 is low, the difference d may be increased. That is, the dissimilarity generation unit 160 calculates the dissimilarity d using the distance between the pixel value of the processing target image and the background estimation value when the reliability _Tt is high, but the reliability _Tt is low. Is set to a large difference d. Alternatively, the dissimilarity generation unit 160 generates the dissimilarity so that the dissimilarity d increases as the reliability _Tt decreases. A large difference d means a moving subject. The reason why the process according to the reliability T _t is performed when the difference is generated is that the risk of misjudging a moving object as a background is greater than the risk of misjudging the background as a moving object. For example, if the moving object is erroneously determined as the background, the moving object disappears without being reflected in the trajectory synthesis, but if the background is erroneously determined as the moving object, the background is synthesized on the background in the trajectory synthesis. However, it has little effect on the output result. The dissimilarity d generated by the dissimilarity generation unit 160 is supplied to the trajectory synthesis unit 170 via the signal line 169.

The trajectory synthesis unit 170 generates a synthesis ratio based on the dissimilarity d generated by the dissimilarity generation unit 160, and is held in the processing target image It at time _t and the trajectory synthesis result holding unit 180 based on the synthesis ratio. The locus synthesis result at time t-1 is synthesized with the locus synthesis result at time t-1. The generated trajectory synthesis result at time t is supplied to the trajectory synthesis result holding unit 180 and the trajectory composite image display unit 190 via the signal line 179.

  The trajectory synthesis result holding unit 180 holds the trajectory synthesis result by the trajectory synthesis unit 170. The locus synthesis result held in the locus synthesis result holding unit 180 is supplied to the locus synthesis unit 170 via the signal line 189 at the next time. That is, when the input image at time t is used as the processing target image, the locus synthesis result at time t−1 is supplied to the locus synthesis unit 170.

  The trajectory composite image display unit 190 displays a trajectory composite image based on the trajectory composite result supplied from the trajectory composite unit 170 via the signal line 179. The trajectory composite image display unit 190 may use the display unit 43 as it is, or may be realized by another display device connected to the imaging device.

FIG. 3 is a diagram illustrating a mode of pixel value selection in the background pixel selection unit 130 according to the embodiment of the present invention. Background pixel selection unit 130 of the input image stored in the input image holding unit 110 _(I 1 ~I _n), as the processing target image the input image _{I t} at time t, p × q of the input image _{I t} A pixel value is selected for each pixel position of the pixel.

In order to estimate the background value for the pixel value V _t of a pixel position in the input image I _t, the pixel value of the corresponding pixel position in the other input image centered on the input image I _t is used. For example, the pixel values V _t-i and V _{t + i} of the corresponding pixel positions in the input images I _t-i and I _{t-i at} the time t-i and the time _{t + i} are used.

  As the number of pixel values selected by the background pixel selection unit 130 increases, it is advantageous to improve the accuracy of background value estimation, but the calculation cost increases. Therefore, it is desirable to adjust the pixel value selection interval in consideration of the accuracy of background value estimation as follows.

FIG. 4 is a diagram illustrating an example of a pixel value selection interval according to the embodiment of the present invention. Here, the time direction of the input images (I _{1 to} I _n ) is shown on the horizontal axis. FIG. 4A shows an example in which all pixel values within a certain range centered on time t are selected. FIG. 5B shows an example in which pixel values within a certain range centered on time t are selected by being thinned out at equal intervals. FIG. 6C shows an example in which pixel values are selected more densely as the time is closer to time t and coarser as the time is farther within a certain range centered on time t. That is, in FIG. 3C, pixel values are selected with a higher frequency in time as the temporal distance from the input image is closer.

  In this way, the number of pixel values can be reduced without changing the time range to be selected by selecting the pixel values as shown in (b) or (c) of FIG. It is possible to suppress degradation of the accuracy of background value estimation.

FIG. 5 is a diagram showing an aspect of background value estimation in the background estimation unit 140 according to the embodiment of the present invention. One of the m pixel values selected by the background pixel selection unit 130 is sampled and used as a reference value. The number of pixel values that fall within a range in which the background threshold is added to each of the high and low pixel values with the reference value as the center is counted. In the example of the figure, the count number is seven. After that, the other one of the m pixel values is sequentially changed as a reference value and counted in the same manner. As a result, the count number is most often made background reference value is the estimate Be _t. That is, the mode value of the pixel value is estimated as the background value. Further, the count number when estimated as the background value is the maximum count number c. Here, the reference value that maximizes the number of counts is estimated as the background value, but an average value of pixel values that fall within the background threshold range at that time may be estimated as the background value.

  FIG. 6 is a diagram illustrating a configuration example of the trajectory synthesis unit 170 in the embodiment of the present invention. The trajectory synthesis unit 170 includes a fade-out control parameter holding unit 171, a synthesis ratio generation unit 172, and a trajectory synthesis value calculation unit 173.

The fade-out control parameter holding unit 171 holds the fade-out control parameter α _bg . The fade-out control parameter α _bg represents the background synthesis ratio. If the fade-out control parameter α _bg is set large, the fade-out control parameter α _bg is buried in the background and fades out in a short period of time.

Based on the difference d supplied from the difference generation unit 160 via the signal line 169 and the fade-out control parameter α _bg held in the fade-out control parameter holding unit 171, the combination ratio generation unit 172 The composition ratio α is generated. This synthesis ratio α is calculated by the following equation.
α = α _bg + d · (1−α _bg )

When the degree of difference d is large, α also increases, and the contribution ratio of the pixel value of the original image (processing target image I _t ) at the time of composition increases, so that the pixel value of the original image is easily reflected as a trajectory in the composition result. . On the other hand, when the degree of difference d is small, α is also small, and the contribution rate of the past trajectory image is high with respect to the composite result, so that the pixel value of the original image is hardly reflected as a trajectory in the composite result.

As described above, α _bg is a parameter representing the background synthesis ratio. Since the difference d is 0 in the background area, the pixel value of the original image is reflected in the trajectory synthesis result by α = αbg. In other words, even if the pixels of the original image are the background, the pixel values of the original image are synthesized with the past trajectory, so that it becomes possible to produce a fade-out effect that is buried in the background as the old trajectory is. In this manner, the degree of trajectory fade-out can be controlled by the fade-out control parameter α _bg .

Locus synthesis value calculation unit 173, the locus synthesis result immediately before supplied via the signal line 189 from the target image I _t and the locus synthesis result storage unit 180 supplied via the signal line 129 from the trajectory pixel selection unit 120 S _t-1 is synthesized by the synthesis ratio α generated by the synthesis ratio generation unit 172. Here, if the input image at the coordinates (x, y) at time t is I (x, y, t) and the locus synthesis result at time t-1 is S (x, y, t-1), The locus synthesis result S (x, y, t) is calculated by the following equation.
S (x, y, t) = α · I (x, y, t) + (1−α) · S (x, y, t−1)

The trajectory image at time t is synthesized by repeating the above processing at all pixel positions of coordinates (1, 1) to (p, q). As described above, since the degree of difference d is higher the greater the synthesis ratio α greater, the processing target image I _t is a moving subject is likely to be reflected in the composition result as the locus.

Here, in order to explain the effect of the fade-out control parameter α _bg , images are taken in the order of FIG. 7A, FIG. 7B, FIG. 7C, FIG. The captured image is assumed as an input image. When “0” is set as the fade-out control parameter α _bg , the composition ratio α matches the dissimilarity d. If the dissimilarity d is “1” in the moving subject region and the dissimilarity d is “0” in the background region, the background and the moving subject are not mixed at the time of composition. In the image, the locus of the moving subject is all darkly displayed as shown in FIG.

On the other hand, when about “0.3” is set as the fade-out control parameter α _bg , the composition ratio α is a value between “0.3” and “1”. If the dissimilarity d is “1” in the moving subject region and the dissimilarity d is “0” in the background region, the background is mixed with the moving subject every time the composition is performed. In the image, as the locus of the moving subject is older, the lighter is displayed as shown in FIG.

Further, the trajectory composite image is not only displayed as one trajectory composite image as shown in FIGS. 8A and 8B, but also a plurality of states during the movement of the moving subject as shown in FIG. 9 and FIG. It may be displayed in time series as a single trajectory composite image. Also in this case, the degree of fade-out can be controlled using the above-described fade-out control parameter α _bg . For example, when it is desired to leave a locus clearly, if “0” is set as the fade-out control parameter α _bg , the composition ratio α matches the difference d. If the dissimilarity d is “1” in the moving subject region and the dissimilarity d is “0” in the background region, the background and the moving subject are not mixed at the time of composition. As shown in FIG. 9, the locus of the moving subject is all darkly displayed.

On the other hand, when it is desired to fade out the trajectory, the synthesis ratio α becomes a value between “0.3” and “1” by setting about “0.3” as the fade-out control parameter α _bg . If the dissimilarity d is “1” in the moving subject region and the dissimilarity d is “0” in the background region, the background is mixed with the moving subject every time the composition is performed. As shown in FIG. 10, the older the moving subject trajectory is, the lighter is displayed.

FIG. 11 is a diagram illustrating a modification of the first example of the image processing circuit 23 of the imaging device according to the embodiment of the present invention. In this modification, a background information holding unit 220 is provided instead of the background update value holding unit 221. This background information holding unit 220 is to hold the reliability T _t of the background value generated by the background update value Bu _t and background reliability determining unit 150 updated by the background update section 210 as background information. By providing the background information holding unit 220, the background pixel selection unit 130, the background estimation unit 140, the background reliability generation unit 150 and the background update unit 210 perform background update processing, and the trajectory pixel selection unit 120 and the difference degree generation unit 160. It is possible to execute the moving object detection process according to the above or the trajectory synthesis process by the trajectory synthesis unit 170 at different timings.

  FIG. 12 is a diagram illustrating an example of processing timing of the imaging apparatus according to the embodiment of the present invention. The background pixel selection unit 130, the background estimation unit 140, the background reliability generation unit 150, and the background update unit 210 perform the background update process, and the trajectory pixel selection unit 120 and the difference generation unit 160 perform the moving object detection process and the trajectory synthesis unit. Assuming that the processing by 170 is the trajectory synthesis processing, the timing is as shown in FIG. That is, the moving object detection process is performed after the background update process is performed, and the locus synthesis process is performed after the moving object detection process is performed.

  When the background information holding unit 220 is provided as shown in FIG. 11, the result of the background update process is held as background information in the background information holding unit 220. Therefore, the moving object detection process and the trajectory synthesis process as shown in FIG. It is possible to perform the next background update process without waiting for the end of. That is, the background update process and the moving object detection process or the trajectory synthesis process can be executed in an overlapping manner.

  In addition, since the background update process is heavier than the moving object detection process and the trajectory synthesis process, the frequency of the background update process, the moving object detection process, and the trajectory synthesis process is set to one pair in order to further improve the overall throughput. FIG. 12C shows the ratio of 2. According to this, a moving body detection and locus | trajectory synthesis | combination can be performed twice by the result of having performed background update once. When the background fluctuates gently, it is not necessary to update the background every time a moving object is detected, and the overall throughput can be improved by reducing the frequency of background updating, which is heavy processing.

  When the background information holding unit 220 is provided under the same conditions as in FIG. 12C, the next background update process can be performed without waiting for the end of the moving object detection process and the trajectory synthesis process as shown in FIG. . That is, one background update process and two moving object detection processes or trajectory synthesis processes can be executed in an overlapping manner, further improving the throughput.

  Next, the operation of the first example of the imaging apparatus according to the embodiment of the present invention will be described with reference to the drawings.

  FIG. 13 is a flowchart illustrating an example of a background update processing procedure by the imaging apparatus according to the embodiment of the present invention.

  First, in the background pixel selection unit 130, m pixel values are selected from the corresponding pixel positions in the input image at the time before and after the processing target image in the time series order (step S911). The relationship between input images at the time of pixel value selection is as described with reference to FIGS.

On the basis of the selected pixel values, the background values Be _t is estimated in the background estimation unit 140 (step S912). The process for estimating the background value is as described with reference to FIG.

Based on the maximum count number (c) and the total sampling number (m) obtained at the time of estimating the background value, the background reliability generation unit 150 generates the background value reliability T _t (step S913).

Then, based on the estimated background value Be _t, background update value _{Bu t-1} is updated by the background update section 210 (step S914). During the background value update is considered the reliability T _t of the background value, the higher the reliability T _t of the background value Be _t becomes high proportion of the background value Be _t, reliability T background value Be _{t If t} is low, the ratio of the background update value But _-1 is high.

When the above-described processing for each pixel position of the p × q pixels in the target image I _t is completed, the process relates to the processing target image I _t is completed (step S915). The processes in steps S911 to S914 can be performed independently for each pixel position.

  After the processing of steps S911 to S915 is completed with the input image at time t as the processing target image, the trajectory composite image is sequentially repeated by repeating the processing of steps S911 to S915 with the input image after time t + 1 as the processing target image. It can be generated in time series. These processing target images can be processed independently between the processing target images.

  FIG. 14 is a flowchart showing an example of a moving object detection and trajectory synthesis processing procedure by the imaging apparatus according to the embodiment of the present invention.

First, the locus pixel selection unit 120, as a processing target image I _t the one input image, a pixel value of each pixel position is selected (step S921). Then, the pixel value of the selected input image, the dissimilarity d between the updated background value Bu _t is generated by the degree-of-difference generation unit 160 by the background update process (step S923).

Based on thus generated dissimilarity d, the generated synthesis ratio α in the locus synthesis unit 170, the process target image I _t and the previous locus synthesis result of S _t-1 in accordance with the synthesis ratio α is The result is synthesized and the result is output (step S924).

  When the above-described processing is completed for each pixel position of p × q pixels in the processing target image, the processing regarding the processing target image ends (step S925). The processes in steps S921 to S924 can be performed independently for each pixel position.

  After the processing of steps S921 to S925 is completed using the input image at time t as the processing target image, the trajectory composite image is sequentially repeated by repeating the processing of steps S921 to S925 using the input image after time t + 1 as the processing target image. It can be generated in time series. These processing target images can be processed independently between the processing target images.

  Further, by providing the background information holding unit 220 as in the modified example of FIG. 11, the background update process of FIG. 13 and the moving object detection and trajectory synthesis process of FIG. 14 can be executed in an overlapping manner.

  Thus, according to the embodiment of the present invention, the background estimation unit 140 estimates the background value from the time-series input image, and is stored in the background update value holding unit 221 based on the estimated background value. By sequentially updating the background value, stable moving subject detection can be performed even in a situation where the background fluctuates. That is, an image processing apparatus capable of generating a highly accurate moving subject trajectory composite image without being influenced by the background state of the captured image and the number, size, deformation, color, etc. of the moving subject is realized. be able to.

  Further, according to the embodiment of the present invention, with respect to the problem of erroneous detection of a moving subject, it is determined that the subject is a moving subject in an area with low background reliability, thereby reducing both the influence on the trajectory synthesis result and the calculation cost. Can be suppressed.

  Moreover, according to the embodiment of the present invention, it is possible to effectively improve the throughput by controlling the background update frequency for moving object detection.

  The embodiment of the present invention has been described assuming a plurality of images continuously captured by an imaging apparatus such as a digital still camera or a digital video camera. However, the present invention is not limited to this, and playback is performed. Video data or the like acquired via a broadcast network or the Internet in the device can also be applied as an input image.

  Next, a second example of the imaging apparatus according to the embodiment of the present invention will be described with reference to the drawings.

FIG. 15 is a diagram illustrating a second example of the image processing circuit 23 of the imaging device according to the embodiment of the present invention. The imaging apparatus according to the second embodiment has a configuration in which a background information holding unit 220 is provided instead of the background update unit 210 and the background update value holding unit 221 in the configuration of the first example. In this case, the background pixel selection unit 130 selects m pixel values from the entire input image (I _{1 to} I _n ) as a range. Based on the selected pixel value, the background estimation unit 140 estimates the background value B, and the reliability T is generated by the background reliability generation unit 150. The background value B estimated by the background estimation unit 140 and the reliability T generated by the background reliability generation unit 150 are held in the background information holding unit 220 as background information. Unlike the case of the first embodiment, this background information is not updated. Then, the difference generation unit 160 generates a difference between the pixel value of the input image and the background value B based on the background information held in the background information holding unit 220. At that time, the reliability T of the background value B is also considered.

  FIG. 16 is a flowchart illustrating an example of a processing procedure performed by the imaging apparatus according to the embodiment of the present invention.

First, the background pixel selection unit 130, m pieces of pixel values are selected from the entire input image _(I 1 ~I _{n) (step} S931). Based on the selected pixel value, the background estimation unit 140 estimates the background value B (step S932). In addition, based on the maximum count number (c) and the total sampling number (m) obtained in the background value estimation, the background reliability generation unit 150 generates the reliability T of the background value (step S933). The processes in steps S931 to S933 can be performed independently for each pixel position.

Further, in the trajectory pixel selection unit 120, as a processing target image I _t the one input image, a pixel value of each pixel position is selected (step S 941). Then, a difference degree d between the pixel value of the selected input image and the background value B updated by the background update process is generated by the difference degree generation unit 160 (step S943).

Based on thus generated dissimilarity d, the generated synthesis ratio α in the locus synthesis unit 170, the process target image I _t and the previous locus synthesis result of S _t-1 in accordance with the synthesis ratio α is The result is synthesized and the result is output (step S944).

When the above-described processing for each pixel position of the p × q pixels in the target image I _t is completed, the process relates to the processing target image I _t is completed (step S945). The processes in steps S941 to S944 can be performed independently for each pixel position.

  After the process of steps S941 to S945 is completed with the input image at time t as the processing target image, the process of steps S941 to S945 is successively repeated with the input image after time t + 1 as the processing target image. It can be generated in time series. These processing target images can be processed independently between the processing target images.

  Thus, according to the second embodiment, the cost required for background calculation can be reduced by continuing to use the background value once calculated without updating.

  The embodiment of the present invention is an example for embodying the present invention and has a corresponding relationship with the invention-specific matters in the claims as shown below, but is not limited thereto. However, various modifications can be made without departing from the scope of the present invention.

  That is, in claim 1, the pixel selection unit corresponds to, for example, the background pixel selection unit 130. The background value holding unit corresponds to the background update value holding unit 221, for example. The background value estimating means corresponds to the background value estimating unit 140, for example. A background update unit corresponds to the background update unit 210, for example. Further, the difference generation unit corresponds to the difference generation unit 160, for example. The output image generation means corresponds to the trajectory synthesis unit 170 and the trajectory synthesis result holding unit 180, for example.

  Further, in claim 2, the background reliability generation means corresponds to the background reliability generation unit 150, for example.

  Further, in claims 11 and 12, the pixel selection procedure corresponds to, for example, step S911. The background value estimation procedure corresponds to, for example, step S912. The background update procedure corresponds to, for example, step S914. The difference degree generation procedure corresponds to, for example, step S923. The output image generation procedure corresponds to step S924, for example.

  Further, in claim 13, the pixel selection means corresponds to, for example, the background pixel selection unit 130. The background value estimating means corresponds to the background value estimating unit 140, for example. A background reliability generation unit corresponds to the background reliability generation unit 150, for example. Further, the difference generation unit corresponds to the difference generation unit 160, for example. The output image generation means corresponds to the trajectory synthesis unit 170, for example.

  Further, in claims 14 and 15, the pixel selection procedure corresponds to, for example, step S931. The background value estimation procedure corresponds to, for example, step S932. The background reliability generation procedure corresponds to, for example, step S933. The difference degree generation procedure corresponds to, for example, step S943. The output image generation procedure corresponds to step S944, for example.

  The processing procedure described in the embodiment of the present invention may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program May be taken as

It is a figure which shows an example of the imaging device in embodiment of this invention. It is a figure which shows the 1st Example of the image processing circuit 23 of the imaging device in embodiment of this invention. It is a figure which shows the aspect of the pixel value selection in the background pixel selection part 130 by embodiment of this invention. It is a figure which shows the example of the selection space | interval of the pixel value in embodiment of this invention. It is a figure which shows the aspect of the background value estimation in the background estimation part 140 by embodiment of this invention. It is a figure which shows the example of 1 structure of the locus | trajectory synthetic | combination part 170 in embodiment of this invention. It is a figure which shows an example of an input image. It is a figure which shows an example of a locus | trajectory composite image. It is a figure which shows the example of a time series output in a locus | trajectory synthetic | combination moving image. It is a figure which shows the other output example of the time series in a locus | trajectory synthetic | combination moving image. It is a figure which shows the modification of the 1st Example of the image processing circuit 23 of the imaging device in embodiment of this invention. It is a figure which shows the example of a process timing of the imaging device in embodiment of this invention. It is a flowchart which shows an example of the background update process sequence by the imaging device in embodiment of this invention. It is a flowchart which shows an example of the moving body detection and locus | trajectory synthesis processing procedure by the imaging device in embodiment of this invention. It is a figure which shows the 2nd Example of the image processing circuit 23 of the imaging device in embodiment of this invention. It is a flowchart which shows an example of the process sequence by the imaging device in embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Lens 13 Image pick-up element 21 Sampling circuit 22 A / D conversion circuit 23 Image processing circuit 31 Encoding / decoding device 32 Memory 41 D / A conversion circuit 42 Video encoder 43 Display part 51 Timing generator 52 Operation input reception part 53 Driver 54 Control unit 59 Bus 110 Input image holding unit 120 Trajectory pixel selection unit 130 Background pixel selection unit 140 Background estimation unit 150 Background reliability generation unit 160 Dissimilarity generation unit 170 Trajectory synthesis unit 171 Fade-out control parameter storage unit 172 Composition ratio generation unit 173 Locus synthesis value calculation unit 180 locus synthesis result holding unit 190 locus synthesis image display unit 210 background update unit 220 background information holding unit 221 background update value holding unit

Claims (15)

Pixel selection means for selecting a plurality of pixel values around the input image to be processed at each of the corresponding pixel positions of the plurality of input images arranged in time series;
Background value holding means for holding a background value at the pixel position;
Background value estimation means for estimating a background value at the pixel position of the input image to be processed based on the plurality of selected pixel values;
A background updating unit that updates the background value held in the background value holding unit based on the estimated background value and holds the background value in the background value holding unit as a new background value;
A difference generation unit that generates a difference between a pixel value of the input image to be processed and the updated background value;
An image processing apparatus comprising: output image generation means for generating a pixel value of an output image by reflecting a pixel value of the input image to be processed according to the degree of difference for each of the pixel positions. . Further comprising background reliability generation means for generating the reliability of the estimated background value based on the selected plurality of pixel values;
The image processing apparatus according to claim 1, wherein the difference generation unit generates the difference according to the reliability.   The image processing apparatus according to claim 2, wherein the dissimilarity generation unit generates the dissimilarity such that the dissimilarity increases as the reliability decreases.   The image processing apparatus according to claim 2, wherein the background update unit updates a background value held in the background value holding unit according to the reliability.   The image processing according to claim 4, wherein the background update unit updates the background value held in the background value holding unit so that the specific gravity of the estimated background value increases as the reliability increases. apparatus.   The image processing apparatus according to claim 1, wherein the background value estimation unit estimates a mode value of the selected plurality of pixel values as a background value. The output image generation means includes
Output image holding means for holding pixel values of the output image;
A composition ratio generating means for generating a composition ratio according to the degree of difference for each of the pixel positions;
For each of the pixel positions, the pixel value of the input image that is the processing target is combined with the pixel value of the output image held in the output image holding unit according to the combination ratio as a pixel value of a new output image The image processing apparatus according to claim 1, further comprising: a composite value calculation unit that is held in the output image holding unit.   The image processing apparatus according to claim 1, wherein the pixel selection unit selects the plurality of pixel values at equal intervals in time.   The image processing apparatus according to claim 1, wherein the pixel selection unit selects the plurality of pixel values with a higher frequency in time as a temporal distance from the input image to be processed is closer.   The image processing apparatus according to claim 1, wherein the frequency of the background value update by the background update unit and the difference generation by the difference generation unit are different from each other. A pixel selection procedure for selecting a plurality of pixel values around the input image to be processed at each of the corresponding pixel positions of the plurality of input images arranged in time series; and
A background value estimation procedure for estimating a background value at the pixel position of the input image to be processed based on the selected plurality of pixel values;
A background update procedure for updating the background value held in the background value holding unit based on the estimated background value and holding the background value in the background value holding unit as a new background value;
A difference generation procedure for generating a difference between a pixel value of the input image to be processed and the updated background value;
An image processing method comprising: an output image generation procedure for generating a pixel value of an output image by reflecting a pixel value of the input image to be processed according to the degree of difference for each of the pixel positions . A pixel selection procedure for selecting a plurality of pixel values around the input image to be processed at each of the corresponding pixel positions of the plurality of input images arranged in time series; and
A background value estimation procedure for estimating a background value at the pixel position of the input image to be processed based on the selected plurality of pixel values;
A background update procedure for updating the background value held in the background value holding unit based on the estimated background value and holding the background value in the background value holding unit as a new background value;
A difference generation procedure for generating a difference between a pixel value of the input image to be processed and the updated background value;
A program causing a computer to execute an output image generation procedure for generating a pixel value of an output image by reflecting a pixel value of an input image to be processed according to the degree of difference for each of the pixel positions. . Pixel selection means for selecting a plurality of pixel values at each of corresponding pixel positions of a plurality of input images arranged in time series,
Background value estimating means for estimating a background value at the pixel position based on the selected plurality of pixel values;
Background reliability generation means for generating reliability of the estimated background value based on the selected plurality of pixel values;
A difference generation means for generating a difference between the pixel value of the input image to be processed and the estimated background value according to the reliability;
An image processing apparatus comprising: output image generation means for generating a pixel value of an output image by reflecting a pixel value of the input image to be processed according to the degree of difference for each of the pixel positions. . A pixel selection procedure for selecting a plurality of pixel values at each of corresponding pixel positions of a plurality of input images arranged in time series, and
A background value estimation procedure for estimating a background value at the pixel position based on the selected plurality of pixel values;
A background reliability generation procedure for generating the reliability of the estimated background value based on the selected plurality of pixel values;
A difference generation procedure for generating a difference between the pixel value of the input image to be processed and the estimated background value according to the reliability;
An image processing method comprising: an output image generation procedure for generating a pixel value of an output image by reflecting a pixel value of the input image to be processed according to the degree of difference for each of the pixel positions . A pixel selection procedure for selecting a plurality of pixel values at each of corresponding pixel positions of a plurality of input images arranged in time series, and
A background value estimation procedure for estimating a background value at the pixel position based on the selected plurality of pixel values;
A background reliability generation procedure for generating the reliability of the estimated background value based on the selected plurality of pixel values;
A difference generation procedure for generating a difference between the pixel value of the input image to be processed and the estimated background value according to the reliability;
A program causing a computer to execute an output image generation procedure for generating a pixel value of an output image by reflecting a pixel value of an input image to be processed according to the degree of difference for each of the pixel positions. .

Images