JP6800681B2 - Image processing equipment, image processing methods and computer programs - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and a computer program for processing a background image having distance information for each pixel.

In recent years, background distance images representing background distance information have been used for many purposes such as mask image generation for object detection and foreground extraction, distance information coding, and distance information correction. Many methods for automatically extracting a background image from a plurality of images have been proposed, but most of them are methods for extracting a background color image from a plurality of color images. That is, there are few that utilize the characteristics of distance information.

Utilizing the characteristics of distance information means that the context of the distance is reflected in the background creation. For example, Patent Document 1 discloses a method of holding a distance value representing the farthest distance in each pixel in a background image for a plurality of distance images input one after another, assuming a fixed-point camera. In this way, by using the context of the distance, for example, even in the case where an animal body invades the screen and stops for a certain period of time, the animal body can be excluded and the background image can be maintained. It will be possible.

In fact, when a background image is generated using a distance image obtained from a distance measurement sensor as a reference, noise included in the distance image may become a problem. In Patent Document 1, it is assumed that the noise is generated. As a countermeasure, when the distance information is in the range of small noise that frequently occurs with respect to the background image obtained at the present time, this distance information is not referred to. By doing so, it is possible to prevent the distance information larger than the original accidentally generated by the small noise from being reflected in the background image.

Japanese Unexamined Patent Publication No. 2011-133320

Patent Document 1 attempts to suppress the influence of small noise by assuming small noise that occurs frequently and not referring to distance information in the noise range with respect to the current background. However, this method alone may not be enough to suppress the influence of noise. For example, when the distance to the background acquired first is a value different from the true value within the assumed noise range, then even if the true value is acquired, that value is regarded as the noise range and the background is updated. Not done. In addition, this method has low resistance to large noise that rarely occurs. For example, if distance information that exceeds the assumed noise range is observed, the background value is updated with that value. If this value is incorrect due to noise, a background image different from the actual one will be generated. Further, if a distant distance is included in the background image, the value cannot be corrected to the front side, and it is difficult to obtain a stable value. The present invention has been made in view of the above problems, and is an image processing device that processes a background image having distance information for each pixel, and an object of the present invention is to increase resistance to noise.

In order to solve the above problems, the image processing apparatus of the present invention includes a holding means for holding a background image having distance information of the background of the shooting space for each pixel, an input means for inputting a distance image of the shooting space, and the above. It has an updating means for updating the distance information of the background image for each pixel based on the distance image, and the updating means sets a range of the distance indicating the background of the background image, and the distance indicated by the distance image. If it is determined that the information is on the front side of the range, the distance information of the background image is not changed, and if it is determined that the distance information indicated by the distance image is within the range, the distance image is displayed. When the distance information of the background image is updated based on the indicated distance information and the distance information of the background image and it is determined that the distance information indicated by the distance image is behind the range, the background image is displayed. It is characterized in that the distance information is replaced with the distance information indicated by the distance image.

According to the present invention, it is an image processing device that processes a background image having distance information for each pixel, and can be made highly resistant to noise.

Block diagram of a distance moving image encoder with reference to a background distance image Flowchart of distance moving image coding means with reference to background distance image Detailed block diagram of the background image update section Flowchart of background image update means Flow chart of background pixel processing in the first embodiment Specific Example of Numerical Processing of Flow Chart for Background Pixel Processing in First Embodiment Flow chart of background pixel processing in the second embodiment Specific Example of Numerical Processing of Flow Chart for Background Pixel Processing in Second Embodiment Specific Example of Numerical Processing of Flow Chart for Background Pixel Processing in Third Embodiment Block configuration diagram inside the computer

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In addition, each embodiment described below is one of the specific embodiments showing an example in which the present invention is specifically implemented.

(First Embodiment)
In the present embodiment, when a distance moving image (continuous distance image) is acquired and the distance images are predicted and encoded one by one according to a time series, the distance image at the previous time encoded / decoded immediately before that is used. An embodiment of referring to the background distance image generated by referring to the decoded distance image will be described. The distance image refers to an image in which a value (distance value) expressing the distance from the camera that shoots the shooting space to the subject is stored in each pixel. The distance image can be triangulated using a plurality of cameras, or acquired by an infrared distance sensor or the like. In addition to the method of directly expressing the distance as a physical quantity, there are various methods of expressing the distance represented by the distance value of the distance image, such as parallax expression and an index for referring to a quantization table representing the distance. It can be used in any distance expression method. In the present embodiment, the distance value represents a distance in mm for the sake of simplicity. Further, it is desirable that the background image is a distance image showing the distance to the closest subject among the still subjects in the moving image. Hereinafter, unless otherwise specified in the present embodiment, the moving image represents a distance moving image and the image represents a distance image. The image processing apparatus of this embodiment is composed of blocks 101 to 106 of FIG. FIG. 1 is a block diagram of a distance moving image coding device using a background distance image as a reference. The elements constituting the present embodiment will be described below together with the flowchart of FIG. 2 showing the processing procedure of the block of FIG.

The image input unit 101 acquires input images one after another in image units (S201), and sequentially sends them to the prediction coding unit 102.

The predictive coding unit 102 refers to the background image from the background image storage unit 106 for the decoded image, which is the image immediately before being decoded from the decoded image storage unit 105, and performs predictive coding of the input image (S202). ). Note that the prediction coding is a method of generating a prediction image of an input image and encoding the difference between the input image and the prediction image. This includes, for example, H. There is a motion compensation prediction used in 264. At this time, by using the background image, it is possible to obtain information that is not in the image acquired immediately before, such as when a part hidden in the image acquired immediately before appears in the input image due to the movement of the subject in the screen. .. As a result, the accuracy of the prediction is increased, and the difference between the predicted value and the true value is reduced, so that the coding efficiency is increased. When the predictive coding is completed, the generated code is decoded (S203), and the decoded input image is written in the decoded image storage unit 105 (S204).

The code output unit 103 writes the code generated by the predictive coding unit to the external storage device or transmits it to the communication path (S205).

The background image updating unit 104 updates the background image of the background image storage unit 106 with reference to the decoded image from the decoded image storage unit (S206).

Next, the details of the background image updating unit are shown in FIG. The image input unit 301 inputs the decoded image of the decoded image storage unit 105 of FIG. At this time, a signal indicating whether or not the input image is the first image of the continuous image group for generating the background image is also input. When predictive coding of a moving image as shown in the present embodiment is performed, it is common to insert an image called an intraframe that does not refer to another decoded image at regular intervals. This enables playback from the middle of the moving image. Since there must be no dependency between the images before and after the intra frame, it is necessary to initialize the background image each time the intra frame is input even when the background image is generated.

When it is determined (S401) that the input image is the first image of the image group, the background image initialization unit 302 initializes the background image of the background image storage unit 106 using the input image (S402). ).

If the input image is not the first image, the background pixel determination unit 303 and the background pixel update unit 304 process the background image for each pixel (S403). In FIG. 4, N represents the number of pixels of the image, and n represents the index when the images are referenced in the order of raster scan.

Further, the details of the background pixel processing S403 are shown in FIG. The background pixel processing S403 is executed by the background pixel determination unit 303 and the background pixel update unit 304. When the distance value is input, the background pixel determination unit 303 determines the input distance value as one of the background, the foreground, and the new background (S501). According to the determination, the background pixel update unit 304 updates the background pixel S502. The new background represents a case where the background behind the moving object newly appears due to the movement of the moving object that was provisionally treated as the background by resting on the screen.

Here, the concept of background image generation in consideration of noise will be described as described at the beginning of this embodiment. Therefore, it is first necessary to assume what kind of distribution the actually measured values will occur with respect to the true values that accurately represent the background. For example, if a Gaussian distribution centered on the true value is assumed, the average of the pixels determined to be the background may be used as the background image. Further, it is sufficient to acquire the variance σ ^{2 of the} input image in advance for the background range and set the range to ± 3σ with respect to the provisional background value. If the acquired image can be used as it is for background image generation, only ranging noise needs to be considered. However, it should be noted that in the present embodiment, it is necessary to additionally define the background range including the noise due to coding.

For example, a predicted image of the distance image input by motion compensation prediction is generated, and the difference image is near-lossless coded called JPEG-LS (guaranteeing that the error of the pixel value due to coding is equal to or less than the parameter δ). Consider the case of encoding using standard technology. At this time, δ may be added to the noise range 3σ originally included in the distance image, and ± (3σ + δ) may be set in the noise range. By doing so, it is possible to adjust appropriate background image parameters according to the coding parameters.

Next, an example of specific numerical processing is shown. It is assumed that the input image is I, and the background image storage unit holds the background image B and the number of updates C of the background. I, B, and C are vectors having N elements each.

Initialization of background image S402 initializes the background image with the input image I. That is, C ← 1 (all elements are 1) and B ← I.

FIG. 6 shows a specific flowchart of the processing S403 of the nth pixel of the background image. However, in FIG. 6, the index n is omitted. The determination of the background pixel, a range of ± (3σ + δ) _{I n} is relative to _{B n,} i.e. a _{δ 1 = δ 2 = 3σ +} δ,
_{B n -δ 1 ≦ I n ≦} B n + δ 2
If it is, it is the background, if it is closer than this range (front side), it is the foreground, and if it is far (back side), it is the new background. If it is determined to be the background, the background image is updated in S602. The update formula at that time is as shown in FIG. If it is determined that the new background, replaced with the input _{I n} the distance value _{B n} of the background as S603. With the above configuration, it is possible to generate a robust background image in consideration of the noise distribution model included in the image.

Each part of the block diagram shown in FIG. 1 may be configured by hardware, but may be implemented as software (computer program). In this case, this software will be installed in the memory of a general computer such as a PC (personal computer). Then, when the CPU of this computer executes the installed software, this computer realizes the functions of the above-mentioned image processing device. That is, this computer can be applied to the above-mentioned image processing apparatus. An example of a computer hardware configuration applicable to the image processing apparatus of the first embodiment will be described with reference to the block diagram of FIG.

The CPU 2001 uses the computer programs and data stored in the RAM 2002 and the ROM 2003 to control the entire computer and execute each of the above-described processes described as those performed by the image processing apparatus.

RAM 2002 is an example of a computer-readable storage medium. The RAM 2002 has an area for temporarily storing computer programs and data loaded from the external storage device 2007, the storage medium drive 2008, and the network interface 2009. Further, the RAM 2002 has a work area used by the CPU 2001 to execute various processes. That is, the RAM 2002 can appropriately provide various areas. The ROM 2003 is an example of a computer-readable storage medium, and stores computer setting data, a boot program, and the like.

The keyboard 2004 and the mouse 2005 can be operated by a computer operator to input various instructions to the CPU 2001. The display device 2006 is composed of a CRT, a liquid crystal screen, or the like, and can display the processing result by the CPU 2001 with images, characters, or the like.

The external storage device 2007 is an example of a computer-reading storage medium, and is a large-capacity information storage device typified by a hard disk drive device. The external storage device 2007 stores an OS (operating system), computer programs and data for realizing each process shown in FIG. 1 in the CPU 2001, the above-mentioned various tables, a database, and the like. The computer programs and data stored in the external storage device 2007 are appropriately loaded into the RAM 2002 according to the control by the CPU 2001, and are processed by the CPU 2001.

The storage medium drive 2008 reads out computer programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and outputs the read computer programs and data to an external storage device 2007 or RAM 2002. It should be noted that a part or all of the information described as being stored in the external storage device 2007 may be recorded in this storage medium and read by the storage medium drive 2008.

The I / F 2009 is an interface for inputting a color image, a distance image, etc. from the outside, and if shown as an example, it is a USB (Universal Serial Bus). 2010 is a bus connecting the above-mentioned parts.

In the above configuration, when the power of the computer is turned on, the CPU 2001 loads the OS from the external storage device 2007 into the RAM 2002 according to the boot program stored in the ROM 2003. As a result, the information input operation can be performed via the keyboard 2004 and the mouse 2005, and the GUI can be displayed on the display device 2006. When the user operates the keyboard 2004 or the mouse 2005 and inputs the start instruction of the coding application or the decoding application stored in the external storage device 2007, the CPU 2001 loads the program into the RAM 2002 and executes it. As a result, the computer functions as an image processing device.

The coding application program executed by the CPU 2001 basically includes functions corresponding to each part of FIG. Similarly, the decoding application program includes functions corresponding to each part of FIG. Here, the image processing result is stored in the external storage device 2007. It is clear from the following description that this computer can be similarly applied to the image processing apparatus according to each subsequent embodiment.

(Second Embodiment)
In the first embodiment, when a value significantly different from the original distance appears in the range to be determined as the new background due to a large noise, the background distance value is replaced by this value, and a background image significantly different from the actual situation is generated. There was a problem. Therefore, a method for dealing with such a problem will be described.

The constituent blocks of this embodiment are the same as those of the first embodiment, but the detailed processing flow of the background pixel processing (S403) in the flowchart of FIG. 4 is different. The flowchart shown in FIG. 5 is replaced with FIG. 7. In FIGS. 5 and 7, S701 and S501 and S702 and S502 correspond to each other. The detailed processing flow of S403 of the present embodiment will be described below.

When the background pixel determination S701 determines that the background is new, this value indicates whether the new background has really appeared or whether the new background has not appeared but is farther than the distance value of the current background due to noise. A process S703 for determining whether a value has been observed has been added. Here, if it is determined that the value is normal, the background pixel is updated S702, and if it is determined that the value is incorrect, the background pixel is not updated.

Next, an example of specific numerical processing is shown. It is assumed that the input image is I, and the background image storage unit holds a background image B, a background update number C, and a new background continuous number F indicating the number of consecutive appearances of the new background. Like I and B, F is also a vector having N elements, and corresponds to the pixels of the distance image. Initialization of background image S402 initializes the background image with the input image I.
That is, C ← 1, B ← I, and F ← 0.

A specific flowchart of the background pixel processing S403 is shown in FIG. S801 corresponds to the background pixel determination S701, S802, and S803, and the background pixel update S702 and S804 correspond to the measurement error determination S703. If it is determined by the determination S801 of the background pixel and the _{B n + δ 2 <I n} ( new background), determination S804 of measurement error. In the determination of measurement error, if the new background continuous number F is ω times or more given as a parameter, the distance value of the background image is replaced with the input distance value as YES, and if it is less than ω, the update is not performed as NO. The update of the new background continuous number F _n ends by adding 1 (S806) when the input value is determined to be a new background and determining a measurement error, and resetting to 0 (S805) otherwise. If ω = 0 in this configuration, the behavior is the same as that in the first embodiment.

By doing so, the replacement with the new background can be executed only when the new background is continuously determined for a certain period of time, and the background image can be updated with higher reliability.

With the above configuration, it is possible to realize an image processing device that generates a background image having enhanced robustness to large noise in addition to small noise.

(Third Embodiment)
In the first embodiment, the discussion proceeded ignoring the influence of the rounding error associated with the generation of the image. However, practically, it is desirable that the background image expressing the background has the same format as the reference distance image, and since these distance images are often expressed by integers, they are naturally affected by the integer rounding error. .. In this embodiment, a rounding error caused by background image generation and a method for reducing the rounding error will be described.

In the first embodiment, in S602, B ← (C _n B _n + I) / (C _n +1) was set, but if the condition that B is an integer is provided, rounding is necessary. Accumulation of values that are truncated (raised) at this time may become a problem. Therefore, a method of reducing this problem will be described by preparing a buffer E for storing a rounding error of several bits in the background distance information. FIG. 9 shows a specific flow of numerical processing for processing the background pixels.

Assuming that the bit depth of E is ^d , the number of gradations r is 2 ^d . Then, S902 updates the background image by holding a rounding error in E as follows. _{That, b ← (C n B n} + I n) / (C n +1) + E n / r, B n ← ROUND (b), C n ← C n + 1, E n ← FLOOR (r (b-B n) ).

However, in the above equation, ROUND (X) represents the value rounded to X, and FLOOR (X) represents the maximum integer not exceeding X. When the above E is calculated, the rounding error b-ROUND (b) takes a value of -0.5 to 0.5, so the FLOOR (r (b-B _n )) is -2 ^d-1 to 2 ^d-1. It does not cause overflow when entering -1 next E _n. By reflecting the rounding error information held here at the next update, the error due to integerization can be reduced. At this time, if d = 0, the behavior is the same as that of the first embodiment.
This is the end of the description of this embodiment. With the above configuration, it is possible to generate a background image robust against noise as in the first embodiment. Further, the rounding error due to the update process can be suppressed to a small value.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

Claims (9)

A holding means for holding a background image having distance information of the background of the shooting space for each pixel,
An input means for inputting a distance image of the shooting space and
It has an updating means for updating the distance information of the background image for each pixel based on the distance image.
The updating means sets a range of distances indicating the background of the background image.
When it is determined that the distance information indicated by the distance image is on the front side of the range, the distance information of the background image is not changed.
When it is determined that the distance information indicated by the distance image is within the range, the distance information of the background image is updated based on the distance information indicated by the distance image and the distance information of the background image.
An image processing apparatus characterized in that when it is determined that the distance information indicated by the distance image is behind the range, the distance information of the background image is replaced with the distance information indicated by the distance image. When the updating means determines that the distance information indicated by the distance image is behind the range, it determines whether or not the distance information indicated by the distance image is a measurement error, and determines that the measurement error. If it is determined, the distance information of the background image is not changed, and if it is not determined that the measurement is incorrect, the distance information of the background image is replaced with the distance information indicated by the distance image. The image processing apparatus according to claim 1. 2. The updating means is characterized in that it determines whether or not the measurement error is based on the number of times that the distance information indicated by the distance image is continuously determined to be on the back side of the range. The image processing apparatus according to. A rounding means for rounding the value obtained as a value for updating the distance information of the background image to an integer, and
The image processing apparatus according to any one of claims 1 to 3, further comprising a second holding means for quantizing and holding a rounding error caused by the rounding of the rounding means. The image processing apparatus according to any one of claims 1 to 4, further comprising a coding means for predictively coding the distance image using the background image as a reference for predictive coding. The image processing apparatus according to claim 5, wherein the coding means sets a noise range according to a magnitude of ranging noise when measuring the distance image and a parameter of predictive coding. .. The image processing apparatus according to claim 6, wherein the coding means sets a range of the noise according to the magnitude of the ranging noise and a coding parameter. The holding step is a holding step of holding a background image having distance information of the background of the shooting space for each pixel.
The input means is an input process for inputting a distance image of the shooting space, and
The updating means has an updating step of updating the distance information of the background image for each pixel based on the distance image.
In the update step, a range of distances indicating the background of the background image is set.
When it is determined that the distance information indicated by the distance image is on the front side of the range, the distance information of the background image is not changed.
When it is determined that the distance information indicated by the distance image is within the range, the distance information of the background image is updated based on the distance information indicated by the distance image and the distance information of the background image.
An image processing method characterized in that when it is determined that the distance information indicated by the distance image is on the back side of the range, the distance information of the background image is replaced with the distance information indicated by the distance image. Computer,
A holding means for holding a background image having distance information of the background of the shooting space for each pixel,
An input means for inputting a distance image of the shooting space and
It has an updating means for updating the distance information of the background image for each pixel based on the distance image.
The updating means sets a range of distances indicating the background of the background image.
When it is determined that the distance information indicated by the distance image is on the front side of the range, the distance information of the background image is not changed.
When it is determined that the distance information indicated by the distance image is within the range, the distance information of the background image is updated based on the distance information indicated by the distance image and the distance information of the background image.
When it is determined that the distance information indicated by the distance image is on the back side of the range, the image processing device functions as a feature of replacing the distance information of the background image with the distance information indicated by the distance image. Computer program.

Images