JP6632134B2 - Image processing apparatus, image processing method, and computer program - Google Patents

Description

  The present invention relates to an image processing device, an image processing method, and a computer program.

  2. Description of the Related Art Conventionally, there has been proposed a technique for generating an image from a free viewpoint other than a camera viewpoint (hereinafter, referred to as a free viewpoint image) for a sports scene or the like. In this technology, based on images taken by multiple cameras, images of virtual viewpoints where they are not arranged are synthesized, and the results are displayed on the screen, so that image viewing from various viewpoints can be viewed. It is possible.

  Accurate extraction of objects by background subtraction is the first step in obtaining high quality free viewpoint video. Conventionally, techniques described in Non-Patent Documents 1 and 2 have been known for background subtraction. Non-Patent Document 1 discloses a technique for automatically extracting an object. Non-Patent Document 2 discloses a technique for accurately extracting an object by the Grabcut method.

Elgammal, Ahmed, David Harwood, and Larry Davis. "Non-parametric model for background subtraction," Computer Vision-ECCV 2000.Springer Berlin Heidelberg, 2000.751-767. Rother, Carsten, Vladimir Kolmogorov, and Andrew Blake. "Grabcut: Interactive foreground extraction using iterated graph cuts." ACM transactions on graphics (TOG). Vol. 23. No. 3. ACM, 2004 Shinji Morita, Kazumasa Yamazawa, Masahiko Terasawa, Naokazu Yokoya: "Network-enabled Remote Monitoring System Using Omnidirectional Image Sensor", IEICE Transactions on Information and Systems (D-II), Vol. J88-D-II, No. . 5, pp. 864-875, (2005.5)

  However, in the technology described in Non-Patent Document 1, it is necessary to manually set various parameters in advance. Therefore, there is no guarantee that the parameters that are troublesome for the user and that are manually set are optimal. Further, the Grabcut method described in Non-Patent Document 2 requires input by a user. Therefore, if the Grabcut method is applied to the extraction of an object from a moving image, an input is required from the user for each frame of the moving image, which is not realistic.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide a background subtraction technique that can achieve both object extraction accuracy and user convenience.

One embodiment of the present invention relates to an image processing device. The image processing apparatus includes: a unit that obtains a first mask obtained by performing a first background difference on a target frame of a moving image in the target frame; and a second background difference between the target frame and the frame. Means for obtaining a second mask obtained by performing the above , means for generating a combined mask by combining the first mask and the second mask, and both of the first background difference and the second background difference Means for generating a reference mask by performing a different third background difference; and at least one of a parameter used in the first background difference and a parameter used in the second background difference, based on the generated reference mask. Means for setting one of them .

  It should be noted that any combination of the above-described constituent elements, and those in which the constituent elements and expressions of the present invention are replaced with each other among apparatuses, methods, systems, computer programs, recording media storing computer programs, and the like, are also included in the present invention. This is effective as an embodiment.

  According to the present invention, it is possible to provide a background subtraction technique that can achieve both the accuracy of object extraction and user convenience.

1 is a schematic diagram illustrating a free viewpoint image distribution system including an image processing device according to an embodiment. FIG. 2 is a block diagram illustrating functions and configurations of the mobile terminal of FIG. 1. FIG. 2 is a block diagram illustrating functions and configurations of the image processing apparatus of FIG. 1. FIG. 4 is a data structure diagram illustrating an example of a parameter holding unit in FIG. 3. FIGS. 5A and 5B are representative screen diagrams of the region of interest setting screen. FIGS. 6A, 6B, and 6C are representative screen diagrams of the weight setting screen. It is a typical screen figure of a free viewpoint image reproduction screen. It is a typical screen figure of an error screen. FIG. 4 is an explanatory diagram for describing parameter setting processing in a parameter setting unit in FIG. 3. FIGS. 10A, 10B, and 10C are explanatory diagrams for explaining the difference in the combination mask due to the difference in the weight α. FIGS. 11A and 11B are graphs for explaining evaluation based on the improved F value. FIGS. 12A to 12F are graphs showing fluctuations in the average pixel intensity of a frame. 2 is a flowchart illustrating a flow of a series of processes in the image processing apparatus of FIG. 1. 14 is a flowchart illustrating a flow of processing in a threshold setting processing step of FIG. 13; 14 is a chart showing a flow of processing in a synthetic mask generation processing step of FIG. 13.

  Hereinafter, the same or equivalent components, members, and processes illustrated in each drawing are denoted by the same reference numerals, and the repeated description will be omitted as appropriate. In each drawing, some of the members that are not important for the description are omitted.

  When extracting an object from a frame of a moving image, the image processing apparatus according to the embodiment generates a mask from the frame based on a background difference in a spatial domain. Since the background difference in the spatial domain is a background difference performed within the frame to be processed, it is hereinafter referred to as an intra background difference. A mask generated by the intra background difference is called an intra mask. The image processing apparatus generates a mask from a frame based on the background difference in the temporal domain in parallel with the intra background difference. Since the background difference in the time domain is a background difference performed between frames including the frame to be processed, the background difference is hereinafter referred to as an inter background difference. The mask generated by the inter background subtraction is called an inter mask. The image processing apparatus generates a combined mask by combining the two generated masks, the intra mask and the inter mask, and uses the generated combined mask for extracting an object. As a result, highly accurate object extraction can be realized while suppressing the necessity of instructions and inputs by the user.

  In the present embodiment, a background difference using a normal mixture model (Gaussian Mixture Model) (see Non-Patent Document 1) is adopted as the intra background difference. In this background subtraction, a background is modeled using a mixture normal distribution (Mixture of Gaussian Distribution, MoG). First, a background frame without objects is prepared, and a mixed normal distribution model of the background is obtained from the background frame. If the pixel of the processing target frame does not belong to the mixture normal distribution model, the pixel is extracted as a foreground. A threshold is used when determining whether or not the item belongs. Hereinafter, this threshold is referred to as an intra threshold (gmm_th).

In the present embodiment, a background difference (see Non-Patent Document 3) in which one normal model is applied to each pixel is adopted as an inter background difference. In this background subtraction, when the coordinates of a pixel in a frame are expressed as (i, j), one normal distribution model having an average value μ _{i, j} and a standard deviation σ _{i, j} is represented by the pixel (i, j). Assign to strength. The average value μ _{i, j} and the standard deviation σ _{i, j} are calculated using a set of intensities of the pixels (i, j) over a plurality of frames included in the moving image as a population. When the absolute value of the deviation of the intensity of the pixel (i, j) of the frame to be processed is smaller than the value obtained by adding a threshold to the standard deviation σ _{i, j} for all the channels Y, U, and V, the pixel ( i, j) are determined to belong to the background. Otherwise, pixel (i, j) is determined to belong to the foreground. Hereinafter, the thresholds for each of Y, U, and V are referred to as inter thresholds (Y_th, U_th, V_th).

  FIG. 1 is a schematic diagram illustrating a free viewpoint image distribution system 110 including an image processing device 200 according to the embodiment. The free viewpoint image distribution system 110 includes a plurality of cameras 116, 118, and 120, an image processing apparatus 200 connected to the cameras, and a mobile phone, tablet, smartphone, HMD (Head Mounted Display), and notebook PC. And a terminal 114. The image processing device 200 and the portable terminal 114 are connected via a network 112 such as the Internet. In the free viewpoint image distribution system 110, for example, a plurality of cameras 116, 118, and 120 arranged indoors image a person 124 standing on a floor 126. The plurality of cameras 116, 118, and 120 transmit the captured images to the image processing device 200, and the image processing device 200 processes the images. The user of the portable terminal 114 designates a desired viewpoint to the image processing apparatus 200, and the image processing apparatus 200 synthesizes an image when the person 124 is viewed from the designated viewpoint (virtual viewpoint), and To the portable terminal 114.

Although FIG. 1 illustrates the case where an indoor person is imaged, the present invention is not limited to this case. For example, when an image of a fitness instructor, an image of a dancer's dance, or an image of a soccer match is taken, the present invention is not limited to this. The technical idea of the embodiment can be applied. Further, instead of the portable terminal 114, a stationary terminal such as a desktop PC, a laptop PC, a TV receiver, and a set-top box may be used. In addition, the distribution form of the image processing apparatus 200 may be such that the whole is downloaded in advance and then reproduced, or may be streaming or progressive. When the distribution mode is not real-time, the image processing apparatus 200 may calculate the average μ _{i, j} and the standard deviation σ _{i, j} of the moving image acquired and held from the camera in advance. When the distribution mode is real-time, the image processing apparatus 200 calculates the average value μ _{i, j} and the standard deviation σ _{i, j} from the moving images obtained up to the present time, and every time a predetermined amount of a new moving image is obtained. May be updated by recalculating those values.

  FIG. 2 is a block diagram illustrating functions and configurations of the portable terminal 114 in FIG. Each block shown here can be realized by hardware or other elements or mechanical devices such as a CPU (Central Processing Unit) of the computer, and is realized by a computer program or the like in software. The functional blocks realized by their cooperation are drawn. Therefore, it will be understood by those skilled in the art referred to in this specification that these functional blocks can be realized in various forms by a combination of hardware and software.

  The portable terminal 114 includes a communication unit 246, a display control unit 216, a display 218, and an input unit 220. The communication unit 246 transmits information specifying the viewpoint specified by the user via the input unit 220 to the image processing device 200 via the network 112. The communication unit 246 acquires a moving image from the image processing device 200 via the network 112. The display control unit 216 controls the display 218 and causes the display 218 to display various screens. The display control unit 216 causes the display 218 to display the moving image acquired by the communication unit 246. The display 218 may be an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. The input unit 220 receives an input from a user. The input unit 220 may be a mouse, a keyboard, a touch panel, buttons, or a remote controller.

  FIG. 3 is a block diagram illustrating functions and configurations of the image processing apparatus 200 in FIG. Each block shown here can be realized by hardware or other elements or mechanical devices such as a CPU (Central Processing Unit) of the computer, and is realized by a computer program or the like in software. The functional blocks realized by their cooperation are drawn. Therefore, it will be understood by those skilled in the art referred to in this specification that these functional blocks can be realized in various forms by a combination of hardware and software.

  The image processing apparatus 200 extracts an object, for example, an image of the person 124 from a moving image obtained from the camera using a synthesis mask, and synthesizes an image of an arbitrary virtual viewpoint based on the extracted object. The image processing device 200 includes a moving image acquisition unit 202, a parameter setting unit 204, a combined mask generation unit 206, an object extraction unit 208, a reset determination unit 210, an update determination unit 212, a parameter update unit 214 , A moving image distribution unit 244, a moving image holding unit 222, and a parameter holding unit 224.

  The moving image acquisition unit 202 acquires a moving image from each of the cameras 116, 118, and 120 connected to the image processing device 200. The moving image acquisition unit 202 stores the acquired moving image in the moving image holding unit 222. The moving image holding unit 222 holds a moving image. The moving image may be a time series of a plurality of frames.

  When starting extraction of an object from a moving image, the parameter setting unit 204 sets parameters used for an intra background difference and an inter background difference. The parameters of the intra background difference include gmm_th, and the parameters of the inter background difference include Y_th, U_th, and V_th. The parameter setting unit 204 includes a region of interest setting unit 226, a reference mask generation unit 228, a weight setting unit 230, a test mask generation unit 232, an improved F value calculation unit 234, and a parameter determination unit 236.

  The region-of-interest setting unit 226 acquires, from the moving image holding unit 222, a reference frame used for generating a reference mask, for example, the first frame of the moving image to be processed. The region-of-interest setting unit 226 generates a region-of-interest setting screen 400 (to be described later with reference to FIGS. 5A and 5B) including the acquired first frame, and transmits the screen data to the mobile terminal 114 via the network 112. Send. The display control unit 216 of the mobile terminal 114 causes the display 218 to display the region of interest setting screen 400 based on the received screen data. The mobile terminal 114 receives designation of a region of interest (Region Of Interest) in the displayed first frame from the user via the input unit 220. The communication unit 246 of the mobile terminal 114 transmits information of the designated region of interest to the image processing device 200 via the network 112. The region of interest setting unit 226 of the image processing device 200 receives the information of the designated region of interest.

  The reference mask generation unit 228 generates a reference mask by performing a background difference different from any of the intra background difference and the inter background difference on the first frame in which the region of interest is specified. The reference mask generation unit 228 generates a reference mask by the Grabcut method (see Non-Patent Document 2) that inputs a first frame and a region of interest specified for the first frame. The reference mask generation unit 228 may accept editing (painting, brushing, etc.) by the user of the reference mask generated by the Grabcut method.

  The Grabcut method requires more time for processing than the intra background difference or the inter background difference, and requires user input, that is, the user to specify a region of interest. However, a more accurate result can be obtained than the background difference. Objective criteria are important in the search for optimal parameters, described below. In the present embodiment, a reference mask is obtained from the first frame by the Grabcut method, and the reference mask is used as a reference when searching.

  The weight setting unit 230 receives designation of a weight α in a calculation of an improved F value described later from a user. The weight setting unit 230 generates a weight setting screen 500 (described later in FIGS. 6A, 6B, and 6C), and transmits the screen data to the portable terminal 114 via the network 112. The display control unit 216 of the mobile terminal 114 causes the display 218 to display the weight setting screen 500 based on the received screen data. The portable terminal 114 acquires a value input or specified by the user via the input unit 220 to the weight setting screen 500. The communication unit 246 of the mobile terminal 114 transmits the obtained value to the image processing device 200 via the network 112. The weight setting unit 230 of the image processing device 200 sets the received value as the weight α.

  The test mask generation unit 232 selects one inter threshold from a set of inter thresholds (Y_th, U_th, V_th) for which an inter background difference can be set. The test mask generation unit 232 generates an intertest mask by performing an inter background difference using the selected inter threshold on the first frame acquired by the region of interest setting unit 226.

…（１）
ここで、重みαは０以上２以下の値であり、重み設定部２３０により設定される。重みαは、オブジェクトの抽出の際のユーザの好みを反映する。適合率（Precision）および再現率（Recall）はそれぞれ以下の式２、式３により算出される。

…（２）

…（３）
ここで、ＴＰ（True Positive）はインターテストマスクおよび基準マスクの両方で背景に属する画素の総数であり、ＦＮ（False Negative）はインターテストマスクでは前景に属するが基準マスクでは背景に属する画素の総数であり、ＦＰ（False Positive）はインターテストマスクでは背景に属するが基準マスクでは前景に属する画素の総数である。 The improved F value calculation unit 234 evaluates the inter-test mask generated by the test mask generation unit 232 as an evaluation target, and sets the precision and recall when the reference mask generated by the reference mask generation unit 228 is regarded as a correct answer. A weighted harmonic average with (Recall) is calculated as an improved F value (Modified F-Measure). The improved F-number is given by Equation 1 below.

… (1)
Here, the weight α is a value of 0 or more and 2 or less, and is set by the weight setting unit 230. The weight α reflects the user's preference when extracting an object. The precision (Precision) and the recall (Recall) are calculated by the following equations 2 and 3, respectively.

… (2)

… (3)
Here, TP (True Positive) is the total number of pixels belonging to the background in both the intertest mask and the reference mask, and FN (False Negative) is the total number of pixels belonging to the foreground in the intertest mask but belongs to the background in the reference mask. FP (False Positive) is the total number of pixels belonging to the background in the intertest mask but belonging to the foreground in the reference mask.

  The test mask generator 232 and the improved F value calculator 234 repeat the selection of the inter threshold and the calculation of the improved F value until all the settable inter thresholds are selected. The parameter determining unit 236 determines the inter threshold so that the improved F value increases. The parameter determination unit 236 extracts a set having the largest improved F value from a set of sets of the inter threshold and the improved F value obtained by the test mask generation unit 232 and the improved F value calculation unit 234. The parameter determination unit 236 determines the extracted set of inter thresholds as optimal inter thresholds (Y_th_opt, U_th_opt, V_th_opt), and registers them in the parameter holding unit 224.

  The test mask generation unit 232 selects one intra threshold from a set of intra thresholds (gmm_th) for which an intra background difference can be set. The test mask generation unit 232 generates an intra test mask by performing an intra background difference using the selected intra threshold on the first frame acquired by the region of interest setting unit 226. The test mask generation unit 232 generates an inter-optimal mask by performing an inter-background difference using an optimal inter-threshold (Y_th_opt, U_th_opt, V_th_opt) held in the parameter holding unit 224 for the first frame. The test mask generation unit 232 synthesizes the intra-test mask and the inter-optimal mask by the same synthesis method as the synthesis method in the synthesis unit 242 described later or a similar method, and generates a synthesized test mask.

  The improved F value calculation unit 234 calculates an improved F value when the synthesized test mask generated by the test mask generation unit 232 is to be evaluated and the reference mask generated by the reference mask generation unit 228 is a correct answer. The test mask generator 232 and the improved F value calculator 234 repeat the selection of the intra threshold and the calculation of the improved F value until all the intra thresholds are selected. The parameter determination unit 236 determines the intra threshold so that the improved F value increases. The parameter determining unit 236 extracts a set having the largest improved F value from a set of sets of the intra threshold value and the improved F value obtained by the test mask generating unit 232 and the improved F value calculating unit 234. The parameter determination unit 236 determines the extracted set of intra thresholds as the optimal intra threshold (gmm_th_opt), and registers the determined intra threshold in the parameter holding unit 224.

  The synthesis mask generation unit 206 generates a synthesis mask by referring to the threshold value stored in the parameter storage unit 224 and applying an inter background difference and an intra background difference to a frame included in a moving image to be processed. . The synthesis mask generation unit 206 includes an intra background difference unit 238, an inter background difference unit 240, and a synthesis unit 242.

  The intra background difference unit 238 refers to the parameter holding unit 224 and specifies an optimal intra threshold (gmm_th_opt). The intra-background difference unit 238 generates an intra-mask by performing an intra-background difference on the frame using the specified intra threshold.

  The inter-background difference unit 240 refers to the parameter storage unit 224 and specifies the optimum inter-threshold values (Y_th_opt, U_th_opt, V_th_opt). The inter-background difference unit 240 generates an inter-mask by performing an inter-background difference on the frame using the specified inter threshold.

The synthesis unit 242 generates a synthesis mask by synthesizing the intra mask generated by the intra background subtraction unit 238 and the inter mask generated by the inter background subtraction unit 240. The combining unit 242 generates a combined mask by performing a logical product (AND operation, &&) between the intra mask and the inter mask for each pixel. When represented by RGB, the pixel value of the background portion of the mask is (0, 0, 0), that is, black, and the pixel value of the foreground portion is (2 ⁿ -1,2 ⁿ −1) ^where n is the sampling bit of the pixel. , 2 ⁿ -1), that is, white. Therefore, the background portion of the composite mask is a background portion of at least one of the intra mask and the inter mask, and the foreground portion of the composite mask is a foreground portion of both the intra mask and the inter mask. In other words, the background portion of the mask is increased by the combination. Thereby, a more accurate and sharp outline of the object is extracted.

The object extraction unit 208 extracts an object from the frame using the synthesis mask generated by the synthesis unit 242.
The moving image distribution unit 244 combines the moving images from the viewpoint specified by the user of the portable terminal 114 using the extraction result of the object extraction unit 208. The moving image distribution unit 244 transmits the synthesized moving image obtained by the synthesis to the portable terminal 114 via the network 112.

…（４）
ここで、Ｍはフレームの幅、Ｎはフレームの高さである。再設定判定部２１０は、現在のフレームから得られた合成マスクの平均画素強度と一つ前のフレームから得られた合成マスクの平均画素強度との差の絶対値｜Δｍｓｋ｜を算出する。再設定判定部２１０は、算出された｜Δｍｓｋ｜が所定の閾値ｔｈ＿ｍｓｋを上回る場合、パラメータの再設定が必要であると判定し、そうでなければ再設定は不要と判定する。 The reset determination unit 210 determines whether or not it is necessary to generate a reference mask by the parameter setting unit 204 and reset parameters using the generated reference mask. The reset determination unit 210 determines whether resetting is necessary using the amount of change in the average pixel intensity of the combined mask generated by the combining unit 242 as an index. The intensity of the pixel (i, j) in the synthesis mask generated by the synthesis unit 242 is denoted as Mout (i, j). The reset determination unit 210 calculates the average pixel intensity Ave_msk of the combined mask by the following Expression 4.

… (4)
Here, M is the width of the frame, and N is the height of the frame. The reset determination unit 210 calculates the absolute value | Δmsk | of the difference between the average pixel intensity of the composite mask obtained from the current frame and the average pixel intensity of the composite mask obtained from the immediately preceding frame. When the calculated | Δmsk | exceeds the predetermined threshold th_msk, the reset determination unit 210 determines that the parameter needs to be reset, and otherwise determines that the reset is unnecessary.

  Since the contents of two consecutive frames are usually very similar to each other, | Δmsk | is small when the error in object extraction is small. If the error of the object extraction is large, | Δmsk | becomes large due to the error. According to the reset determination unit 210, when | Δmsk | is larger than th_msk, it is determined that the error is large, and the parameters are reset. This error detection is performed automatically.

…（５）
更新判定部２１２は、現在のフレームの平均画素強度と一つ前のフレームの平均画素強度との差の絶対値｜Δｉｍｇ｜を算出する。更新判定部２１２は、算出された｜Δｉｍｇ｜が所定の閾値ｔｈ＿ｉｍｇを上回る場合、更新が必要であると判定し、そうでなければ更新は不要と判定する。ｔｈ＿ｉｍｇがゼロに設定される場合、フレームごとに更新が行われる。 The update determination unit 212 determines whether the update of the inter threshold held in the parameter storage unit 224 is necessary. The update determination unit 212 determines whether update is necessary based on the difference in average pixel intensity between frames. The intensity of the pixel at the position (i, j) is denoted by I (i, j). The update determination unit 212 calculates the average pixel intensity Ave of the frame by the following Expression 5.

… (5)
The update determination unit 212 calculates the absolute value | Δimg | of the difference between the average pixel intensity of the current frame and the average pixel intensity of the immediately preceding frame. When the calculated | Δimg | exceeds a predetermined threshold th_img, the update determination unit 212 determines that update is necessary, and otherwise determines that update is unnecessary. If th_img is set to zero, an update is performed for each frame.

  The above-described average pixel intensity of the mask or frame is calculated for each of the Y, U, and V channels, and is compared with a threshold. In particular, Δimg includes Δimg_Y for the Y channel, Δimg_U for the U channel, and Δimg_V for the V channel.

  When the update determination unit 212 determines that the update is necessary, the parameter update unit 214 updates the inter threshold according to the average pixel intensity difference Δimg obtained by the update determination unit 212. The parameter updating unit 214 accesses the parameter holding unit 224, and adds the difference Δimg to the held inter threshold. In particular, Δimg_Y is added to the Y-channel inter threshold Y_th_opt. The same applies to the U channel and the V channel.

  FIG. 4 is a data structure diagram showing an example of the parameter holding unit 224. The parameter holding unit 224 holds gmm_th_opt, which is an intra threshold, and Y_th_opt, U_th_opt, V_th_opt, which are inter thresholds, in association with each other.

  FIGS. 5A and 5B are representative screen diagrams of the region of interest setting screen 400. FIG. The region of interest setting screen 400 includes a frame display region 402, an OK button 404, and a cancel button 406. The frame display area 402 displays the first frame. The user operates the touch panel while viewing the region of interest setting screen 400 shown in FIG. FIG. 5B shows a state in which a rectangle 408 is drawn for the first frame displayed in the frame display area 402. When the OK button 404 is tapped in this state, the portable terminal 114 acquires a region surrounded by the rectangle 408 as a region of interest.

  FIGS. 6A, 6B, and 6C are representative screen diagrams of the weight setting screen 500. FIG. The weight setting screen 500 has a weight setting area 502, a representative mask display area 504, and an OK button 506. The weight setting area 502 is a slider bar, and accepts designation of a weight value by a user. In the representative mask display area 504, a composite mask when the weight value specified by the user in the weight setting area 502 is used is displayed. The user checks the composite mask displayed in the representative mask display area 504 while changing the weight value variously in the weight setting area 502, and selects a weight that gives a composite mask that matches his / her preference. When the OK button 506 is tapped, the portable terminal 114 acquires the value set in the weight setting area 502 at that time as the weight α.

  FIG. 7 is a representative screen diagram of the free viewpoint image reproduction screen 600. The display control unit 216 receives the combined moving image transmitted from the moving image distribution unit 244, and causes the display 218 to display the free viewpoint image reproduction screen 600. The free viewpoint image reproduction screen 600 has a free viewpoint image display area 602, a progressive bar 604, a viewpoint specification area 606, an operation area 608, an ROI specification button 610, and a weight specification button 612. In the free viewpoint image display area 602, a synthesized moving image is displayed. An operation area 608 is an area for performing operations such as playing, pausing, and fast-forwarding the synthesized moving image. The viewpoint designation area 606 is a slider bar for designating the vertical position of the viewpoint. When the ROI designation button 610 is tapped, the portable terminal 114 notifies the image processing device 200 of the fact. Upon receiving the notification, the image processing apparatus 200 starts a process of accepting designation of a region of interest on the region of interest setting screen 400. When the weight designation button 612 is tapped, the portable terminal 114 notifies the image processing device 200 of the fact. Upon receiving the notification, the image processing apparatus 200 starts a process of accepting the designation of the weight α on the weight setting screen 500.

  FIG. 8 is a representative screen diagram of the error screen 800. When determining that the resetting is necessary, the resetting determination unit 210 generates an error screen 800 and transmits the screen data to the portable terminal 114 via the network 112. The display control unit 216 of the mobile terminal 114 displays an error screen 800 on the display 218 based on the received screen data. When the OK button 802 is tapped, the portable terminal 114 notifies the image processing device 200 of the fact. Upon receiving the notification, the image processing apparatus 200 starts a process of accepting designation of a region of interest on the region of interest setting screen 400.

  FIG. 9 is an explanatory diagram for explaining the parameter setting process in the parameter setting unit 204. In the first frame 902, a region of interest 904 is specified. By applying the Grabcut method to this, a reference mask 906 is generated. Various intertest masks are obtained by applying inter background differences using various inter thresholds to the first frame 902. From these inter-test masks, the inter-optimal mask 908 that best matches the reference mask 906 (that is, has the largest improved F value) is selected. Next, an intra test mask obtained by applying an intra background difference using various intra threshold values to the first frame 902 and an inter-optimal mask 908 are synthesized to obtain various synthesized test masks. From these combined test masks, the combined optimal mask 910 that best matches the reference mask 906 (ie, has the largest improved F value) is selected. The inter threshold that gives the inter optimal mask 908 and the intra threshold that gives the combined optimal mask 910 are registered in the parameter holding unit 224 as optimal thresholds. When the reference mask 906 is compared with the combined optimum mask 910, the region between the right leg and the left leg of the person is determined to be the foreground in the reference mask 906, but the combined optimum mask after combining the inter optimal mask 908 is used. At 910, that area is also correctly recognized as the background.

  In the Grabcut method, the outside of the region of interest 904 is treated as a background portion. Therefore, the background portion of the reference mask 906 includes the outside of the designated region of interest 904. Comparing the right foot of the image of the person in the inter-optimal mask 908 with the right foot of the image of the person in the composite optimal mask 910, it can be seen that the latter extracts the contour of the foot more accurately.

  FIGS. 10A, 10B, and 10C are explanatory diagrams for explaining the difference in the combination mask due to the difference in the weight α. FIGS. 10A, 10B, and 10C show combined masks when α = 0.3, 1.0, and 1.7, respectively. As described above with respect to Equation 1, when α is different, the determined optimal inter thresholds (Y_th_opt, U_th_opt, V_th_opt) and the optimal intra thresholds (gmm_th_opt) are different, and therefore, the appearance of the resultant combined mask is different. Specifically, when the weight α is increased, it is possible to reduce a portion that is blurred but not a background but is set as a background. When the weight α is reduced, the outline becomes sharper, but the number of portions which are not the background but become the background increases. The user may set the weight α according to the application or preference while understanding the difference.

  FIGS. 11A and 11B are graphs for explaining evaluation based on the improved F value. FIG. 11A is a graph in which the numbers specifying the inter thresholds (Y_th, U_th, V_th) selected by the test mask generation unit 232 are on the horizontal axis, and the improved F value calculated by the improved F value calculation unit 234 is the vertical axis. It is. The maximum of the improved F-number is obtained at the point denoted by reference numeral 150, and the corresponding inter-threshold is determined as the optimum inter-threshold (Y_th_opt, U_th_opt, V_th_opt). FIG. 11B is a graph in which 1/10 of the intra threshold value gmm_th selected by the test mask generation unit 232 is the horizontal axis, and the improved F value calculated by the improved F value calculation unit 234 is the vertical axis. The maximum of the improved F-number is obtained at a position denoted by reference numeral 152, and the corresponding intra threshold is determined as the optimal intra threshold (gmm_th_opt).

  FIGS. 12A to 12F are graphs showing fluctuations in the average pixel intensity of a frame. FIGS. 12A, 12B, and 12C respectively show a frame number on the horizontal axis, a Y channel, a U channel, and a V axis on a 600-frame moving image obtained by capturing a certain scene with a camera. It is plotted as the average pixel intensity of the channel. FIGS. 12D, 12E, and 12F respectively show a frame number on a horizontal axis, a Y channel, and a U channel on a 600-frame moving image obtained by capturing the same scene with another camera. , V-channel. As can be seen from these graphs, the variation in average pixel intensity between frames is generally not very large. Therefore, the difference between the average pixel intensities between frames can be said to be a good index for determining whether or not the threshold needs to be updated.

The operation of the image processing apparatus 200 having the above configuration will be described.
FIG. 13 is a flowchart illustrating a flow of a series of processes in the image processing apparatus 200. The image processing device 200 acquires one frame from the moving image to be processed held in the moving image holding unit 222 (S12). The image processing device 200 determines whether or not the frame acquired in step S12 is the first frame (S14). When the frame is the first frame (YES in S14), the image processing apparatus 200 executes a threshold setting process (S16) requiring input by the user. The image processing apparatus 200 generates a combined mask from the first frame acquired in step S12 using the inter threshold and the intra threshold set in step S16 (S18). When the first frame is handled, the image processing apparatus 200 skips the following steps S20 and S22. The image processing device 200 calculates Δmsk for the second and subsequent frames using the composite mask generated in step S18 (S20). The image processing apparatus 200 compares Δmsk calculated in step S20 with th_msk to evaluate a mask error (S22). If Δmsk> th_msk (YES in S22), the process returns to step S16. If Δmsk ≦ th_msk (NO in S22), the image processing apparatus 200 outputs the combined mask generated in step S18 (S24). After the output of the composite mask, the image processing apparatus 200 determines whether the frame being handled is the last frame of the moving image to be processed (S26). If it is the last frame (YES in S26), the process ends. If it is not the last frame (NO in S26), the process returns to step S12.

  If the frame acquired in step S12 is not the first frame (NO in S14), the image processing device 200 calculates Δimg (S28). The image processing device 200 compares Δimg with th_img in order to determine whether the threshold needs to be updated (S30). If Δimg> th_img (YES in S30), the image processing apparatus 200 updates the inter threshold (S32). After step S32, the process proceeds to step S18, and the image processing apparatus 200 generates a combined mask using the updated inter threshold. If Δimg ≦ th_img (NO in S30), the process proceeds to step S18 without updating the inter threshold.

  FIG. 14 is a flowchart showing the flow of the process in the threshold setting process step S16 of FIG. The image processing device 200 accepts the designation of the region of interest via the portable terminal 114 used by the user (S36). The image processing device 200 generates a reference mask by the Grabcut method based on the designated region of interest and the first frame (S38). The image processing apparatus 200 selects an inter threshold for testing (S40). The image processing device 200 generates an intertest mask from the first frame using the inter threshold value selected in step S40 (S42). The image processing apparatus 200 calculates an improved F value from the reference mask generated in step S38 and the intertest mask generated in step S42 (S44). If there is an unselected inter threshold value (NO in S46), the process returns to step S40. When all the inter thresholds have been selected (YES in S46), the image processing apparatus 200 determines the inter threshold that gives the maximum improved F value as the optimum inter threshold (S48). The image processing apparatus 200 selects a test intra threshold value (S50). The image processing device 200 generates an intra test mask from the first frame using the intra threshold value selected in step S50 (S52). The image processing apparatus 200 combines the intra-test mask generated in step S52 with the inter-optimal mask corresponding to the optimal inter-threshold determined in step S48 to generate a combined test mask (S54). The image processing apparatus 200 calculates an improved F value from the reference mask generated in step S38 and the combined test mask generated in step S54 (S56). If there is an unselected intra threshold (NO in S58), the process returns to step S50. When all the intra thresholds have been selected (YES in S58), the image processing apparatus 200 determines the intra threshold that gives the maximum improved F value as the optimal intra threshold (S60).

  FIG. 15 is a chart showing a flow of processing in the synthetic mask generation processing step S18 in FIG. The image processing device 200 acquires a frame to be processed (S62). The image processing device 200 executes the intra background difference and the inter on the frame acquired in step S62 in parallel. With the intra background subtraction, the image processing device 200 acquires a background frame (S64), and reads an intra threshold from the parameter storage unit 224 (S66). The image processing device 200 performs an intra-background difference on the frame acquired in step S62 using the intra threshold read in step S66 and the background frame acquired in step S64, and generates an intra mask ( S68). For the inter background difference, the image processing apparatus 200 reads out the inter threshold from the parameter holding unit 224 (S70). The image processing device 200 performs an inter background subtraction on the frame acquired in step S62 using the inter threshold read out in step S70, and generates an inter mask (S72). The image processing device 200 combines the intra mask generated in step S68 with the inter mask generated in step S72 to generate a combined mask (S74).

  In the above embodiment, examples of the holding unit are a hard disk and a semiconductor memory. Further, based on the description in this specification, each unit is realized by a CPU (not shown), a module of an installed application program, a module of a system program, and a semiconductor memory that temporarily stores the content of data read from a hard disk. It will be understood by those skilled in the art referred to herein that this can be achieved.

  According to the image processing apparatus 200 according to the present embodiment, basically, the user can enjoy a free viewpoint video based on a more accurately extracted object simply by first specifying a region of interest. After designating the region of interest, parameters necessary for the intra background difference and the inter background difference are automatically set and updated automatically. Therefore, user convenience is improved.

  For example, even if a user who does not have knowledge of the meaning of the background difference parameter or the background difference itself is requested to set the parameter, it is unlikely that the parameter is appropriately set, and the user will be embarrassed. On the other hand, in the image processing device 200 according to the present embodiment, such parameters are automatically set and updated, so that the user can enjoy the free viewpoint video without having knowledge of the background difference. be able to.

  The image processing apparatus 200 according to the present embodiment generates a combined mask by combining the result of the intra background difference and the result of the inter background difference. Therefore, the level of accuracy required for each of the intra background difference result and the inter background difference result can be reduced, and the processing speed can be increased instead. As a result, it is possible to provide a background difference that can extract an object more accurately while being fast enough to be applicable in real time.

  Further, in image processing apparatus 200 according to the present embodiment, a mask error is automatically detected. Therefore, the user does not need to find the mask error by himself, and the user convenience is improved.

  In the image processing apparatus 200 according to the present embodiment, the quality of the test mask is evaluated using the improved F value. By changing the weight α of the improved F value, the user can adjust the extraction result to his / her preference. Therefore, a highly flexible evaluation method that can reflect user preferences is realized. For example, if the user wants to extract a more accurate contour, α may be set to be smaller than 1. If the user wants to reduce omission of object extraction, α may be set to be larger than 1.

  Further, in the image processing apparatus 200 according to the present embodiment, it is determined whether or not the update of the inter threshold is necessary for each frame, and when it is determined that the inter threshold is required, the threshold is automatically set based on a difference in average pixel intensity between frames. Be updated. Therefore, robustness (Robustness) of object extraction is improved.

  The configuration and operation of the image processing device 200 according to the embodiment have been described above. This embodiment is an exemplification, and it is understood by those skilled in the art that various modifications can be made to the combination of each component and each processing, and such modifications are also within the scope of the present invention.

  In the embodiment, the distribution of the free viewpoint video has been described as an example, but the present invention is not limited to this, and the technical idea according to the present embodiment is applied to analysis of general moving images such as analysis of a moving image obtained from a surveillance camera. May be applied.

  In the embodiment, the case where the necessity of the update of the inter threshold is determined for each frame has been described, but the present invention is not limited to this, and the determination of the necessity of the update may be made for any number of frames, It may be done at random timing. In addition, whether or not at least one of the intra threshold and the inter threshold is required to be updated may be determined based on an arbitrary criterion, not limited to Δimg.

  In the embodiment, the first frame is adopted as the reference frame. However, the present invention is not limited to this, and any frame of the moving image to be processed may be adopted as the reference frame.

  In the embodiment, the case where the image processing apparatus 200 performs the process of generating the combined mask by combining the result of the intra background difference and the result of the inter background difference is described. However, the present invention is not limited thereto. A combining process may be performed. As described above, a mode in which the mobile terminal 114 realizes part or all of the functions of the image processing apparatus 200 is also possible.

  In the embodiment, the case where the intra test mask is generated and the synthesized test mask is generated by synthesizing the intra test mask with the inter-optimal mask has been described. However, the present invention is not limited to this. Then, the intra threshold that gives the maximum improved F value may be determined as the optimal intra threshold. In this case, the inter threshold and the intra threshold are individually optimized for the reference mask.

The following is an example of a program code for determining the optimum inter thresholds (Y_th_opt, U_th_opt, V_th_opt). The optimal intra threshold (gmm_th_opt) may be determined by a similar program code.

  110 free viewpoint image distribution system, 112 network, 114 portable terminal, 200 image processing device.

Claims (11)

Means for obtaining a first mask obtained by performing a first background difference on a target frame of a moving image in the target frame;
Means for obtaining a second mask obtained by performing a second background difference between frames for the target frame;
Means for generating a combined mask by combining the first mask and the second mask;
Means for generating a reference mask by performing a third background difference different from any of the first background difference and the second background difference;
Means for setting at least one of a parameter used in the first background difference and a parameter used in the second background difference based on the generated reference mask .   The image processing apparatus according to claim 1, wherein a background portion of the composite mask is a background portion of at least one of the first mask and the second mask.   The image processing apparatus according to claim 1, further comprising a unit configured to extract an object from the target frame using the generated composite mask. The apparatus further includes means for receiving designation of a region of interest in a reference frame of the moving image from a user,
4. The image processing apparatus according to claim 1 , wherein a background portion of the reference mask includes an area outside the designated region of interest. 5. 5. The parameter setting unit according to claim 1 , wherein the setting unit sets the parameter such that a weighted harmonic average of a precision and a recall when the reference mask is set as a correct answer is large. 6. An image processing apparatus according to claim 1. The image processing apparatus according to claim 5 , further comprising a unit configured to receive designation of a weight in the weighted harmonic mean from a user. The image processing apparatus according to any one of claims 1 to 6 , further comprising: a unit configured to determine whether it is necessary to generate a reference mask and reset parameters using the generated reference mask. The apparatus according to any one of claims 1 to 7 , further comprising: a unit configured to determine whether at least one of a parameter used in the first background difference and a parameter used in the second background difference needs to be updated. Image processing device. The determining means determines whether or not updating is necessary based on a difference in average pixel intensity between frames,
The image processing apparatus further updates at least one of a parameter used in the first background difference and a parameter used in the second background difference according to the difference when it is determined that the update is necessary. The image processing apparatus according to claim 8 , further comprising a unit. Obtaining a first mask obtained by performing a first background difference on a target frame of a moving image in the target frame;
Obtaining a second mask obtained by performing a second background difference between frames for the target frame;
Generating a combined mask by combining the first mask and the second mask;
Generating a reference mask by performing a third background difference different from any of the first background difference and the second background difference;
Setting at least one of a parameter used in the first background difference and a parameter used in the second background difference based on the generated reference mask . A function of obtaining a first mask obtained by performing a first background difference on a target frame of a moving image in the target frame;
A function of obtaining a second mask obtained by performing a second background difference between frames for the target frame;
A function of generating a combined mask by combining the first mask and the second mask;
A function of generating a reference mask by performing a third background difference different from any of the first background difference and the second background difference;
A computer program for causing a computer to implement , based on the generated reference mask, a function of setting at least one of a parameter used in the first background difference and a parameter used in the second background difference .

Images