JP2006190207A - Image generating method, device and program, image processing method, device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To present an object image hidden in a moving image. <P>SOLUTION: A series of image frames are input, region information showing a region occupied by an object in the series of image frames is input, a target point in a first image frame of the series of image frames is input, and a hidden object region of the object corresponding to the target point is estimated in accordance with the region information, and a brightness value for the estimated hidden object region is found from the series of image frames to generate an image frame to be presented. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image generation method for estimating and presenting a concealment area of a subject when there is a subject concealed by another object in a part of a video scene, for example, in a moving image captured by a two-dimensional camera. The present invention relates to an image generation apparatus, an image generation program, an image processing method, an image processing apparatus, and an image processing program.

  There is a demand to view a subject hidden in a captured video scene.

In realizing this, conventionally, as described in Non-Patent Document 1 below, for example, a large number of videos including areas concealed in advance are captured and data corresponding to a three-dimensional space is stored. When there is an instruction to change the position (that is, the viewpoint) in the dimensional space, an image in the three-dimensional space with the viewpoint changed according to the instruction is presented.

The general procedure of the image presentation process described in Non-Patent Document 1 below is as follows. In order for the user to see the concealed area, first, a user instruction to change the viewpoint in the three-dimensional space is accepted. In other words, the viewpoint is changed so as to reveal the hidden area. In accordance with the received instruction, the image presentation processing program generates a two-dimensional display image based on the held three-dimensional image information. The generation of the two-dimensional display image is performed by searching for an image closest to the viewpoint instructed by the user from multi-viewpoint images taken in advance and performing interpolation as necessary. By presenting the generated two-dimensional display image, the user can see the hidden area in the three-dimensional image.
SE Chen, "QuickTime VR: an image-based approach to virtual environment navigation", Proceedings of the 22nd annual conference on Computer graphics and interactive techniques (ACM SIGGRAPH 95), pp.29-38, 1995.

  When presenting an object hidden in a three-dimensional space as an image, the conventional technique as described in Non-Patent Document 1 is effective only when image information corresponding to an arbitrary viewpoint is already obtained. is there. However, an image such as that captured by a two-dimensional moving image capturing camera has a fixed viewpoint or lacks viewpoint information. That is, an image corresponding to an arbitrary viewpoint in the three-dimensional space is not always prepared. Therefore, it is inherently impossible to reconstruct an image to be presented in accordance with a viewpoint change instruction from the user.

  Further, it is demanded to realize a technique capable of presenting a concealed subject as an image with respect to an image captured by the above-described two-dimensional moving image capturing camera (for example, a security camera).

  Therefore, the present invention uses a hidden subject as an image even when the viewpoint is fixed or the viewpoint information is missing, such as an image captured by a two-dimensional moving image capturing camera. An object is to provide an image generation method, an image generation apparatus, an image generation program, an image processing method, an image processing apparatus, and an image processing program for enabling presentation.

  An image generation method according to an aspect of the present invention includes: a step of inputting a series of image frames; a step of inputting region information indicating an area occupied by an object in the series of image frames; A step of inputting an attention point in one image frame, a step of estimating a concealing object region of an object corresponding to the attention point based on the region information, and a luminance value of the estimated concealing object region Generating a presentation image frame by obtaining from the image frame.

  According to the present invention, a subject image concealed in a moving image can be presented.

(Definition of terms)
In describing embodiments of the present invention, the following terms are defined.

  A subset of the two-dimensional Euclidean metric space is referred to as a “screen”. A unit area (two-dimensional set included in the screen) constituting the screen is referred to as a “pixel”. The pixels are, for example, regions in which the screen is divided at equal intervals on the left and right and top and bottom. The method of dividing the screen is not limited to this. For example, the screen may be distorted or may be divided into a honeycomb shape using a regular hexagon. A vector associated with a pixel is referred to as a “luminance value”. The luminance value corresponds to, for example, a three-dimensional RGB value or YUV value in a digital camera, and represents a color. The entire screen in which the luminance value is associated with each pixel is referred to as “image” or “still image”. A plurality of images arranged in time series are referred to as “moving images”. One image in the moving image is referred to as an “image frame”. That is, a moving image is composed of a series of image frames.

  Determining unknown information by image processing based on input information or the like is referred to as “estimation”. Data that distinguishes whether each pixel in the image is the subject or the background (for example, binary values of 0 and 1 or decimal values in the range of 0 to 1) is referred to as a “mask image”. In addition, data indicating which region each pixel in the image belongs to (for example, data assigned region numbers of 1, 2, 3, 4,..., N corresponding to N regions for each pixel). This is called “label image”.

  FIG. 1 is a block diagram of an image generation apparatus according to an embodiment of the present invention. As illustrated in FIG. 1, the image generation apparatus 100 includes a moving image storage unit 101, a region information storage unit 102, a hidden region estimation unit 103, and an image generation unit 104. A display device 105 and a mouse 106 are connected to the image generation device 100. The moving image storage unit 101 stores a series of image frames of a moving image input via the moving image input unit 107. The area information storage unit 102 stores area information of an object input via the area information input unit 108. The area information of the object is area information indicating an area occupied by the object in each of a series of image frames of the input moving image.

  The user can input a point of interest in the first image frame in a series of image frames by operating the mouse 106 while observing the display device 105. The hidden area estimation unit 103 estimates the hidden object area of the object corresponding to the input point of interest based on the area information stored in the area information storage unit 102. The image generation device 100 generates an image frame for presentation by obtaining the estimated luminance value of the hidden object region from the moving image data stored in the moving image storage unit 101. The generated presentation image frame is displayed by the display device 105, for example.

  The image generation apparatus 100 may include a region information estimation unit that estimates region information of an object based on a series of image frames stored in the moving image storage unit 101. The estimation of the object area information will be described in detail later.

  Note that the present invention can also be implemented as a program that causes a computer to function as the image generation apparatus 100. In this case, the program according to the present invention is stored in a program storage device in the computer. The program storage device is composed of, for example, a nonvolatile semiconductor storage device or a magnetic disk device. The computer can be caused to function as the image generation apparatus 100 by being read into a random access memory (RAM) under the control of a CPU (not shown) and executed by the CPU. Note that an operating system that manages various computer resources and provides a file system, various communication functions, a graphical user interface (GUI), and the like is also installed in this computer.

  An overview of an image generation method according to an embodiment of the present invention will be described with reference to FIG.

  In a moving image, an image of a hidden area is often included in another frame. An example of this is shown in FIG. FIG. 2 shows an image frame of a moving image in which the subject 201 and the obstacle 202 with respect to the subject 201 when viewed from the viewpoint are simultaneously photographed. In the center frame of FIG. 2, the vicinity of the center of the subject 201 is blocked by the obstacle 202 and cannot be seen. However, the part that cannot be seen in the center frame is visible in the left and right frames. For example, when the central frame is a frame of interest, the image generation method according to the embodiment of the present invention estimates a hidden object region that cannot be seen by the obstacle 202 or the like in the central frame from an image that is visible in another frame. The presentation image frame is generated by obtaining the luminance value of the hidden object region for the central frame.

(Processing procedure when object area information is known)
When object area information is given to a moving image and it can be used, the object area information is input to the apparatus 100 via the area information input unit 108 and stored in the area information storage unit 102. Hereinafter, the processing procedure when the object region information is known in this way will be described with reference to the flowchart shown in FIG. Specifically, the object area information includes data representing an area where an object exists in the image. For example, the object area information of the subject 201 in the left frame shown in FIG. 2 can be shown as a black area 701 in FIG.

  As shown in FIG. 3, first, an input of a point of interest, for example, an input of object designation by a user is accepted (S101). As described above, the attention point can be input using the mouse 106 or the like. Next, based on the object area information stored in the area information storage unit 102, for the object (designated object) including the point of interest, the concealed object area is estimated for a plurality of frames (S102). The image of the concealed object region in the frame of interest is obtained from the image of the object region of other frames, and the image of the frame of interest is updated (S103). Finally, the updated attention frame image including the hidden object region is displayed (S104). Note that in S104, the entire frame of interest including the subject 201 may be displayed, or an image consisting only of the subject 201 with the updated luminance value of the image portion corresponding to the concealed object region may be extracted and presented. Also good.

(Preview display)
It is preferable to present feedback (preview) to the user when selecting an image by designating a point of interest using object region information. This will be described with reference to the flowchart of FIG. Consider the case where the user moves the mouse cursor 301 over an object in the image as shown on the left side of FIG. 6 (not clicked). Here, the object region corresponding to the mouse cursor 301 can be known by referring to the object region information (S601). Which region is about to be selected by the mouse cursor 301 is displayed on the screen as a schematic shape mark 401 as shown on the right side of FIG. 6 (S602). This is applicable even when the object region information includes only an outline like MPEG-7 metadata. Even when object area information is not given, the same processing can be realized by obtaining individual object areas before presenting an image to the user.

  (Modification) In this embodiment, an operation example using the mouse 106 is used. However, for the operation to be determined by moving the mouse cursor 301, a keyboard, a joystick, or a remote controller (moving up / down / left / right, keyboard buttons for determination) ), Voice recognition (up / down / left / right, instructing the decision by voice, if the object is known, prepare the object order in advance and move the mouse cursor up / down / left / right to move to the next object. Etc.) may be used.

(Hidden area estimation method)
A specific processing procedure in S102 (estimation of the object region corresponding to the point of interest) in FIG. 3 will be described with reference to the flowchart in FIG. When the object area information is given, the object area for each frame is known as indicated by reference numeral 801 in FIG. 8, but the hidden area is an unknown area because it is treated in the same manner as the background. For example, it is not known at this time that the lower part of the subject 801 is missing in FIG. However, it is known that the subject 801 in FIG. 8 and the subject in another frame (subject 701 in FIG. 4) are the same subject. Therefore, given the shape 801 (S1501) of the target frame and the shape (S1502) of the reference frame, the subject 701 is subjected to, for example, affine transformation (deformation by linear transformation such as translation and rotation, see FIG. 9). (S1503) That is, a large amount of deformation area information is prepared (for example, translation in the range of −16 to +16 pixels, rotation from −15 degrees to +15 degrees, and size change from 0.80 to 1.20 are separately defined. A pattern changed at intervals is prepared), and the similarity H is calculated by comparing each deformed subject and the subject 801 (S1504), and the one with the closest similarity H to the subject is searched (S1505). Thereby, affine transformation parameters for associating the subject 701 in FIG. 4 with the subject 801 in FIG. 8 are obtained.

  As described above, the affine transformation parameter is a parameter representing the transformation from the reference frame region to the current frame region. For example, “the previous frame is shifted by 1 pixel to the left and 4 pixels downward, and rotated 10 degrees to the left. This is a parameter that specifically specifies the affine transformation (reduction of other things by 0.9). According to the affine transformation parameters obtained as described above in the present embodiment, it can be determined which part of the reference frame corresponds to the part missing in the current frame. That is, the luminance value (copy source) in the reference frame can be determined.

  For example, Reference 1 (Haifeng Xu et al., “Automatic Moving Object Extraction for Content-Based Applications”, IEEE Transactions on Circuits and Systems for Video Technology, pp. 796-812, vol.14, no.6, June 2004.) or the number of pixels in different regions between the deformed subject and the subject 801.

  Here, the extended Hausdorff distance will be described. First, the distance between points is defined as the Euclidean distance or the city area distance. At this time, the distance between one point x on the region X and the region Y is “the minimum value when the distances from the point x to all the points y on the region Y are calculated”. At this time, the extended Hausdorff distance from the region X to the region Y is the Kth (0) when the “distance between one point x on the region X and the region Y” is obtained for N points on the region X. ≦ K ≦ N). The extended Hausdorff distance from the region X to the region Y and the extended Hausdorff distance from the region Y to the region X are not necessarily the same, and different values may be obtained. In particular, when one region X is included in the other region Y, the distance from the region X (smaller region) to the region Y (larger region) is zero. Using this extended Hausdorff distance (with direction), for example, the maximum value of these two distances or one of the values is used as the similarity.

  A black area shown in FIG. 9 represents a label corresponding to the object area. When the frame having the closest similarity to the target frame (X03 in FIG. 9) is searched from the reference frame X01 and the reference frame X02 and overlapped with the image of X03 (OR region is calculated), X04 and X05, respectively, An image in which a part of the subject is missing can be generated. When these are overlapped (OR area is calculated), the shape (X06) of the subject before being hidden by the obstacle can be found. A portion that exists in the obtained shape X06 but does not exist in X03 is set as a target for subsequent restoration. In this example, the OR region of all reference frames is calculated. However, if there are many shapes (shapes such as X04 and X05) that overlap the target frame and individual reference frames, the calculation of all these OR regions is performed. This increases the possibility of including noise and the like. In this case, a threshold value is determined in advance, and for each pixel, the number of times a shape in which the target frame and individual reference frames are overlapped is counted. Only when the number is equal to or greater than the threshold value, the object It is preferable to determine that the region is a region because the influence of noise can be suppressed.

(Another method for estimating hidden areas)
In addition, in the estimation of the object area in S102 (estimation of the object area corresponding to the point of interest), the method of using the similarity based on the extended Hausdorff distance to obtain the affine transformation parameters has been described. For example, Reference 2 (Kaneko) Toshimitsu, Osamu Hori, “Robust Object Tracking Method Using Multiple Block Matching of Small Areas”, IEICE Transactions D-II, Vol. J85-D-II, No. 7, pp. 1188-1200, The parameters of the affine transformation can also be estimated by using the method described in July 2002). Further, the type of deformation is not limited to affine transformation. For example, in addition to the second-order transformation, that is, the affine deformation, a transformation considering the square of x, the square of y and the product of x and y may be used.

(Method of obtaining the brightness value of the image of the missing area)
A specific processing procedure in S103 (estimation / update of the object region of the image frame) in FIG. 3 will be described. In order to obtain the luminance value of the image of the missing region, only the image of the region not included in 801 but included in the deformed subject among the regions associated in step S102 may be overwritten on the original image.

  However, if overwriting is performed as it is, unnaturalness remains in the vicinity of the boundary between the subject 801 and the deformed subject. This unnaturalness can be reduced by blurring the boundary portion in the spatial direction or the time direction. “Blur in the spatial direction” means, for example, replacing the luminance value of each pixel with an average value of luminance values of 3 × 3 pixels including the pixel for one pixel around the boundary between the subject 801 and the deformed subject. It is. As a result, the vicinity of the boundary is moderately blurred, so that unnaturalness due to overwriting is reduced. “Blur in time direction” means, for example, that the luminance value of the subject 801 is estimated in a similar manner in a plurality of frames to obtain a plurality of deformed subjects, and the subject 801 and those deformed subjects (hidden by each pixel). The luminance value is obtained by calculating an average value (for a portion not present).

(Method 1 when object area information is not given: When the background is stationary)
The processing procedure in the case where the object area information is known has been described above. Next, the case where the object area information is not given because it is not given will be described. Even when the object area information is not given, a step of estimating the object area information is added before S102 in the flowchart of FIG. Here, when it is assumed that the background is still (this is an appropriate assumption for a video photographed by a surveillance camera, for example), object information is estimated in three steps as described below, for example.

Step 1: Moving object detection
Step 2: Matching the same object in multiple image frames
Step 3: Calculation of object area information in each frame
Steps 1 and 2 include, for example, Reference 3 (Hiroyoshi Fujiyoshi, Yasuyoshi Enomoto, Osamu Hasegawa, Takeo Kanade, "Activity Monitoring-Summary of Outdoor Monitoring Video and Display / Search System on WWW-" 7th Image Sensing Symposium. The method described in the collection of lecture papers, pp. 423-428, June 2001) can be used. In the method of Reference 3, a moving image is input and a rectangular area including an object is obtained. A moving object is detected on the assumption that the background is stationary, and the same object is associated by object tracking based on the Kalman filter. At this time, a three-dimensional tube-like region is obtained for each object when the moving image (spatio-temporal image) is regarded as a three-dimensional image. In Step 3, Reference 4 (Hidenori Takeshima, Takashi Ida, Osamu Hori, Nobuyuki Matsumoto, “Extraction of Object Contour Using Self-Similarity of Spatio-Temporal Image”, IEICE Technical Report (Science Technical Report) OIS 2003- 37, pp. 39-44, Sep. 2003.) or Reference 5 (T. Yoshida and T. Shimosato, “Motion Image Segmentation Using 3-d Watershed Algorithm”, in Proceedings of IEEE International Conference on Image Processing 2001 vol II. , pp.773-776.), an accurate region based on a rough region in the spatiotemporal image space is calculated. In Step 3, for each frame, the Watershed described in Reference 1 and Reference 6 (Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour models”, International Journal of Computer Vision, vol. 1, no. 4, pp. 321-331, 1987. and Reference 7 (Takashi Ida and Yoko Sambonsugi, "Self-Affine Mapping System and Its Application to Object Contour Extraction", IEEE Transactions on Image Processing, Vol. 9, No. 11, November 2000.) A true shape may be obtained from a rough shape by using the method described in No. 11, November 2000.). If the object area information is obtained in this way, S102 and subsequent steps in the flowchart of FIG. 3 can be performed.

(Method 2 when object area information is not given: no assumption of background stillness)
An example using a spatio-temporal image will be described as another method when object area information is not given. This method consists of the three steps described below and does not require the assumption that the background is stationary.

Step 1: Region division is performed on the spatiotemporal image.
Step 2: A feature amount is calculated for each image frame for each object (in many cases, overdivision is performed in Step 1, but Step 2 is performed in this state).
Step 3: Integrate objects considered identical from the result of Step 2.
In step 1 above, reference 8 (MA El Saban and BS Manjunath, “Video region segmentation by spatio-temporal watersheds”, IEEE International Conference on Image Processing (ICIP), Barcelona, Spain, vol. 1, pp. 349-352 , Sep. 2003.) and Reference 9 (Y. Ueshige, Y. Kuroki, and T. Ohta, "Region Segmentation for Video by Using 3D-IFS Coding", in Proceedings of 22nd Picture Coding Symposium, pp.378-380 Using the space-time image region division described in.), The region division is performed on the moving image. In step 2, feature quantities such as an average value and variance of RGB colors are calculated for each object. In step 3, absolute difference values are obtained for each of the six parameters (R average, G average, B average, R variance, G variance, B variance), and a weighted sum (weight is determined separately) is separately determined. If they are below the threshold, they are determined to be the same and integrated. Thereby, a region divided for each object in the moving image is obtained. When the object region information is obtained in this way, the processing from S102 onward in the flowchart of FIG. 3 can be performed as in the previous case.

(3D-IFS region segmentation by multi-label reduction mapping-positioning)
The spatio-temporal region segmentation shown in Reference 9 above is based on an enlarged map obtained by expanding Reference Literature 10 (Japanese Patent Laid-Open No. 8-329255 (fractal region segmentation)), which is a still image technique, into spatiotemporal space. Yes, it is strong against noise, but the region boundary obtained by the magnified mapping becomes unstable. Furthermore, since the region is determined by the K-Means method or periodic point classification after processing the entire spatio-temporal image, sequential processing is difficult. Therefore, in the embodiment of the present invention, it is described in Reference Document 11 (Japanese Patent Laid-Open No. 2001-188910 (multi-label fitting and area division, hereinafter referred to as “2D-IFS area fitting”)) for still images. A new method (hereinafter referred to as “3D-IFS region fitting”) obtained by improving the above method is used. Since 2D-IFS region fitting uses reduced mapping, the region boundary is stable, but there is no prior art suggesting application of this method to spatiotemporal images. Therefore, in the embodiment of the present invention, the region segmentation required when using a spatio-temporal image is performed by using a method that targets a moving image (a spatio-temporal image) but does not use an enlarged mapping as in the above-described Reference Document 11. Do it stably. Furthermore, a method of high accuracy and sequential processing of spatiotemporal images is provided by combination with Watershed.

(3D-IFS region segmentation method by multi-label reduction mapping)
A spatiotemporal image is a kind of three-dimensional image composed of three-dimensional pixels. Hereinafter, the region division of the spatio-temporal image will be described with reference to the flowchart of FIG. 10 (this method can be applied to a three-dimensional image as it is). First, a spatio-temporal image and an initial label are given as inputs (S1601). Next, a three-dimensional processing box is arranged in a three-dimensional space-time (S1602). Next, for each processing box, an area similar to the image in the processing box is searched for in the image (S1603). The search of S1603 is known as fractal coding. For example, if the processing box of the child box 1201 of FIG. 12 is given in the spatiotemporal image of FIG. 11, the parent of FIG. This is a process of searching for the box 1202. As a result, each processing box (child box) and a similar region (parent box) corresponding thereto are obtained. Here, the shape of the parent box can be determined so that the mapping from the parent box to the child box is a reduced map, but in the following, the parent box corresponds to a shape that doubles each side of the child box. It will be explained as being. In this case, the obtained mapping from the parent box to the child box maps the contents of 4 pixels to 1 pixel (see FIG. 13). When this mapping is applied to a label image, the label value after mapping is determined by majority decision of the contents of four pixels (if the two labels are the same, it is determined by a predetermined criterion, for example, the maximum value or the minimum In the same manner as in Reference Document 11, the same label is assigned to the same area. As an initial label in S1601, for example, an independent label value may be given to all pixels. Further, the output of another region dividing method may be used as an input for 3D-IFS region fitting. An example of using Watershed output as input for 3D-IFS region fitting will be described later. The processing boxes in S1602 may be arranged so as to cover the entire pixels of the spatiotemporal image as in fractal coding. In addition, the method described in Reference Document 7 and Reference Document 12 (Japanese Unexamined Patent Application Publication No. 2004-240913 (spatiotemporal fractal contour extraction method)) for mask images is extended to a general label image, and the label image is scanned. However, the processing box may be arranged at a place having a different label value in the vicinity of the target pixel and where the processing box is not arranged.

  An advantage of 3D-IFS region fitting that is not found in the above-mentioned reference 9 is that sequential processing is possible. FIG. 14 shows the relationship between the time axis and the frame to be processed. The portion described as the frame being processed in FIG. 14 substantially corresponds to the entire 3D-IFS region fitting image described above. A flowchart of this method is shown in FIG.

  First, the head frame is set to the head of the moving image (S2301), and the end frame is a frame shifted by a separately defined number of frames, for example, head frame + 20 frames (S2302). Next, 3D-IFS region fitting is performed for 1801 in FIG. 14 (S2303). It is checked whether processing has been performed up to the end of the moving image (S2304). If it is not the end, the steps after S2301 are repeated with the first frame shifted by one frame. Specifically, for example, if 1801 in FIG. 14 is processed first, 3D-IFS region fitting is performed on 1802 shifted by one frame. Thereafter, 3D-IFS region fitting is repeated in the same manner while shifting by one frame as indicated by 1803 and 1804. As a result, the number of frames for which image processing is performed at a time becomes constant, so that even if the time is long enough, even if the number of input frames is large, it is possible to obtain a label image in consideration of the connection in the time direction with a limited memory.

Here, the outline of the search range limiting method for enhancing the effect of 3D-IFS region fitting will be described (the similarity ratio between the parent box and the child box is 1/2). In the three-dimensional fractal coding, for example, all boxes including child boxes are candidates as parent box candidates. An example will be described with reference to FIG. FIG. 16 shows that when the child box 2401 is in the frames 9 to 14, the frames 4 to 14, the frames 5 to 15,..., 9 to 19 are searched as candidates for the parent box 2402. -17 is shown in the selected state. On the other hand, as shown in FIG. 17, when the search is always performed so that the end frame of the child box and the end frame of the parent box overlap (hereinafter referred to as “search range limiting method”), in this example, the search range is 9 When limited to only -19, the following two effects can be obtained.
(Effect A) Area division in which the number of labels does not increase can be realized.
(Effect B) Label data corresponding to the end frame is always obtained.
The area division used here is for associating a hidden portion with another frame. Therefore, a method in which no extra area is generated in the association is desirable, and the effect A is important. First, the reason why the effect A is obtained will be described. In the child box / parent box obtained by the search range limiting method, when mapping in S1604, frames 4 to 5 are to frame 9, frames 6 to 7 to frame 10,. Each frame 14 is copied. Therefore, all the frames other than the frame 14 are copied from the past frame. Here, assuming that there is no data of the frame 14 before mapping, when determining the label value of the frame 14 in the mapping of S1604, the two pixels of the frame 13 instead of the majority of the four pixels of the frame 13 and the frame 14 are determined. If the maximum value (or the minimum value) is determined, the effect A can be obtained. Alternatively, even if the contents of the frame 13 are copied to the frame 14 before S1604, the effect A can be obtained in the same manner. Alternatively, if the minimum value (maximum value) is used as the determination method in the case of the same number, all the pixels in the frame 14 have different unused label values that are larger (or smaller) than all the pixels in the frame 13. The above-mentioned effect A can be obtained even if assigned.

  Next, the reason why the effect B is obtained will be described. If the search range limiting method is used, the image required for the search does not exceed the end frame of the child box. Therefore, if the end frame of the child box overlaps the end frame, label data corresponding to the child box can be obtained. Note that the search range limiting method itself is not limited to 3D-IFS region fitting, and can also be applied to the case where the input is not a label image but a mask image (the tracking method described in Reference Document 12 above). In the search range limiting method, the mask image of the end frame is copied from a known mask image, so that the process of predicting a future frame required in the reference document 12 is not necessary. Since this prediction process is not easy, it is also useful for the method of the above-mentioned reference 12 that prediction is not required by the search range limiting method.

  The flow of the search range limiting method in both the mask image and the label image (hereinafter referred to as region information image) will be described with reference to FIG. 10 and FIGS. In this example, the label image shown in FIG. 10 is described as a region information image. An image / region information image to be processed is given in S1601. FIG. 21 shows an area information image in space-time, and 2901 is a known subject area (area 1 in the label image). When this is cut along a cutting plane perpendicular to the x-axis, the area boundary of the area information image is, for example, 2603 in FIG. Here, reference numeral 2601 denotes a subject in the luminance value image. Thus, the position of the subject of the area information image and the subject of the luminance value image are greatly shifted at the beginning. One of the child boxes arranged in accordance with the arrangement of the child box (S1602) is shown in 2602. When the parent box search (S1603) is performed on 2602 so that the rightmost frames overlap with each other by the search range limiting method, 2701 in FIG. 19 is found. When the mapping of the area information image (S1603) is applied using the set of 2602 and 2701, the area boundary moves closer to the rightmost frame (frame where two boxes overlap) as shown by 2801 in FIG. To do. Here, since the similarity ratio between the child box and the parent box is ½, the area on the left side is copied when S1603 is applied. When mapping S1603 is performed once, the frame from the left end frame of the child box to the (old region boundary frame + right end frame / 2) frame (that is, the new region boundary frame) is a copy from the frame before the old region boundary frame. The copy from the frame before the old region boundary frame is not copied from the new region boundary frame to the rightmost frame, but, as in the previous example, (1) when the majority decision (or the method of Reference 12 above) is performed, the new region boundary frame If the data from the frame to the rightmost frame is excluded, and (2) the data before the old boundary frame is copied before the mapping S1603, all the frames up to the rightmost frame will be copied. It becomes a copy from the old region boundary frame, and the effect of the search range limiting method can be utilized. In order to simplify the processing, the data after the old region boundary frame is a predetermined pattern (for example, a pattern in which the subject region and the background region appear alternately for each pixel in the frame) or a predetermined single region. You can fill it.

  In the previous example, the case where the end frame of the child box time and the end frame of the parent box time overlap as the search range of the search range limiting method has been described. It is not limited to. Generally, if the difference between the center frame of the parent box and the center frame of the child box is fixed to a predetermined value, the label image of the desired frame (or mask image, the same applies hereinafter) is determined from the label images of the surrounding frames. Can be estimated. For example, if the search range is set so that the center frame of the parent box and the center frame of the child box overlap (the center frame of the parent box is mapped to the center frame of the child box by mapping S1604), the overlapped center frame The label image is always copied from another frame like the end frame in the example of FIG.

(Fitting an image segmented with Watershed)
By combining Watershed and 3D-IFS region fitting, more accurate region division can be realized. This method can be applied not only to the combination of Watershed and 3D-IFS region fitting, but also to the combination of Watershed and 2D-IFS region fitting. For convenience of explanation, a two-dimensional case is illustrated. FIG. 22 is an example of an input image. FIG. 23 shows a 2D-IFS region fitting, and FIG. 24 shows an example of region division by watershed (white lines are region boundaries). The properties of each region boundary in this example are summarized in FIG. Watershed is over-textured, and IFS region fitting is over-partitioned in regions with less texture (regional boundaries are not generated). Since 2D-IFS fitting and 3D-IFS fitting can receive other region segmentation results as an input, it is easy to combine with other region segmentation results. In particular, when combined with a region division method having properties different from IFS region fitting, such as Watershed, there is a possibility that the method has both advantages. FIG. 26 shows the result when the label image obtained by Watershed is used as an input for 2D-IFS fitting. Unnecessary region boundaries are reduced compared to the case where each method is applied alone. Similarly, in the combination of watershed and 3D-IFS region fitting, watershed is performed in two or three dimensions, and region division is realized by applying 3D-IFS region fitting to the result.

For example, Reference 13 (L. Vincent and P. Soille, “Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.13, no. .6, June 1991.), and specifically consists of the following steps.
(Step 1) The edge strength of each pixel is detected.
(Step 2) Place seeds at the minimum edge strength.
(Step 3) The water level (threshold value) is initialized to the minimum value of the edge strength, for example.
(Step 4) When there is no label, it is a point around the label, and the edge strength is below the water level,
(Step 4-a) If there are only one kind of surrounding labels, the labels are set.

    (Step 4-b) Or, if there are two or more kinds of surrounding labels, the pixel is set as a region boundary.

  (Step 5) Repeat (Step 4) while raising the water level until label values are set for all pixels.

  FIG. 27 is a flowchart showing the procedure of area division using Watershed and 2D-IFS fitting.

  First, a watershed threshold is set (S1701). Next, the region is grown according to the watershed (step 4) described above (S1702). Next, S1702 is repeated while raising the threshold according to the Watershed (step 5) described above. As a result, a label image is obtained, and this is used as input for 2D-IFS fitting or 3D-IFS fitting, and the same processing as S1602 and subsequent steps in the flowchart of FIG. 10 is performed. Although only 2D and 3D have been described here, 4D or more may be targeted, for example, a 4D image obtained by adding time to a 3D space image. In addition, since each label spreads only to adjacent pixels in Watershed, Reference 14 (T. Yoshida and T. Shimosato, “Motion Image Segmentation Using 3-D Watershed Algorithm”, in Proceedings of IEEE International Conference on Image Processing 2001, pp. (.773-776, 2001.), it is possible to perform sequential calculation as well (this reference 14 describes binary labels, but it can also be applied to multi-value labels as it is). Accordingly, a method combining watershed and 3D-IFS region fitting can be sequentially processed by sequentially providing the output as the input of 3D-IFS region fitting while sequentially performing watershed. As shown in Reference 15 (D. Zhong et al. “Video object model and segmentation for content-based video indexing” in Proceedings of IEEE Symposium on Circuits and Systems 1997, pp.1492-1495) Integration may be performed. In this case, 2D or 3D IFS fitting may be applied after Watershed, and then region integration may be performed, or IFS fitting may be performed first.

  As described above, when the improved region segmentation is used, when the hidden region of the image is displayed by the user's instruction with respect to the image captured by the two-dimensional moving image capturing camera, this is used. It becomes possible to display more accurately. Specifically, for example, when the subject is a car, the part displayed by clicking becomes only a part of the missing part, or it is considered that all the areas instructed by clicking are displayed. Therefore, the problem that the hidden area is not displayed is improved.

  According to the embodiment of the present invention described above, a device in which a hidden region of an image is displayed according to a user instruction on an image captured by a two-dimensional moving image capturing camera, and the method of the present invention is mounted. The operability can be improved.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

1 is a block diagram of an image generation apparatus according to an embodiment of the present invention. The figure for demonstrating the outline | summary of the image generation method which concerns on embodiment of this invention, Comprising: The figure which shows the example of the image frame in an input moving image Flow chart showing processing procedure when object area information is known The figure which shows the object area | region information corresponding to the left image frame of the moving image shown by FIG. The flowchart which shows the processing procedure of the feedback presentation to the user of the selection area Diagram showing examples of feedback presentation A flowchart showing a processing procedure for estimating an object region corresponding to a point of interest The figure which shows the object area | region information corresponding to the center image frame of the moving image shown by FIG. The figure for explaining the estimation method of the hidden area Flow chart showing processing procedure of 3D-IFS region segmentation method by multi-label reduction mapping The figure which shows an example of a spatiotemporal image Diagram showing child box processing box and parent box in spatio-temporal image Diagram showing similarity between child box and parent box Diagram showing the frame being processed along the time axis Flow chart showing the sequential processing procedure in 3D-IFS region fitting Diagram showing selection range of child box and parent box Diagram showing selection range of child box and parent box Diagram for explaining the search range limiting method Diagram for explaining the search range limiting method Diagram for explaining the search range limiting method It is a figure for demonstrating the search range limitation method, Comprising: The figure which shows the area | region information image in space time Diagram showing an example of an input image The figure which shows the result of having performed 2D-IFS area | region fitting about the input image of FIG. The figure which shows the result of having divided the area | region by Watershed about the input image of FIG. The figure which shows the comparison result of the property which the area boundary has regarding IFS area fitting and watershed The figure which shows the processing result when the label image obtained by Watershed is used as the input of 2D-IFS fitting Flow chart showing the procedure of region segmentation using Watershed and 2D-IFS fitting

Explanation of symbols

100: Image generation device;
101 ... moving image storage unit;
102 ... area information storage unit;
103 ... Hidden region estimation unit;
104: Image generation unit;
105 ... display device;
106 ... mouse;
107: moving image input unit;
108 ... area information input section

Claims (19)

Inputting a series of image frames;
Inputting region information indicating a region occupied by an object in the series of image frames;
Inputting a point of interest in a first image frame in the series of image frames;
Estimating a hidden object region of an object corresponding to the attention point based on the region information;
Generating an image frame for presentation by obtaining a luminance value of the estimated hidden object region from the series of image frames.   The image generation method according to claim 1, further comprising a step of estimating the region information from the series of image frames instead of the step of inputting the region information. Estimating a schematic shape of the hidden object region;
3. The image generation method according to claim 1, further comprising a step of presenting the estimated schematic shape on the first image frame. Estimating a luminance value of the hidden object region for a plurality of image frames;
The image generation method according to claim 1, further comprising: determining one luminance value from a plurality of luminance values estimated for a plurality of image frames. Inputting first area information indicating an area occupied by an object in the first image frame;
Inputting second area information indicating an area occupied by the object in the second image frame;
Generating a plurality of modified second region information by performing different conversions while changing the conversion parameters for the second region information;
Calculating a similarity between the first area information and each of the plurality of modified second area information;
Selecting the conversion parameter based on the calculated similarity;
An image processing method for outputting selected conversion parameters as information for associating the same object between frames. The step of calculating the similarity includes
Either the urban area distance or Euclidean distance from the point on the first area information to the modified second area information, or the urban area distance or Euclidean distance from the point on the second area information to the modified first area information A distance calculating step for calculating
Executing the distance calculating step for N points on the first area information or N points on the second area information;
The image processing method according to claim 5, further comprising: outputting a Kth (0 ≦ K ≦ N) value of the distances from the N points obtained by the distance calculating step as the similarity. An image processing method for estimating region information of an object according to claim 1,
Providing a three-dimensional label image giving information for distinguishing regions for each three-dimensional pixel of the spatiotemporal image based on the series of image frames;
Placing a three-dimensional processing box in the spatiotemporal image;
Obtaining a parent box corresponding to a region having a volume larger than that of the three-dimensional processing box and having similar image data with respect to the three-dimensional processing box;
Mapping the three-dimensional label image from the parent box to the three-dimensional processing box.   8. The step of switching the frame of interest by inputting a spatio-temporal image and a three-dimensional label image having an arbitrary frame of interest as a head frame and a frame shifted by a predetermined number of frames from the head frame as an end frame. Image processing method. inputting n-dimensional image data (n is an integer of 2 or more);
An area dividing step of generating an n-dimensional label image by area division on the n-dimensional image data;
Placing an n-dimensional processing box in the n-dimensional image data;
Obtaining a parent box for the n-dimensional processing box having a measure larger than the n-dimensional box and having similar image data;
Mapping the n-dimensional label image from the parent box to the n-dimensional box. The region dividing step includes:
Inputting the n-dimensional image data and its edge strength;
Inputting labels for some pixels of the n-dimensional image data;
Setting an initial value of the threshold T of the edge strength;
A label region growing step for setting the label if the label is set in an adjacent pixel for each pixel in which the label is not set and the edge strength is equal to or less than a threshold T;
The image processing method according to claim 9, further comprising the step of repeating the label region growing step while satisfying a separately defined condition while increasing the threshold value T.   The image processing method according to claim 10, wherein the edge strength is an absolute value of a value obtained by a weighted sum of a pixel of interest and surrounding pixels. Arbitrary attention frame is the first frame,
The image processing method according to claim 9, wherein the frame of interest is switched with n-dimensional image data and an n-dimensional label image having a frame shifted by a predetermined number of frames from the first frame as inputs. A first step of inputting data in which an arbitrary frame of interest in spatio-temporal image data in which image frames of moving images are arranged in time series is set as a start frame, and a frame shifted by a predetermined number of frames from the start frame is set as a end frame When,
A spatio-temporal space corresponding to either a spatio-temporal label image that gives information for distinguishing a region or a spatio-temporal mask image that gives information for distinguishing between a subject region and a background region for each pixel in the spatio-temporal image data A second step of preparing an area information image;
A third step of arranging a child box in the spatiotemporal image data;
A fourth step for determining a parent box having a volume larger than that of the child box and having a similar space-time image data in the child box;
A fifth step of mapping the three-dimensional region information image from the parent box to the child box;
Performing the first to fifth steps while switching the frame of interest;
In the fourth step, the frame that is a candidate for the parent box is limited to a frame in which a difference between a center frame of the parent box and a center frame of the child box is a separately determined value. . Means for inputting a series of image frames;
Means for inputting region information indicating a region occupied by an object in the series of image frames;
Means for inputting a point of interest in a first image frame in the series of image frames;
Means for estimating a concealed object region of an object corresponding to the attention point based on the region information;
An image generation apparatus comprising: means for generating a presentation image frame by obtaining an estimated luminance value of the concealed object region from the series of image frames. Means for inputting first area information indicating an area occupied by the object in the first image frame;
Means for inputting second area information indicating an area occupied by the object in the second image frame;
Means for generating a plurality of modified second area information by performing different conversions while changing the conversion parameters for the second area information;
Means for calculating similarity between the first area information and each of the plurality of modified second area information;
Means for selecting the conversion parameter based on the calculated similarity;
An image processing apparatus comprising: means for outputting the selected conversion parameter as information for associating the same object between frames. means for inputting n-dimensional image data (n is an integer of 2 or more);
Area dividing means for generating an n-dimensional label image by area division on the n-dimensional image data;
Means for arranging an n-dimensional processing box in the n-dimensional image data;
Means for obtaining a parent box whose area is similar to that of the n-dimensional processing box and whose image data is similar to the n-dimensional processing box;
An image processing apparatus comprising: means for mapping the n-dimensional label image from the parent box to the n-dimensional box. A procedure for entering a series of image frames;
A procedure for inputting region information indicating a region occupied by an object in the series of image frames;
Entering a point of interest in a first image frame in the series of image frames;
A procedure for estimating a concealed object region of an object corresponding to the attention point based on the region information;
An image generation program for causing a computer to execute a procedure of generating a presentation image frame by obtaining an estimated luminance value of a hidden object region from the series of image frames. Inputting first area information indicating an area occupied by the object in the first image frame;
A procedure for inputting second area information indicating an area occupied by the object in the second image frame;
A procedure for generating a plurality of modified second area information by performing different conversions while changing conversion parameters for the second area information;
Calculating a similarity between the first region information and each of the plurality of modified second region information;
Selecting the conversion parameter based on the calculated similarity;
An image processing program for causing a computer to execute a procedure for outputting selected conversion parameters as information for associating the same object between frames. a procedure for inputting n-dimensional image data (n is an integer of 2 or more);
An area division procedure for generating an n-dimensional label image by area division on the n-dimensional image data;
Placing an n-dimensional processing box in the n-dimensional image data;
A procedure for obtaining a parent box whose area is similar to that of the n-dimensional processing box and whose image data is similar to the n-dimensional processing box;
An image processing program for causing a computer to execute a procedure for mapping the parent box to the n-dimensional box on the n-dimensional label image.

Images