WO2021117233A1

WO2021117233A1 - Moving image separation device, program, and moving image separation method

Info

Publication number: WO2021117233A1
Application number: PCT/JP2019/048991
Authority: WO
Inventors: 愛梨守谷; 秀明前原
Original assignee: 三菱電機株式会社
Priority date: 2019-12-13
Filing date: 2019-12-13
Publication date: 2021-06-17
Also published as: JPWO2021117233A1; JP7003342B2

Abstract

This moving image separation device is provided with: an image acquisition unit (102) that acquires two images including the foreground from moving images captured at a moving viewpoint; a misalignment amount identification unit (103) that identifies a misalignment amount regarding the position of the foreground between the two images; a correlation calculation unit (104) that corrects a positional misalignment between the two images in accordance with the misalignment amount, and calculates a correlation, in a predetermined area, between each two pixels corresponding one another in the two corrected images or between each two groups of pixels corresponding one another in the two corrected images; and a threshold value processing unit (105) that sets pixels where the calculated correlation is larger than a predetermined threshold value as pixels of the foreground, and generates an image having pixel values of the pixels of the foreground extracted as the foreground image from one image included in the moving images.

Description

Video separator, program and video separation method

This disclosure relates to a video separation device, a program, and a video separation method.

As a technique for separating the background and the foreground in the frame included in the moving image, there is a moving image processing apparatus described in Patent Document 1.
The conventional moving image processing device divides each frame into small areas, calculates a feature amount based on motion for each divided area, and classifies the small area according to the calculated feature amount, thereby foreground and background. And separate.

Japanese Unexamined Patent Publication No. 2005-176339

In the conventional technology, each frame must be divided into small areas, and the feature amount based on the movement must be calculated for each divided area. Therefore, the processing cost for separating the foreground and the background in each frame is very high. It's getting higher.

Therefore, one or a plurality of aspects of the present disclosure aims to reduce the processing cost when separating the foreground and the background in the image included in the moving image data.

In the moving image separation device according to one aspect of the present disclosure, the position of the foreground in the two images is deviated from the image acquisition unit that acquires two images including the foreground from the moving image captured from the moving viewpoint. The deviation amount specifying part that specifies the deviation amount and the deviation amount of the two images are corrected according to the deviation amount, and each of the corresponding pixels of the two images after correction, or the two after correction. A correlation calculation unit that calculates the correlation of a predetermined area for each group of a plurality of corresponding pixels of one image, and a pixel whose calculated correlation is larger than a predetermined threshold are defined as pixels in the foreground. It is characterized by including a threshold processing unit that generates an image obtained by extracting pixel values of the pixels of the foreground as a foreground image from one image included in the moving image.

In the program according to one aspect of the present disclosure, the image acquisition unit that acquires two images including the foreground from the moving viewpoint captured by the computer, and the positions of the foreground in the two images are deviated from each other. The deviation amount specifying unit for specifying the deviation amount, the deviation of the positions of the two images is corrected according to the deviation amount, and each of the corresponding pixels of the two images after correction, or the two after correction. A correlation calculation unit that calculates the correlation of a predetermined area for each group of a plurality of corresponding pixels of an image, and a pixel whose calculated correlation is larger than a predetermined threshold are defined as pixels in the foreground. It is characterized in that it functions as a threshold processing unit that generates an image obtained by extracting the pixel values of the pixels of the foreground from one image included in the moving image as a foreground image.

In the moving image separation method according to one aspect of the present disclosure, two images including a foreground are acquired from a moving image captured from a moving viewpoint, and the amount of deviation in the position of the foreground in the two images is specified. Then, according to the amount of deviation, the displacement of the positions of the two images is corrected, and each of the corrected pixels of the two images or a group of a plurality of corresponding pixels of the corrected two images. For each, the correlation of the predetermined area is calculated, and the pixel whose calculated correlation is larger than the predetermined threshold is defined as the foreground pixel, and from one image included in the moving image, the foreground pixel is used. It is characterized in that an image obtained by extracting the pixel value of is generated as a foreground image.

According to one or more aspects of the present disclosure, it is possible to reduce the processing cost when separating the foreground and the background in the image included in the moving image.

It is a block diagram which shows schematic structure of the moving image separation apparatus which concerns on Embodiments 1 to 3. (A) and (B) are schematic views showing the two acquired frames. It is a schematic diagram for demonstrating the specification of the deviation amount in the deviation amount specifying part in Embodiment 1. FIG. It is a schematic diagram which shows the correlation image which classified the calculated correlation by the value of the predetermined range for each pixel. It is a block diagram which shows the hardware configuration example of the moving image separation apparatus. It is a flowchart which shows the processing in a moving image separation apparatus. (A) to (H) are schematic diagrams for explaining the process of dividing the first frame and the second frame and calculating the deviation amount. (A) to (C) are schematic views showing one frame included in the moving image, and the first frame and the second frame acquired from the one frame. (A) and (B) are schematic views showing a first frame and a mask image.

Embodiment 1.
FIG. 1 is a block diagram schematically showing the configuration of the moving image separation device 100 according to the first embodiment.
The moving image separation device 100 includes a moving image input unit 101, an image acquisition unit 102, a deviation amount specifying unit 103, a correlation calculation unit 104, a threshold value processing unit 105, and an image output unit 106.

The video input unit 101 receives video input from an external camera. The moving image input unit 101 gives the input moving image to the image acquisition unit 102. The moving image contains a plurality of images as a plurality of frames.
The moving image here is a moving image of an object in the foreground while the camera is moving. In other words, by moving the camera, it is assumed that the viewpoint of the camera is also moving. Specifically, it is assumed that the moving image in the first embodiment is a copy of an electric wire from a helicopter. Therefore, the moving image in the first embodiment is a moving image captured from a moving viewpoint.

In the first embodiment, it is premised that the foreground does not change significantly with time in the moving image, and the foreground and the background change with time.

The image acquisition unit 102 acquires two images including the foreground from the moving image.
For example, the image acquisition unit 102 acquires two frames that are close in time from a plurality of frames included in the moving image. Here, of the two acquired frames, the earlier frame in time is referred to as the first frame, and the later frame is referred to as the second frame.

Specifically, the image acquisition unit 102 is included in two frames included in a predetermined time or within a predetermined number of frames among the plurality of frames included in the moving image. All you have to do is get two frames. Here, it is assumed that the image acquisition unit 102 acquires two consecutive frames.

2A and 2B are schematic views showing two frames acquired by the image acquisition unit 102. Here, FIG. 2 (A) shows the first frame 110, and FIG. 2 (B) shows the second frame 111.
The foreground electric wire 112A is reflected in the first frame 110, and a portion other than the electric wire 112A is the background. Here, it is assumed that the focus is on the electric wire 112A, which is the foreground, and the forest in the background is blurred. Since the background is blurred, it is assumed that the forest is reflected as

patterns

113A and 114A.

Further, the electric wire 112B as the foreground is also reflected in the second frame, and the portion other than the electric wire 112B is the background. In the background, the forest is reflected as

blurred patterns

113B and 114B.
Here, it is assumed that the camera is moving in the D direction.

The deviation amount specifying unit 103 specifies the deviation amount in which the positions of the foregrounds in the two acquired frames are displaced.
For example, the deviation amount specifying unit 103 specifies the deviation amount in which the positions of these two frames are displaced so that the portions reflected in the two acquired frames match.
Specifically, the deviation amount specifying unit 103 may specify the deviation amount of the positions of these two frames so that the foreground parts included in these two frames match. Specifically, the deviation amount specifying unit 103 specifies the deviation amount by obtaining the correlation of the entire two frames. The deviation amount specifying unit 103 may specify the deviation amount by matching the feature points.

FIG. 3 is a schematic diagram for explaining the specification of the deviation amount by the deviation amount specifying unit 103.
The first frame 110 and the second frame 111 are imaged by a camera moving in the D direction. Therefore, when the portion included in the first frame 110 and the corresponding portion included in the second frame 111 are overlapped with each other, as shown in FIG. 3, the second frame 111 Will be displaced in the D direction with respect to the first frame 110. In addition, the frame shifts in the vertical direction depending on the vibration or the moving direction of the camera.

In order to correct the above deviation, the deviation amount specifying unit 103 specifies the deviation amount X in the horizontal direction and the deviation amount Y in the vertical direction. For example, as described above, when the foreground

electric wires

112A and 112B are in focus and the background is blurred, the correlation of the entire image of the first frame 110 and the second frame 111 can be calculated. The correlation is high when the corresponding portion of the electric wire 112A and the corresponding portion of the electric wire 112B match. Therefore, as shown in FIG. 3, when the corresponding portions of the electric wire 112A shown in the first frame 110 and the electric wire 112B shown in the second frame 111 overlap each other, The correlation is high.

Further, even when matching feature points, the degree of matching is high when the focused objects match each other. Therefore, the same deviation amounts X and Y as in FIG. 3 should be specified. Can be done.

The correlation calculation unit 104 corrects the deviation of the positions of the two frames according to the deviation amount specified by the deviation amount specifying unit 103, and has a predetermined area for each of the corresponding pixels of the two corrected frames. Calculate the correlation.
For example, the correlation calculation unit 104 shifts either one of the first frame 110 and the second frame 111 by the amount of deviation specified by the deviation amount specifying unit 103, and determines in advance for each matching pixel. Calculate the correlation of size windows.

FIG. 4 is a schematic view showing a correlation image 114 in which the correlation calculated by the correlation calculation unit 104 is classified for each value in a predetermined range for each pixel.
As shown in the correlation image 114 of FIG. 4, the portion corresponding to the electric wire in the foreground has a relatively high correlation, and the background, which is the other portion, has a relatively low correlation.

Therefore, the threshold processing unit 105 identifies the pixels having a higher correlation than the predetermined threshold as the pixels in the foreground by comparing the correlation for each pixel with the predetermined threshold, and other than that. Is specified as a background pixel.

Then, the threshold processing unit 105 is an image obtained by extracting the pixel values of the pixels of the foreground in at least one of the frames included in the moving image, here, the first frame 110 and the second frame 111. An image and a background image, which is an image obtained by extracting the pixel values of the background pixels, are generated.
The foreground image is an image in which the pixel values of the foreground pixels are extracted without extracting the pixel values of the background pixels, and the background image is an image in which the pixel values of the foreground pixels are not extracted. It is an image obtained by extracting the pixel value of.

The image output unit 106 outputs at least one of the foreground image and the background image.

FIG. 5 is a block diagram showing a hardware configuration example of the moving image separator 100.
In the moving image separation device 100, the I / O (In or Out) interface 120, the auxiliary storage device 121 that functions as a non-volatile memory, the CPU (Central Processing Unit) 122 that functions as a processor, and the main memory 123 are bus 124. It can be realized by the computer connected to.

Specifically, the moving image input unit 101 and the image output unit 106 can be realized from the I / O interface 120.
The image acquisition unit 102, the deviation amount specifying unit 103, the correlation calculation unit 104, and the threshold processing unit 105 read the program stored in the auxiliary storage device 121 into the main memory 123 and execute the program. It can be realized.

Note that such a program may be provided through a network, or may be recorded and provided on a recording medium. That is, such a program may be provided as, for example, a program product.

The camera 125 as an imaging device outputs a moving image to the moving image separating device 100.
The moving image separation device 100 outputs at least one of the generated foreground image and the generated background image to the display 126 as a display device, so that at least one of the foreground image and the background image is displayed on the display 126. To display.

Further, in the example shown in FIG. 5, the camera 125 and the display 126 are not included in the moving image separator 100, but at least one of them may be included in the moving image separating device 100.
For example, when the camera 125 is included in the moving image separating device 100, the camera 125 functions as an imaging unit for capturing a moving image.
Further, when the display 126 is included in the moving image separating device 100, the display 126 functions as a display unit for displaying at least one of the foreground image and the background image.

FIG. 6 is a flowchart showing the processing in the moving image separator 100 according to the first embodiment.
First, the video input unit 101 receives video input from an external camera (S10). The moving image input unit 101 gives the input moving image to the image acquisition unit 102.

Next, the image acquisition unit 102 acquires two frames that are close in time from the plurality of frames included in the moving image as the first frame and the second frame (S11). Here, of the two acquired frames, the earlier frame in time is referred to as the first frame, and the later frame is referred to as the second frame. Then, the image acquisition unit 102 gives the acquired first frame and the second frame to the deviation amount specifying unit 103.

Next, the deviation amount specifying unit 103 specifies the deviation amount in which the positions of these two frames are displaced so that the portions reflected in the two acquired frames match (S12). The deviation amount specifying unit 103 gives the specified deviation amount and the first frame and the second frame to the correlation calculation unit 104.

Next, the correlation calculation unit 104 shifts either one of the first frame 110 and the second frame 111 by a given amount of deviation, for each matching pixel, and for each window of a predetermined size. The correlation is calculated in (S13). Then, the correlation calculation unit 104 gives the correlation data indicating the calculated correlation for each pixel, the amount of deviation, and the first frame and the second frame to the threshold value processing unit 105.

Next, the threshold processing unit 105 compares the correlation shown in the correlation data with a predetermined threshold for each pixel, so that the pixel having a higher correlation than the predetermined threshold is in the foreground. Is specified as the pixel of the above, and the other pixels are specified as the background pixel (S14).

Then, the threshold processing unit 105 in at least one of the first frame 110 and the second frame 111, an image composed of pixel values of the pixels in the foreground and an image composed of pixel values of the pixels in the background. A background image is generated (S15). The threshold processing unit 105 gives at least one of the generated foreground image and background image to the image output unit 106.

Next, the image output unit 106 outputs at least one of the foreground image and the background image (S16).

As described above, according to the moving image separating device 100 according to the first embodiment, the processing cost is low and the background and the foreground can be separated with high accuracy.

In the first embodiment, as shown in FIG. 3, the deviation amount specifying unit 103 specifies the deviation amounts X and Y from the entire first frame 110 and the second frame 111. Form 1 is not limited to such an example. For example, the deviation amount specifying unit 103 may divide the first frame 110 and the second frame into a plurality of divided frames in the same manner, and calculate the deviation amount for each divided frame.

7 (A) to 7 (H) are schematic views for explaining a process of dividing the first frame 110 and the second frame to calculate the amount of deviation.
For example, the deviation amount specifying unit 103 divides the first frame 110 shown in FIG. 7 (A) in the middle in the horizontal direction, thereby dividing the first frame 110 shown in FIG. 7 (B). The frame 110A and the second divided frame 110B shown in FIG. 7C are generated.

Similarly, the deviation amount specifying unit 103 divides the second frame 111 shown in FIG. 7 (D) in the middle in the horizontal direction, so that the third frame 111 shown in FIG. 7 (E) is divided. The division frame 111A and the fourth division frame 111B shown in FIG. 7 (F) are generated.

Then, as shown in FIG. 7 (G), the deviation amount specifying unit 103 sets the displacement amount between the first division frame 110A and the third division frame 111A in the same manner as in FIG. By specifying to, the division deviation amounts X ₁ and Y ₁ , which are the deviation amounts of these positions, are specified.
Similarly, as shown in FIG. 7 (H), the deviation amount specifying unit 103 identifies the displacement amount between the second divided frame 110B and the fourth divided frame 111B, and these _{The division deviation amounts X 2} and Y ₂ , which are the deviation amounts of the positions of, are specified.

In such a case, the correlation calculation unit 104 specifies the corresponding pixels by using the _{division deviation amounts X 1} and Y ₁ for the pixels included in the first division frame 110A or the third division frame 111A. Then, the correlation may be calculated.
Similarly, the correlation calculation unit 104, for the pixels included in the second divided frame 110B or the fourth divided frame 111B, using a split shift amount X _2, Y _2, to identify the corresponding pixel , The correlation may be calculated.

Further, the image acquisition unit 102 or the deviation amount specifying unit 103 may convert the first frame and the second frame into grayscale when they are in color. Then, the deviation amount specifying unit and the correlation calculation unit 104 may perform the above-mentioned processing using the first frame and the second frame converted into gray scale.

In the first embodiment, of the two acquired frames, the earlier frame is the first frame and the later frame is the second frame, but the earlier frame is the earlier frame. May be the second frame, and the slower frame may be the first frame.

Embodiment 2.
As shown in FIG. 1, the moving image separation device 200 according to the second embodiment includes a moving image input unit 101, an image acquisition unit 202, a deviation amount specifying unit 103, a correlation calculation unit 104, and a threshold processing unit. It includes 205 and an image output unit 106.

The moving image input unit 101, the deviation amount specifying unit 103, the correlation calculation unit 104, and the image output unit 106 of the moving image separation device 200 according to the second embodiment are the moving image input units 101 and the deviation amount of the moving image separating device 100 according to the first embodiment. This is the same as the quantity specifying unit 103, the correlation calculation unit 104, and the image output unit 106.

In the second embodiment, the moving image input to the moving image input unit 101 is assumed to be a moving image captured by the interlace method. The interlace method is a kind of image transmission method, and is a method in which every other scanning line is transmitted in two times.

The image acquisition unit 202 acquires two frames from one frame included in the moving image. Here, since it is an interlaced moving image, the image acquisition unit 202 extracts the pixel values of the pixels of the even-numbered rows and the first frame from which the pixel values of the pixels of the odd-numbered rows included in one frame are extracted. The second frame is acquired.

8 (A) to 8 (C) show, in the second embodiment, one frame 215 included in the moving image, and the first frame 210 and the second frame 211 acquired from the one frame 215. It is a schematic diagram which shows.
In the second embodiment, from one frame 215 shown in FIG. 8 (A) to the first frame 210 in which only the pixels of the odd-numbered rows are extracted as shown in FIG. 8 (B). , As shown in FIG. 8C, the second frame 211 in which only the pixels of even-numbered rows are extracted is acquired.

The threshold processing unit 205 compares the correlation calculated by the correlation calculation unit 104 with a preset threshold value for each pixel, and sets pixels having a higher correlation than the preset threshold value as foreground pixels. It is specified, and the other pixels are specified as background pixels.

Then, the threshold processing unit 205 extracts the foreground image, which is an image obtained by extracting the pixel values of the pixels in the foreground, and the pixel values of the pixels in the background from the original frame from which the first frame and the second frame are acquired. Generates a background image that is an image.

As described above, in the second embodiment, the first frame having only odd-numbered lines and the second frame having only even-numbered lines are acquired from one frame of the interlaced moving image. Since the amount of deviation of the first frame and the second frame acquired in this way is specified by the deviation amount specifying unit 103, the blurring due to the temporal deviation between the odd-numbered rows and the even-numbered rows in the interlace method is eliminated. Will be done.

Therefore, according to the second embodiment, it is possible to separate the foreground and the background from one frame while eliminating the blurring of that one frame.

Embodiment 3.
As shown in FIG. 1, the moving image separation device 300 according to the third embodiment includes a moving image input unit 101, an image acquisition unit 102, a deviation amount specifying unit 303, a correlation calculation unit 104, and a threshold processing unit. It includes 105 and an image output unit 106.

The video input unit 101, the image acquisition unit 102, the correlation calculation unit 104, the threshold processing unit 105, and the image output unit 106 of the video separation device 300 according to the third embodiment are used to input the video of the video separation device 100 according to the first embodiment. This is the same as the unit 101, the image acquisition unit 102, the correlation calculation unit 104, the threshold processing unit 105, and the image output unit 106.

The deviation amount specifying unit 303 specifies the deviation amount in which the positions of these two frames are displaced so that the portions reflected in the two acquired frames match.
In the third embodiment, the amount of deviation from the other frames is specified by using a partial image obtained by extracting only the portion matching the features of the foreground from one frame selected from these two frames. Here, one frame selected from the two frames is also referred to as a selected image, and the remaining frame is also referred to as a non-selected image.

9 (A) and 9 (B) are schematic views for explaining the processing of the deviation amount specifying unit 303 in the third embodiment.
Here, a specific example will be described by taking the first frame of the two acquired frames as an example, but the second frame may be used.
The deviation amount specifying unit 303 is, for example, a mask as shown in FIG. 9 (B) by using the characteristics of the electric wire 112A which is the foreground from the first frame 110 shown in FIG. 9 (A). Generate image 316.

Since the electric wire 112A has, for example, a color close to gray, the deviation amount specifying unit 303 specifies the saturation range corresponding to the gray color of the electric wire 112A from the first frame 110, and therefore, FIG. A mask image 316 as shown in B) can be generated.

Similar to the first embodiment, the first frame 110 shown in FIG. 9A shows the electric wire 112A as the foreground, the portion other than the electric wire 112A is the background, and the forest as the background is blurred. The patterns are shown as 113A and 114A.
The mask image 316 shown in FIG. 9B is divided into an area 317 corresponding to the foreground and an area 318 corresponding to the background.

Then, the deviation amount specifying unit 303 uses the mask image 316 shown in FIG. 9 (B) to form the region 317 corresponding to the foreground from the first frame 110 shown in FIG. 9 (A). A partial image that is a partial image can be extracted.

The deviation amount specifying unit 303 specifies the deviation amount by obtaining the overall correlation between the extracted partial image and the second frame.
Here, since the extracted partial image does not include the

patterns

113A and 114A shown in FIG. 9A, the deviation amount specifying unit 303 can accurately identify the deviation in the foreground.

As described above, according to the third embodiment, the influence of the background can be suppressed and the deviation of the foreground can be accurately identified.

In the above-described first to third embodiments, the threshold processing units 105 and 205 generate the foreground image and the background image, but only the foreground image may be generated.

In the first to third embodiments described above, the correlation calculation unit 104 calculates the correlation of a predetermined area for each corresponding pixel of the two corrected frames, but the first embodiment 1 ~ 3 is not limited to such an example.
For example, the correlation calculation unit 104 may calculate the correlation of a predetermined area for each group of a plurality of corresponding pixels of the two corrected frames. Specifically, the correlation calculation unit 104 performs a correlation in a window having a predetermined size such as 5 × 9 in each pixel included in a group of a plurality of predetermined pixels in the window. It can also be a correlation. As a result, the calculation load when calculating the correlation can be reduced. It is desirable that the group of a plurality of pixels includes the pixel in the center of the window.

100, 200, 300 video separator, 101 video input unit, 102, 202 image acquisition unit, 103, 303 deviation amount identification unit, 104 correlation calculation unit, 105, 205 threshold processing unit, 106 image output unit.

Claims

An image acquisition unit that acquires two images including the foreground from a moving image captured from a moving viewpoint,
A shift amount specifying unit that specifies the shift amount in which the foreground position is shifted in the two images, and a shift amount specifying unit.
According to the amount of deviation, the displacement of the positions of the two images is corrected for each corresponding pixel of the two images after correction, or for each group of a plurality of corresponding pixels of the two images after correction. In addition, a correlation calculation unit that calculates the correlation of a predetermined area,
Pixels whose calculated correlation is larger than a predetermined threshold value are defined as the foreground pixels, and an image obtained by extracting the pixel values of the foreground pixels from one image included in the moving image is generated as the foreground image. A moving image separator characterized by having a processing unit.
The threshold processing unit is characterized in that pixels other than the pixels in the foreground are used as background pixels in the one image, and an image obtained by extracting the pixel values of the background pixels from the one image is generated as the background image. The moving image separation device according to claim 1.
The moving image separation device according to claim 1 or 2, wherein the image acquisition unit acquires two frames within a predetermined time in the moving image as the two images.
The moving image separation device according to claim 1 or 2, wherein the image acquisition unit acquires two frames within a predetermined number of frames in the moving image as the two images.
The moving image separation device according to any one of claims 1 to 4, wherein the one image is one of the two images.
The deviation amount specifying unit extracts a partial image which is an image of a portion corresponding to the foreground from a selected image which is one image selected from the two images, and in the partial image and the two images. The moving image separation device according to any one of claims 1 to 5, wherein the amount of deviation is specified by using a non-selected image which is one image different from the selected image.
Any one of claims 1 to 5, wherein the deviation amount specifying unit divides each of the two images into a plurality of images, and specifies the deviation amount from each of the divided plurality of images. The video separator described in the section.
The moving image is a moving image captured by an interlaced method.
The image acquisition unit acquires, as the two images, an image obtained by extracting the pixel values of the pixels of the odd-numbered rows and an image obtained by extracting the pixel values of the pixels of the even-numbered rows from one frame included in the moving image. The moving image separator according to claim 1 or 2.
The moving image separation device according to claim 8, wherein the one image is the one frame.
The moving image separation device according to any one of claims 1 to 9, further comprising an imaging unit for capturing the moving image.
The moving image separating device according to any one of claims 1 to 10, further comprising a display unit for displaying the foreground image.
Computer,
An image acquisition unit that acquires two images including the foreground from a moving image captured from a moving viewpoint.
A shift amount specifying unit that specifies a shift amount in which the positions of the foregrounds in the two images are shifted,
According to the amount of deviation, the displacement of the positions of the two images is corrected for each corresponding pixel of the two images after correction, or for each group of a plurality of corresponding pixels of the two images after correction. In addition, a correlation calculation unit that calculates the correlation of a predetermined area, and
Pixels whose calculated correlation is larger than a predetermined threshold value are defined as the foreground pixels, and an image obtained by extracting the pixel values of the foreground pixels from one image included in the moving image is generated as the foreground image. A program characterized by functioning as a processing unit.
Two images including the foreground are acquired from the video captured from the moving viewpoint.
The amount of deviation in the foreground position in the two images is specified, and the amount of deviation is specified.
According to the amount of deviation, the deviation between the positions of the two images is corrected.
A predetermined area correlation is calculated for each corresponding pixel of the two corrected images or for each group of a plurality of corresponding pixels of the two corrected images.
Pixels whose calculated correlation is larger than a predetermined threshold value are defined as the pixels in the foreground.
A moving image separation method, characterized in that an image obtained by extracting the pixel values of the pixels of the foreground from one image included in the moving image is generated as a foreground image.