JP2022182835A - Image processing system, and control method and program thereof - Google Patents

Abstract

To reduce a load of foreground extraction in producing a virtual viewpoint image.SOLUTION: An image processing system generates a virtual viewpoint image showing a view from a virtual viewpoint according to photographic images obtained by imaging a space including a subject with a plurality of imaging apparatuses. The image processing system includes: a projection part configured to project an image in a prescribed color with brightness different from that of a background, onto the background in a prescribed color different from the subject of a foreground; and a separation part configured to separate the foreground from the photographic image according to a color difference.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to foreground/background separation processing techniques.

In filming studios such as television broadcasting stations, the performers are usually notified of the lines and the progress of the program by presenting a tool called a so-called kanpe, which is a piece of paper written by the performers and the remaining time of the program. . In recent years, the performer is placed in front of a background with a specific color such as green or blue, and the image area of the background and the performer is separated from the image captured by the camera by image processing, and the background image area is replaced with another image. There are many occasions when chromakey processing is performed to create a composite image by replacement, and performer support tools such as campaigns play an important role in the progress of programs even in such shooting.

On the other hand, there is a virtual viewpoint (also called free viewpoint) video technology that shoots the performer with multiple cameras surrounding the performer and generates video from an arbitrary viewpoint (virtual viewpoint) from the captured image (patent Reference 1). Virtual viewpoint video technology also incorporates chromakey processing and synthesizes an arbitrary background image with the performer, making it possible to view the performer in a three-dimensional background space from an arbitrary viewpoint.

JP 2010-020487 A

In a shooting studio for producing virtual viewpoint video, a plurality of cameras surrounding the performer are used to shoot the performer. Furthermore, when combining a background subtraction method such as chromakey processing, the entire closed space around the performer must have a specific color of green or blue. As a result, the competition in the photography studio, which is not originally the object of photography, becomes the foreground area by the background subtraction method, and a burden such as processing for removing the foreground other than the performers is generated.

In order to solve this problem, for example, the image processing system of the present disclosure has the following configuration. i.e.
An image processing system that generates a virtual viewpoint image representing a view from a virtual viewpoint based on captured images obtained by capturing a space including a subject with a plurality of capturing devices,
Projecting means for projecting an image of a predetermined color with a brightness different from that of the background onto a background of a predetermined color different from a foreground subject;
separating means for separating the foreground from the captured image based on color differences.

According to the present disclosure, it is possible to reduce the load related to foreground extraction in producing a virtual viewpoint video.

FIG. 2 is a configuration diagram of a shooting studio for virtual viewpoint video according to the first embodiment; FIG. 2 is a configuration diagram of an image processing system for virtual viewpoint video according to the first embodiment; FIG. 4 is a diagram showing a camera shooting area including a studio wall surface in the first embodiment; 4A and 4B are diagrams showing a color space diagram and a separation result of foreground/background separation in the first embodiment; FIG. 4A and 4B are diagrams showing a processing procedure and an example of grayscale according to the first embodiment; FIG. FIG. 11 is a block diagram of a shooting studio for virtual viewpoint video according to the second embodiment; FIG. 11 is a configuration diagram of an image processing system for virtual viewpoint video according to the second embodiment; FIG. 11 is a configuration diagram of an image processing system for virtual viewpoint video according to the third embodiment; FIG. 11 is a block diagram of a shooting studio for virtual viewpoint video according to the third embodiment; 2 is a hardware configuration diagram of an image processing apparatus or system control apparatus according to the embodiment; FIG.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

[First embodiment]
First, referring to FIGS. 1A and 1B, a studio configuration of an image processing system for generating a virtual viewpoint image according to the first embodiment will be described. For the sake of simplification, the studio configuration of the image processing system is simply referred to as "studio".

FIG. 1(a) is a top view of the studio, and FIG. 1(b) is a side view of the studio. Reference numeral 100 in the drawing represents a studio wall surface, and virtual viewpoint video shooting is performed within a closed space bounded by this studio wall surface 100 .

In this embodiment, the performer, who is the photographed subject, is the foreground, and the studio wall surface 100 is the background, and the background subtraction method using the color information of the foreground and the background is used to extract the photographed performer who is the foreground. The background (studio wall 100) for applying the background subtraction method is painted in a specific color. Green and blue, which are generally used in chromakey technology, are preferred, but are not limited to these.

In addition, the particular color is not limited to being directly painted on the studio wall surface 100, and a curtain of a predetermined color such as green or blue is placed on the studio wall surface 100 so that the color can be changed according to the filmed program (program). It may be hung and used by changing the color by changing the curtains as necessary. To simplify the explanation, it is assumed that the studio wall surface (including the curtain) 100 in this embodiment is green (green screen).

Reference numerals 101 to 116 are arranged to surround the entire studio as shown in FIG. are multiple cameras. Cameras 101 to 116 are arranged on the upper part of the wall surface of the studio as shown in FIG.

In this embodiment, all the cameras are arranged on the upper part of the studio wall surface as shown in FIG. 1(b). You can shoot from behind the wall of the studio. Hereinafter, the image captured by the camera will simply be referred to as a captured image or a captured image.

Image processing apparatuses 1001 to 1016, which will be described later, are connected to the cameras 101 to 116, respectively. In this embodiment, the number of cameras and the number of image processing devices are set to 16 respectively, but the number of installed cameras and image processing devices is not limited to 16, and the number of installed cameras and image processing devices can be increased or decreased according to the size of the studio and the resolution of the captured image. Also good. The illustrations should be understood to be exemplary only.

The image processing apparatuses 1001 to 1016 are connected to each other in a daisy chain by network cables, and the end of the chain is connected to the system controller 120 . An image captured by the camera 101 is transferred to the image processing device 1016 . The image processing device 1016 separates the foreground by foreground/background separation processing, which will be described later, and transmits an image representing the separated foreground to the system control device 120 via a daisy chain network cable. Other cameras 102 to 116 and image processing apparatuses 1002 to 1016 also perform the same processing as the camera 101 and the image processing apparatus 1001. FIG.

Reference numeral 130 is a video projector installed on the ceiling of the studio. The range surrounded by the dotted line in the drawing indicates the projection range of this video projector 130. As shown in FIG. In this embodiment, for the sake of convenience, the upper part of FIG. 1A indicates the azimuth of "north". Accordingly, the examples of FIGS. 1(a) and 1(b) show the image projector 130 projecting an image onto the west wall of the studio. It should be noted that the image projector 130 can produce a monochrome image (achromatic image) with only the lightness component.

Reference numeral 140 is the performer (one person) in the studio. The illustration shows a performer 140 facing the west side of the studio and looking at the projected image of the video projector 130 projected on the west side wall.

The above is the configuration of the equipment and the performers when the studio for the virtual viewpoint video shooting of the present embodiment is viewed from above. Next, referring to FIG. I will explain the composition of the equipment and performers when looking at the studio for video shooting from the side.

FIG. 1(b) is a side view of the studio standing on the south side and looking toward the north side.

Here, image processing devices 1001 to 1016 connected to cameras 101 to 116, respectively, are not shown in FIG. Alternatively, they may be installed on the floor vertically below the cameras 101-116.

The image projector 130 projects an image on the west side wall of the studio within the range indicated by the dotted line as in the description of FIG. 1(a). A performer 140 in the studio is reading video information from the video projector 130 . More specifically, FIGS. 1(a) and 1(b) show the virtual viewpoint video being captured, and the performer 140 facing west in the studio during the performance. The performer 140 is in a state of reading the projector image of the progress instruction given by the program staff (not shown).

FIG. 3(a) is a diagram showing the vicinity of the northwest of the studio wall surface 100. A video projector 130 projects an image 2 of a progress instruction issued by a program staff member (not shown) onto the studio wall surface 100 as a projected image 302. Performer 140 is from the program staff "2 minutes remaining. Roll it! ] is reading the progress instructions.

Reference numeral 301 represents the imaging field range of the camera 101, and the camera 101 simultaneously photographs the performer 140, the studio wall surface 100, and the projector projection image 302. FIG.

Here, the color scheme of the captured image of the camera 101 shown in FIG. 3A will be described. The studio wall surface 100 and the studio floor surface are green. The performer 140 with the background of the studio wall 100 is wearing a costume other than green.

Projector projection image 302 is the same color green and bright image displayed on studio wall surface 100 by projection of white light by image projector 130 . The performer 140 recognizes the difference in brightness of the projected image on the wall surface of the studio and reads the contents of the progress instruction from the program staff.

The image processing apparatuses 1001 to 1016 and the system control apparatus 120 have the configuration of an information processing apparatus represented by a PC (personal computer) in terms of hardware. Then, the information processing apparatus functions as one of the image processing apparatuses 1001 to 1016 or as the system control apparatus 120 according to the program executed by the CPU in the information processing apparatus.

FIG. 10 shows the basic hardware configuration of the information processing device 200. As shown in FIG. The information processing apparatus 1200 has a CPU 1211, a ROM 1212, a RAM 1213, an auxiliary storage device 1214, a display section 1215, an operation section 1216, a communication I/F 1217, and a bus 1218 connecting these components to each other.

The CPU 1211 controls the entire information processing apparatus 1200 using computer programs and data stored in the ROM 1212 and RAM 1213 . Depending on this program, the information processing device 1200 functions as one of the image processing devices 1001 to 1016 in FIGS. The information processing apparatus 1200 may have one or a plurality of pieces of dedicated hardware different from the CPU 1211, and the dedicated hardware may execute at least part of the processing by the CPU 1211. FIG. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). ROM 1212 stores programs that do not require modification. The RAM 1213 temporarily stores programs and data supplied from the auxiliary storage device 1214 and data externally supplied via the communication I/F 1217 . The auxiliary storage device 1214 is composed of, for example, a hard disk drive, etc., and stores various data such as image data and audio data. Note that the system control device 120 must process more information than the image processing device 1001 does. Therefore, the system control device 120 is required to have higher resources (mainly CPU processing power) than the image processing device 1001 .

FIG. 2 shows a configuration diagram of an image processing system for virtual viewpoint video in the first embodiment. As described above, image processing apparatuses 1001 to 1016 are connected to cameras 101 to 116, respectively, and images captured by each camera are transferred to the connected image processing apparatuses. Each image processing device extracts the foreground area from the camera-captured image from the corresponding camera by the background subtraction method, and transfers it to the downstream image processing device connected by the daisy chain. Then, the foreground image group extracted by the image processing devices 1001 to 1016 is recorded and held in the foreground recording unit 210 of the system control device 120 from the image processing device 1001 at the end of the daisy chain.

The inside of the studio is photographed from all directions by the cameras 101 to 116 surrounding the studio, and the image area of the performer 140 from all directions is also extracted as a foreground image. Each extracted foreground image is recorded and held in foreground recording section 210 of system control device 120 .

The three-dimensional shape derivation unit 211 uses the foreground image group recorded and held in the foreground recording unit 210 to generate a three-dimensional model of the foreground object. The virtual viewpoint video generation unit 212 generates a virtual viewpoint image by mapping a texture that matches the virtual viewpoint to the three-dimensional model of the foreground object according to the set position and orientation of the virtual viewpoint and performing rendering. .

The virtual viewpoint video output unit 213 converts the virtual viewpoint video created by the virtual viewpoint video generation unit 212 into a video signal for display on a display and outputs the video signal, or converts the data format for writing to an external recording device. , converts to a data format suitable for the output device.

The virtual viewpoint input unit 214 inputs virtual viewpoint information representing the position and direction of the virtual viewpoint according to the user's operation. The input virtual viewpoint information is supplied to the virtual viewpoint video generation unit 212 . The virtual viewpoint input unit 214 may be input by the user operating the system control device 120, or may be input via another terminal on the network. In this case, the virtual viewpoint input unit 214 has a function of receiving virtual viewpoint information from other terminals on the network.

Next, the image processing apparatuses 1001-1016 will be described with reference to FIG. Although FIG. 2 illustrates the configuration of one of the image processing apparatuses 1002, it should be understood that the other image processing apparatuses have the same configuration.

In the background subtraction method, a background image is stored in advance, and at the time of shooting, the background image stored in advance and the shot image are compared pixel by pixel, and the area having a difference from the background image is taken as the foreground image area. is. Therefore, in the background video shooting, the cameras 101 to 116 including the camera 102 simultaneously shoot the inside of the studio without anyone in the studio.

A studio background image captured by the camera 102 is recorded and held in the background image storage unit 205 of the image processing device 1002 .

FIG. 3(b) shows the same studio area as in FIG. 3(a), where the studio wall surface 100 and the floor surface are photographed in the camera photographing field of view 301, and recorded and held in the background image storage unit 206 as, for example, an rgb color image. be done.

Once the recording of the background image is completed, the performers are put into the studio and the virtual viewpoint video is started to be shot. As described above, the video projector 130 projects progress instructions from the program staff (not shown) on the studio wall surface 100 according to the progress of filming, and the performers 140 are projected as projection images 302 in FIG. 3(a). Confirm the progress instructions and continue the performance.

Here, from the photographed image photographed by the camera 102, the performer 140 which is the foreground area, the studio wall surface 100 which is the background area, and the projector projected onto the studio wall surface 100 which is the projected image area projected on the background area Processing for separating each region of the projection image 302 is called region separation processing. This area separation processing is performed by the foreground/background separation unit 204 of the image processing device 1002 .

First, the foreground/background separation unit 204 that separates and extracts only the performer 140, the projected image 302 from the projector, and the performer 140, which is the foreground, from the studio wall surface 100 will be described.

The color space conversion unit 201 of the image processing device 1002 converts the input photographed image into a signal form consisting of a brightness signal and a color signal. In this embodiment, the camera output signal is converted from an RGB signal consisting of red, green and blue to an LCH signal in a color space consisting of lightness (L), saturation (C) and hue (H). Lightness (L), saturation (C), and hue (H) are also called lightness (L), saturation (S), and hue (H). Without change, the conversion formula is, for example:
If MIN = B then H = 60(GR)/(MAX-MIN)+60
If MIN = R then H = 60(BG)/(MAX-MIN)+180
If MIN = G then H = 60(RB)/(MAX-MIN)+300
If MIN = MAX then H = undefined
L = (MAX + MIN)/2
C = (MAX - MIN)/(1-|MAX + MIN -1|)
In addition, the above formula is an example, and is not limited to this.

The color space conversion unit 201 supplies the captured image converted into the LCH signal to the foreground/background separation unit 204 in order to extract the area of the performer 120, which is the foreground.

On the other hand, the background image recorded and held in the background image storage unit 205 is also converted from lightness (l), saturation (c), and hue (h) to is converted to the lch signal. Although the color space conversion unit 201 and the color space conversion unit 202 are shown independently in FIG. 2, they may be common.

The lch signal converted by the color space converter 202 is input to the threshold generator 203 . The threshold generation unit 203 generates a new threshold l ₂ , which will be described later, and supplies the lch and l ₂ signals to the foreground/background separation unit 204 .

A foreground/background separation unit 204 performs background separation processing using the captured image (LCH), the background image (lch), and the threshold _l2 . The state of foreground/background separation will be described below with reference to FIGS.

FIG. 4(a) represents a color space consisting of lightness (L), saturation (C), and hue (H). The LCH color space, which is a polar coordinate system, is a color space that takes lightness (L) on the vertical axis, saturation (C) on the axis orthogonal to the lightness axis, and hue (H) on the angle from the saturation axis. , and the chromaticity coordinates (lch) of green, which is the studio background, are located at reference numeral 401 in FIG. 4(a).

The foreground/background separation unit 204 determines the degree of matching between the captured image and the background image for each pixel of the captured image shown in FIG. . That is, when the captured image signal (LCH) and the background signal (lch) match within a predetermined allowable error range, the corresponding pixel of the captured image signal (LCH) is determined to be "background". At this time, the foreground/background separation unit 204 sets the permissible errors for lightness (L), saturation (C), and hue (H) to dl, dc, and dh, and sets them to the external variable setting unit 209 through the UI unit 208 of the system control device 120. and input to the foreground/background separation unit 204 .

Here, the separation extraction of the background area excluding the area of the performer 140 and the projector projection image 302 from the photographed image of FIG. 4B will be described. When the chromaticity coordinates (lch) of each pixel of the background image (reference numeral 301 in FIG. 3(b)) are positioned at reference numeral 401 in FIG. Compare the chromaticity coordinates (LCH) and size of pixels at the same position in the captured image.

When the absolute difference between the signals at the coordinates (L, C, H) and (l, c, h) is less than the permissible error (dl, dc, dh), the foreground/background separator 204 separates the captured image (Fig. 4 The pixel at the target position of (b)) is judged to be the background.
If (|Ll| < dl and |Cc| < dc and |Hh| < dh) then target pixel = background pixel

The black-painted portion in FIG. 4(c) is the "background" area of the captured image in FIG. 4(a). Actor 104 and projector projection image 302 are not "background" regions because they are different from the background color. As described above, the background area is extracted from the photographed image of FIG. 4(a).

Next, a method for separating and extracting the projector projection image 302 area from the photographed image of FIG. 4B will be described.

As previously mentioned, projector 130 projects white light of constant brightness onto the studio background. At this time, the projector projection image 302 projected onto the studio wall surface is not affected by the color of the studio wall surface due to the projection of the white light, and becomes brighter according to the intensity of the projector projection light. That is, (L, C, H) of the projector projection image 302 drawn on the background (l, c, h) differs only in brightness (L) in the LCH color space (brighter than the background L > l). , saturation (C=c), and hue (H=h) are the same as the background color (green).

The relationship between the brightness increment ΔL of the background projected image due to the light emission intensity of the video projector 130 and the drawing image signal level (usually 0 to 255 gradations) of the video projector 130 is measured in advance. Externally input from this system according to the signal level.

Since the projector projection image 302 has a portion that is brighter than the background brightness "l" by the increment ΔL, the bright portion of the projector projection image 302 is "l + ΔL". Therefore, the threshold generation unit 203 generates a threshold for Campe region separation.

When the background signal (lch) is input to the threshold generation unit 203, the brightness increment δL due to the projector drawing is added to the background brightness (l), and the threshold generation unit 203 determines the threshold _l2 for separating the projector projection image. It is generated according to the following formula.
_l2 = l + δL
Reference numeral 402 in FIG. 4A indicates the chromaticity coordinates of the projected image of the projector in the LCH color space. As described above, if the color coordinates (l, c, h) of each pixel of the background image are indicated by reference numeral 401, the projector projection image projected on the background is obtained by changing the brightness l from the color coordinates 401 of the background image to ΔL It is located at coordinates (l + δL,c,h) indicated by reference numeral 402 where it is only bright and the saturation (C) and hue (H) are the same as the background.

Therefore, when the absolute difference between the signals (L, C, H) and (l ₂ , c, h) is less than the allowable error (dl, dc, dh), the target position in the photographed image FIG. It is determined that the pixel at is the projected image of the projector.

If (|Ll ₂ | < dl and |Cc| < dc and |Hh|< dh) then the pixel of interest belongs to the projector projection image area

FIG. 4(d) shows the projector projection image area separated and extracted in this way.

As described above, by ORing the background area (FIG. 48c) and the projector projection image area (FIG. 4D), the background area and the projector projection image area represented by the black area in FIG. 4E are extracted. The foreground/background separation unit 204 collectively performs the above processes.
(|Ll|< dl and |Cc|< dc and |Hh|< dh) OR (|Ll ₂ |< dl ₂ and |Cc|< dc and |Hh|< dh) then belong to a region or projector projection image region

That is, in order to extract the performer 140 which is the foreground, the foreground/background separation unit 204 extracts the area excluding the background area and the projector projection image area as the foreground image area as follows.
NOT ((|Ll|<dl or | _Ll2 |< _dl2 ) and |Cc|<dc and |Hh|<dh) then The pixel of interest in the captured image belongs to the foreground image. , represents the foreground region that is the performer 104 thus extracted and separated.

The processing flow will be described below with reference to FIGS.

FIG. 5(a) is a flowchart of background image acquisition in the background subtraction method. When acquiring a background image, first remove the installed objects and people in the studio to make the studio empty (S5a01). Next, without projecting the video projector 103, the studio lighting (not shown in the present embodiment) is turned on under the same conditions as at the time of shooting (S5a02). In this state, cameras 101 to 116 simultaneously photograph the inside of the studio (S5a03). The photographed studio background image is transferred to the image processing apparatuses 1001 to 1016 and recorded in the background image storage section 206 in each image processing apparatus (S5a04).

In FIG. 5(b), the brightness increment δL of the background projection image due to the light emission of the video projector 130 is determined for each drawing image signal level of the video projector 130 (8-bit gradation 0 to 255 in this embodiment). 1 is a flow diagram for FIG. The measurer prepares in advance a gray scale chart for projection on the projector.

FIG. 5(c) is an example of a grayscale chart 5C01, in which stripes of drawing image signal levels (0 to 255 gradations) are created. The projector projects from a PC (not shown) onto the studio wall surface 100 (S5b01). The camera captures the grayscale projected on the studio wall (S5b502). The grayscale photographed image projected on the studio wall is converted into an image signal (LCH) in the LCH color space (S5b502). On the other hand, the background image recorded and held in the flow of FIG. 5(a) is read from the background image storage unit 206 (S5b504) and similarly converted into an image signal (lch) in the LCH color space (S5b505). For each signal (LCH, lch), the brightness difference is calculated (S5b505), and the brightness difference (difference) corresponding to the light emission level of projector 130 is calculated (S5b506).
δL = L - l (δL, L, l, have different L values for each brightness of the grayscale chart, and δL is the calculated brightness increment for 256 shades)

FIG. 5(d) is a flow for recognizing the foreground, which is the performer, and the area including the competition portion projected by the projector 130 and the background from the image captured by the camera.

When camera shooting starts in S5d01, a campe image is projected from projector 130 as needed (S5d02). A photographed image 301 including a competition as shown in FIG. 3(b) is converted from an RGB image signal to an LCH signal (S5d03). On the other hand, the rgb background image recorded and held in the background image storage unit is read out and converted into an lch signal. The brightness increment ΔL at the time of projector projection is added to the background lch signal, and the brightness threshold value _L2 of the rough portion projected by the projector 130 is calculated.
_L2 = l + δL

The foreground/background separation unit 204 performs foreground/background separation processing using the captured image LCH and the thresholds l, c, h, and L2 of the background image and the rough portion (S5d04).

After the foreground, the background, and the projected image of the performer are separated from the image captured by the camera as described above, only the foreground area image is transferred to the transmission unit 207 . The other image processing apparatuses 1001, 1003-1016 also transmit only the foreground area via transmission units in a similar manner. Ultimately, the foreground recording unit 210 of the system control device 120 records and holds the foreground image group of each camera.

The three-dimensional shape derivation unit 211 generates a three-dimensional model of the foreground object from the foreground image group recorded and held in the foreground recording unit 210. FIG. The virtual viewpoint video generation unit 212 maps a texture that matches the virtual viewpoint to the three-dimensional model of the generated foreground object, and performs rendering to generate a virtual viewpoint image.

The virtual viewpoint video output unit 213 converts the virtual viewpoint video created by the virtual viewpoint video generation unit 212 into a video signal for display on a display and outputs the video signal, or converts the data format for writing to an external recording device. , converted to a data format suitable for the output device
[Application example 1 of the first embodiment]
FIG. 6A is a configuration diagram of a virtual viewpoint video studio in another application of the first embodiment. In the studio, there are multiple projectors (four units 130, 601, 602, and 603 in the illustration) are arranged so that they can project onto the walls in any direction in the east, west, south, or north, and the performers face each other according to the performance. By displaying a projected image of the progress instruction issued by the program staff in front, it is possible to confirm the progress instruction without hindrance to the performance.

[Application example 2 of the first embodiment]
In the above application example 1, a plurality of projectors (130, 601, 602, 603) shown in FIG. An application example for confirming instructions was shown, but the background images to be synthesized in the production of the final image of the program were created in advance instead of the instructions given by the program staff to project the images projected by the projectors, and multiple projectors (130, 601, 602, 603 ) is used to project onto the north, south, east, and west walls of the studio, allowing the performers to act as if they were immersed in the final image.

[Second embodiment]
In the first embodiment, in order to separate the projector projected image by the foreground/background separation unit 204, the threshold generation unit 203 generates the threshold l ₂ for separating the projector projected image from the studio background image recorded in the background image recording unit 205. generated.

FIG. 7 shows a configuration diagram of an image processing system for virtual viewpoint video in the second embodiment. In the second embodiment, instead of calculating the threshold value _l2 for separating the projected image from the projector, an image for separating the projected image from the projector is captured separately and used by the foreground/background separating unit 204 .

That is, this is the second background image in which the projector is turned off for the recording of the studio background image in the first embodiment, the studio background image is photographed, the recording is held in the background image recording unit 205, and then the projector is turned on. A studio background image 2 is photographed, recorded and held in the background image recording unit 705, and recognition processing of the projector projection image area is performed.

Since each image processing device connected to each camera has a second background image, there is no need to calculate the threshold l ₂ =l + ΔL for projector projection image area recognition.
In this embodiment, in addition to the flow of background image acquisition in the background subtraction method described in the first embodiment, the background image acquisition flow during projection shown in FIG. 6B is added. .

Steps S5a01 to S5a04 in the background image acquisition flow during projector projection in FIG. 6(b) are the same as in FIG. 5(a). In the second embodiment, after step S5a04 is finished, studio lighting and projectors (not shown) are turned on (Fig. 6(a) and Fig. 6(b) S6b01), and camera shooting is performed in this state (S6b02 ), the photographed image is recorded in the ruler 2 for the studio background image.

The background image recorded in steps S5a01 to S5a04 is recorded in the studio background image recording section 1 in FIG. Also, the background image recorded in steps S6b01 to S6b03 is recorded in the studio background image recording section 2 in FIG.

For the two background images, an lch signal and an _l2ch signal are generated from the rgb image by the color space conversion section and input to the foreground/background separation section.

In addition, dl, dl ₂ , dc, and dh are input from the UI unit to the foreground/background separation unit, and the discriminant
NOT (|Ll|<dl or | _Ll2 |< _dl2 ) and |Cc|<dc and |Hh|<dh) then the pixel of interest in the captured image belongs to the foreground image.

[Third embodiment]
FIG. 8 shows a configuration diagram of an image processing system for virtual viewpoint video in the third embodiment. In the third embodiment, a plurality of projectors (130, 601, 602, 603) as shown in FIG. This is an embodiment of an image processing system for virtual viewpoint video in which recognition processing of a projector projected image area is performed only on the projected image.

In FIG. 6, it is assumed that the projector projects a projected image onto two walls, the north wall and the west wall of the studio. In FIG. 9, of the four projectors, the projector 601 that projects on the north wall of the studio and the projector 130 that projects on the west wall of the studio project the projected images in the area indicated by the dotted line, and the projectors 602 and 603 are turned off. It shows how it is. At this time, among the cameras 101 to 126, the cameras 112 to 116 capture images including the projector projected image on the north wall of the studio by the projector 601, and the cameras 101 to 103 capture the projector projected images on the west wall of the studio by the projector 130. Shoot a video that contains Other cameras 104 to 111 photograph the studio walls and performers without the projected image of the projector.

In such a shooting environment, the system control device 120 of FIG. 8 transmits a projected image of the projector to the projector (130, 601) through the user's operation of the system (not shown) from its UI unit. In general, the UI section has a menu for selecting a projector for image output, and the system user designates a projector from the menu and outputs an image.

When a system user designates a projection projector, the system control device 120 connects to each camera a signal indicating that the image captured by the camera includes the projected image of the projector. sent to the specified image processing device. For example, a 1-bit signal, which is 1 when the projected image of the projector is included in the image captured by the camera and 0 when the projected image of the projector is not included, is sufficient. A 1-bit projector projection image presence/absence signal is input to the foreground/background separator of the image processing device connected to each camera. When a camera-captured image includes a projector-projected image, the corresponding image processing device recognizes three types of foreground, background, and projector-projected images. Also, the corresponding image processing device when the image captured by the camera does not include the projected image of the projector recognizes two types of images: the foreground image and the background image.

In this way, the presence or absence of a projector projected image is known in advance in the foreground/background separation process, and if the projector projected image is not included, the foreground/background separation process load can be reduced by not executing the projector projected image recognition process in the foreground/background separation process. In addition, an effect of avoiding erroneous determination is obtained.

In the above embodiment, the image processing device and the system control device 120 are daisy chain connected, but the communication connection may be a star type with a hub intervening, as long as a high communication speed can be secured. For example, wireless LAN communication may be used.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

101-116... Camera, 120... System control device, 130... Projector, 1001-1016... Image processing device

Claims (9)

An image processing system that generates a virtual viewpoint image representing a view from a virtual viewpoint based on captured images obtained by capturing a space including a subject with a plurality of capturing devices,
Projecting means for projecting an image of a predetermined color with a brightness different from that of the background onto a background of a predetermined color different from a foreground subject;
a separation means for separating the foreground from the captured image based on color difference;
An image processing system comprising: The separating means separates the foreground and the background based on the coordinates of the predetermined color in a predetermined color space and the coordinates represented by the color obtained by adding the lightness in the image projected by the projecting means. The image processing system according to claim 1, wherein: 3. The image processing system according to claim 2, further comprising generating means for generating a virtual viewpoint image based on a set virtual viewpoint from the foreground separated by said separating means. 4. The image processing system according to claim 3, wherein said projection means includes a plurality of projectors for projecting onto a plurality of positions of said background. The separating means is
a first determination means for determining, as a pixel belonging to the background, a pixel having a color within a range separated from the coordinates representing the color of the background by a preset allowable error in a predetermined color space;
A pixel having a color in a range from the coordinate representing the color of the background in the predetermined color space to the coordinate spaced apart by the lightness increment of the image projected by the projection means is defined as a pixel belonging to the image projected by the projection means. and a second determination means for determining,
5. The image processing system according to claim 4, wherein pixels excluding pixels of colors determined by said first and second determining means are separated as pixels belonging to said foreground. a generation device having the generation means;
6. The image processing according to claim 5, further comprising a plurality of image processing devices connected to each of said photographing devices and having said separation means and transmission means for transmitting the separated foreground images to said generation device. system. selecting means for selecting which one of the plurality of projectors is used for projection;
The image processing device connected to the photographing device having the range projected by the projector selected by the selecting means as the photographing range is configured to separate the background image and the projected image from the photographed image and generate a foreground image. Set,
setting means for setting an image processing device connected to an imaging device whose projection range of the projector selected by the selection is outside the imaging range to separate a background image and generate a foreground image;
7. The image processing system according to claim 6, comprising: A control method for an image processing system for generating a virtual viewpoint image representing a view from a virtual viewpoint based on captured images obtained by capturing a space including a subject with a plurality of capturing devices, comprising:
a projecting step of projecting an image of a predetermined color with a brightness different from that of the background onto a background of a predetermined color different from the foreground subject;
a separation step of separating the foreground from the captured image based on color differences;
A control method for an image processing system, comprising: A program that is read and executed by a computer to cause the computer to execute each step of the control method according to claim 8.

Images