JP2018116421A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable the production of a high-quality free-viewpoint video using a billboard.SOLUTION: An image processing device includes: means for acquiring multiple images acquired by picking up images of an object in a real space from multiple viewpoints; means for calculating coordinates that indicate the position of the object in the real space from the acquired multiple images; and means for creating a synthesized image corresponding to a viewpoint not included in the multiple viewpoints by disposing, with reference to the calculated coordinates, a billboard of the object created from at least one image of the multiple acquired images.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus and an image processing method.

  2. Description of the Related Art Conventionally, a technique for generating a video from a free viewpoint other than the camera viewpoint (hereinafter referred to as a free viewpoint video) has been proposed for a sports scene or the like. This technology synthesizes videos from virtual viewpoints that are not arranged based on videos taken by multiple cameras, and displays the results on the screen for viewing videos from various viewpoints. It is possible.

  Here, among techniques for synthesizing free viewpoint videos, there is a technique for synthesizing free viewpoint videos at high speed using a simple model called a billboard (see Non-Patent Document 1). In the technology using this billboard, the texture of the object to be modeled is accurately cut out from the video, and it is made to stand on the ground in a virtual space as a thin billboard model, thereby generating a free viewpoint video.

  Here, in general, in the billboard system, the billboard is arranged such that the lowest point (for example, a person's foot) of a certain billboard is in contact with the ground of the virtual space. When the virtual viewpoint moves in the horizontal direction, the billboard is rotated in accordance with the movement of the virtual viewpoint, and when the virtual viewpoint moves in the vertical direction, the direction of the billboard is not changed.

Hayashi, K .; Saito, H., "Synthesizing Free-Viewpoing Images from Multiple View Videos in Soccer Stadium ADIUM," in Computer Graphics, Imaging and Visualisation, 2006 International Conference on, vol., No., Pp.220-225, 26-28 July 2006

  The method described in Non-Patent Document 1 is superior in that a high-quality free viewpoint video can be synthesized at a high speed and the size of the synthesized content data is smaller than other methods. However, due to the restriction that the billboard stands vertically to the ground, a situation may occur where the subject in real space and the billboard do not correspond. In this case, the obtained free viewpoint video may become unnatural.

  The present invention has been made in view of these problems, and an object thereof is to provide a technique that enables generation of a higher-quality free viewpoint video using a billboard.

  One embodiment of the present invention relates to an image processing apparatus. The image processing apparatus calculates a coordinate representing the position of the subject in the real space from the plurality of acquired images and means for acquiring a plurality of images obtained by imaging the subject in the real space from a plurality of viewpoints. And a viewpoint that is not included in multiple viewpoints by placing the billboard of the subject generated from at least one of the acquired multiple images with reference to the calculated coordinates Generating a composite image.

  It should be noted that any combination of the above-described constituent elements, or those obtained by replacing the constituent elements and expressions of the present invention with each other between apparatuses, methods, systems, computer programs, recording media storing computer programs, and the like are also included in the present invention. It is effective as an embodiment of

  According to the present invention, it is possible to generate a higher-quality free viewpoint video using a billboard.

It is a schematic diagram which shows arrangement | positioning of the billboard in the conventional billboard system. It is a schematic diagram which shows a free viewpoint image delivery system provided with the image processing apparatus which concerns on embodiment. It is a block diagram which shows the function and structure of the image processing apparatus of FIG. It is explanatory drawing which shows the correspondence of the coordinate on the image plane of a camera, and a field coordinate. 5A to 5C are explanatory diagrams illustrating an example of processing in the background difference unit of FIG. It is a schematic diagram which shows the three-dimensional model produced | generated by the three-dimensional process part of FIG. 3, and its model representative point. It is a schematic diagram which shows a billboard reference plane. FIGS. 8A to 8C are schematic diagrams for explaining billboard generation processing in the billboard generation unit of FIG. 3. It is a flowchart which shows the flow of a series of processes in the image processing apparatus of FIG. It is explanatory drawing of the method which concerns on the modification which determines the coordinate which should arrange | position a billboard from the several image image | photographed with the some camera.

  Hereinafter, the same or equivalent components, members, and processes shown in the drawings are denoted by the same reference numerals, and repeated description is appropriately omitted. In addition, in the drawings, some of the members that are not important for explanation are omitted.

  In the conventional billboard method, the billboard is arranged such that the lowest point (for example, a person's foot) touches the ground (hereinafter referred to as a field) of the virtual space. The inventor has uniquely recognized the following problems associated with this restriction.

  FIG. 1 is a schematic diagram showing the arrangement of billboards 10 in a conventional billboard system. The camera 14 takes a picture (also referred to as imaging) where the subject person 12 jumps high from the field. The billboard 10 of the person 12 is generated from the image obtained from the camera 14, but the generated billboard 10 is the actual person 12 because of the restriction that the lowest point of the billboard 10 must touch the field. It is arranged not at the position 16 but at the intersection 18 between the line of sight 20 of the camera 14 and the field. In this case, since the billboard 10 viewed from the virtual viewpoint is arranged at coordinates completely different from the position 16 of the original person 12, the synthesized video quality can be degraded. Such a problem may frequently occur in a scene where a person jumps high near the camera, such as volleyball or basketball (hereinafter, the action of the subject leaving the field is referred to as air movement).

  In the conventional billboard method, when a video from a virtual viewpoint is generated, a billboard of a subject cut out from a corresponding video from one fixed camera is arranged in a three-dimensional space. However, it is difficult to specify the true position of the subject from the video from one fixed camera if there is no precondition such as being in contact with the field. Further, in shooting performed at a relatively short distance such as volleyball or basketball shooting, since the ratio of the person's air movement in the screen is large, the video quality is likely to deteriorate.

  On the other hand, the image processing apparatus according to the embodiment estimates the position of the subject in real space from a plurality of images obtained from a plurality of photographing apparatuses (for example, cameras). The estimated position is not limited to the field and may be in the air. The image processing apparatus arranges the billboard with reference to the estimation result, thereby enabling a more natural display when the billboard free viewpoint video technology is applied to the video accompanied by the air movement. The estimation of the position of the subject is performed, for example, by generating a three-dimensional model of a thick subject.

  FIG. 2 is a schematic diagram illustrating a free viewpoint image distribution system 110 including the image processing apparatus 200 according to the embodiment. The free-viewpoint image distribution system 110 includes a plurality of cameras 116, 118, and 120, an image processing device 200 connected to the cameras, and a mobile terminal 114 such as a mobile phone, a tablet, a smartphone, or an HMD (Head Mounted Display). . The image processing apparatus 200 and the portable terminal 114 are connected via a network 112 such as the Internet. In the free-viewpoint image distribution system 110, for example, a plurality of cameras 116, 118, 120 arranged in the arena shoots a volleyball player 124 standing on the field 126 or jumping high. The plurality of cameras 116, 118, and 120 transmit captured images to the image processing apparatus 200, and the image processing apparatus 200 processes these images. The user of the portable terminal 114 designates a desired viewpoint with respect to the image processing apparatus 200, and the image processing apparatus 200 synthesizes an image when the player 124 is viewed from the designated viewpoint (virtual viewpoint) via the network 112. To the mobile terminal 114.

  In addition, although FIG. 2 demonstrated the case where the volleyball player 124 in an arena was image | photographed, it is not restricted to this, For example, when shooting a fitness instructor, when shooting a tennis game, or when shooting a soccer game The technical idea of the present embodiment can be applied when shooting a subject that can move in the air. In addition, the present invention is not limited to photographing sports scenes, and the technical idea of the present embodiment can be widely applied to any application that can photograph the same subject on a plurality of videos obtained from a plurality of cameras. Further, a stationary terminal such as a desktop PC, a laptop PC, or a TV receiver may be used instead of the portable terminal 114.

  FIG. 3 is a block diagram showing functions and configuration of the image processing apparatus 200 according to the embodiment. Each block shown here can be realized by hardware such as a computer (CPU) (Central Processing Unit) and other elements and mechanical devices, and software can be realized by a computer program or the like. The functional block realized by those cooperation is drawn. Therefore, it is understood by those skilled in the art who have touched this specification that these functional blocks can be realized in various forms by a combination of hardware and software.

  The image processing apparatus 200 synthesizes an image of an arbitrary virtual viewpoint from images taken by a plurality of cameras 116, 118, and 120. The image processing apparatus 200 generates a free viewpoint video in accordance with a billboard system based on a plurality of camera videos obtained by photographing the same subject. In the conventional billboard method, only the video from one camera is used when generating the billboard, but the image processing apparatus 200 according to the present embodiment uses a three-dimensional model calculated from a plurality of camera videos. Use coordinate data.

  In the following, it is assumed that time synchronization between the plurality of cameras 116, 118, 120 is performed in advance. In the following, a person is assumed as a subject, but the technical idea of the present embodiment can be applied to other subjects. The virtual viewpoint is a virtual viewpoint that can be arbitrarily designated by the user, for example, and has a different meaning from an actual viewpoint in which a plurality of cameras 116, 118, and 120 are arranged.

[Outline of Image Processing Device 200]
The image processing apparatus 200 includes a calibration unit 202, a background difference unit 204, a three-dimensional processing unit 206, a reference plane calculation unit 208, a billboard generation unit 210, a projection point calculation unit 212, and a free viewpoint video. A generation unit 214. The calibration unit 202 receives a plurality of images captured by the plurality of cameras 116, 118, and 120 as inputs. The calibration unit 202 associates a field in the real space with an image photographed by the camera for each camera and outputs it as calibration data. If it is assumed that the camera is fixed, this calibration operation in the calibration unit 202 need only be performed once at the beginning. The background difference unit 204 classifies an image into a background and a foreground using a known background difference method, and outputs a binarized image as mask data.

  The three-dimensional processing unit 206 generates a three-dimensional model of the subject from a plurality of mask data obtained from a plurality of images captured by the plurality of cameras 116, 118, and 120. The three-dimensional processing unit 206 estimates the position of the subject in real space from the generated three-dimensional model. The three-dimensional processing unit 206 outputs coordinates obtained as a result of estimation. The 3D processing unit 206 includes a 3D model construction unit 216, a model representative point calculation unit 218, and a smoothing unit 220.

  The three-dimensional model construction unit 216 uses the foreground mask generated by the background difference unit 204 and the calibration data generated by the calibration unit 202 to generate a three-dimensional model having a thickness different from that of the billboard. The model representative point calculation unit 218 calculates the coordinates of a model representative point that is a representative point (for example, robust to a change in the posture of a person) representing the three-dimensional model. The smoothing unit 220 smoothes the coordinates of the model representative points calculated by the model representative point calculation unit 218 in the time axis direction.

  The reference plane calculation unit 208 sets a billboard reference plane that is a plane on which the billboard is arranged. The reference plane calculation unit 208 sets the billboard reference plane so as to include the model representative points. The projected point calculation unit 212 calculates coordinates in the image plane that are referred to when the billboard is generated. The billboard generator 210 generates and outputs a billboard for the subject. The free viewpoint video generation unit 214 generates and outputs a free viewpoint video using the billboard output from the billboard generation unit 210, the billboard reference plane, and the model representative point.

[Calibration unit 202]
The calibration unit 202 acquires a plurality of videos from the plurality of cameras 116, 118, and 120. For each of the plurality of cameras 116, 118, and 120, the calibration unit 202 uses the characteristic points of the field (such as the intersection of the white lines of the coat) in the image taken at a certain time and the actual field in the real space. Are associated with the points and calculated as camera parameters. For example, when shooting a general sports game, since the size of the court is standardized, the calibration unit 202 corresponds to which coordinate in the real space (world coordinate system) the point on the image plane corresponds to Can be calculated. Camera parameters include external parameters and internal parameters. The external parameter is a parameter indicating the relationship between the real space and the camera, and includes, for example, a parameter indicating the camera position (viewpoint position) and the camera posture (rotation). Internal parameters are camera-specific parameters and include, for example, lens distortion.

FIG. 4 is an explanatory diagram showing the correspondence between the coordinates on the image plane of the camera 402 and the field coordinates. When the coordinates on the two-dimensional image plane of the camera 402 are (u, v) and the coordinates (field coordinates) on the field plane of the world coordinate system are (x ′, y ′), the correspondence between the two is homography. matrix
And a scalar value s can be expressed as follows.
... (Formula 1)

  S and H can be obtained by inputting the set of corresponding points in Equation 1, and mutual conversion between the coordinates of an arbitrary pixel on the image plane and the field coordinates becomes possible. The camera calibration method for the field is not limited to the above.

  Returning to FIG. 3, the calibration of the camera in the calibration unit 202 may be performed using a technique related to a known automatic calibration in addition to the manual operation. As a manual method, for example, there is a method of estimating camera parameters by selecting an intersection of white lines on the screen by a user operation and associating it with a field model measured in advance. If the screen is distorted, the internal parameters are estimated as follows. As an example of a method of automatically performing the same operation, a method of extracting a white line in the screen using threshold processing and estimating a coordinate in the screen of an intersection point by extracting a linear component by Hough transform, etc. There is.

  On the other hand, when a camera including a wide-angle lens such as a fisheye lens is used for shooting, the calibration unit 202 estimates the internal parameters of the camera individually and corrects the screen distortion. This estimation may be executed by photographing a geometric pattern such as a checkerboard with a camera used for photographing in advance. When shooting with a fixed camera is assumed, the calibration unit 202 may perform camera calibration once at the beginning of video generation. In addition, when it is assumed that shooting is performed with a moving camera, the calibration unit 202 performs the above-described known automatic calibration for each frame. The calibration unit 202 generates calibration data including the calculated camera parameters and outputs the calibration data.

Hereinafter, a process for an image captured by each of the plurality of cameras 116, 118, 120 at a certain time t will be described.
[Background difference unit 204]
The background difference unit 204 classifies each pixel of an image at a certain time t from each camera into a background and a foreground, thereby dividing the image into a background and a foreground. In the present embodiment, this separation may be realized using a known background subtraction method, for example. The background difference unit 204 generates a foreground mask by assigning binary values such as 0 and 1 to the pixel values set as the background and the foreground, respectively. By separating the background and the foreground, a rough area including the subject can be extracted.

  FIGS. 5A to 5C are explanatory diagrams illustrating an example of processing in the background difference unit 204. FIG. 5A shows an image 502 at time t taken by a certain camera. The background difference unit 204 processes this image 502 as an original image. FIG. 5B shows a foreground mask 504 obtained as a result of applying the background difference method to the original image of FIG. In the foreground mask 504, the black portion is determined to be the background and 0 is assigned. The white part is determined to be the foreground, and 1 is assigned. FIG. 5C shows the texture of the person 506 obtained from the original image of FIG. 5A and the foreground mask 504 of FIG.

[Three-dimensional model construction unit 216]
Returning to FIG. 3, the three-dimensional model construction unit 216 acquires the calibration data generated by the calibration unit 202 and the foreground mask generated by the background difference unit 204. The three-dimensional model construction unit 216 uses the acquired information to construct a three-dimensional model that schematically represents the shape of the subject in a virtual three-dimensional space that imitates the real space. When the subject is a person, the three-dimensional model may represent a general shape of the person's body. This three-dimensional model is represented by a polygon mesh model or a voxel model. The three-dimensional model has a thickness different from the billboard model.

In order to extract a three-dimensional model from a plurality of foreground masks derived from a plurality of cameras, a known visual volume intersection method may be used. In this method, a space consisting of a group of cubes uniformly divided in the vertical, horizontal, and depth directions, called a voxel space, is defined in a three-dimensional space. By projecting the silhouette of the foreground mask derived from each camera into the voxel space, a three-dimensional outline of the actual subject is obtained. If the foreground mask derived from the i-th camera is M _i ^t and the silhouette mask obtained by projecting the three-dimensional model onto the image plane of the i-th camera is N _i ^t , theoretically
It becomes.

  The foreground mask derived from each camera generally includes a foreground that is erroneously detected due to noise. However, in the above method, information of a plurality of foreground masks projected onto the voxel space is integrated, so that such excessive foreground detection can be reduced. Therefore, in the subsequent processing, a silhouette mask obtained by projecting the three-dimensional model onto the image plane of the camera may be used as the foreground mask.

[Model representative point calculation unit 218]
The model representative point calculation unit 218 calculates the coordinates of the model representative point representing the position of the three-dimensional model (voxel data) of the subject generated by the three-dimensional model construction unit 216 as coordinates representing the position of the subject in real space. . Any model representative point can be used as long as it represents the position of the three-dimensional model. However, from the viewpoint of video quality including the connection of movements in the time axis direction, it is desirable to adopt a point that does not largely depend on a change in the posture of a subject (for example, a person or an animal). That is, since the hand and feet are highly likely to change rapidly depending on the posture of a person or animal, it is desirable to adopt points included in the skeleton such as the trunk and waist as model representative points.

  Alternatively, the model representative point calculation unit 218 may determine a model representative point by statistically processing the generated three-dimensional model in the voxel space, and calculate the coordinates thereof. For example, the model representative point calculation unit 218 may calculate the coordinates of the center of gravity of all voxels included in the three-dimensional model of the subject as the coordinates of the model representative point. In this case, model representative points that are robust against changes in the posture of the subject can be calculated. Alternatively, the coordinates of the model representative point may be obtained by other statistical processing as long as the property of being robust to changes in the posture of the subject is obtained. For example, the three-dimensional model is sequentially cut along a plane parallel to the xy plane, and the plane having the largest number of voxels included in the cut surface is specified as the most xy plane. Similarly, for the yz plane and the zx plane, the maximum yz plane and the maximum zx plane where the voxels included in the cut surface are the maximum are specified. The model representative point calculation unit 218 may determine an intersection point at which the most xy plane, the most yz plane, and the most yz plane intersect as a model representative point. As a more precise method, the model representative point calculation unit 218 performs tracking for each part of the person to generate a bone of a three-dimensional model, and determines a point corresponding to the waist or spine as the model representative point among the generated bones. May be. Alternatively, the model representative point calculation unit 218 may estimate the position of the person's waist from the image by machine learning, and determine the point obtained by the estimation as the model representative point.

[Smoothing unit 220]
The smoothing unit 220 statistically processes the coordinates of the model representative points of the three-dimensional model in the time axis direction. For example, the smoothing unit 220 smoothes the coordinates of the model representative points calculated by the model representative point calculation unit 218 in the time axis direction. When calculating the coordinates of model representative points, even if a calculation method that is robust against changes in the posture of the person is selected, unnatural motion ( (Coordinate movement). In order to reduce this, the smoothing unit 220 smoothes the coordinates in the time axis direction.

  For example, the smoothing unit 220 performs smoothing by applying a low-pass filter to the coordinates of model representative points in n frames before and after the current frame. In addition, the smoothing unit 220 generates a smooth trajectory using a spline curve or a B-spline curve with the model representative point sampled discretely as a control point, and the model representative point is determined to follow the generated trajectory. The coordinates may be moved. In this case, natural movement can be realized. In addition, the smoothing unit 220 may apply a time series filter such as a Kalman filter to the time series data of the coordinates of the model representative points. In this case, natural movement can be realized. Hereinafter, the model representative point is a model representative point smoothed by the smoothing unit 220.

  FIG. 6 is a schematic diagram showing a three-dimensional model 602 generated by the three-dimensional processing unit 206 and its model representative point 604. The three-dimensional processing unit 206 generates a three-dimensional model 602 of a subject (a person who has jumped high from the field 610) from images obtained from the plurality of cameras 606 and 608. The three-dimensional processing unit 206 calculates the coordinates of the center of gravity of the three-dimensional model 602 as the coordinates of the model representative point 604. As will be described later, the billboard of the subject is arranged with reference to the model representative point 604.

[Reference plane calculation unit 208]
Returning to FIG. 3, the reference plane calculation unit 208 determines the billboard reference plane based on the calculated model representative point coordinates and the external parameter (for example, the camera elevation angle α) generated by the calibration unit 202. FIG. 7 is a schematic diagram showing the billboard reference plane 702. The reference plane calculation unit 208 sets a plane that includes the model representative point 704 and forms the angle θ with the field 706 as the billboard reference plane 702. A free viewpoint video generation unit 214 described later arranges the billboard 710 so as to be superimposed on the set billboard reference plane 702. Therefore, θ is an angle formed by the billboard 710 of the subject and the field 706.

  The reference plane calculation unit 208 acquires the elevation angle α of the camera 708 from the external parameter generated by the calibration unit 202. The reference plane calculation unit 208 calculates θ by calculating θ = 90 (degrees) −α with respect to the elevation angle α (unit: degrees) of the camera 708. In this case, natural display can be realized. The external parameter represents the relationship between the absolute position T and the posture R (rotation matrix) between the world coordinate system and the camera coordinate system, and a desired α can be obtained by using a component of R among them. Can be calculated. The elevation angle α of the camera 708 also depends on the position of the camera 708. For example, when the camera 708 is installed at a higher position, the elevation angle α of the camera 708 also becomes larger.

  For example, in the case of a scene shot at a relatively close distance such as volleyball, it is more natural to arrange the billboard 710 by changing the angle θ formed with the field 706 in accordance with the elevation angle α of the camera 708. This is because the image of a person is actually taken obliquely from above. In particular, it is suitable from the viewpoint of perspective. That is, when a person is photographed at a close distance from obliquely above, the head near the camera appears relatively larger than the torso far from the camera. When this image is cut out and used as a billboard, and standing vertically on the field, when the billboard is viewed from a virtual viewpoint on the field (the line of sight is parallel to the field), it looks as if it is tilted toward the camera, which is unnatural. It is. Therefore, by tilting the billboard to the opposite side by the elevation angle of the camera, it is possible to express the proportion of the head and torso with a more uncomfortable feeling.

[Projected point calculation unit 212]
Returning to FIG. 3, the projected point calculation unit 212 determines the projected point on the billboard side using the calculated model representative point coordinates and the camera parameters generated by the calibration unit 202. In the conventional billboard method, the lowest point of the billboard is projected onto the field as a projection point. In the image processing apparatus 200 according to the present embodiment, since the 3D model side is used as a reference, the projected point in the billboard is different for each frame. The projected point calculation unit 212 determines the projected point by projecting the model representative point (coordinates thereof) onto the image plane of the camera. The free viewpoint video generation unit 214 arranges the billboard so that the projected point is on the billboard reference plane and coincides with the model representative point.

[Billboard generator 210]
The billboard generation unit 210 generates a billboard using the image from the camera, the foreground mask generated by the background difference unit 204, and the projection point calculated by the projection point calculation unit 212. FIGS. 8A to 8C are schematic diagrams for explaining billboard generation processing in the billboard generation unit 210. FIG. In each of FIGS. 8A to 8C, a texture 802 obtained by applying a foreground mask to an image from a camera is shown. As shown in FIG. 8A, when the projection point 806 is inside the subject and included in the circumscribed rectangle 804 of the mask area of the subject, the billboard generation unit 210 cuts out the circumscribed rectangle 804 and creates a billboard. To do. As shown in FIG. 8B, when the projection point 810 is outside the subject but is still included in the circumscribed rectangle 804 of the mask area of the subject, the billboard generation unit 210 cuts out the circumscribed rectangle 804 and extracts the billboard. And On the other hand, as shown in FIG. 8C, when the projection point 814 is not included in the circumscribed rectangle of the mask area, the billboard generator 210 generates a minimum rectangular area 812 including the projection point 814 and the mask area. Cut out to make a billboard. Here, the billboard generation unit 210 assigns the pixel values of the actual image to the pixels included in the mask area, and sets the other areas to have no pixel values (transparency treatment).

  In addition, as a situation where the model representative points are set outside the three-dimensional model as shown in FIGS. 8B and 8C, for example, as a result of smoothing in the time axis direction in the smoothing unit 220, the model A situation where the representative point protrudes from the three-dimensional model can be considered. Also, in order to place the projected point of the billboard on the field, the model representative point calculation unit 218 sets the point on the field directly below the 3D model as the model representative point. A model representative point is set at the bottom of the original model.

[Free viewpoint video generation unit 214]
Returning to FIG. 3, the free viewpoint video generation unit 214 uses the camera parameters generated by the calibration unit 202 to place the billboard generated by the billboard generation unit 210 at the model representative point, thereby generating a virtual viewpoint. A composite image corresponding to is generated. The free viewpoint video generation unit 214 acquires information on the virtual viewpoint designated by the user, for example, the coordinates of the virtual viewpoint. This synthesized image is one frame of a free viewpoint video. The billboard arranged by the free viewpoint video generation unit 214 is not necessarily perpendicular to the field, and maintains the angle θ. That is, the billboard keeps the angle θ with the field for the vertical movement of the virtual viewpoint, and faces the virtual viewpoint by rotating around the axis perpendicular to the field for the horizontal movement. The free viewpoint video generation unit 214 generates the free viewpoint video by performing the above processing continuously for each frame.

The operation of the image processing apparatus 200 having the above configuration will be described.
FIG. 9 is a flowchart showing a flow of a series of processes in the image processing apparatus 200. The image processing apparatus 200 acquires an image including the image of the subject from the cameras arranged at each of a plurality of viewpoints set around the subject on the field (S902). The image processing apparatus 200 generates a foreground mask by applying the background difference method to each of the acquired images (S906). The image processing apparatus 200 generates a three-dimensional model of the subject from the generated foreground masks (S908). The image processing apparatus 200 calculates the coordinates of the model representative point of the generated three-dimensional model (S910). The image processing apparatus 200 generates a billboard for the subject using the image acquired in step S902 and the corresponding foreground mask (S912). The image processing apparatus 200 synthesizes the image viewed from the virtual viewpoint by placing the generated billboard at the coordinates calculated in step S910 (S914).

  Based on the description in this specification, each unit can be realized by a CPU (not shown), a module of an installed application program, a module of a system program, a semiconductor memory that temporarily stores the contents of data read from a hard disk, or the like. Will be understood by those of ordinary skill in the art having touched this specification.

  According to the image processing apparatus 200 according to the present embodiment, it is possible to eliminate an unnatural display caused by restrictions imposed by the conventional billboard method. In this embodiment, since the position where the billboard should be placed is determined using a three-dimensional model (thick model) representing the outline of the subject, the advantage of the billboard that the display load is light is utilized. Above, a more natural display is possible.

  In addition, according to the image processing apparatus 200 according to the present embodiment, the position information during the air movement of the subject is estimated in a manner that is robust to the posture of the subject, so that a display with smooth movement is realized when a moving image is created. be able to. Furthermore, smoother expression is possible by smoothing the coordinates obtained as a result of estimation in the time axis direction.

  The configuration and operation of the image processing apparatus 200 according to the embodiment have been described above. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to each component and combination of processes, and such modifications are within the scope of the present invention.

  In the embodiment, a case has been described in which a three-dimensional model of a subject is generated and coordinates where a billboard is arranged are calculated using the generated three-dimensional model. However, the present invention is not limited to this, and is acquired from a plurality of cameras. The coordinates representing the position of the subject in the real space may be calculated from the plurality of images.

  FIG. 10 is an explanatory diagram of a method according to a modified example for determining coordinates where a billboard is to be arranged from a plurality of images 150a, 152a taken by a plurality of cameras 150, 152, 154. The image processing apparatus according to this modification acquires a plurality of images 150a and 152a taken by a plurality of cameras 150 and 152. The image processing apparatus identifies the first light ray 150c of the first camera 150 that passes through the image 150b of the target portion 154a of the subject 154 that is captured in the first image 150a captured by the first camera 150. The image processing device identifies the second light ray 152c of the second camera 152 that passes through the image 152b of the target portion 154a of the subject 154 that appears in the second image 152a photographed by the second camera 152. Similarly, the image processing apparatus specifies the third light beam 156c of the third camera 156.

  The image processing apparatus sets the spatial coordinates closest to the light rays 150c, 152c, and 156c of the plurality of cameras 150, 152, and 156 as the coordinates of the target portion 154a. The image processing apparatus specifies the determined coordinates of the target portion 154a as the coordinates where the billboard of the subject 154 should be placed. In this case, it is not necessary to generate a three-dimensional model in order to specify the coordinates where the billboard is to be arranged, and the processing amount can be reduced.

  110 free viewpoint image distribution system, 112 network, 114 mobile terminal, 200 image processing apparatus.

Claims (10)

Means for acquiring a plurality of images obtained by imaging a subject in real space from a plurality of viewpoints;
Means for calculating coordinates representing the position of the subject in real space from the plurality of acquired images;
Corresponding to viewpoints not included in the plurality of viewpoints by arranging the billboard of the subject generated from at least one of the acquired images with reference to the calculated coordinates And an image processing apparatus. The means for calculating is
Means for generating a three-dimensional model of the subject from the plurality of acquired images;
The image processing apparatus according to claim 1, further comprising: means for calculating coordinates of the representative point of the generated three-dimensional model as coordinates representing the position of the subject in real space.   The image processing apparatus according to claim 2, wherein the means for calculating the coordinates of the representative point calculates the coordinates of the representative point by statistically processing the generated three-dimensional model in a space.   The image processing apparatus according to claim 2, wherein the means for calculating the coordinates of the representative point statistically processes the coordinates of the generated representative point of the three-dimensional model in a time axis direction. When the subject is a person or an animal,
5. The image processing apparatus according to claim 2, wherein the means for calculating the coordinates of the representative point calculates the coordinates of the representative point based on a skeleton of the subject. Means for generating a mask for separating a foreground and a background for each of the plurality of acquired images;
Means for generating a billboard of the subject using the generated mask;
Further comprising
The image processing apparatus according to claim 2, wherein the unit that generates the three-dimensional model of the subject generates the three-dimensional model of the subject based on the generated mask.   6. The image processing according to claim 1, further comprising means for generating a billboard of the subject based on a point on the image plane of the at least one image corresponding to the calculated coordinates. apparatus.   The image processing apparatus according to any one of claims 1 to 7, further comprising means for setting an angle formed by a billboard of the subject with a field according to a position of a viewpoint corresponding to the at least one image. Acquiring a plurality of images obtained by imaging a subject in real space from a plurality of viewpoints;
Calculating coordinates representing the position of the subject in real space from the acquired plurality of images;
Corresponding to viewpoints not included in the plurality of viewpoints by arranging the billboard of the subject generated from at least one of the acquired images with reference to the calculated coordinates Generating a composite image to be processed. A function of acquiring a plurality of images obtained by imaging a subject in real space from a plurality of viewpoints;
A function of calculating coordinates representing the position of the subject in real space from the acquired plurality of images;
Corresponding to viewpoints not included in the plurality of viewpoints by arranging the billboard of the subject generated from at least one of the acquired images with reference to the calculated coordinates A computer program for causing a computer to realize a function of generating a composite image.