JP2021056679A - Image processing apparatus, method and program - Google Patents

Abstract

To provide an image processing apparatus for generating a three dimensional model that can suppress calculation costs for generating a model by considering user visual field information and can realize a natural display even when used for drawing.SOLUTION: An image processing apparatus 10 comprises: an extraction unit 12 for extracting an object area being photographed as a mask image from an image of each viewpoint in a multi-viewpoint image; and a generation unit 13 for applying a visual volume crossing method to the mask image to generate a three dimensional model for the object by determining whether or not each voxel belongs to the model. The generation unit 13 arranges the acquired depth information of a user viewpoint in a voxel space with reference to a virtual camera viewpoint to determine a spatial position given by the depth information before applying the visual volume crossing method, then determines whether each voxel is closer to or farther from the virtual camera viewpoint than the spatial position given by the depth information, and applies the visual volume crossing method only to the voxel determined that it is closer.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus, method and program for generating a 3D model that can suppress the calculation cost of model generation by considering the user's visibility information and can realize a natural display even when used for drawing. ..

Research and development is underway on AR (Augmented Reality) technology that superimposes virtual objects on physical objects in real space and presents them to users. By using HMDs (Head Mounted Display) such as smartphones and smart glasses, users can perform AR expression using the video see-through method or optical see-through method. In order to enhance the expressiveness in AR, it is important that the displayed virtual objects such as contexts and the physical objects in the user's surrounding environment are natural.

A system has been proposed for displaying a virtual object in consideration of geometrical consistency with a real object around the user (Patent Document 1). On the other hand, regarding the method of generating a virtual object, there is a method of generating a 3D (three-dimensional) model using a camera image. For example, a 3D model can be generated from an image taken by arranging a plurality of cameras so as to surround the subject (Patent Document 2). Patent Document 2 describes a method of overwriting a distant object with a near object in order to maintain geometrical consistency when there are a plurality of subjects, or when a distant object is hidden by a near object. There is.

JP-A-2018-106262 Japanese Unexamined Patent Publication No. 2019-101795 JP-A-2018-163467

A. Laurentini, `` The visual hull concept for silhouette-based image understanding,'' IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 2, Feb 1994.

In the AR experience, it is expected that the experience quality will be improved by displaying virtual objects according to the user's viewpoint position that changes in real time while maintaining consistency with the physical objects included in the user's field of view. The object. However, when generating a 3D model using the above-mentioned plurality of camera images, the calculation cost may increase depending on the number of cameras and the density of the area to be 3D modeled.

As a countermeasure against this increase in calculation cost, by considering the user's visibility information, there is a possibility that the target area to be 3D modeled can be reduced with respect to the area unnecessary for display. However, this possibility has not been examined in the prior art, and there is a problem in this respect.

Patent Document 1 describes a display mechanism that matches a physical object and a virtual object in consideration of user's visibility information, but does not describe a reduction in calculation cost in generating a virtual object. That is, in order to confirm the geometrical consistency between the virtual object and the real object, it is necessary to obtain the virtual object as point cloud data even in an area unnecessary for display, and the calculation cost cannot be reduced.

Further, Patent Document 2 describes model generation in consideration of occlusion between a plurality of virtual objects, but does not describe reduction of calculation cost at the time of this generation, and also describes the physical environment around the user and the like. Is not considered. That is, in patent document 2 as well as patent document 1, in order to maintain geometrical consistency and overwrite with a near view object for drawing, it is necessary to obtain a distant view object even in an area unnecessary for display. , The calculation cost cannot be reduced.

Non-Patent Document 1 proposes a method of generating a shape model from a plurality of images, but the user's view information and the like are not taken into consideration when the model is generated.

In view of the above problems of the prior art, the present invention generates a 3D model that can realize a natural display even when used for drawing, while suppressing the calculation cost of model generation by considering the user's view information. It is an object of the present invention to provide an image processing device, a method and a program.

In order to achieve the above object, the present invention applies an extraction unit that extracts a region of an object being photographed as a mask image from an image of each viewpoint of a multi-viewpoint image, and a visual volume crossing method to the mask image. An image processing device including a generation unit that generates a 3D model of the object by determining whether or not each voxel of a predetermined voxel set belongs to the 3D model, and the generation unit is the above-mentioned generation unit. Before applying, the depth information acquired from the user's viewpoint is placed in the voxel space with reference to the virtual camera viewpoint to determine the spatial position given by the depth information, and then each voxel determines the depth information. Whether the voxels that are closer to or farther from the virtual camera viewpoint than the spatial position given by the voxels belong to the 3D model as the application target of the visual volume crossing method. It is characterized in determining whether or not it is. Further, it is characterized in that it is a method or program corresponding to the image processing apparatus.

According to the present invention, by using the depth information acquired from the user's viewpoint, the calculation cost is suppressed by suppressing the application of the visual volume crossing method to the region determined to correspond to the occlusion region from the user's viewpoint. Then, it is possible to generate a 3D model that can realize a natural display even when used for drawing.

It is a figure which shows the functional structure of the image processing system which concerns on one Embodiment. It is a schematic diagram of telepresence as a use case of an image processing system. It is a table (including the schematic illustration of each information) which compared the image processing system which concerns on one Embodiment of this invention, and the server-side rendering of the prior art. It is a functional block diagram of the acquisition part which concerns on one Embodiment. It is a figure which shows typically the visual volume crossing method as an existing technique. It is a figure which shows typically in comparison with the prior art that a calculation load is reduced by applying a visual volume crossing method excluding an occlusion region in a generation part. It is a flowchart of the visual volume crossing method by the generation part which concerns on one Embodiment. It is a flowchart of the determination process whether or not it is in the occlusion area in step S12 which concerns on one Embodiment. It is a figure which shows typically the problem which can occur in 1st Embodiment. FIG. 9 is a diagram schematically showing a solution provided by the second embodiment to the problem of the first embodiment schematically shown in FIG. It is a flowchart which shows an example of the procedure of model generation of the generation part 13 by 2nd Embodiment. It is a figure which shows the schematic example of the drawing information at time t = 0 second and the drawing information at time t = 0.1 second corresponding to the explanatory example. It is a figure which shows the example of the hardware configuration in a general computer apparatus.

FIG. 1 is a diagram showing a functional configuration of an image processing system according to an embodiment. The image processing system 100 includes an image processing device 10 and a terminal device 20 capable of communicating with each other via a network NW. The image processing device 10 includes a photographing unit 11, an extracting unit 12, a generating unit 13, and a drawing unit 14 as a functional block configuration. The terminal device 20 includes an acquisition unit 21, a display side shooting unit 22, and a display unit 23 as a functional block configuration.

As a use case of the image processing system 100, as shown in the schematic diagram in FIG. 2, a remote person is separated by utilizing the free viewpoint video technology that can create a view of an arbitrary viewpoint from a plurality of camera images. There is a telepresence that allows you to experience as if you were in the place of. In this case, the image processing device 10 as a server device exists on the shooting environment PE side, and a 3D model of the subject OB is created by using multiple cameras C1, C2, ..., CN of multiple cameras (N units, N ≧ 2). create. The created 3D model of the subject is drawn (rendered) by the virtual camera, and the virtual object VOB of the drawing result is transmitted to the user.

Then, as further shown in FIG. 2 as a schematic diagram, on the viewing environment WE side, the user U wears a terminal device 20 configured as a device capable of viewing AR such as smart glasses, and the user's viewing The physical object POB of the environment and the drawn virtual object VOB are displayed in a superimposed state. (When the user U directly visually looks at the terminal device 20 such as smart glasses, the physical object POB exists as a real object, but the virtual object VOB drawn as a display for AR on the terminal device 20 is The viewing environment WE has a physical object POB such as a table, and when a person as a virtual object VOB is placed behind the table, it is schematically shown in FIG. As described above, it is desirable that the display in consideration of occlusion is made so that a part of the area of the virtual object VOB is blocked by the physical object POB.

In the image processing system 100 according to the embodiment of the present invention, it is possible to display in consideration of occlusion on the viewing user side in this way, and the area related to occlusion is calculated without generating a 3D model. It is possible to reduce the load.

The framework itself of the image processing system 100 as shown in the system configuration in FIG. 1 also exists in the prior art as server-side rendering in AR, but the image processing system 100 according to the embodiment of the present invention is described above. As described above, it has an effect not found in the prior art in relation to occlusion. FIG. 3 is a table (including a schematic illustration of each information) comparing the image processing system 100 according to the embodiment of the present invention with the server-side rendering of the prior art.

As shown in FIG. 3, as the information acquired from the user side who is the viewer of the AR display, only the line-of-sight information of the user is acquired through smart glasses or the like in the prior art, whereas one embodiment of the present invention is implemented. In the form (hereinafter, abbreviated as "this method" in the description of FIG. 3), depth information is acquired in addition to line-of-sight information. In the conventional technique, the generated 3D model is generated without considering the occlusion location, whereas in this method, the occlusion can be excluded and the calculation load can be reduced. Therefore, even in the drawing result using this generated 3D model, the occlusion part is not excluded in the conventional technique, but is excluded in the drawing by this method, and the display image as the AR display for the user is also the conventional technique. The occlusion is not reflected in this method, but it is reflected in this method.

Thus, as shown in the schematic illustration of FIG. 3, in the prior art, the physical object (physical object POB such as the table of FIG. 2) in the user's viewing environment is not considered when generating the 3D model. If the subject that generates the 3D model is a person, the whole body is the target of model generation. Then, since the view image in which the whole body is rendered is sent, the virtual object of the whole body is displayed regardless of the physical object unless the viewing device performs occlusion processing or the like. In the prior art, a virtual person is displayed on top of a physical table.

As described above, unlike the conventional technique, the calculation load of 3D model generation is reduced by considering occlusion, and the operation of the image processing system 100 according to the embodiment capable of drawing in consideration of occlusion. Details will be described below as details of each functional unit of the functional block shown in FIG.

<Shooting section 11>
As shown in the schematic example in the shooting environment PE of FIG. 2, the shooting unit 11 surrounds and shoots an object OB such as a person to be generated as a 3D model in the shooting environment PE (for example, a shooting studio). Multiple N (N ≧ 2) cameras C1, C2, ..., CN are configured as hardware. The photographing unit 11 outputs an image (multi-viewpoint image of N viewpoints) obtained by photographing the object OB with the camera of each viewpoint to the extraction unit 12 and the drawing unit 14.

Here, the camera parameters (internal parameters and external parameters) of each camera C1, C2, ..., CN of N units (N ≧ 2) that configure the photographing unit 11 as hardware are estimated by known or prior calibration. Therefore, it is assumed that the image processing device 10 can be used by referring to the information of this camera parameter. (For example, the processing of the generation unit 13 and the drawing unit 14, which will be described later, can be performed with reference to this camera parameter.)

The photographing unit 11 photographs the object OB in real time as an image to acquire a multi-viewpoint image of N viewpoints, and the image processing device 10 draws this object OB as a virtual object VOB in real time and draws the result as a virtual object VOB in the terminal device 20. It can be displayed in real time on the side of. The following description of each functional part of the image processing system 100 relates to any one arbitrary time in such real-time processing, unless otherwise specified for processing on the time axis. That is, the multi-viewpoint image obtained by the photographing unit 11 is assumed to be an arbitrary one-time frame in the multi-viewpoint video.

<Extractor 12>
In the extraction unit 12, the mask image (corresponding to the foreground of the silhouette) is obtained by extracting the captured object OB as the foreground silhouette for each of the N images of the N viewpoints in the multi-viewpoint image obtained by the shooting unit 11. A binary mask image in which the value "1" is given to the pixels and the value "0" is given to the pixels corresponding to the other backgrounds), and the extracted N mask images are sent to the generation unit 13. Output.

Any existing technique may be used for the method of extracting the mask image by the extraction unit 12. For example, for each of the N images of each viewpoint of the multi-viewpoint image, a background image taken in a state where the object OB does not exist is prepared in advance, and a region determined to be different from this background image by the background subtraction method is the foreground. By determining that, the extraction unit 12 may extract the mask image.

<Acquisition department 21>
The acquisition unit 21 provided on the side of the terminal device 20 (viewing device for AR display) in the viewing environment PE (Fig. 2) in which the user U who views the AR display exists provides the environment information of the user who uses the terminal device 20. Is acquired in real time, and this environmental information is transmitted to the generation unit 13 of the image processing device 10 via the network NW. Here, when the acquisition unit 21 acquires the environment information, the acquisition time is linked as a time stamp and then transmitted to the generation unit 13.

FIG. 4 is a functional block diagram of the acquisition unit 21 according to the embodiment, and the acquisition unit 21 includes a position / posture acquisition unit 211 and a depth acquisition unit 212. In the acquisition unit 21, as the user's environmental information, the position / posture information (line-of-sight information) regarding the user's viewpoint acquired by the position / attitude acquisition unit 211 and the depth information (depth information) in the viewing environment as seen by the user acquired by the depth acquisition unit 212. Information of the depth image representing (spatial distribution information) can be transmitted to the generation unit 13. When the position / orientation acquisition unit 211 and the depth acquisition unit 212 acquire the position / orientation information and the depth image, respectively, the image captured by the display side photographing unit 22 described later may be used, or this image may be used. You may try to acquire the environmental information without doing so.

(Position / Posture Acquisition Unit 211)
The position / orientation acquisition unit 211 can acquire the position / orientation information of the terminal device 20 by any existing method. This position / orientation information corresponds to the information of the external parameters in the camera parameters, and the position and orientation of the terminal device 20 in the three-dimensional world coordinate system defined in the viewing environment PE (FIG. 2) in which the user U exists. To give.

For example, the position / orientation acquisition unit 211 is configured to include sensors (accelerometers, gyro sensors, orientation sensors, etc.) that acquire the position / attitude as hardware, and obtains position / attitude information from the measurement output of the sensor. It may be acquired in real time. Further, the position / orientation acquisition unit 211 may acquire the position / orientation by analyzing the image acquired by the display side photographing unit 22. For example, in the viewing environment PE (FIG. 2) in which the user U exists, a predetermined marker (such as a square marker used in AR technology) that can be used to detect the position and orientation of the camera is arranged in advance on the display side. The position and orientation may be acquired as an external parameter after detecting the marker region in the image by performing corner detection or SIFT feature amount detection on the image obtained by the photographing unit 22.

(Depth acquisition section 212)
In the depth acquisition unit 212, the depth image seen from the terminal device 20 in the viewing environment PE (FIG. 2) in which the user U exists (the depth camera or the like is used for the position / orientation acquired by the position / orientation acquisition unit 211) by an arbitrary existing method. It is possible to acquire a certain depth image).

For example, the depth image acquisition unit 212 is configured to include a depth camera (any existing depth camera such as a TOF (light arrival time) method or a pattern irradiation method) as hardware, so that the depth image can be captured by the depth camera. May be obtained. Further, the depth image acquisition unit 212 may acquire the depth image by analyzing the image acquired by the display side photographing unit 22. For example, the depth image may be obtained by acquiring the depth by stereo matching after obtaining the same point correspondence between the image of the current time and the image of the past time taken by the display side shooting unit 22. A depth image may be obtained by applying a neural network trained in advance to output a depth image from a captured image by deep learning.

As a schematic illustration for the "depth information" in the table of FIG. 3 described above, a schematic example of the depth image obtained by the depth image acquisition unit 212 is shown. This schematic depth information relates to a table as an example of the physical object POB in FIG. 2, and the part corresponding to the table surface (two vertical and horizontal surfaces) is closer to white as the depth value is small, and the table. The parts other than the surface are closer to black as the depth value is large, and the depth image is expressed as a grayscale image.

<Generator 13>
The generation unit 13 excludes the occlusion area from the N mask images of the N viewpoints obtained by the extraction unit 12 by considering the environment information of the user on the terminal device 20 side transmitted from the acquisition unit 21. The visual volume crossing method is applied in the form, and the 3D model (virtual object VOB) of the generated object OB is output to the drawing unit 14. The N mask images obtained from the extraction unit 12 are associated with the common shooting time (synchronized by N cameras) in the shooting unit 11, and the time is also included in the environmental information in the acquisition unit 21. Since the acquisition time is linked as a stamp, the generation unit 13 can generate a 3D model after referring to the environment information at the same time as the time of the N mask images.

FIG. 5 is a diagram schematically showing a visual volume crossing method as an existing technique. As is known, the principle of the visual volume crossing method is that the mask images M1, M2, ..., MN are obtained from the positions of N cameras C1, C2, ..., CN (the camera is shown as the center of the camera in FIG. 5), respectively. A 3D model is obtained as a common volume portion (visual hull VH) through which all of the visual cones V1, V2, ..., VN obtained by performing three-dimensional back projection onto the foreground of the camera. In FIG. 5, only the first two cameras C1 and C2, their mask images M1 and M2, and the pyramidal cones V1 and V2 are shown as schematic examples.

When actually generating a 3D model by the visual volume crossing method (using voxels) based on the principle schematically shown in FIG. 5, the 2D on the mask image is opposite to the 3D back projection. Projections can be used. That is, a predetermined set of voxels (set of discrete lattice points in the three-dimensional model space) is defined and arranged in advance in the model space, and N units of each voxel point (X, Y, Z) are arranged. Mask images of cameras C1, C2, ..., CN M1, M2, ..., Projection positions on each mask image by performing two-dimensional projection on MN (x, y) _[1] , (x, y) _[2] …, (x, y) _[N] is obtained, and in all of the N mask images, the voxel points (X, Y, Z) as projected on the foreground as a silhouette are the 3D model. Regarding other voxel points (X, Y, Z) (boxel points (X, Y, Z) projected on the background in at least one mask image) that are determined to belong to the inside (or surface), It can be determined that the point is an external point that does not belong to the 3D model. The set of voxels for which affirmative judgment is obtained in this way becomes the obtained 3D model.

As described above, in the visual volume crossing method as an existing technique using voxels, a set of voxels (X, Y, Z) having a predetermined density is placed in a predetermined range (for example, a rectangular parallelepiped range) in a three-dimensional model space. By definition, for all voxel points (X, Y, Z) as the lattice points, the projection result on the foreground / background on N mask images by the visual volume crossing method will be determined. ..

The generation unit 13 does not apply the visual volume crossing method as the existing technology as it is, but by referring to the environmental information obtained by the acquisition unit 21, it is defined in advance in the model space with a predetermined density in a predetermined range. Of all the points (X, Y, Z) of the existing voxel set, only those that are judged not to be affected by occlusion are projected onto the foreground / background on N mask images by the visual volume crossing method. Is judged. As a result, the generation unit 13 omits the projection process from the beginning for all the voxel points (X, Y, Z) that are determined to be affected by occlusion, and constructs the generated 3D model. Since it is possible to obtain a determination to exclude from the points, it is possible to reduce the calculation load as compared with the conventional technique.

FIG. 6 is a diagram schematically showing that the calculation load is reduced by excluding the occlusion region and applying the visual volume crossing method in the generation unit 13 in comparison with the prior art. In the prior art shown in the first panel PL1, the visual volume crossing method is applied to all points in the predefined voxel set VS, and the virtual object VOB'of the object OB as the set of voxels corresponding to the foreground is used as a 3D model. can get. On the other hand, in the method of the generation unit 13 shown in the second panel PL2, the visual volume crossing method is applied by excluding all the points in the voxel set VS that are judged to be affected by occlusion. A virtual object VOB of an object OB is obtained as a 3D model as a set of corresponding voxels in the foreground.

While the conventional virtual object VOB'shown in FIG. 6 models the whole body of the object OB to be modeled such as a person, the virtual object VOB of the generation unit 13 shows a schematic example in FIG. The occlusion area ROC by the physical object POB such as a table (a part of which is shown as the occlusion area ROC in a certain visual cone area as seen from the virtual camera VC) is excluded in advance, and only a part of the whole body is modeled. Has been done. In addition, in FIG. 6, as a distinction from the occlusion region ROC, a part of the region RD to be determined by the visual volume crossing method in a certain visual cone region as seen from the virtual camera VC is also shown.

FIG. 7 is a flowchart of the visual volume crossing method by the generation unit 13 according to the embodiment, and shows the details of the method for excluding the occlusion region as described above.

In step S10, as an overall setting when applying the visual volume crossing method, a voxel set or the like to be determined whether or not to configure the obtained 3D model is set, and then the process proceeds to step S11. _{In step S10, let {v i} | i = 1,2, ..., M} be a set of voxels set in a predetermined range in the 3D model space with a predetermined density. _{For each voxel v i} (i = 1,2, ..., M) to be set, it is determined whether or not the point belongs to the 3D model by the iterative processing of the subsequent steps S11 to S17. It is assumed that the voxel v _{i is judged.} The determination order of the voxels v _i may be arbitrary, and may be determined by, for example, the raster scan order in the three-dimensional space.

In step S10, _{after setting the voxel set {v i} | i = 1,2, ..., M}, the judgment result is set as the initial value of the binary judgment result of whether or not _{each voxel v i belongs to the 3D model.} Set E (v _i ) = 0 (representing "does not belong to 3D model"). As described below, the voxel v of _i, which has reached the step S16 in the repetitive processing subsequent steps S11~S17 determination result is E (v _i) = 1 (representing "belonging to the 3D model") will be rewritten to, without being the rewriting not to have reached the step S16, ( "does not belong to the 3D model") determination result E (v _i) = 0 as an initial value is determined as the actual results Will be done.

Further, in step S10, a user who performs AR viewing as a reference position for determining the occlusion area in the three-dimensional model space in which _{the voxel set {v i | i = 1,2, ..., M} is set.} The virtual viewpoint VC (virtual camera VC schematically shown in FIG. 6) is set. The position of this virtual viewpoint may be set in the three-dimensional model space as being in line with the position / orientation acquired and transmitted by the position / orientation acquisition unit 211 of the acquisition unit 21 of the terminal device 20 on the user side. (Note that this three-dimensional model space may be set to match the world coordinate system of the shooting environment PE shot by the cameras constituting the shooting unit 11.)

That is, the position / orientation acquisition unit 211 acquires the position / orientation in the world coordinate system of the viewing environment WE in which the user exists, and by performing predetermined conversion (translation and rotation) set in advance, the three-dimensional model space. The position and orientation of the virtual viewpoint VC within can be obtained. In this predetermined conversion, an administrator or the like who prepares the AR viewing content provided by the image processing system 100 configures the shooting unit 11 by arranging the object OB to be modeled in the shooting side environment PE (including the movement range and the like). Arrangement based on the camera) and arrangement of the virtual object VOB in the viewing environment WE (arrangement based on the terminal device 20 as a viewing device including the movement range), and these alignments are performed. It may be set in advance as information for performing.

Further, to the predetermined conversion, a conversion that adjusts the display position / orientation of the virtual object by the input specified by the user who uses the terminal device 20 is applied (combined conversion of the predetermined conversion and the position adjustment conversion is applied). Then, the position and orientation of the virtual viewpoint VC in the three-dimensional model space may be obtained. The information for this adjustment may be linked to the time, included in the environmental information, and transmitted to the generation unit 13.

As described above, the entire flow of FIG. 7 can be performed for each time in real time, but the position / orientation of the virtual viewpoint VC set in step S10 of each time is subjected to a predetermined conversion. As a result, it moves in the 3D model space with the same behavior as the position / orientation acquired by the position / attitude acquisition unit 211.

_{In step S11, an unprocessed voxel v i} that has not yet been determined whether or not the point belongs to the 3D model is selected, and then the process proceeds to step S12. In step S12, _{it is determined whether or not the voxel v i} is in the occlusion area, and then the process proceeds to step S13.

FIG. 8 is a flowchart of a process for determining whether or not the device is in the occlusion area in step S12 according to the embodiment. Main processing in the procedure illustrated in FIG. 8, by matching the depth values of the obtained depth image the voxel v _i is transmitted from the depth acquiring unit 212, from the position shown the voxel v _i is the depth value If it is on the front side (closer to the virtual camera VC), it is judged that it is not in the occlusion area, and conversely, if it is on the back side (farther side than the virtual camera VC), it is judged to be in the occlusion area.

In step S20, _{the distance dist (v i , VC) between the voxel v i} and the virtual camera VC is calculated, and then the process proceeds to step S21. (This distance dist (v _i , VC) is referred to as the distance e _i as being linked to the voxel v _i . The voxel vi is set in advance in the 3D model space, and the virtual camera VC is also set in step S10 in FIG. Since the position (and orientation) in the 3D model space is obtained in, the distance e _i can be calculated in the 3D model space.)

In step S21, the voxel v _i is converted from the voxel space coordinate system (that is, the 3D model space coordinate system) to the coordinate system of the virtual camera VC, and then the process proceeds to step S22. The coordinate transformation voxel v is denoted as' _i. The position and orientation of the virtual camera VC is so is a need in the 3D model space in step S10 in FIG. 7, it is possible to coordinate transformation of the "v _i → v _'i'.

In step S22, the mapping coordinate transformation voxel v _'i = obtained in step S21 (X, Y, Z) the pixel position of the "foreground depth image" as defined to the image coordinate system of the virtual camera VC (x, y) _{Then, the distance e i} = e _i (x, y) obtained in step S20 is assigned as the pixel value of this pixel position (x, y), and then the process proceeds to step S23. (Here, by _{assigning this distance e i} as a pixel value, a “foreground depth image” defined for each _{voxel v i} is obtained as an image of the image coordinate system of the virtual camera VC. The mapped pixel position (x, Other than y), the pixel value of the "foreground depth image" is not defined.)

In step S22, transformation into the spatial coordinates of the voxel _{v 'i (X, Y,} Z) position of the two-dimensional image coordinate system of the virtual camera VC from (x, y) are set in advance for the virtual camera VC It can be performed as a two-dimensional projection using internal parameters. (It should be noted that even when AR display is performed on the display unit 23 of the terminal device 20, the drawing result made by the drawing unit 14 (described later) of the image processing device 10 is displayed using the internal parameters of this virtual camera VC. Become.)

In step S23, coordinate transformation voxels obtained in step _{S21 v 'i = (X,} Y, Z) is into the angle of view of the virtual camera VC (image area of the image plane from the position of the virtual camera VC (usually rectangular) After checking whether or not it is included in the range of the three-dimensional back-projected visual cone), the process proceeds to step S24. As the range of the angle of view of the virtual camera VC in step S23, a predetermined range may be used as the camera parameter of the virtual camera VC. (Note that the drawing unit 14 (described later) will also be drawn within the range.)

In step S24, the result of examining in step S23 is, if the voxel v _'i proceeds to step S25 in the case was within the angle range (if so), was outside the range (if negative) Proceeds to step S27.

In step S25, at the pixel position (x, y) mapped in step S22, the pixel value d (x, y) of the “background depth image” is obtained from _{the pixel value (distance e i) of the foreground depth image obtained in step S22.} ) Is subtracted to obtain the difference D = e _i (x, y) -d (x, y), and the process proceeds to step S26. Here, as the "background depth image" to be subtracted, the depth image acquired by the depth acquisition unit 212 of the terminal device 20 is used. (I.e., the "background depth image", as the image depth of background present in the viewing environment WE user (physical object POB, etc.), give the depth of the voxel v _'i those distinguished as "foreground depth image" is there.)

The depth acquisition unit 212 has been calibrated in advance so that the depth image can be acquired assuming that the depth d (x, y) is given at the pixel position (x, y) of the image plane of the virtual camera VC. Then, the depth shall be obtained.

In step S26, it is determined whether or not the difference D obtained in step S25 is positive (D> 0), and if it is affirmative (D> 0), the process proceeds to step S27, and if it is negative (D ≤ 0). Then proceed to step S28.

In step S27, _{the determination result that the voxel v i} corresponds to the occlusion area is obtained, and the flow of FIG. 8 (determination process of step S12 of FIG. 7) is terminated. In step S28, _{the determination result that the voxel v i} does not correspond to the occlusion area is obtained, and the flow of FIG. 8 is terminated.

Here, when going from step S26 to step S27, the difference D> 0, and _{the spatial position of the voxel v i} is behind the spatial position given by the corresponding depth value d (x, y) (from the virtual camera VC). Since it is on the far side), it is judged to be in the occlusion area. On the other hand, in the case from step S26 to step S28, the opposite is true, so it is determined that the region is not an occlusion region.

It should be noted that step S27, which is determined to be in the occlusion region, may not be reached from step S26, but may be reached after obtaining a negative determination in step S24. If a negative judgment is obtained in step S24, it means that the voxel v _i is out of the range of the angle of view of the virtual camera VC, and in this case, the voxel v _i is out of the range where AR display can be performed. (The frame is out of the range of AR drawing and display by the drawing unit 14 and the display unit 23). Therefore, for convenience of omitting 3D model generation, AR drawing, etc., a determination result of the occlusion area is given. It becomes.

Since each step of FIG. 8 has been described above as an embodiment of step S12 of FIG. 7, the description of each step of FIG. 7 will be returned again.

In step S13, if the determination result in step S12 is negative (not in the occlusion area), the process proceeds to step S14, and if it is affirmative (in the occlusion area), the processing for the _{voxel v i is completed.} Proceed to step S17 as if it had been done.

In step S14, the _{visual volume crossing method is applied to the voxel v i , and as described in FIG. 5, the voxel v i} is projected on the foreground silhouette in all of the N mask images obtained by the extraction unit 12. After checking whether or not the points included in the 3D model are applicable, the process proceeds to step S15.

In step S15, if the result in step S14 is affirmative and corresponds to the point included in the 3D model, the process proceeds to step S16, and if it is negative, the processing for the _{voxel v i is completed.} Proceed to step S17 as if it had been done. In step S16, _{the judgment result of the voxel v i} is rewritten from the initial value so that E (v _i ) = 1 (“belonging to the 3D model”), and _{the process for the voxel v i} is assumed to be completed, and the process proceeds to step S17. ..

In step S17, it is determined whether or not the processing is completed for all the voxels of the voxel set {v _{i | i = 1,2,…, M} set in step S10, and if so, the process proceeds to step S18.} , If unprocessed voxels remain, the process returns to step S11. By the iterative processing of steps S11 to S17, one of the following three determination results can be obtained for each voxel _{of the voxel set {v i | i = 1,2, ..., M}.}

(First case) makes a transition to as "S16 → S17", in terms of applying the volume intersection method (S14), E (v _i) = 1 (does not correspond to the occlusion region, "belong to the 3D model" ) Is determined.
By (second case) transits "S15 → S17", in terms of applying the volume intersection method (S14), does not correspond to E (v _i) = 0 (occlusion area, do not belong to the "3D model ") Is determined.
By (third case) transits "S13 → S17", without applying the volume intersection method _{(S14), E (v i} ) = 0 ( for corresponding to the occlusion region "does not belong to the 3D model" ) Is determined.

In step S18, performs post-processing such as to obtain E a (v _i) = 1 ( "belonging to the 3D model") and the determined surface shape by polygons against set of voxels, enables drawing The 3D model in this state is output from the generation unit 13 to the drawing unit 14, and the flow of FIG. 7 ends. Any existing technology may be used for post-processing such as polygonization.

In addition, when applying the visual volume crossing method in step S14 after giving the distinction between the first to third cases described above according to the flow of FIG. 7 _{so as to obtain the judgment result of each voxel v i.} May be combined with any existing method.

<Drawing unit 14>
The drawing unit 14 draws the 3D model generated by the generation unit 13 by rendering it from the viewpoint of the virtual camera VC using the texture of the multi-viewpoint image obtained by the shooting unit 11, and the obtained virtual viewpoint image ( (It becomes a mask image in which the pixel value is not defined except for the portion where the drawing is made) is transmitted to the display unit 23 of the terminal device 20.

For rendering in the drawing unit 14, any existing method (for example, the method of Patent Document 3 described above) used in synthesizing a free-viewpoint image may be used, and a polygon which is an element of a 3D model is converted into a virtual camera VC. It may be projected onto the image plane of the viewpoint, and the corresponding texture may be selected from the multi-view image obtained by the photographing unit 11 on the projected polygon, and the texture may be pasted after reflecting the deformation due to the projection. Here, the texture may be selected from one or more images in a position and orientation close to the virtual camera VC among the multi-viewpoint images of N viewpoints. When two or more images are used, a weighted sum or the like may be used.

<Display unit 23 and display side shooting unit 22>
The display unit 23 displays the virtual viewpoint image transmitted from the drawing unit 14 to the user, thereby enabling the user to view the AR. The display unit 23 as hardware can be realized as, for example, an optical see-through type HMD or a video see-through type HMD. In the case of the former (optical see-through type HMD), only the virtual viewpoint image (mask image) transmitted from the drawing unit 14 is superimposed and displayed on the actual background of the shooting environment WE that is visible to the naked eye of the user. do it. In the case of the latter (video see-through type HMD), the hardware is a virtual viewpoint image (mask image) transmitted from the drawing unit 14 with respect to the background image of the shooting environment WE taken by the display side shooting unit 22 composed of the camera. ) May be superimposed and displayed. If it is not necessary to take an image on the terminal device 20 side, the display side shooting unit 22 may be omitted.

As described above, according to one embodiment of the present invention, the model generation can be simplified in the server (image processing device 10) that generates and draws the model. Therefore, the model generation and the simplified model in the server are used. It is expected that the rendering cost will be reduced, and the viewing device (terminal device 20) will be able to display the virtual object in consideration of occlusion with the physical object, and a more natural AR display can be realized.

The various three-dimensional coordinate systems themselves used by the image processing device 10 and the terminal device 20 may use the same relationship as the existing server-side rendering to realize AR display. It's a street.

In the terminal device 20, the position / orientation information is acquired in the world coordinate system (X, Y, Z) _{[viewing side world] on the viewing side} , and this position / orientation information is used in the depth image, and the position of the depth camera from which the depth image is acquired is obtained. As a representation of the posture, it is linked as if it was acquired at the same time. In the image processing device 10, in the photographing unit 11, each camera Ck (k) of the N viewpoints the object OB to be modeled arranged in _{the world coordinate system (X, Y, Z) [shooting side world] on the shooting side.} Take a picture at the camera coordinates (X, Y, Z) _[Ck] of = 1,2,…, N). In the generation unit 13, a voxel is defined in advance in the voxel space (X, Y, Z) _{[voxel] as the 3D model space.} With known camera parameters, mutual conversion between N camera coordinate systems and the shooting side world coordinate system "(X, Y, Z) _[Ck] ⇔ (X, Y, Z) _{[shooting side world]} " is possible. is there. In addition, since the voxel space (X, Y, Z) _[voxel] is set by the administrator etc. in the world coordinate system (X, Y, Z) _{[shooting side world], which is the stage of shooting, these mutual} The conversion "(X, Y, Z) _[voxel] ⇔ (X, Y, Z) _{[shooting world]} " is also possible. (It may be set as "(X, Y, Z) _[voxel] = (X, Y, Z) _{[shooting world]" as the same thing.)}

In addition, as explained in step S10, the world coordinate system on the terminal device 20 side and the voxel coordinate system on the image processing device 10 side for _{viewing are changed to "(X, Y, Z) [viewing side world]} ⇔ (X, Y, Z) _[Voxel] ”can be converted. In particular, the position / orientation of the terminal device 20 acquired in the viewing side world coordinates by this conversion is converted into the position / orientation of the voxel coordinate system, which gives the position / orientation of the virtual camera VC in the voxel coordinate system. Since the position and orientation of the virtual camera VC are given in the voxel coordinate system (X, Y, Z) _[boxel] in this way, the conversion between the voxel coordinate system and the coordinate system of the virtual camera VC "(X, Y, Z) Z) _{[voxel] ⇔ (X, Y, Z} ) [VC] "(in step S21" v _i → v _'i', etc.) are also possible in the conversion of the direction of the home position and the coordinate axes.

Then, when the image processing device 10 draws the 3D model obtained by the coordinate system (X, Y, Z) _[VC] of the virtual camera VC and displays it on the terminal device 20 side, the viewpoint of the virtual camera VC is displayed. As it is, it may be treated as matching the viewpoint of the user who performs AR viewing on the terminal device 20. (For example, when the display on the display unit 23 is realized by the video see-through method and the background image captured by the display side shooting unit 22 is superimposed and displayed, the internal parameters of the virtual camera VC are set to the display side shooting unit 22. By setting the same as the internal parameters of the camera as the hardware to be configured, it is possible to draw in the drawing unit 14 so that the display is consistent. The same applies to the optical see-through method.)

The embodiment described above will be referred to as the first embodiment, and the second to sixth embodiments, which are modified examples of the above, will be described below.

<Second embodiment>
The second embodiment has the following problems that may occur when a light source is arranged in the 3D model space using the virtual object obtained in the first embodiment and then drawing is performed including the shadow of the virtual object. It is possible to deal with.

That is, when a light source is installed and drawn in the 3D model space, a shadow is generated from the positional relationship between the light source and the virtual object. However, in the first embodiment, the model generation is omitted for some voxels in consideration of occlusion, so that the shadow for the omitted region is not generated. The occlusion area itself is a part that is not visible to the user's view, but its shadow may be a part that is visible to the user. In this case, it is a problem that can occur in the first embodiment that the shadow is interrupted or disappears from the user's point of view and looks unnatural.

Therefore, in the second embodiment, instead of skipping modeling for all voxels in the occlusion area, the voxels in the part of the voxels in the occlusion area that affect the shadow are modeled, and only the part that does not affect the shadow is modeled. Skip modeling.

FIG. 9 is a diagram schematically showing the above-mentioned problems that may occur in the first embodiment. (Note that, in FIG. 9, the 3D model space is schematically shown by its two-dimensional cross section.) In the first embodiment, the region shielded by the physical object POB when viewed from the virtual camera VC forms an occlusion region ROC. However, the generated virtual object VOB is generated only outside this occlusion area ROC, and in the example of FIG. 9, it is generated in the form of being divided into two parts, the upper part up and the lower part dp of the original object OB. There is. Here, assuming that the virtual object VOB is generated by applying the conventional technique described in FIG. 3, it is not divided into two parts, the upper part up and the lower part dp, but the intermediate part mp (occlusion area). The entire original object OB will be generated, including (in the ROC).

When the virtual light source VL is placed on the virtual object VOB generated in the first embodiment and the shadow is drawn on the ground GR defined in the virtual space, the upper shadow derived from the upper portion up is obtained. A divided shadow is drawn in two areas, the area us and the lower shadow area ds derived from the lower part dp, but the virtual camera VC is in a position where the shadow in this divided state can be seen, and the AR viewing user Gives an unnatural impression to.

FIG. 10 is a diagram schematically showing a solution provided by the second embodiment to the problem of the first embodiment schematically shown in FIG. Since the same reference numerals as those in FIG. 9 in FIG. 10 represent the same contents, duplicate description will be omitted. As shown in FIG. 10, with respect to the intermediate part mp in which the virtual object generation is omitted, which was the cause of giving an unnatural impression, in the second embodiment, the first region m1 and the second region m2 are distinguished. be able to.

The first region m1 is in the occlusion region ROC but is calculated by applying the visual volume crossing method as a region constituting the virtual object VOB, and is ignored in the texture drawing of the virtual object VOB, but the shadow It is treated as a consideration in drawing. As a result, the second region m2 is treated as an exemption from the application of the visual volume crossing method as in the first embodiment. (In the example of FIG. 10, the second region m2 is composed of two regions, one closer to the upper portion up and the other closer to the lower portion dp.)

In the second embodiment, it is possible to distinguish the regions that affect the shadow even within the occlusion region ROC, as in the first region m1 described above, and when drawing the shadow as shown in FIG. Is drawn as an intermediate shadow area ms1 generated by the first area m1 so that the shadow area is continuous with us, ms, ds without interruption, and is the first embodiment when viewed from the virtual camera VC. It is possible to eliminate the unnatural impression of.

Specifically, in the second embodiment, the processes of the generation unit 13, the drawing unit 14, and the display unit 23 are changed or added from the first embodiment as follows. (Since the processing contents of the other functional parts are the same in the second embodiment and the first embodiment, no duplicate explanation will be given.)

<Generation unit 13 ... Second embodiment>
FIG. 11 is a flowchart showing an example of a procedure for generating a model of the generation unit 13 according to the second embodiment. The flowchart of FIG. 11 is obtained by adding steps S131 and S132 as additional procedures in the second embodiment to steps S10 to S18 similar to those shown in the flowchart of FIG. 7. Steps S10 to S18 are the same as the steps of the same reference numerals in FIG. 7 unless otherwise specified. Therefore, duplicate explanations are omitted, and steps S13, S131, S132, S16, and S18 as differences are described below. explain.

In step S13 of FIG. 11, as in step S13 of FIG. 7, the destination of the flow branches depending on the case according to the determination result of whether or not the _{voxel v i corresponds to the occlusion region.} In the case of a negative determination in step S13 of FIG. 11 (when it does not correspond to the occlusion area), the process proceeds to step S14 as in FIG. 7, but in the case of an affirmative determination (when it corresponds to the occlusion area), the process proceeds to step S131. move on.

In step S131, _{it is determined whether or not the voxel v i} affects the shadow, and the process proceeds to step S132. The details of the determination in step S131 will be described later together with the details of the additional processing in step S16. In step S132, if the judgment result in step S131 is affirmative (affects), the process proceeds to step S14, and if it is negative (does not affect), _{the process for the voxel v i} is assumed to be completed and the process proceeds to step S17. ..

By adding steps S131 and S132 of the branch destination in the affirmative determination in step S13 as described above, in step S18 of the second embodiment of FIG. 11, it is shown as the first region m1 of the schematic example of FIG. As described above, a 3D model can be obtained including voxels that correspond to the occlusion area but affect shadows. That is, in the first embodiment, _{the determination result of each voxel v i} was any of the first to third cases, but in the second embodiment, the voxels determined to have an influence on the shadow also in this third case. If this is the case, the visual volume crossing method is applied, and a determination result of whether or not it is a point constituting the 3D model can be obtained.

(Details of judgment processing in step S131)
In step S131, the voxel v _i is mapped to the image plane of the camera coordinate system in the virtual light source VL in which a predetermined position in the virtual space is defined, the pixel position (x, y) is obtained, and the virtual light source VL and the voxel are obtained. Calculate the distance to v _{i e2 i} = dist (v _i , VL). Here, due to the additional processing of step S16 described later, the minimum value of the distance to the voxels belonging to the 3D model as viewed from the virtual light source VL is "light source depth image" on the image plane of the camera coordinate system in the virtual light source VL. It is recorded as and is continuously updated.

Therefore, in step S131, the value L (x, y) of the pixel position (x, y) in the light source depth image is further referred to, _{and the magnitude relationship between the distance e2 i} and the light source depth value L (x, y). If "e2 _i > L (x, y)", 3D is on the side closer to the light source VL than the _{voxel v i (and on the line connecting the voxel v i} and the light source VL). Since the voxels that make up the model already exist, it _{is determined that the voxels v i} do not affect the shadow. (This is because the shadow of the voxel v _i that is closer to the existing shadow can also be generated.) On the contrary, in the case of a negative judgment, that is, "e2 _i ≤ L (x, y)", the voxel v _It is determined that i affects the shadow.

(Details of additional processing in step S16)
As an additional process in step S16, the pixel value of the "light source depth image" to be referred to in step S131 is updated. Specifically, as in step S131, the voxels v _i determined to be the points constituting the 3D model are mapped to the image plane of the camera coordinate system in the virtual light source VL, and the pixel positions (x, y) are mapped. ), And the distance e2 _i = dist (v _i , VL) between the virtual light source VL and the voxel v _{i is calculated.} Then, with respect to the calculated distance e2 _i , the magnitude relationship with the pixel value L (x, y) of the light source depth image at the same position (x, y) at the present time is investigated, and the distance e2 _i is smaller ("e2"). _{If i} <L (x, y) ”), the pixel value L (x, y) of the light source depth image is overwritten with the value of the _{smaller distance e2 i and updated.} Negation If "e2 _i ≥ L (x, y)", the value is not updated. As the initial value of the pixels of the light source depth image, the light source depth value L (x, y) = ∞ (infinity) at each pixel position (x, y) may be set in step S10. Regardless of the calculated value of the distance e2 _i , the magnitude relation is always "e2 _i <∞".

As described above, the significance of the determination process in step S131 using the light source depth image to be updated as the additional process in step S16 is as follows.

_{That is, when it is determined that each voxel v i} , which is determined in a predetermined order (raster scan order, etc.) whether or not it belongs to the 3D model by the iterative process of the flow shown in FIG. 11, is in the occlusion area (in step S131). _{(When it arrives), another voxel v j} (j <i) for which the processing such as the determination has already been completed, exists at a position that shields the light source VL when viewed from the _{voxel v i, and} If there is something that belongs to the 3D model (including the one in the occlusion area), it is _{determined that the voxel v i} does not affect the shadow. On the other hand, in the _{absence of such another voxel v j} , the voxel v _i is shielded by any voxel that constitutes the already determined 3D model _{(before vi} itself in the processing order). Since it is in a position where the light source VL can be seen directly, it is _{judged that the voxel v i} affects the shadow.

The light source depth image is an example of a means for determining whether or not the _{voxel v i} is shielded by another voxel v _j (which constitutes a 3D model) when viewed from the virtual light source VL as described above. Yes, by setting the internal parameters of the light source camera, the determination can be made simply. (It is not for drawing something using the light source depth image as it is.)

<Drawing unit 14 ... Second embodiment>
The drawing unit 14 uses the 3D model obtained by the generation unit 13 to perform drawing reflecting the light source. Here, in the entire 3D model, the portion that is not determined to be the occlusion region is drawn after reflecting the light source in the same manner as in the first embodiment, and the presence of the light source causes the texture to be drawn. The shadow generated by the area may also be drawn. On the other hand, in the entire 3D model, the part determined to be the occlusion area is not visible from the user's viewpoint (virtual camera), so the texture drawing is omitted, but the drawing related to the shadow generated in relation to the light source by the area is You can do it. (That is, the texture drawing may be performed using only the portion of the entire 3D model that is not the occlusion area, and the shadow drawing may be performed using the entire 3D model regardless of whether or not it is the occlusion area. ) For drawing shadows with a light source effect, any existing method used in the field of 3DCG (3D computer graphics) may be used.

<Display unit 23 ... Second embodiment>
The display unit 23 may display the texture of the virtual object and its shadow as the drawing result obtained by the drawing unit 14.

<Third Embodiment>
The third embodiment performs the following as additional processing for the generation unit 13 in the first embodiment. That is, in the generation unit 13, the voxels constituting the 3D model are obtained in advance by excluding the occlusion area according to the first embodiment, and as an additional process, the voxel density is also determined for the voxel set determined to correspond to the occlusion area. After lowering by a predetermined ratio, the visual volume crossing method is applied to obtain the voxels constituting the 3D model.

In addition to the voxel set (referred to as the first voxel set) constituting the 3D model obtained from outside the occlusion region at high density in the first embodiment, the 3D obtained from within the occlusion region at low density in the third embodiment. As an example of the use of the voxel set (referred to as the second voxel set) constituting the model, the following is possible in the third embodiment.

The prerequisites for explaining this application will be described first. In the image processing system 100, as described above, the time is synchronized between the image processing device 10 and the terminal device 20, and then the terminal device 20 acquires the environmental information and transmits it to the image processing device 10 in real time. Then, the image processing device 10 takes a multi-viewpoint image, generates a 3D model, draws in a virtual viewpoint, sends the drawing result to the terminal device 20, and sends the drawing result to the user on the terminal device 20 side. On the other hand, AR viewing is possible.

Here, the acquisition of environmental information (only position / orientation information or both position / orientation information and depth image) on the terminal device 20 side is low-load and high-speed using a dedicated device such as a position / orientation sensor. Compared to what can be achieved at the processing rate, 3D model generation and drawing from multi-viewpoint video on the image processing device 10 side is image processing because the data size is larger than the environmental information and the amount of calculation is also large. Even if the device 10 is mounted on a dedicated server or the like having abundant computing resources, there may be a limit to the processing rate that can be realized. As a countermeasure against this, the acquisition and transmission of environmental information on the terminal device 20 side is performed at a high speed rate (for example, every 0.1 seconds), and the 3D model generation by the generation unit 13 on the image processing device 10 side is slow. By performing at a rate (for example, every second) and matching the drawing by the drawing unit 14 and the transmission of this drawing information to the high-speed rate, it is possible to realize the display at the high-speed rate on the terminal device 20 side. is there.

At this time, the 3D model generated by the generation unit 13 at a low speed rate (for example, every 1 second) on the image processing apparatus 10 side is MD (1) at the time t = 0, 1, 2, ... Every 1 second. Assuming 0), MD (1), MD (2), ..., the drawing unit 14 should perform drawing by interpolating the most recently generated 3D model at a high-speed rate (for example, every 0.1 seconds). Just do it. For example, MD (0) can be used as it is for the 3D model at time t = 0, but for time t = 0.1, 0.2, 0.3, ..., the model MD'(0.1) that interpolates this latest model MD (0). , MD'(0.2), MD'(0.3), ... may be used.

In the third embodiment, it is possible to generate a 3D model capable of such interpolation while suppressing the amount of calculation. For example, the interpolation model MD'(0.1) at time t = 0.1 changes the environmental information from time t = 0 to time t = 0.1 (the position and orientation of the virtual camera) with respect to the model MD (0). The coordinates may be moved (moved by rotation and translation) to reflect the change) to give a distinction between the inside and outside of the occlusion area. (That is, the interpolation model MD'(0.1) remains the same shape as the model MD (0), and only the visible position and orientation change as the position and orientation of the virtual camera move. If the transformation (rigid transformation) that represents the change in position and orientation between t = _{0 and 0.1 is T [0 → 0.1]} , then "MD'(0.1) = T _{[0 → 0.1]} · MD (0)". .) Including the following explanation, t = 0.2, 0.3, ..., etc. can be interpolated in the same way.

The distinction between the inside and outside of the occlusion area in the interpolation model MD'(0.1) and the drawing in the drawing unit 14 based on this may be given as follows. _{For the sake of explanation, MD (0) [visible]} is the visible part of the model MD (0) outside the occlusion area, which is required at high density, and the shielded part inside the occlusion area, which is required at low density. MD (0) _[Shield] . Similarly, the visible and occluded parts of the interpolated model MD'(0.1) as a result of the distinction are MD'(0.1) _[visible] and MD'(0.1) _[occluded] , respectively. Further, when the model MD (0) at time t = 0 seconds is obtained by the processing of the first embodiment (corresponding to the preprocessing in this third embodiment), it is transmitted from the depth acquisition unit 122 and obtained. The surface area in the space determined by the depth image is defined as the occlusion surface OC (0). The occlusion surface OC (0) is the depth d (x, y) from the position of the virtual camera on a straight line that performs three-dimensional back projection from the position of the virtual camera to each position (x, y) of the depth image. It is a plane in the model space that passes through points (X, Y, Z) that are only apart.

This occlusion surface may be obtained as a surface passing through each discrete position (X, Y, Z) obtained by three-dimensionally back-projecting each discrete position (x, y) of the depth image. For example, it may be obtained by curve fitting, or it may be obtained as a polygon having each discrete position (X, Y, Z) as a vertex. After finding the surface as the polygon, etc., for each surface element (individual polygon, etc.) that composes the surface, even at the position of the virtual camera (t = 0.1, etc.) at the time when the model was obtained, t = 0.1, When the angle between the direction of the straight line extending from the position of the time to be interpolated, such as 0.2, ...) to the position of the surface element and the direction of the normal of the surface element is determined to be close to a right angle by the threshold judgment. , The surface element may be excluded to obtain the occlusion surface. (The surface element to be determined may be excluded because the surface of the corresponding actual physical object may not exist.)

Here, the occlusion surface OC (0) is stationary in the world coordinates on the shooting side (thus, even in the virtual space for drawing) as illustrated as a table in the schematic example of the physical object POB shown in FIG. (Similarly stationary). That is, the relationship between the occlusion region OC (0) at time t = 0 seconds and the occlusion surface OC (0.1) at time t = 0.1 seconds is the relationship between model MD (0) and model MD'(0.1). Similarly, it is assumed that "OC (0.1) = T _{[0 → 0.1]} · OC (0)". (In other words, the 3D model generated at time t = 0 seconds and the obtained occlusion surface are stationary at the same position in the model space even at time t = 0.1, and only the virtual viewpoint in the model space moves. It is assumed that

_{Based on this assumption, the distinction between the visible part MD'(0.1) [visible]} and the shielding part MD'(0.1) _{[shielding] in} the interpolation model MD'(0.1) is seen from the position of the virtual camera at time t = 0.1 seconds. Therefore, it can be judged by the presence or absence of shielding by the stationary occlusion surface OC (0.1). That is, when projecting each polygon constituting the interpolation model MD'(0.1) to the virtual camera position at time t = 0.1 seconds, if it passes through the occlusion surface OC (0.1), the shielding portion MD'(0.1) _{[ It belongs to [shield]} , and if it does not pass, it can be judged to belong to the visible part MD'(0.1) _[visible]. The same method as in step S13 of FIG. 7 may be used for this determination, and the depth information seen from the position of the virtual camera at time t = 0.1 seconds (occlusion surface stationary at the position at time t = 0 seconds). ), Depending on whether the model MD'(0.1) is shielded by this depth information, the shielded part MD'(0.1) _[shielded] if it is shielded, and the visible part MD'if it is not shielded. (0.1) Can be judged as _[visible].

As another method, if the depth image is obtained in real time even at time t = 0.1, 0.2, ..., etc., interpolation is performed without using the occlusion surface OC (0.1) as a 3D model as described above. For each polygon constituting the model MD'(0.1), the same processing as in step S13 of FIG. 7 is applied with reference to the depth image at time t = 0.1, and the visible part MD'(0.1) _[visible] and the shielding part MD are applied. '(0.1) You may want to get the distinction of _[shielding].

The drawing by the drawing unit 14 with respect to the obtained visible portion MD'(0.1) _[visible] may be the same as in the first embodiment. FIG. 12 is a diagram showing a schematic example of drawing information G (0) at time t = 0 seconds and drawing information G (0.1) at time t = 0.1 seconds corresponding to the above description example. The drawing information G (0) and G (0.1) are the drawing of the visible part MD (0) _[visible] and MD'(0.1) _[visible] shown in gray, respectively. Other shielding parts and occlusion surfaces shown on a white background are not drawn, but are shown as schematic examples of the above explanatory examples.

In the example of FIG. 12, the object OB drawn on the shooting environment PE side and the physical object POB that generates the occlusion region on the viewing environment WE side are a person and a table, respectively, as in the schematic example of FIG. In the drawing information G (0) at time t = 0 seconds, the user on the viewing environment WE side is drawn as if they were viewed from the front, and in the drawing information G (0.1) at time t = 0.1 seconds, these are drawn. It is drawn as if it was seen from slightly above. (That is, the virtual camera is moving slightly upward from the front.)

<Fourth Embodiment>
The fourth embodiment is a modification of the third embodiment. In the third embodiment, a voxel set (second voxel set) constituting a 3D model was obtained by applying the visual volume crossing method at a low density for the entire occlusion region, but in the fourth embodiment, the voxel set (second voxel set) is obtained. A voxel set (referred to as a third voxel set) that constitutes a 3D model is obtained by applying the visual volume crossing method at a low density for only a part of the occlusion region.

That is, the relationship between the second voxel set of the third embodiment and the third voxel set of the fourth embodiment is "the second voxel set ⊃ the third voxel set", and is subject to the visual volume crossing method at low density. As the range of the occlusion area is narrowed, the calculation speed can be expected to be increased in the fourth embodiment.

In the fourth embodiment, as a method of determining a part of the occlusion region to which the visual volume crossing method is applied at a low density from the entire occlusion region, it is determined by the threshold value that it is close to the region determined not to be the occlusion region. It may be determined as a region to be (referred to as a "threshold proximity region"). The threshold value determination for determining the threshold value proximity region is performed so that the larger the time variation of the position / orientation of the virtual viewpoint corresponding to the user viewpoint obtained by the position / attitude acquisition unit 211, the wider the threshold value proximity area. It may be relaxed.

This threshold proximity area serves as an extension as an area treated as having no occlusion (including an area that is actually an occlusion area) by adding a buffer area to an area that does not actually have occlusion. It will be fulfilled.

<Fifth Embodiment>
In the fifth embodiment, as a modification of the third embodiment, the time variation of the position / posture of the virtual viewpoint corresponding to the user viewpoint obtained by the position / posture acquisition unit 211 (similar to the above-mentioned concept of easing the threshold value determination). When it is determined that the movement of the virtual viewpoint) is large in the threshold value determination, the third embodiment is applied, and when the determination is not made, the first embodiment is applied.

That is, when it is determined that the movement of the virtual viewpoint corresponding to the user viewpoint is large, in addition to obtaining a high-density 3D model outside the occlusion region by applying the third embodiment, it is also possible to obtain the 3D model inside the occlusion region. When the 3D model is obtained at a low density and it is not determined that the movement of the virtual viewpoint corresponding to the user viewpoint is large, only the 3D model is obtained at a high density outside the occlusion region by applying the first embodiment. To carry out.

<Sixth Embodiment>
As a modification of the fourth embodiment, the sixth embodiment is the same as the above-mentioned concept of easing the threshold value determination, and the time variation of the position / posture of the virtual viewpoint corresponding to the user viewpoint obtained by the position / posture acquisition unit 211 ( When it is determined that the movement of the virtual viewpoint) is large in the threshold value determination, the fourth embodiment is applied, and when the determination is not made, the first embodiment is applied.

That is, when it is determined that the movement of the virtual viewpoint corresponding to the user viewpoint is large, by applying the fourth embodiment, in addition to obtaining a 3D model with high density outside the occlusion area, one in the occlusion area. The 3D model is obtained at a low density even in the partial area, and when it is not determined that the movement of the virtual viewpoint corresponding to the user viewpoint is large, the application of the first embodiment is performed to obtain a high density 3D model outside the occlusion area. Only ask for.

<Hardware configuration>
FIG. 13 is a diagram showing an example of a hardware configuration in a general computer device 70. Each of the image processing device 10 and the terminal device 20 can be realized as one or more computer devices 70 having such a configuration. When each of the image processing device 10 or the terminal device 20 is realized by two or more computer devices 70, information necessary for processing may be transmitted and received via a network. The computer device 70 is a CPU (central processing unit) 71 that executes a predetermined instruction, and a GPU (graphics calculation device) as a dedicated processor that executes a part or all of the execution instructions of the CPU 71 on behalf of the CPU 71 or in cooperation with the CPU 71. ) 72, RAM73 as the main storage device that provides the work area to the CPU71, ROM74 as the auxiliary storage device, GPU memory 78 that provides the memory space for the GPU72, communication interface 75, display 76, mouse, keyboard, touch panel, etc. It includes an input interface 77 sensor 78 that accepts user input, a camera 79, and a bus BS for exchanging data between them.

Each part of the image processing device 10 and the terminal device 20 can be realized by a CPU 71 and / or a GPU 72 that reads and executes a predetermined program corresponding to the function of each part from the ROM 74. Both CPU71 and GPU72 are a type of arithmetic unit (processor). Here, when the display-related processing is performed, the display 76 further operates in conjunction with the display 76, and when the communication-related processing related to data transmission / reception is performed, the communication interface 75 further operates in conjunction with the display. The sensor 78 can be used as one or more types of dedicated sensors when the terminal device 20 acquires environmental information with the dedicated sensor. The display unit 23 of the terminal device 20 can be realized by the display 76 (optical see-through method or video see-through method). The photographing unit 11 and the display side photographing unit 22 can be realized by the camera 79.

100 ... Image processing system
10 ... Image processing device, 11 ... Shooting unit, 12 ... Extraction unit, 13 ... Generation unit, 14 ... Drawing unit
20 ... Terminal device, 21 ... Acquisition unit, 22 ... Display side shooting unit, 23 ... Display unit

Claims (10)

An extraction unit that extracts the area of the object being photographed as a mask image from the image of each viewpoint of the multi-viewpoint image,
An image including a generation unit that applies the visual volume crossing method to the mask image and generates a 3D model of the object by determining whether or not each voxel of a predetermined voxel set belongs to the 3D model. It ’s a processing device,
Before the application, the generation unit arranges the depth information acquired from the user's viewpoint in the voxel space with reference to the virtual camera viewpoint, determines the spatial position given by the depth information, and then determines the spatial position. It is determined whether each voxel is closer to or farther from the virtual camera viewpoint than the spatial position given by the depth information, and only the voxels determined to be closer are subject to the visual volume crossing method. An image processing device for determining whether or not it belongs to the 3D model. The image according to claim 1, further comprising a drawing unit that projects the generated 3D model onto an image plane with reference to the virtual camera viewpoint and draws the image using the texture of the multi-view image. Processing equipment. A light source at a predetermined position is set in the voxel space, and the light source is set.
The generation unit determines in a predetermined order whether each voxel in a predetermined voxel set is on the near side or the far side, and then determines whether the voxel is on the near side and the 3D model for the voxel determined to be on the near side. Judge whether it belongs to or not,
With respect to the voxels determined to be on the distant side, it is further determined whether or not the voxels already determined to belong to the 3D model by the determination in the predetermined order are shielded from the light source.
The image processing apparatus according to claim 1, wherein when it is determined that the voxel is not shielded, it is determined whether or not the voxel belongs to the 3D model as an application target of the visual volume crossing method. Of the generated 3D model, only the part corresponding to the voxel determined to be on the near side is projected onto the image plane based on the virtual camera viewpoint and drawn using the texture of the multi-view image. And then
The image processing apparatus according to claim 3, further comprising a drawing unit that draws a shadow of the 3D model by the light source on the image plane using the entire generated 3D model. The generation unit applies the visual volume crossing method at the first voxel density to the voxels determined to be on the near side, and further, from the first voxel density to the voxels determined to be on the far side. The image processing apparatus according to claim 1, wherein the visual volume crossing method is applied at a low second voxel density. The generation unit has a second voxel density lower than that of the first voxel density only for voxels determined to be on the distant side and which are determined to be close to the voxel determined to be on the near side. The image processing apparatus according to claim 5, wherein the visual volume crossing method is applied. The generator generates the 3D model by applying the visual volume crossing method at the first voxel density and the second voxel density at the first time.
The generated 3D model is further provided with a drawing unit that projects the generated 3D model onto an image plane with reference to the virtual camera viewpoint and draws it using the texture of the multi-view image.
At the first time, the drawing unit draws only the portion of the 3D model having the first voxel density.
At the second time after the first time, the depth information is arranged in the voxel space based on the virtual camera viewpoint at the second time, and the spatial position given by the depth information is determined. It is determined whether each voxel is closer to or farther from the virtual camera viewpoint than the spatial position given by the depth information, and only the part corresponding to the voxel determined to be near is the part of the 3D model. The image processing apparatus according to claim 5 or 6, wherein drawing is performed. The time change of the position and posture of the user's viewpoint has been acquired.
When it is determined that the time change is larger than the threshold value determination,
The generation unit applies the visual volume crossing method at the first voxel density to the voxels determined to be on the near side, and further applies the visual volume crossing method to all or a part of the voxels determined to be on the distant side. Applying the visual volume crossing method at a second voxel density lower than one voxel density,
If it is not determined that the time change is greater than the threshold determination,
The image processing according to claim 1, wherein the generation unit determines whether or not only the voxels determined to be on the near side belong to the 3D model as an application target of the visual volume crossing method. apparatus. An extraction stage that extracts the area of the object being photographed as a mask image from the image of each viewpoint of the multi-viewpoint image,
An image including a generation stage in which a visual volume crossing method is applied to the mask image to generate a 3D model of the object by determining whether or not each voxel of a predetermined voxel set belongs to the 3D model. It ’s a processing method,
In the generation stage, the depth information acquired from the user's viewpoint is arranged in the voxel space with reference to the virtual camera viewpoint in advance before the application, and the spatial position given by the depth information is determined. It is determined whether each voxel is closer to or farther from the virtual camera viewpoint than the spatial position given by the depth information, and only the voxels determined to be closer are subject to the visual volume crossing method. An image processing method characterized by determining whether or not it belongs to the 3D model. An extraction unit that extracts the area of the object being photographed as a mask image from the image of each viewpoint of the multi-viewpoint image,
An image including a generation unit that applies the visual volume crossing method to the mask image and generates a 3D model of the object by determining whether or not each voxel of a predetermined voxel set belongs to the 3D model. It ’s a processing device,
Before the application, the generation unit arranges the depth information acquired from the user's viewpoint in the voxel space with reference to the virtual camera viewpoint, determines the spatial position to be given by the depth information, and then determines the spatial position. It is determined whether each voxel is closer to or farther from the virtual camera viewpoint than the spatial position given by the depth information, and only the voxels determined to be closer are subject to the visual volume crossing method. An image processing program characterized in that a computer functions as an image processing device for determining whether or not it belongs to the 3D model.

Images