JP2023066705A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an image processing device capable of performing high-speed volume rendering.SOLUTION: An image processing device (100) includes acquisition means (105) for acquiring a teacher image and a camera position corresponding to the teacher image, ray-of-light calculation means (106) for calculating a ray of light corresponding to each pixel of the teacher image using the camera position, and learning parameter calculation means (107) for performing machine learning by sampling points on the ray of light and using the teacher image to calculate learning parameters. The sampling density of the ray of light within the depth of field of the teacher image is higher than the sampling density outside the depth of field.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus that performs three-dimensional modeling by machine learning.

2. Description of the Related Art Conventionally, a technique of performing three-dimensional modeling using images of an object photographed from various angles is known. Japanese Patent Application Laid-Open No. 2002-200002 discloses a technique for generating an image of an object viewed from an angle different from that at the time of photographing with a small amount of calculation. However, since the color of light reflected from an object changes depending on the angle at which it is viewed, the technique disclosed in Patent Document 1 may cause a sense of discomfort when the angle for reconstructing an image is changed.

Non-Patent Document 1 describes a technique for reconstructing an image that does not feel strange like a real shot by taking into account the direction of light rays in addition to the three-dimensional position in space, sampling points on the light rays, and performing volume rendering. is disclosed.

JP 2018-205863 A JP 2008-15754 A

Ben Mildenhall, Pratul P.; Srinivasan, Matthew Tancik, Jonathan T.; Barron, Ravi Ramamoorthi, and Ren Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", In ECCV, 2020. TIANYE LI, MIRA SLAVCHEVA, MICHAEL ZOLLHOEFER, SIMON GREEN, CHRISTOPH LASNER, CHANGIL KIM, TANNER SCHMIDT, STEVEN LOVEGROVE, MICHAEL GOSELLE, ZHAOYANG LV, "Neural 3D Video Synthesis", arXiv:2103.02597, 2021

However, with the technique disclosed in Non-Patent Document 1, it is necessary to perform volume rendering by sampling points on rays from end to end of the target space, which increases the amount of computation and requires a great deal of processing time.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of performing volume rendering at high speed.

An image processing apparatus as one aspect of the present invention comprises a teacher image, an acquisition unit for acquiring a position of a camera corresponding to the teacher image, and light rays corresponding to each pixel of the teacher image using the position of the camera. and a learning parameter calculation means for calculating a learning parameter by performing machine learning by sampling points on the light ray and using the teacher image, wherein the depth of field of the teacher image is higher than the sampling density outside the depth of field.

Other objects and features of the invention are described in the following embodiments.

According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program capable of performing volume rendering at high speed.

1 is a block diagram of a personal computer according to a first embodiment; FIG. 4 is a flowchart of 3D model learning in the first embodiment; FIG. 4 is an explanatory diagram of capturing a teacher image in the first embodiment; 4A and 4B are explanatory diagrams of a teacher image and a focus map in the first embodiment; FIG. 4 is a flowchart of free-viewpoint image rendering in the first embodiment; FIG. 2 is an explanatory diagram of a free viewpoint camera in the first embodiment; FIG. 4A and 4B are explanatory diagrams of a teacher image and a focus map in the first embodiment; FIG. FIG. 4 is an explanatory diagram of subject depth calculation in the first embodiment; FIG. 4 is an explanatory diagram of coordinate calculation of a three-dimensional point in the first embodiment; FIG. 4 is an explanatory diagram of a teacher image and a low-resolution focus map in the first embodiment; FIG. FIG. 10 is an explanatory diagram of a teacher image taken and a free-viewpoint camera according to the second embodiment; FIG. 11 is an explanatory diagram of a surrounding teacher image and a focus map in the second embodiment; FIG. 10 is an explanatory diagram of a teacher image and a focus map in the second embodiment; FIG. 11 is a block diagram of a personal computer in a third embodiment; FIG. 10 is a flow chart of dynamic 3D model learning in the third embodiment; FIG. 11 is an explanatory diagram of capturing a teacher image in the third embodiment; FIG. 11 is an explanatory diagram of capturing a teacher image in the third embodiment; FIG. 11 is an explanatory diagram of a teacher image and a focus map in the third embodiment; FIG. 11 is an explanatory diagram of a teacher image and a focus map in the third embodiment; 10 is a flow chart of free-viewpoint video rendering in the third embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
First, a personal computer (image processing apparatus) according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram of a personal computer (image processing apparatus) 100. As shown in FIG. In this embodiment, a personal computer will be described as an example of the image processing apparatus, but the present invention is not limited to this, and can be applied to image processing apparatuses other than personal computers.

The control unit 101 is, for example, a CPU, and controls the operation of each block of the personal computer 100 by reading an operation program of each block of the personal computer 100 from the ROM 102, developing it in the RAM 103, and executing it. The ROM 102 is a rewritable non-volatile memory such as an SSD, and stores an operation program for each block included in the personal computer 100 as well as parameters required for operation of each block. A RAM 103 is a rewritable volatile memory such as a DRAM, and is used as a temporary storage area for data output during the operation of each block of the personal computer 100 . The data storage unit 104 is a recording medium such as a hard disk that reads and writes image data necessary for machine learning, metadata for each image, and the like.

A photographing camera position and orientation estimation unit 105 estimates the position and orientation of the photographing camera for each image from the image data group using a known technique such as SfM (Structure from Motion). That is, the photographing camera position/orientation estimation unit 105 is acquisition means for acquiring a teacher image and the position of the camera corresponding to the teacher image.

The light ray calculator 106 calculates light rays for volume rendering, for example, by a method disclosed in Non-Patent Document 1. In Non-Patent Document 1, a ray for volume rendering is defined as r(t)=o+td. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. The ray direction vector d is obtained by calculating a three-dimensional vector from the principal point of the camera to each pixel on the image plane. Also, the camera principal point o and the ray direction vector d are coordinate-transformed from the camera coordinate system to the world coordinate system by the camera position and orientation parameters. That is, the ray calculation unit 106 calculates a ray corresponding to each pixel of the teacher image using the camera position during learning, and calculates a ray corresponding to each pixel of the arbitrary viewpoint camera using the camera position and orientation during inference. It is a light ray calculation means for calculating.

The neural network unit 107 samples points on the ray for each ray, calculates the color and density of the corresponding points by the neural network, and performs volume rendering over the target space to determine the color of the pixel corresponding to each ray. . For learning and inference, the method disclosed in Non-Patent Document 1 may be used. At the time of learning, the L2 loss between the color calculated by volume rendering and the color of the captured image is used as a loss function, and the learning weight is converged by the error back propagation method. That is, the neural network unit 107 is a learning parameter calculation unit that performs machine learning by sampling points on a ray and using a teacher image during learning to calculate a learning parameter. During inference, the free viewpoint image is rendered by volume rendering on each ray in the free camera position and orientation. That is, the neural network unit 107 is a rendering unit that renders a camera image by machine learning using learning parameters pre-learned by sampling points on a ray by a learning parameter calculation unit during inference.

FΘ: (x, d) → (c, σ) (1)
In equation (1), FΘ is a neural network consisting of multi-layer perceptrons, and inputs are the three-dimensional coordinate x of a point on the sampled ray and the ray direction vector d. FΘ outputs RGB color c and density σ for each sampling point. Volume rendering for ray r is represented by Cvr(r)=ΣTi(1−exp(−σi·δi))ci, i=1˜N. where N is the number of samples, i is the index number for each sampling point, ci and σi are the color and density corresponding to index i, δi=t(i+1)−t(i), t(i) is index i It is the distance from the corresponding camera principal point to the sampling point on the ray. Also, Ti=exp(-Σσj·δj) and j=1 to i−1, thereby excluding the influence of object colors blocked by occlusion.

The free camera position/orientation acquisition unit 108 is acquisition means for acquiring the camera position/orientation (the position of the arbitrary viewpoint camera) for the free viewpoint image. A camera position/orientation parameter calculated in advance by an external device may be obtained, or a camera position/orientation designated by a user via an operation member such as a joystick may be obtained.

Next, 3D model learning by the control unit 101 will be described with reference to FIG. FIG. 2 is a flow chart of 3D model learning. First, in step S201, the photographing camera position/orientation estimation unit 105 estimates the camera position/orientation corresponding to each image used for learning. Learning converges to target learning weights by updating the learning weights through iteration processing. Subsequently, in step S202, the control unit 101 determines whether or not the iteration process has been completed. When the iteration process is completed, this flow ends. On the other hand, if the iteration process has not been completed, the process proceeds to step S203.

In step S203, the control unit 101 randomly selects light beams corresponding to the batch size in each iteration. The batch size may be set, for example, to the number of rays of 4096 shown in Non-Patent Document 1. Further, in this embodiment, at this time, by excluding light rays that are not suitable for 3D modeling and selecting light rays, it is possible to efficiently reduce the calculation cost. The operation will be described with reference to FIGS. 3 and 4. FIG.

FIG. 3 is an explanatory diagram of photographing of the teacher image, and shows a bird's-eye view for explaining the relationship among the subject, the photographing space, and the photographing camera. Reference numeral 301 denotes a photographing camera (imaging device); 302, a 3D modeling target photographing space range; 303, a main subject; and 304, a background subject. The camera 301 is focused on the main subject 303 and is focused within the angle of view and depth of field indicated by hatching 305 . A plurality of imaging cameras 301 are arranged to photograph an object within the imaging space range 302 from various directions, but only the imaging cameras 301 are shown in FIG. 3 for the sake of simplicity.

FIG. 4 is an explanatory diagram of a teacher image and a focus map (distance distribution information), and illustrates an image 401 acquired by a photographing camera 301 and a focus map 402 that is accompanying metadata. In FIG. 4, the degree of focus on the imaging plane is shown in the form of a grayscale map, with white in the front, black in the back, and 50% gray indicating focus. The focus map 402 is configured to obtain a defocus map on the imaging plane as a focus map from a phase difference image obtained from an imaging sensor in which all pixels are composed of phase difference pixels, as disclosed in Patent Document 2, for example. Just do it.

Rays 306 and 307 in FIG. 3 are volume rendering target rays, but since there is no object within the depth of field on the ray 306, they are excluded. Determination as to whether or not it is an unnecessary ray can be made based on the focus map 402 in FIG. The pixel 404 corresponding to ray 307 is a 50% gray in-focus pixel, whereas the pixel 403 corresponding to ray 306 is a dark gray outside the depth of field. It is determined that By excluding unnecessary rays in this way, volume rendering processing can be speeded up. In this embodiment, the acquisition means acquires distance distribution information (focus map) corresponding to the teacher image, and the ray calculation means determines whether the light rays are within the range of depth of field of the teacher image based on the distance distribution information. Determine whether or not there is

In this embodiment, light rays outside the range of the depth of field are excluded, but the invention is not limited to this. The light ray calculation unit 106 changes the density of light rays inside and outside the depth of field to efficiently select light rays (the light ray calculation means preferentially selects light rays within the depth of field of the teacher image). do). For example, for rays within the depth of field, all rays may be selected, and for rays outside the depth of field, no more than 10% of the total rays of the target camera may be selected.

After batch-size rays are selected in step S203 of FIG. 2, the ray calculator 106 calculates rays in step S204. A ray for volume rendering is defined as r(t)=o+td, as described above. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray.

The range of distance t sampled for volume rendering is limited to the range of depth of field indicated by hatching 305 in FIG. The depth range of the hatched portion 305 is represented by Z−Df to Z+Db using the front depth of field Df, the rear depth of field Db, and the in-focus object distance Z. FIG. Df = (r · Av · Z ^ 2) / (f ^ 2 + r · Av · Z), Db = (r · Av · Z ^ 2) / (f ^ 2 - r · Av · Z) be. Here, r is the permissible circle of confusion diameter, Av is the aperture value, and f is the focal length. The permissible circle of confusion radius r is set to twice the pixel pitch. In this way, by limiting the range of distance t sampled for volume rendering to within the range of depth of field, volume rendering processing can be speeded up.

In the present embodiment, the permissible circle of confusion diameter r is set to twice the pixel pitch, but it is not limited to this. You can make it finer. That is, the permissible circle of confusion diameter for determining the depth of field may be determined based on the resolution (rendering resolution) during rendering using the learning parameters.

In the present embodiment, the depth of field is Z−Df to Z+Db, but the depth of field is not limited to this, and Z It may be set to a range with some margin, such as from -2·Df to Z+2·Db. Also, in the present embodiment, the light ray range for volume rendering is limited to within the range of the depth of field, but it is not limited to this. Efficient sampling may be performed by changing the sampling density (sampling density) between within and outside the range of the depth of field. That is, the sampling density of rays within the depth of field range of the teacher image should be higher than the sampling density outside the depth of field range. For example, for the ray 307 in FIG. 3, 32 points are first sampled from the entire area covered by the 3D modeling target area 302, and an additional 128 points are sampled for the area covered by the hatched portion 305. FIG. In this way, the sampling points may be densely arranged only within the depth of field.

After the ray is calculated in step S204 of FIG. 2, the neural network unit 107 updates the learning weights in step S205. The control unit 101 determines the learning weights by repeating steps S202 to S205 until the learning weights converge. It should be noted that determination of completion of iteration in step S202 may be made by determining whether or not the number of iterations of 100 to 300K has been reached, as disclosed in Non-Patent Document 1, for example.

Next, free-viewpoint image rendering by the control unit 101 will be described with reference to FIG. FIG. 5 is a flowchart of free-viewpoint image rendering. First, in step S501, the free camera position/orientation acquisition unit 108 acquires the camera position/orientation of the free viewpoint to be rendered. Subsequently, in step S502, the control unit 101 determines whether or not calculation of pixel values (RGB values) by volume rendering has been completed for all pixels of the rendered image. When the calculation of pixel values for all pixels is completed, this flow ends. On the other hand, if calculation of pixel values has not been completed for all pixels, the process proceeds to step S503.

In step S503, the control unit 101 determines whether the three-dimensional point corresponding to each pixel is within the depth of field of the learning image, that is, whether the ray is within the depth of field. If the 3D point is not within the depth of field, return to step S502. On the other hand, if the 3D point is within the depth of field, the process proceeds to step S503 to perform volume rendering. A fixed pixel value such as black is assigned to a pixel corresponding to a light ray outside the depth of field.

FIG. 6 is an explanatory diagram of the free-viewpoint camera, showing a bird's-eye view explaining the relationship between the subject, the shooting space, and each camera. FIG. 7 is an explanatory diagram of a teacher image and a focus map, showing an image 701 acquired by a photographing camera 601 and a focus map 702 that is attached metadata. 603 in FIG. 6 is a rendering free-viewpoint camera, and 301 and 601 are shooting cameras adjacent to the free-viewpoint camera 603 .

Pixel 404 in FIG. 4 and pixel 704 in FIG. 7 represent the same three-dimensional point 607 in FIG. A light ray 605 of the free viewpoint camera 603 and a light ray 307 of the imaging camera 301 represent the same three-dimensional point 607 in FIG. If the 3D coordinates of the 3D point 607 are known in advance, it can be confirmed that the ray 605 corresponds to the ray 307 and can be obtained from the focus map 402 in FIG.

FIG. 8 is an explanatory diagram of object depth calculation, showing a procedure for calculating the object depth Z+ΔZ from the defocus value def for each pixel position indicated by the focus map. 801 is an imaging optical system, 802 is an imaging plane position, 803 is a focus object distance position, 804 is a defocus image formation position, and 805 is an object distance position. From the lens formula, 1/Z+1/Z'=1/f and 1/(Z+ΔZ)+1/(Z'+def)=1/f hold, so the depth of the subject Z+ΔZ can be calculated from these ( f: focal length).

If the depth Z+ΔZ of the subject is known, x/f=X/(Z+ΔZ) and y/f=Y/(Z+ΔZ) from the triangular geometric relationship shown in FIG. holds. From these, the X coordinate, Y coordinate and Z coordinate (Z+ΔZ) of the three-dimensional point 607 can be obtained.

From the above, it can be confirmed that the ray 605 corresponds to the ray 307, and since it is known that the ray 307 at the time of learning is within the depth of field, the ray 605 is targeted for volume rendering. . On the other hand, pixel 403 in FIG. 4 and pixel 703 in FIG. 7 represent the same three-dimensional point 606 in FIG. 6, corresponding to rays 306 and 604 in FIG. 6, respectively. However, since the ray 306 at the time of learning is known to be outside the depth of field and the imaging camera 601 does not capture the corresponding three-dimensional point 606, the ray 604 is excluded from volume rendering. . In this way, by excluding light rays corresponding to unlearned three-dimensional points from volume rendering, it is possible to speed up the volume rendering processing of free viewpoint images.

In step S504, the light ray calculator 106 calculates the light ray determined to be within the depth of field in step S503 of FIG. A ray for volume rendering is defined as r(t)=o+td, as described above. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. The range of distances t sampled for volume rendering is limited to within the depth of field of adjacent filming cameras 301, 601 indicated by hatching 305 and hatching 602 in FIG. In this way, by limiting the range of the distance t sampled for volume rendering to within the depth of field of the adjacent imaging camera, it is possible to speed up the volume rendering processing of the free viewpoint image.

After the rays are calculated in step S504 of FIG. 5, the neural network unit 107 executes volume rendering processing in step S505 to calculate corresponding pixel values (RGB values).

Note that in the present embodiment, the image (teacher image) of the photographing camera and the focus map (distance distribution information) have the same resolution, but are not limited to this, and may have different resolutions. For example, the distance distribution information may have a lower resolution than the teacher image. A map generated based on a parallax map, such as a focus map, is usually reduced in size by the size of the template because template matching of a predetermined size is performed for searching for stereo corresponding points. For example, if the template size is 16×16 pixels, the normal map size is 1/16 both vertically and horizontally. Note that a focus map is also necessary when rendering a free-viewpoint image. Capacity can be reduced.

FIG. 10 is an explanatory diagram of a teacher image and a low-resolution focus map, and shows an image 401 acquired by a photographing camera 301 and a focus map 1001 that is attached metadata. The focus map 1001 has a lower resolution than the template size (16×16 pixels) for template matching, and one pixel of the focus map corresponds to a 16×16 pixel region of the image. As described with reference to FIG. 3, 303 is the main subject and 304 is the background subject.

A pixel 1003 on the image 401 corresponds to a pixel 1002 on the focus map 1001 . A pixel 1003 shows the main subject 303, but the 16×16 pixel range of the template includes both the main subject 303 and the background ground. Therefore, the focus map pixel value (defocus value) of the pixel 1002 may be between the defocus values of the main subject 303 and the ground. Therefore, with respect to the subject contour area such as the pixel 1003, the light rays are selected widely, and the sampling range for volume rendering is not limited. That is, the light ray calculating means identifies the subject outline area based on the distance distribution information, makes it easier to select light rays for the subject outline area, and sets a wide sampling range.

In the present embodiment, the distortion due to the lens of the photographing camera is very slight and the learning image is not subjected to distortion correction. In that case, distortion correction is performed on the learning image so that the correct ray can be calculated. Furthermore, correct ray selection is made possible by performing distortion correction on the focus map that is referenced in pairs with the image. That is, the acquiring means may process the teacher image and the distance distribution information based on the distortion component of the optical system of the camera.

In this embodiment, the pixel values of the focus map held in the data storage unit 104 are defocus values, but are not limited to this, and may be parallax values or distance values. Also, any format may be used as long as it can be converted into a distance value when determining the range of volume rendering. Further, when storing the defocus values, the defocus values are recorded as a focus map after correcting the defocus value due to the eccentricity of the lens and the tilt of the image pickup device in advance by the imaging camera, and stored in the data storage unit 104 . good too. That is, the distance distribution information may include at least one of a map based on the amount of shift representing parallax (parallax map), a map based on the amount of defocus (defocus map), or a map based on distance (distance map). .

In this embodiment, for example, as shown in FIG. 3, the range of volume rendering is a hatched portion 305 defined by the front depth of field and the rear depth of field, but is not limited to this. do not have. For example, it may be determined from the focus map whether the surface of the object is in front of or behind the object, and further the range of volume rendering may be limited. That is, based on the distance distribution information (the sign of the focus map), it is determined whether the surface of the subject is in front of (front) or behind (back) the focusing distance of the subject. Density may vary. For example, with respect to the object 303 in FIG. 3, the focus map shows that the surface of the object exists in front, so the range of volume rendering may be limited only to the front depth of field.

Also, in the present embodiment, the volume rendering range is determined based on the in-focus subject distance, but it is not limited to this. For example, a predetermined range may be set as the volume rendering range based on the object surface position. That is, the distance to the surface of the subject may be determined based on the distance distribution information, and the sampling density may be determined based on the distance to the surface of the subject. The depth position of the object surface may be determined by the method described with reference to FIG.

In the present embodiment, since volume rendering is performed using only light rays within the depth of field during free-viewpoint image rendering (step S503), pixels corresponding to light rays outside the depth of field have fixed pixel values such as black , but is not limited to this. Only pixels outside the depth of field, such as the background, may be rendered differently. For example, the background may be pasted with a spherical environment texture, or the background may be rendered from an entirely separate background 3D model.

In this embodiment, the focus map can be evaluated over the entire screen, but the present invention is not limited to this, and a low-reliability region in which the focus map cannot be calculated may be defined to prevent problems from occurring in post-processing. In such a case, all target rays should be selected for the low-confidence region. That is, the light ray calculating means can select all light rays for a region (low-reliability region) determined to have low reliability based on the distance distribution information, and set a wide sampling range.

According to this embodiment, it is possible to appropriately limit the rays for volume rendering and the sampling range, thereby speeding up the volume rendering process.

(Second embodiment)
Next, an image processing apparatus according to a second embodiment of the invention will be described. In the present embodiment, the acquisition means further acquires a peripheral teacher image of the surrounding space (a space including the background subject), and the learning parameter calculation means performs machine learning by using the teacher image and the peripheral teacher image, and learns Calculate parameters. The configuration of the image processing apparatus according to this embodiment is the same as the configuration of the personal computer 100 described in the first embodiment with reference to FIG. The flowchart for explaining the 3D model learning operation shown in FIG. 2 and the free viewpoint image rendering operation shown in FIG. 5 executed by the control unit 101 in FIG. 1 is also the same as in the first embodiment.

FIG. 11 is an explanatory diagram of the shooting of the teacher image and the free-viewpoint camera, showing a bird's-eye view for explaining the relationship between the subject, the shooting space, the shooting camera, and the free-viewpoint camera. Reference numerals 1101 and 1105 denote shooting cameras, 1103 a free-viewpoint camera, 302 a shooting space range for 3D modeling, 303 a main subject, and 304 a background subject. The imaging camera 1101 has a wide-angle focal length, is focused on the background subject 304, and is focused within the angle of view and depth of field indicated by hatching 1102 . The imaging camera 1105 has a focal length of a standard angle of view, is focused on the main subject 303, and is focused within the angle of view and depth of field indicated by hatching 1106. FIG.

In this embodiment, in order to photograph the subject within the imaging space range 302 from various directions, a plurality of imaging cameras are arranged in addition to the imaging cameras 1101 and 1105 . Specifically, in order to photograph the main subject 303, a plurality of imaging cameras having focal lengths and imaging distances (focus positions) similar to those of the imaging cameras 1105 are arranged. Also, in order to photograph subjects other than the main subject 303, a plurality of imaging cameras having focal lengths and imaging distances (focus positions) similar to those of the imaging camera 1101 are arranged. In FIG. 11, only the imaging cameras 1101 and 1105 are shown for simplicity.

12 and 13 are explanatory diagrams of surrounding teacher images or teacher images and focus maps, and illustrate images 1201 and 1301 acquired by imaging cameras 1101 and 1105, and focus maps 1202 and 1302, which are accompanying metadata. shows a diagram of A focus map is calculated by a known technique as in the first embodiment.

Since the imaging camera 1101 is focused on the background subject 304 with a wide-angle focal length, multiple subjects are photographed in a wide range as shown in the image 1201, and the background subject 304 is in focus as shown in the focus map 1202. 50% gray. On the other hand, since the camera 1105 is focused on the main subject 303 at the focal length of the standard angle of view, the main subject is mainly photographed as shown in the image 1301, and as shown in the focus map 1302. Main subject 303 shows in-focus 50% gray.

With the configuration as described above, when selecting a batch size of light rays (S204) in the 3D model learning of FIG. Rays 1111 corresponding to are not selected outside the depth of field. On the other hand, the light ray 1103 of the wide-angle imaging camera 1101 is selected as the target light ray because the three-dimensional point 606 on the background subject 304 is within the depth of field, and the background subject 304 as well as the main subject 303 is selected as a 3D model. can be included in

5, the light ray 1110 of the free viewpoint camera 1108 of FIG. 11 is captured within the depth of field by the light ray 1107 of the imaging camera 1105 adjacent to the corresponding three-dimensional point 607. is selected as the ray of interest. In addition, the light ray 1109 of the free-viewpoint camera 1108 in FIG. 11 is selected as the target light ray because it can be seen that the corresponding three-dimensional point 606 is captured within the depth of field by the light ray 1103 of the imaging camera 1101 adjacent thereto. be done. In this way, free-viewpoint images can be rendered not only for the main subject 303 but also for the background subject 304 .

In this embodiment, the imaging cameras 1101 and 1105 have different focal lengths and imaging distances so that both the main subject and the background subject can be rendered in volume. Only the distance may be varied. If the focal length of the imaging camera 1101 for the background subject is lengthened, the angle of view is narrowed, so there is a possibility that the number of imaging cameras to be arranged will increase. Also, if the focal length of the imaging camera 1105 for the main subject is shortened, the angle of view widens. On the other hand, if the focal length is adjusted, there are merits in shooting preparation and camera calibration, so it is suitable for simple shooting.

In this embodiment, both the main subject and the background subject can be rendered in volume by making both the focal length and the shooting distance of the imaging cameras 1101 and 1105 different. The aperture value may be made different. That is, the camera 1105 for the main subject is set to a bright aperture value close to open, while the camera 1101 for the background subject is stopped down to set the aperture value so that the entire shooting space range 302 is within the depth of field. do. As a result, the camera 1105 for the main subject can select a dark shutter speed to capture a sharp image even when the main subject is moving, but the camera 1101 for the background is weak against a moving subject and the sensitivity is low. raises the concern of increasing noise. On the other hand, in a case where a plurality of background subjects are scattered within the imaging space range 302, there is an advantage that a 3D model can be generated without missing any shots. In this way, the peripheral teacher image may be an image that differs from the teacher image in at least one of the focus position (shooting distance), focal length, and aperture value.

In this embodiment, the focus map is used to speed up the volume rendering when rendering the free viewpoint image, but the present invention is not limited to this. For example, the 3D model training may utilize a focus map to speed up, while the free-viewpoint image rendering may be spatially sampled over the imaging space range as in the prior art. This eliminates the need for a focus map during free-viewpoint image rendering. Therefore, even a conventional renderer can render a free-viewpoint image, and the versatility of the renderer can be enhanced.

In this embodiment, the main subject and the background subject are 3D modeled by the same neural network, but the present invention is not limited to this, and separate neural networks may be configured for the main subject and the background subject. That is, the learning parameter calculation means may learn different machine learning models for the teacher image and the surrounding teacher images. This can be expected to improve the learning efficiency and improve the reproducibility of the details of the background subject.

According to this embodiment, it is possible to select the optimum light rays for volume rendering for the main subject and the background subject, and generate a 3D model of not only the main subject but also the background subject at high speed, thereby speeding up the free-viewpoint image. can be rendered.

(Third Embodiment)
Next, an image processing apparatus according to a third embodiment of the invention will be described. FIG. 14 is a block diagram of a personal computer (image processing apparatus) 1400. As shown in FIG. A personal computer 1400 of this embodiment differs from the personal computer 100 having the neural network section 107 described in the first embodiment in that it has a neural network section 1401 . Since other configurations of the personal computer 1400 are the same as those of the personal computer 100, description thereof will be omitted.

The neural network unit 1401 can handle dynamic scenes (moving images) in which the subject moves. In such a dynamic scene, the color of the pixel corresponding to each ray is determined by sampling the points on the ray for each ray, computing the color and density of the corresponding points with a neural network, and performing volume rendering over the target space. . For learning and inference, the technology disclosed in Non-Patent Document 2, for example, can be used. During learning, the L2 loss between the color calculated by volume rendering and the color of the captured image is used as a loss function, and the learning weight is converged by the error backpropagation method. to render a free-viewpoint image.

FΘ: (x, d, zt) → (c, σ) (2)
In equation (2), FΘ is a neural network consisting of a multi-layer perceptron, and inputs the three-dimensional coordinate x of the point on the sampled ray, the ray direction vector d, and the latent code zt in the frame at time t. and FΘ outputs RGB color c and density σ for each sampling point.

The neural network unit 107 of the first embodiment expresses only a single 3D model. On the other hand, the neural network unit 1401 of this embodiment can express a scene in which the 3D shape dynamically changes for each frame by changing the latent code zt for each frame.

Next, dynamic 3D model learning by the control unit 101 will be described with reference to FIG. 15 . FIG. 15 is a flow chart of dynamic 3D model learning. First, in step S201, the photographing camera position/orientation estimation unit 105 estimates the camera position/orientation corresponding to each image used for learning. Since the position of the imaging camera is fixed, the camera position and orientation are estimated using the image acquired in the first frame.

Subsequently, in step S1501, the control unit 101 determines whether processing for all frames has been completed. When the processing for all frames is completed, this flow ends. On the other hand, if processing for all frames has not been completed, the process advances to step S1502. In step S1502, the control unit 101 generates a latent code corresponding to each frame.

Subsequently, in step S1503, the control unit 101 determines whether or not the iteration process has been completed. The learning of each frame converges to the target learning weight by updating the learning weight through iteration processing. If the iteration process is completed, the process returns to step S1501. On the other hand, if the iteration process has not been completed, the process advances to step S1504.

In step S1504, the control unit 101 randomly selects light beams corresponding to the batch size in each iteration. Non-Patent Document 2 discloses a technique for efficiently reducing computational costs by performing ray importance sampling that selects the next ray for learning based on temporal fluctuations of the input video. In this embodiment, the ray importance sampling is performed in consideration of the depth of field of the input video, thereby further improving computational efficiency. The operation will be described below with reference to FIGS. 16 to 19. FIG.

FIG. 16 is an explanatory diagram of shooting a teacher image, and shows a bird's-eye view for explaining the relationship among the subject, shooting space, and shooting camera in the beginning frame of the input video. Reference numeral 1601 denotes a camera, 302 a range of a 3D modeling target shooting space, 303 a main subject, and 304 a background subject. The camera 1601 is focused on the main subject 303, and the entire shooting space range 302 is in focus within a wide angle of view and a pan-focused depth of field indicated by hatching 1605. A plurality of imaging cameras 1601 are arranged to photograph an object within the imaging space range 302 from various directions, but FIG. 16 shows only the imaging cameras 1601 for the sake of simplicity.

In subsequent frames, the imaging camera 1601 changes the focal length to the standard angle of view and changes the aperture to a brighter aperture value (F number) closer to the open, so that the main subject 303 can be captured with higher definition. change the camera parameters to FIG. 17 is an explanatory diagram of photographing of the teacher image, and shows a bird's-eye view explaining the relationship among the subject, the photographing space, and the photographing camera after the camera parameters are changed. The camera 1601 is focused on the main subject 303 , and the photographic space range 302 is partially focused within the standard angle of view and shallow depth of field indicated by hatching 1705 .

FIG. 18 is an explanatory diagram of a teacher image and a focus map, showing an image 1801 captured by a camera 1601 in the beginning frame of an input video and a focus map 1802 as attached metadata. FIG. 18 shows the degree of focus on the imaging plane in the form of a gray scale map, where the front is white, the back is black, and 50% gray indicates focus. FIG. 19 is an explanatory diagram of a teacher image and a focus map, showing an image 1901 acquired by a photographing camera 1601 after camera parameters are changed and a focus map 1902 that is attached metadata.

Rays 1606 and 1607 in FIG. 16 in the first frame of the input video are volume rendering target rays, and the corresponding three-dimensional points are within the depth of field, so both rays are selected. On the other hand, the light rays 1706 and 1707 in FIG. 17 after the camera parameter change are also the light rays for volume rendering, but since there is no object within the depth of field range on the light ray 1706, they are excluded. It is possible to determine whether or not the light beam is an unnecessary ray by using the focus map 1902 in FIG. 19 . Pixel 1904 corresponding to ray 1707 is a 50% gray in-focus pixel, whereas pixel 1903 corresponding to ray 1706 is a dark gray outside the depth of field and is thus considered an excluded ray. judge.

After selecting rays for the batch size in step S1504 of FIG. 15, the ray calculator 106 calculates rays in step S1505. A ray for volume rendering is defined as r(t)=o+td, as described above. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. The range of distances t sampled for volume rendering is limited within the depth of field indicated by hatching 1605 in FIG. 16 and hatching 1705 in FIG. In this way, the ray importance sampling is performed in consideration of the depth of field, thereby speeding up the volume rendering process.

Also, in the present embodiment, even in the case of FIG. 16 where the depth of field is deep, the sampling range t is limited according to the depth of field, but the present invention is not limited to this. In such a case, since the effect of limiting the sampling range is low, a sufficiently wide fixed range may be set to fix the sampling range.

After the ray is calculated in step S1505 of FIG. 15, the neural network unit 1401 updates the learning weights in step S1506. The learning weight is determined by repeating the operations of steps S1503 to S1506 until the learning weight converges. When the processing of all frames of the input video is completed, this flow ends (step S1501).

In the present embodiment, ray importance sampling is based on depth of field, but is not limited to, ray importance sampling based on both time variation of input video and depth of field. may be performed. The input video is often higher resolution than the focus map. Therefore, by observing temporal fluctuations in the input video, it is possible to exclude unnecessary light rays with higher resolution and limit the sampling range based on the focus map, which is expected to further improve computational efficiency.

Next, referring to FIG. 20, free-viewpoint image rendering by the control unit 101 will be described. FIG. 20 is a flowchart of free-viewpoint image rendering in this embodiment. First, in step S501, the free camera position/orientation acquisition unit 108 acquires the camera position/orientation of the free viewpoint to be rendered.

Subsequently, in step S2001, the control unit 101 determines whether or not processing for all frames has been completed. When the processing for all frames is completed, this flow ends. On the other hand, if processing for all frames has not been completed, the process advances to step S2002 to calculate pixel values (RGB values) by volume rendering over all pixels of the rendered image while updating the latent code Zt of each frame. . That is, in step S2002, control section 101 generates a latent code corresponding to each frame. Subsequently, in step S2003, the control unit 101 determines whether calculation of pixel values (RGB values) by volume rendering has been completed for all pixels of the rendered image. When pixel value calculation is completed for all pixels, the process returns to step S2001. On the other hand, if calculation of pixel values has not been completed for all pixels, the process advances to step S2004.

In step S2004, the ray calculator 106 calculates a ray corresponding to each pixel (S2004). A ray for volume rendering is defined as r(t)=o+td, as described above. Here, o is the principal point of the camera in the world coordinate system, d is the direction vector of the ray expressed in the world coordinate system, and t is the distance from the camera principal point to the sampling point on the ray. After the rays are calculated in step S2004, the neural network unit 107 executes volume rendering processing in step S2005 to calculate corresponding pixel values (RGB values).

In this embodiment, the focal length and depth of field are changed at the beginning and after the input video, but the present invention is not limited to this. A plurality of wide-angle and pan-focus shots may be sandwiched between . As a result, when the length of the input video is long, even if the ambient environment such as sunlight changes gradually during the input video, it can be handled.

In this embodiment, the focal length and depth of field are changed at the beginning and the end of the input video, but the present invention is not limited to this, and only the aperture is controlled to change only the depth of field. good too. This makes it possible to shoot even with a single focal length lens that does not have a zoom mechanism. Alternatively, a plurality of wide-angle single-focus cameras and standard-angle single-focus cameras may be arranged. In the case of a zoom lens, even if the focal length is changed from a standard angle of view to a wide angle with the aperture open, the depth of field is increased, so if that is sufficient, only the focal length may be controlled.

In this embodiment, the focal length and depth of field are changed at the beginning of the input video and thereafter, but the invention is not limited to this, and the focus may be gradually changed from the background to the main subject. . As a result, not only the main subject but also other subjects can be photographed with high definition. Since the image magnification changes when the focus is changed, the 3D model can be learned with higher accuracy by further correcting the change in the image magnification.

As described above, in this embodiment, the teacher image is a moving image acquired at a predetermined frame rate. The learning parameter calculation means calculates a code feature amount (latent code) indicating the feature of each frame of the moving image, and performs machine learning by using the code feature amount and the teacher image for each frame of the moving image. Calculate parameters. In the present embodiment, the acquisition means acquires a temporally changing region (a region in which the appearance changes significantly over time) in the teacher image, and the ray calculation means focuses the rays in the region acquired by the acquisition means. to output the corresponding rays. Also, in this embodiment, the teacher image is a moving image acquired so as to include frames in which at least one of the focus position, focal length, or aperture value is different.

According to this embodiment, it is possible to generate a 3D model at high speed by efficiently performing volume rendering even in a dynamic scene in which an object moves.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

According to each embodiment, by appropriately limiting the light rays for volume rendering and the sampling range, it is possible to measure the precise shape of an object at high speed and reconstruct an image that looks like an actual photograph. Therefore, according to each embodiment, it is possible to provide an image processing apparatus, an image processing method, and a program capable of performing volume rendering at high speed.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

100 personal computer (image processing device)
105 Photographing camera position/orientation estimation unit (acquisition unit)
106 ray calculator (ray calculator)
107 neural network unit (learning parameter calculation means, rendering means)
108 Free camera position and orientation acquisition unit (acquisition means)

Claims (20)

a teacher image; an acquisition means for acquiring a position of a camera corresponding to the teacher image;
light ray calculation means for calculating a light ray corresponding to each pixel of the teacher image using the position of the camera;
learning parameter calculation means for performing machine learning by sampling points on the light beam and using the teacher image to calculate learning parameters;
The image processing apparatus according to claim 1, wherein the sampling density of the rays within the range of the depth of field of the teacher image is higher than the sampling density outside the range of the depth of field. Acquisition means for acquiring the position of the camera;
light ray calculation means for calculating a light ray corresponding to each pixel of the camera using the position of the camera;
rendering means for rendering an image of the camera by machine learning using learning parameters pre-learned by sampling points on the light rays by the learning parameter calculation means;
The image processing apparatus according to claim 1, wherein the sampling density of the rays within the depth of field range of the teacher image used in the pre-learning is higher than the sampling density outside the depth of field range. The acquisition means further acquires distance distribution information corresponding to the teacher image,
3. The light ray calculator according to claim 1, wherein the light ray calculator determines whether or not the light ray is within the depth of field of the teacher image based on the distance distribution information. Image processing device. Based on the distance distribution information, it is determined whether the object surface is in front of or behind the object focusing distance,
4. The image processing apparatus according to claim 3, wherein the sampling density is changed based on a determination result regarding whether the surface of the object is in front of or behind the focusing distance of the object. Based on the distance distribution information, the distance to the object surface is determined,
5. The image processing apparatus according to claim 3, wherein said sampling density is determined based on said distance to said object surface. 6. The image processing apparatus according to claim 3, wherein said distance distribution information has a resolution lower than that of said teacher image. 3. The light ray calculating means specifies a subject outline area based on the distance distribution information, makes it easier to select light rays for the subject outline area, and sets a wide sampling range. 9. The image processing apparatus according to any one of items 1 to 8. 8. The light ray calculating means selects all light rays for a region determined to have low reliability based on the distance distribution information, and sets a wide sampling range. The image processing device according to any one of the items. 9. The image processing apparatus according to any one of claims 3 to 8, wherein the acquisition means processes the teacher image and the distance distribution information based on a distortion component of an optical system of the camera. . 10. The distance distribution information according to any one of claims 3 to 9, wherein the distance distribution information includes at least one of a map based on a shift amount representing parallax, a map based on a defocus amount, or a map based on a distance. image processing device. The acquisition means further acquires a surrounding teacher image of the surrounding space,
11. The learning parameter calculation unit according to any one of claims 1 to 10, wherein the learning parameter calculation means performs the machine learning by using the teacher image and the surrounding teacher images to calculate the learning parameter. Image processing device. 12. The image processing apparatus according to claim 11, wherein the peripheral teacher image is an image different from the teacher image in at least one of focus position, focal length, and aperture value. 13. The image processing apparatus according to claim 11, wherein the learning parameter calculating means learns different machine learning models for the teacher image and the surrounding teacher images. The teacher image is a moving image acquired at a predetermined frame rate,
The learning parameter calculation means is
calculating a code feature amount representing a feature of each frame of the moving image;
14. The image processing according to any one of claims 1 to 13, wherein machine learning is performed by using the code feature amount and the teacher image for each frame of the moving image to calculate the learning parameter. Device. The acquiring means acquires a temporally changing region in the teacher image,
15. The image processing apparatus according to claim 14, wherein the light ray calculating means selectively selects the light ray in the area acquired by the acquiring means, and outputs the corresponding light ray. 16. The image processing apparatus according to claim 14, wherein the teacher image is a moving image acquired so as to include frames in which at least one of focus position, focal length, or aperture value is different. 17. The method according to any one of claims 1 to 16, wherein the permissible circle of confusion diameter for determining the depth of field is determined based on the resolution at the time of rendering using the learning parameters. image processing device. obtaining a teacher image and a camera position corresponding to the teacher image;
calculating a ray corresponding to each pixel of the teacher image using the position of the camera;
performing machine learning by sampling points on the light beam and using the teacher image to calculate a learning parameter;
The image processing method, wherein the sampling density of the rays within the range of the depth of field of the teacher image is higher than the sampling density outside the range of the depth of field. obtaining a camera position;
calculating a ray corresponding to each pixel of the camera using the position of the camera;
rendering an image of the camera by machine learning using learning parameters pre-learned by sampling points in the light rays by a learning parameter calculation means;
The image processing method, wherein the sampling density of the rays within the depth of field range of the teacher image used in the pre-learning is higher than the sampling density outside the depth of field range. A program for causing a computer to execute the image processing method according to claim 18 or 19.

Images