JP2018194979A - Three-dimensional information restoration method, restoration program and restoration apparatus - Google Patents

Abstract

To reduce a processing load or memory usage associated with three-dimensional information restoration.SOLUTION: A three-dimensional information restoration method specifies whether an object is present on a straight line in a three-dimensional space connecting respective pixels of an optical center and a first image that is picked up by a first camera and that is selected from among a plurality of images picked up by a plurality of cameras at different photographing positions, defines a region including pixels of the specified first image as a first range where an object search is performed and a second range other than the first range, sets a first interval as an interval where the search is performed for the pixels included in the first range and a second interval longer than the first interval for the pixels included in the second range, matches blocks between a second image among the plurality of images that is not selected as the first image and the first image in accordance with the set ranges and intervals, and estimates a position on the three-dimensional space for each pixel of the first image on the basis of correlation between the matched blocks.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a three-dimensional information restoration method, a restoration program, and a restoration device.

  A technique called free viewpoint video is known. For example, three-dimensional information is restored from multi-viewpoint images captured by a plurality of cameras with different viewpoints. By using such three-dimensional information, it is possible to generate a virtual viewpoint image in which a three-dimensional object is observed from a virtual viewpoint where no camera actually exists.

JP 2012-69090 A JP 2006-215939 A JP 2000-215311 A JP 2002-197443 A

  However, in order to restore the above three-dimensional information, the processing load and memory usage may increase. For example, when restoring three-dimensional information, one of the multi-viewpoint images is used as a reference image while the other is used as a reference image, and the optical center of the camera and the reference image assigned to the shooting of the reference image for each pixel of the reference image The blocks on the standard image set on the straight line in the three-dimensional space connecting the pixels are projected onto the reference image, whereby the blocks are matched between the standard image and the reference image. However, as the interval for matching blocks between the standard image and the reference image is set more finely, the processing load and the memory usage increase, which hinders the restoration of three-dimensional information with high accuracy.

  In one aspect, an object of the present invention is to provide a three-dimensional information restoration method, a restoration program, and a restoration device that can reduce a processing load or a memory usage accompanying restoration of three-dimensional information.

  In one aspect of the three-dimensional information restoration method, a plurality of images captured by a plurality of cameras having different shooting positions are acquired, and a first image captured by a first camera selected from the plurality of images is acquired. For an image, it is specified whether an object exists on a straight line in a three-dimensional space connecting the optical center of the first camera and each pixel of the first image, and an area including the pixel of the specified first image is determined. , Set as a first range in which the search for the object is executed on a straight line in the three-dimensional space, set a range other than the first range in the first image as a second range, For the pixels included in the range, a first interval is set as an interval for executing the search for the object on the straight line in the three-dimensional space, and for the pixels included in the second range, Longer than the first interval 2 is set, and according to the set range and the set interval, a block is matched between the second image that is not selected as the first image and the first image among the plurality of images. The computer executes a process of estimating the position in the three-dimensional space for each pixel of the first image based on the correlation between the matched blocks.

  It is possible to reduce the processing load or memory usage accompanying the restoration of the three-dimensional information.

FIG. 1 is a diagram illustrating a configuration example of a three-dimensional information restoration system according to the first embodiment. FIG. 2 is a block diagram illustrating a functional configuration of the server apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of separation of the foreground and the background. FIG. 4 is a diagram illustrating an example of Visual Hull. FIG. 5 is a diagram illustrating an example of a Visual Hull calculation process. FIG. 6 is a diagram illustrating an example of a Visual Hull calculation process. FIG. 7 is a diagram illustrating an example of setting an object search range. FIG. 8 is a diagram illustrating an example of a search interval. FIG. 9 is a diagram illustrating an example of the relationship between the search interval and the depth. FIG. 10 is a diagram illustrating an example of an arrangement condition between cameras. FIG. 11 is a diagram illustrating an example of block matching. FIG. 12 is a diagram illustrating an example of a depth image. FIG. 13 is a flowchart illustrating the procedure of the three-dimensional information restoration process according to the first embodiment. FIG. 14 is a diagram illustrating an example of a ratio of a region to which each pixel of the reference image belongs. FIG. 15 is a diagram illustrating an example of the ratio of the average value of the search range of each pixel of the reference image. FIG. 16 is a diagram illustrating a hardware configuration example of a computer that executes the restoration program according to the first embodiment and the second embodiment.

  Hereinafter, a three-dimensional information restoration method, restoration program, and restoration apparatus according to the present application will be described with reference to the accompanying drawings. Note that this embodiment does not limit the disclosed technology. Each embodiment can be appropriately combined within a range in which processing contents are not contradictory.

[System configuration]
FIG. 1 is a diagram illustrating a configuration example of a three-dimensional information restoration system according to the first embodiment. The 3D information restoration system 1 shown in FIG. 1 provides, as one aspect, a restoration service that restores 3D information from multi-viewpoint images captured by a plurality of cameras 30A to 30M having different viewpoints.

  As illustrated in FIG. 1, the three-dimensional information restoration system 1 includes a server device 10 and a plurality of cameras 30 </ b> A to 30 </ b> M. Hereinafter, the cameras 30A to 30M may be referred to as “camera 30”.

  The server device 10 and the camera 30 are connected via a predetermined network NW. This network NW can be constructed by any type of communication network, such as the Internet, a LAN (Local Area Network), or a VPN (Virtual Private Network), whether wired or wireless.

  As described above, the three-dimensional information restoration system 1 illustrated in FIG. 1 illustrates a case where a multi-viewpoint image is transmitted from the camera 30 to the server apparatus 10 via the network NW, but this is merely an example of a transmission form. There is no need to perform bidirectional communication between the server device 10 and the camera 30. For example, a multi-viewpoint image may be transmitted from the camera 30 to the server device 10 via a broadcast wave without going through the network NW.

  The server device 10 is a computer that provides the restoration service. The server device 10 is an example of a restoration device.

  As an embodiment, the server apparatus 10 can be implemented by installing a restoration program that realizes the function of the restoration service as package software or online software on a desired computer. For example, the server device 10 may be implemented as a Web server that provides the above restoration service, or may be implemented as a cloud that provides the above restoration service by outsourcing.

  The camera 30 is an image pickup apparatus on which an image pickup device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) is mounted.

  FIG. 1 shows the arrangement of the cameras 30 in a soccer game as an example. For example, a three-dimensional object included in the three-dimensional space on the soccer field F, for example, a game-related person such as a ball, a player, or a referee, is restored. In this case, the plurality of cameras 30 are arranged from the periphery of the field F toward the inside of the field F. At this time, by combining the shooting ranges of the plurality of cameras 30, each camera 30 is installed in an arrangement in which the entire field F is within the shooting ranges of the plurality of cameras 30. Are arranged in a state where a part of the shooting range overlaps between the two. Under such an arrangement, the plurality of cameras 30 shoots in synchronization with each frame, so that a plurality of images shot at the same timing for each different viewpoint can be obtained in units of frames. Hereinafter, a plurality of images captured in the same frame by the cameras 30 </ b> A to 30 </ b> M having different shooting positions are referred to as “multi-viewpoint images”, and a series of images captured in time series by the single camera 30. May be described as “moving image”.

[Configuration of Server Device 10]
Next, a functional configuration of the server device 10 according to the present embodiment will be described. FIG. 2 is a block diagram illustrating a functional configuration of the server apparatus 10 according to the first embodiment. As illustrated in FIG. 2, the server device 10 includes a communication I / F (InterFace) unit 11, a storage unit 13, and a control unit 15. Note that FIG. 2 merely shows the functional units of the server device 10 related to the restoration service, and the functional units other than those illustrated, for example, functions that an existing computer has as a default or an option are provided. Is not prevented from being provided in the server device 10. For example, when the multi-viewpoint image is propagated from the camera 30 to the server device 10 via a broadcast wave or a satellite wave, it may further include a broadcast wave or satellite wave receiver.

  The communication I / F unit 11 is an interface that controls communication with other devices.

  As an embodiment, the communication I / F unit 11 corresponds to a network interface card such as a LAN card. For example, the communication I / F unit 11 receives a multi-viewpoint image from the camera 30, and transmits an instruction related to imaging control, for example, an instruction such as panning and tilting in addition to power ON / OFF, to the camera 30. To do.

  The storage unit 13 is a storage device that stores data used for various programs such as the restoration program described above, including an OS (Operating System) executed by the control unit 15.

  As an embodiment, the storage unit 13 is implemented as an auxiliary storage device in the server device 10. For example, the auxiliary storage device corresponds to an HDD (Hard Disk Drive), an optical disk, an SSD (Solid State Drive), or the like. In addition, a flash memory such as an EPROM (Erasable Programmable Read Only Memory) also corresponds to the auxiliary storage device.

  The storage unit 13 includes, as an example of data used in a program executed by the control unit 15, parameters 13 a including external parameters such as the position and orientation of the camera 30 and internal parameters such as the angle of view of the camera 30 and lens distortion. Remember. In addition to the parameter 13a, other electronic data can be stored. For example, the storage unit 13 can store time-series data of multi-viewpoint images transmitted from the camera 30.

  The control unit 15 is a processing unit that performs overall control of the server device 10.

  As one embodiment, the control unit 15 can be implemented by a hardware processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Here, the CPU and the MPU are illustrated as an example of the processor. However, the processor and the MPU can be mounted by any processor regardless of a general-purpose type or a specialized type. In addition, the control unit 15 may be realized by a hard wired logic such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

  The control unit 15 expands the restoration program described above on a RAM work area such as a DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) mounted as a main storage device (not shown), thereby The processing unit is virtually realized.

  As shown in FIG. 2, the control unit 15 includes an acquisition unit 15a, a separation unit 15b, a first setting unit 15c, a second setting unit 15d, a matching unit 15e, and an estimation unit 15f.

  The acquisition unit 15a is a processing unit that acquires a multi-viewpoint image.

  As one embodiment, the acquisition unit 15a can acquire a multi-viewpoint image transmitted from the camera 30 in units of frames. Here, the source from which the acquisition unit 15a acquires the multi-viewpoint image may be arbitrary, and is not limited to the camera 30. For example, the acquisition unit 15a can also acquire a multi-viewpoint image by reading from an auxiliary storage device such as a hard disk or an optical disk that accumulates the multi-viewpoint image or a removable medium such as a memory card or a USB (Universal Serial Bus) memory. In addition, the acquisition unit 15a can also acquire a multi-viewpoint image by receiving from an external device via the network NW.

  The separation unit 15b is a processing unit that separates the foreground and the background. The “foreground” here refers to an object such as a moving object among objects existing in a three-dimensional space within the shooting range of the camera 30, while the “background” refers to an object that is not a moving object.

  As an embodiment, every time a frame of a moving image is acquired for each camera 30 by the acquisition unit 15a, the separation unit 15b extracts a region corresponding to the foreground from the image of the frame for each camera 30. For example, the separation unit 15b extracts a region corresponding to the foreground by detecting a difference in pixel value between the image of the frame and the image of the frame acquired before that. FIG. 3 is a diagram illustrating an example of separation of the foreground and the background. In FIG. 3, a series of images included in a moving image captured by a camera 30 among the cameras 30 </ b> A to 30 </ b> M are shown in a state of being arranged in time series. For example, the separation unit 15b detects a pixel having a pixel value difference equal to or greater than a predetermined threshold value between the image F0 of the frame in which there is a high possibility that no moving object is observed in the image and the image FL of the latest frame. Here, as the image F0, for example, an image of a frame in which a difference between frames is not detected over a predetermined number of frames can be used. The pixels thus extracted can be extracted as blobs by performing labeling. In addition, the separation unit 15b sets the pixel value of the pixel corresponding to the blob extracted by labeling to “1”, and sets the pixel values of the other pixels to “0”. Accordingly, it is possible to generate the mask image ML in which the area corresponding to the foreground is expressed by “white” and the area corresponding to the background is expressed by “black”. FIG. 3 illustrates a case where a binarized image is generated as an example of a mask image, but it goes without saying that the number of gradations is not limited to “2”.

  The first setting unit 15c connects the optical center of the first camera and each pixel of the first image for the first image captured by the first camera selected from the plurality of images. It is a processing unit that identifies whether an object exists on a straight line in the dimensional space. Furthermore, the first setting unit 15c sets a region including the pixel of the identified first image as a first range in which an object search is performed on a straight line in the three-dimensional space, and includes the first image in the first image. This is a processing unit for setting a region other than the first range to the second range.

  The first setting unit 15c includes, for each pixel of the reference image in the multi-viewpoint image, the object center on the straight line in the three-dimensional space that connects the optical center of the camera assigned to the reference image and the pixel of the reference image. Sets the range in which the search is performed. Hereinafter, a range in which an object search is performed on a straight line in a three-dimensional space may be referred to as a “search range”. The first setting unit 15c is also an example of a specifying unit.

  Here, the standard image refers to an image selected from the multi-viewpoint images, and other images other than the standard image are identified as reference images. The sharing of the reference image is carried around. Eventually, all images of the multi-viewpoint image are selected as the reference image, and block matching and depth map generation are performed. Hereinafter, the camera 30 that captures the reference image may be referred to as “reference camera 30α”, and the camera 30 that captures the reference image may be referred to as “reference camera 30β”. The reference camera is an example of a first camera, and the reference image is an example of a first image.

  As an embodiment, the first setting unit 15 c is formed by referring to the camera parameter 13 a stored in the storage unit 13 and using the foreground silhouette included in the mask image of each camera 30 and the optical center of each camera 30. The product set space of the cones to be formed, that is, “Visual Hull” is obtained. After that, the first setting unit 15c narrows the search range of the object to Visual Hull to the existence range of the object existing in the three-dimensional space. FIG. 4 is a diagram illustrating an example of Visual Hull. FIG. 4 shows a case where the mask images 31A to 31C of the three cameras 30 of the cameras 30A to 30C are used for the calculation of Visual Hull. As shown in FIG. 4, a straight line group connecting the optical center of the camera 30A and the foreground silhouette SA on the mask image 31A, a straight line group connecting the optical center of the camera 30B and the foreground silhouette SB on the mask image 31B, and the camera A region E in the three-dimensional space that is included by the group of straight lines connecting the optical center of 30C and the foreground silhouette SC on the mask image 31C is obtained as Visual Hull. By calculating such Visual Hull, the existence range of the object in the three-dimensional space that is the subject of each camera 30 can be narrowed down to the region E.

  For example, an example of an algorithm for calculating Visual Hull will be described. FIG. 5 is a diagram illustrating an example of a Visual Hull calculation process. As illustrated in FIG. 5, the first setting unit 15c selects one of the cameras 30A to 30M as the reference camera 30α. On the other hand, the camera 30 that has not been selected as the reference camera 30α is hereinafter identified as the reference camera 30β. Subsequently, the first setting unit 15c selects a pixel (u1, v1) that forms a foreground silhouette among the pixels of the mask image 31α of the reference camera 30α. Thereafter, the first setting unit 15c selects one of the reference cameras 30β. The reference camera 30β selected in this way is hereinafter identified as “reference camera 30β1”.

  The first setting unit 15c then projects a straight line passing through the optical center O of the reference camera 30α and the pixel (u1, v1) onto the mask image 31β1 of the reference camera 30β1 in a three-dimensional space, thereby causing the epipolar line EL Draw. Then, the first setting unit 15c determines a range where the epipolar line EL exists on the mask image 31β1 of the reference camera 30β1. For example, the start position of the range corresponds to the point at which the optical center O of the base camera 30α is projected onto the mask image 31β1 of the reference camera 30β1. The end position of the range converges on the mask image 31β1 of the reference camera 30β1 when a straight line extending from the optical center O of the reference camera 30α to the pixel (u1, v1) is extended to infinity in the three-dimensional space. Corresponding to

  Thereafter, the first setting unit 15c calculates a range where the epipolar line EL overlaps the foreground silhouette Sβ on the mask image 31β1 of the reference camera 30β1, that is, the intersection start position and the intersection end position. Then, the first setting unit 15c refers to the camera parameter 13a stored in the storage unit 13, and sets the depth information (from the optical center O of the reference camera 30α to the range where the epipolar line EL and the foreground silhouette Sβ overlap ( z).

  Thus, the derivation of the overlapping range of the epipolar line EL and the foreground silhouette Sβ is repeatedly performed until all the reference cameras 30β are selected. As a result, as shown in FIG. 6, an overlapping range between the epipolar line EL and the foreground silhouette Sβ is obtained for each reference camera 30β other than the base camera 30α. FIG. 6 is a diagram illustrating an example of a Visual Hull calculation process. In FIG. 6, the overlapping range of the epipolar line EL and the foreground silhouette Sβ for each of the four reference cameras 30β is indicated by depth information (z) from the optical center O of the reference camera 30α. That is, in FIG. 6, the overlapping range of the four reference cameras 30β is indicated by a dotted line, a one-dot chain line, a broken line, and a two-dot chain line. For example, the first setting unit 15c uses a portion where the overlapping range is common among the reference cameras 30β, that is, a portion indicated by a thick line in FIG. 6 in the pixel (u1, v1) on the mask image of the reference camera 30α. Filter as the existence range of objects. Here, it is not necessarily required that the overlapping range is common among all the reference cameras 30β. For example, it is possible to narrow down a portion having a common overlapping range among a predetermined number of reference cameras 30β as an object existing range. Thus, the pixels (u1, v1) that have successfully narrowed down the existence range of the object are classified into the Visual Hull region.

  FIG. 6 illustrates the case where there is a portion having a common overlapping range between the reference cameras 30β. However, when there is no portion having a common overlapping range between the reference cameras 30β, the existence range of the object is illustrated. Fails to narrow down. Thus, even if the pixels are included in the foreground silhouette, the pixels that have failed to narrow down the existence range of the object are classified as non-Visual Hull regions. Further, when the pixel corresponding to the background, that is, not included in the silhouette, the Visual Hull is not calculated, and is automatically classified into the non-Visual Hull region.

  After the classification into the Visual Hull area or the non-Visual Hull area is completed, the first setting unit 15c sets an object search range for the pixel. FIG. 7 is a diagram illustrating an example of setting an object search range. FIG. 7 shows a search range of each pixel existing on the horizontal line H <b> 1 of the image 40. The search range set in the Visual Hull area is indicated by black filling, and the search range set in the non-Visual Hull area is indicated by dot filling.

  As illustrated in FIG. 7, the first setting unit 15 c sets the existence range of the foreground object in the search range of the pixels classified into the Visual Hull area. Further, the first setting unit 15c sets a wide search range for searching both the foreground and the background in the search range of the pixels classified into the non-Visual Hull region. As an example of this wide search range, zNear and zFar are set. For example, when shooting a soccer stadium with the camera 30, “60 m” is set in zFar as an example with reference to the position farthest from the camera 30. In addition, as an example, “1 m” is set in zNear under the assumption that the camera is away from the camera 30 by a certain distance or more. These zNear and zFar are sufficiently spaced so that a foreground can exist between them.

  The second setting unit 15d sets the first interval as an interval for searching for an object on a straight line in the three-dimensional space for the pixels included in the first range, and is included in the second range. For a pixel to be processed, the processing unit sets a second interval longer than the first interval.

  That is, the second setting unit 15d sets an interval at which an object search is performed on a straight line in a three-dimensional space connecting the optical center of the reference camera 30α and the pixels of the reference image. Hereinafter, an interval at which an object search is performed on a straight line in a three-dimensional space may be referred to as a “search interval”.

  As an embodiment, the second setting unit 15d sets a different search interval N for each pixel of the reference image depending on whether the pixel is a Visual Hull region or a non-Visual Hull region. For example, FIG. 8 is a diagram illustrating an example of a search interval. FIG. 8 shows a straight line in a three-dimensional space from the optical center of the reference camera 30α to the infinity via pixels of the reference image. As shown in FIG. 8, a search interval finer than the pixels of the reference image classified into the non-Visual Hull region is set for the pixels of the reference image classified into the Visual Hull region. On the other hand, a coarser (longer) search interval is set for the pixels of the reference image classified into the non-Visual Hull region than the pixels of the reference image classified into the Visual Hull region. In other words, the second setting unit 15d sets the search interval for increasing the density at which the search for the object is performed on the straight line in the three-dimensional space for the pixels of the reference image classified into the Visual Hull region. For the pixels of the reference image classified into the non-Visual Hull region, a search interval is set to reduce the density at which the search for the object is executed on a straight line in the three-dimensional space. In the example shown in FIG. 8, the search interval of the Visual Hull region is set to a density that is about four times the search interval of the non-Visual Hull region. Furthermore, the scale of the epipolar line of the reference image in the reference image becomes coarser as the distance from the optical center of the reference camera 30α increases. Therefore, the search interval is set closer to the optical center of the reference camera 30α so that the search interval is substantially equidistant with the epipolar line, while the search interval is decreased as the distance from the optical center of the reference camera 30α is increased. Set to

  Here, the search interval is related to the number of gradations N + 1 for discretizing the depth z of the straight line in the three-dimensional space to the depth d. Here, as an example, the following description will be given by exemplifying a case where the number of scales of the search interval and the number of gradations of the depth are the same. FIG. 9 is a diagram illustrating an example of the relationship between the search interval and the depth. As shown in FIG. 9, the search interval is set to be denser as it is closer to the optical center of the reference camera 30α, while the search interval is set to be sparser as it is farther from the optical center of the reference camera 30α. On the other hand, each gradation of depth is set at equal intervals on a straight line in a three-dimensional space. For example, the depth z of the straight line in the three-dimensional space can be converted into the depth d according to the following equation (1).

  Under such a relationship between search interval and depth, for example, N can be determined according to the following criteria. For example, when the depth d is shifted by 1, N can be determined depending on how many pixels the shift of the pixel position on the epipolar line of the reference image falls within.

  Therefore, in the following, as an example, the displacement of the pixel position is set so that the displacement in the horizontal and vertical directions is larger, within a “1” pixel in the non-Visual Hull region, and within “0.25” pixels in the Visual Hull region. A description will be given under the design of setting each N. The reason why the non-Visual Hull region is determined to be within 1 pixel in this way is to keep the depth accuracy to a minimum if the search interval is about 1 pixel on the reference image. On the other hand, in the Visual Hull area as the foreground, in order to increase the depth accuracy, the search interval is set to ¼ to make it finer.

  At this time, the camera conditions are as follows. That is, it is assumed that the resolution of the camera 30 is 1920 × 1080 pixels and the horizontal angle of view is 66 degrees. Furthermore, the conditions between the cameras are as follows. FIG. 10 is a diagram illustrating an example of an arrangement condition between cameras. As shown in FIG. 10, it is assumed that the distance between the base camera 30α and the reference camera 30β is 3.71 m. Furthermore, the intersection C between the optical axis of the standard camera 30α and the optical axis of the reference camera 30β is 11.33 m from the optical center of the standard camera 30α, and 11.93 m from the optical center of the reference camera 30β.

  Further, when zNear is 3.0 m and zFar is 24.0 m, the search interval N between the Visual Hull region and the non-Visual Hull region is as follows. That is, the search interval N of the Visual Hull region is 4320. The search interval of the non-Visual Hull region is 1080. By setting these search intervals, it is possible to suppress the displacement of the pixel position within “1” pixels in the non-Visual Hull region and within “0.25” pixels in the Visual Hull region.

  The matching unit 15e is a processing unit that matches blocks between the standard image and the reference image.

  As one embodiment, the matching unit 15e generates, for each pixel of the reference image, the reference camera 30α for each search interval set by the second setting unit 15d from the start position of the search range set by the first setting unit 15c. A block of the reference image is arranged on a straight line in a three-dimensional space connecting the optical center of the pixel and the pixel of the reference image. Then, the matching unit 15e projects the block of the standard image onto the reference image for each reference image every time the block of the standard image is arranged. Thereby, a block can be arrange | positioned in the position where the block of a base image is observed on a reference image. For example, when block matching is performed according to the search range shown in FIG. 7 and the search interval shown in FIG. 8, the block matching shown in FIG. 11 can be realized. FIG. 11 is a diagram illustrating an example of block matching. As shown in FIG. 11, when the pixel of the standard image 40α is a non-Visual Hull region, the block of the standard image 40α is matched on an epipolar line indicated by a one-dot chain line of the reference image 40β. On the other hand, when the pixel of the standard image 40α is the Visual Hull region, the block of the standard image 40α is matched on the epipolar line indicated by the two-dot chain line of the reference image 40β. From these comparisons, in the Visual Hull area, it is possible to reduce processing load and memory usage by narrowing block matching to the object existence range. Furthermore, since the block matching is executed more densely in the presence range of the object than in the non-Visual Hull region, it can be expected that the three-dimensional information can be restored with high accuracy.

  A correlation, that is, a so-called similarity is calculated between the block of the reference image and the block of the reference image thus matched. For example, SAD (Sum of Absolute Difference), so-called sum of absolute differences, can be calculated between the block of the standard image and the block of the reference image. In the case of this SAD, the lower the SAD value, the higher the correlation between the block of the base image and the block of the reference image. Therefore, hereinafter, the value of the SAD may be described as “cost”. In each pixel of the standard image, the minimum value among the costs calculated for each reference image can be set as the representative value of the cost of the pixel, or the average value of the average value of the cost calculated for each reference image. It can also be. In this way, the cost cost (x, y, d) for each depth d can be calculated for each pixel of the reference image. In addition, although SAD was illustrated here as an example of a correlation, another correlation can also be calculated. For example, the correlation coefficient can be calculated between the block of the standard image and the block of the reference image. In this case, the larger the correlation coefficient, the higher the correlation between the block of the base image and the block of the reference image.

  The estimation unit 15f is a processing unit that estimates the position of the pixel in the three-dimensional space for each pixel of the reference image.

  As an embodiment, the estimation unit 15f generates the depth image d * (x, y) based on the cost cost (x, y, d) for each depth d calculated by the matching unit 15e for each pixel of the reference image. To do. FIG. 12 is a diagram illustrating an example of a depth image. As illustrated in FIG. 12, the estimation unit 15f sets, as an energy minimization problem, a problem of selecting an optimal depth for each pixel from the cost cost (x, y, d) for each pixel of the reference image 40α. . Then, the estimation unit 15f generates the depth image 50 according to an algorithm that solves the energy minimization problem, for example, an algorithm that minimizes energy from both the local solution and the consistency between adjacent pixels.

[Process flow]
FIG. 13 is a flowchart illustrating the procedure of the three-dimensional information restoration process according to the first embodiment. This process can be executed in real time every time a frame of a multi-viewpoint image is acquired, or can be executed in a batch process when the frame of the multi-viewpoint image has already been acquired as a moving image.

  As illustrated in FIG. 14, when the frame of the multi-viewpoint image is acquired by the acquisition unit 15a (Step S101), the separation unit 15b acquires the image of the frame and the previous image for each camera 30. An area corresponding to the foreground is extracted by detecting a difference in pixel value between the frame image and the foreground and the background are separated (step S102). Thus, a mask image is generated for each camera 30 by separating the foreground and the background.

  Then, the first setting unit 15c selects one of the multi-viewpoint images as a reference image (Step S103). As a result, the other images are identified as reference images. Subsequently, the first setting unit 15c selects one of the pixels included in the reference image selected in step S103 (step S104).

  In addition, the first setting unit 15c uses the mask image generated for each camera 30 in step S102, and the epipolar line EL and the foreground silhouette of the pixel of the reference image selected in step S104 for each reference image. An overlapping range with Sβ is obtained, and a portion where the overlapping range is common is narrowed down as an object existence range (step S105).

  At this time, if the object range is successfully narrowed down (Yes in step S106), the pixels of the reference image selected in step S104 are classified into the Visual Hull area. In this case, the first setting unit 15c sets the presence range of the foreground object in the search range of the reference image pixel selected in step S104 (step S107). Then, the second setting unit 15d sets a first interval shorter than a later-described second interval as a search interval for the pixel of the reference image selected in step S104 (step S108).

  On the other hand, if the object existence range has not been narrowed down (No in step S106), the pixel of the reference image selected in step S104 is classified into a non-Visual Hull region. In this case, the first setting unit 15c sets a wide search range, that is, a start position zNear and an end position zFar as the search range of the pixel of the reference image selected in step S104 (step S109). Then, the second setting unit 15d sets a second interval longer than the first interval as a search interval for the pixel of the reference image selected in step S104 (step S110).

  Thereafter, the matching unit 15e matches a block between the reference image and the reference image for each reference image according to the search range and the search interval, and calculates a correlation between the matched blocks (step S111).

  Then, the processes from step S104 to step S111 are repeated until all the pixels of the reference image are selected (step S112 No). Thereafter, when all the pixels of the reference image are selected (Yes in step S112), the estimation unit 15f calculates the cost cost (x, y, d) for each depth d calculated for each pixel of the reference image by the matching unit 15e. Based on this, a depth image d * (x, y) is generated (step S113).

  Then, the processes from step S103 to step S113 are repeatedly executed until all the images of the multi-viewpoint images are selected as reference images (No in step S114). Thereafter, when all the images of the multi-viewpoint images are selected as the reference images (Yes at Step S114), the process is terminated.

[One aspect of effect]
As described above, the server device 10 according to the present embodiment searches for an object on a straight line in a three-dimensional space connecting the optical center of the reference camera and the pixels of the reference image for each pixel of the reference image in the multi-viewpoint image. The range is narrowed down with Visual Hull, and the block matching interval is set finely. Therefore, according to the server device 10 according to the present embodiment, it is possible to reduce the processing load or the memory usage accompanying the restoration of the three-dimensional information.

[Comparison of increase in memory usage]
FIG. 14 is a diagram illustrating an example of a ratio of a region to which each pixel of the reference image belongs. FIG. 15 is a diagram illustrating an example of the ratio of the average value of the search range of each pixel of the reference image. As shown in FIG. 14, it is assumed that the ratio of the Visual Hull area in the reference image is 7% while the ratio of the non-Visual Hull area is 93%. Further, as shown in FIG. 15, “(average of search depth range of pixels with Visual Hull region) / (search depth range of pixels with non-Visual Hull region)”, that is, the dot filling shown in FIG. Assume that the average percentage of black fill is 1.3%. This means that the Visual Hull area only needs to have 1.3% of memory per pixel as compared to the non-Visual Hull area.

  At this time, the ratio of the memory used for calculating the cost in each area is as follows. That is, the non-Visual Hull region becomes 93 by calculation of 0.93 × 100. The Visual Hull area is 0.091 by the calculation of 0.07 × 1.3. Overall, it is 93.091 by the calculation of 93 + 0.091.

  Further, when the search interval of only the Visual Hull area is set to ¼, only the memory of the Visual Hull area is quadrupled and becomes 0.364 by the calculation of 0.091 × 4. In total, 93.364 is obtained by the calculation of 93 + 0.364. Therefore, the memory increase degree is about 0.29% by the calculation of 93.364 / 93.091.

  Although the embodiments related to the disclosed apparatus have been described above, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

[Stand-alone]
In the first embodiment, the implementation example is shown by the client server system by exemplifying the server device 10 that provides the restoration service. However, the server apparatus 10 may be implemented in a stand-alone manner. In this case, the restoration program may be installed on a computer that operates stand-alone.

[Change search interval]
In the first embodiment, the example in which the search interval is set depending on whether or not the pixel of the reference image is the Visual Hull region has been described. However, a different search interval may be set based on a further reference. For example, when the pixel of the reference image is a Visual Hull region, different search intervals can be set according to the importance of the pixel. As an example, the importance can be set depending on whether or not the pixels of the reference image correspond to an important area detected from the reference image, for example, a face important for display or a back number important for tracking. For example, when the pixel of the reference image corresponds to the face or the spine number, higher importance is given than when the pixel does not correspond to the face or the spine number. In addition, when the pixel of the reference image is the Visual Hull region, the second setting unit 15d can set a finer search interval as the importance of the pixel of the reference image is higher.

[Distribution and integration]
In addition, each component of each illustrated apparatus does not necessarily have to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. For example, the acquisition unit 15a, the separation unit 15b, the first setting unit 15c, the second setting unit 15d, the matching unit 15e, or the estimation unit 15f may be connected as an external device of the server device 10 via a network. In addition, another device has an acquisition unit 15a, a separation unit 15b, a first setting unit 15c, a second setting unit 15d, a matching unit 15e, or an estimation unit 15f, and is connected to a network to cooperate. You may make it implement | achieve the function of said server apparatus 10. FIG.

[Restore program]
The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. In the following, an example of a computer that executes a restoration program having the same function as that of the above embodiment will be described with reference to FIG.

  FIG. 16 is a diagram illustrating a hardware configuration example of a computer that executes the restoration program according to the first embodiment and the second embodiment. As illustrated in FIG. 16, the computer 100 includes an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Further, the computer 100 includes a CPU 150, a ROM 160, an HDD 170, and a RAM 180. These units 110 to 180 are connected via a bus 140.

  As shown in FIG. 16, the HDD 170 is similar to the acquisition unit 15a, the separation unit 15b, the first setting unit 15c, the second setting unit 15d, the matching unit 15e, and the estimation unit 15f described in the first embodiment. A restoration program 170a that exhibits the above function is stored. This restoration program 170a is integrated or separated in the same manner as the components of the acquisition unit 15a, separation unit 15b, first setting unit 15c, second setting unit 15d, matching unit 15e, and estimation unit 15f shown in FIG. It doesn't matter. That is, the HDD 170 does not necessarily have to store all the data shown in the first embodiment, and data used for processing may be stored in the HDD 170.

  Under such an environment, the CPU 150 reads the restoration program 170 a from the HDD 170 and expands it in the RAM 180. As a result, the restoration program 170a functions as a restoration process 180a as shown in FIG. The restoration process 180a develops various data read from the HDD 170 in an area allocated to the restoration process 180a in the storage area of the RAM 180, and executes various processes using the decompressed various data. For example, the process shown in FIG. 13 is included as an example of the process executed by the restoration process 180a. Note that the CPU 150 does not necessarily operate all the processing units described in the first embodiment, and the processing unit corresponding to the process to be executed may be virtually realized.

  Note that the restoration program 170a is not necessarily stored in the HDD 170 or the ROM 160 from the beginning. For example, the restoration program 170 a is stored in a “portable physical medium” such as a flexible disk inserted into the computer 100, so-called FD, CD-ROM, DVD disk, magneto-optical disk, IC card or the like. Then, the computer 100 may acquire and execute the restoration program 170a from these portable physical media. In addition, the restoration program 170a is stored in another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, etc., and the computer 100 acquires and executes the restoration program 170a therefrom. You may make it do.

DESCRIPTION OF SYMBOLS 1 Restoration system 10 Server apparatus 11 Communication I / F part 13 Storage part 13a Camera parameter 15 Control part 15a Acquisition part 15b Separation part 15c 1st setting part 15d 2nd setting part 15e Matching part 15f Estimation part

Claims (5)

Acquire multiple images captured by multiple cameras with different shooting positions,
On a first image captured by a first camera selected from the plurality of images, on a straight line in a three-dimensional space connecting the optical center of the first camera and each pixel of the first image To determine if the object exists
A region including the pixel of the identified first image is set as a first range in which the search for the object is executed on a straight line in the three-dimensional space, and other than the first range in the first image Set the area of to the second range,
For the pixels included in the first range, a first interval is set as an interval for performing the search for the object on a straight line in the three-dimensional space, and for the pixels included in the second range Then, a second interval longer than the first interval is set,
According to a set range and a set interval, a block is matched between the second image that is not selected as the first image among the plurality of images and the first image,
Estimating a position in the three-dimensional space for each pixel of the first image based on a correlation between matched blocks;
A method for restoring three-dimensional information, wherein a computer executes processing.   3. The process according to claim 1, wherein the process of setting the first interval sets the first interval that is finer as the importance of the pixel of the first image that has succeeded in specifying the range is higher. How to restore dimension information.   The three-dimensional information restoration method according to claim 1, wherein the identifying process uses Visual Hull to identify a range in which the object exists. Acquire multiple images captured by multiple cameras with different shooting positions,
On a first image captured by a first camera selected from the plurality of images, on a straight line in a three-dimensional space connecting the optical center of the first camera and each pixel of the first image To determine if the object exists
A region including the pixel of the identified first image is set as a first range in which the search for the object is executed on a straight line in the three-dimensional space, and other than the first range in the first image Set the area of to the second range,
For the pixels included in the first range, a first interval is set as an interval for performing the search for the object on a straight line in the three-dimensional space, and for the pixels included in the second range Then, a second interval longer than the first interval is set,
According to a set range and a set interval, a block is matched between the second image that is not selected as the first image among the plurality of images and the first image,
Estimating a position in the three-dimensional space for each pixel of the first image based on a correlation between matched blocks;
A three-dimensional information restoration program which causes a computer to execute processing. An acquisition unit for acquiring a plurality of images captured by a plurality of cameras having different shooting positions;
On a first image captured by a first camera selected from the plurality of images, on a straight line in a three-dimensional space connecting the optical center of the first camera and each pixel of the first image A specific part that identifies whether an object exists in
A region including the pixel of the identified first image is set as a first range in which the search for the object is executed on a straight line in the three-dimensional space, and other than the first range in the first image A first setting unit for setting the area of the second range to a second range;
For the pixels included in the first range, a first interval is set as an interval for performing the search for the object on a straight line in the three-dimensional space, and for the pixels included in the second range Then, a second setting unit that sets a second interval longer than the first interval;
A matching unit that matches a block between the first image and the second image that is not selected as the first image among the plurality of images according to a set range and a set interval;
An estimation unit that estimates a position in the three-dimensional space for each pixel of the first image based on a correlation between matched blocks;
An apparatus for restoring three-dimensional information, comprising:

Images