JP4706068B2 - Image information processing method and image information processing system - Google Patents

Description

  The present invention relates to an image information processing method and an image information processing system for generating a free viewpoint image viewed from a viewpoint (virtual camera) where no camera actually exists based on a plurality of camera images, and in particular, a plurality of camera images. The present invention relates to image information processing in a transmission-side system that acquires image and a reception-side system that displays a free viewpoint image.

Free viewpoint TV (FTV) has been proposed that allows a viewer to freely change the viewpoint and view a three-dimensional scene as if the viewer is on the spot (for example, Non-Patent Documents 1 and 2). reference). FIG. 1 is a diagram conceptually showing the basic structure of an FTV system. The FTV system shown in FIG. 1 captures by a plurality of cameras (step ST1), interpolation processing (step ST2 or 2a) for interpolating images between discretely existing camera images, and an input viewpoint (virtual) The image viewed from the camera) is cut out and displayed (steps ST4 and ST5). In the FTV system, image information of the subject 101 existing in the three-dimensional real space is shown by a plurality of cameras (FIG. 1 shows _five cameras 102 _{1 to} 102 ₅ , but more cameras are actually used. ), And a plurality of acquired camera images (five images of reference numerals 103 _{1 to} 103 ₅ are shown in FIG. 1 but more images are actually used) are arranged in the light space 103. And an FTV signal. In the figure, x is the horizontal viewing direction, y is the vertical viewing direction, and u (= tan θ) is the viewing zone direction.

In the ray space method, one ray in a three-dimensional real space is represented by one point in a multidimensional space with parameters representing the coordinates as coordinates. This virtual multidimensional space is called a ray space. The entire ray space expresses all rays in the three-dimensional space without excess or deficiency. Ray space is created by collecting images taken from many viewpoints. Since the value of the point in the light space is the same as the pixel value of the image, the conversion from the image to the light space is a simple coordinate conversion. As shown in FIG. 2A, the light beam 107 passing through the reference plane 106 in the real space can be uniquely expressed by four parameters of the passing position (x, y) and the passing direction (θ, φ). it can. In FIG. 2A, X is a horizontal coordinate axis in the three-dimensional real space, Y is a vertical coordinate axis, and Z is a depth coordinate axis. In addition, θ is an angle in the horizontal direction with respect to the normal line of the reference surface 106, that is, an output angle in the horizontal direction with respect to the reference surface 106, and φ is an angle in the vertical direction with respect to the normal line of the reference surface 106, that is, the reference surface. The emission angle in the vertical direction with respect to 106. Thereby, the light ray information in the three-dimensional real space can be expressed as luminance f (x, y, θ, φ). Here, in order to make the explanation easy to understand, the parallax (angle φ) in the vertical direction is ignored. As shown in FIG. 2 (a), images taken by a large number of cameras arranged toward the reference plane 106 and horizontally are shown in FIG. 2 (b) as x, y, In the three-dimensional space having the axis of u (= tan θ), the cross sections are 103 _{1 to} 103 ₅ drawn by dotted lines. By cutting an arbitrary surface from the light beam space 103 shown in FIG. 2B, an image viewed from an arbitrary viewpoint in the horizontal direction in the real space can be generated. For example, when the cross section 103a is cut out from the light beam space 103 shown in FIG. 3A, an image as shown in FIG. 3B is displayed on the display, and the cross section from the light beam space 103 shown in FIG. When 103b is cut out, an image as shown in FIG. 3C is displayed on the display. In addition, since there is no data between the images (cross sections 103 _{1 to} 103 ₅ ) arranged in the light space 103, this is created by interpolation. In some cases, the interpolation is performed not for the entire light space but only for a necessary portion.

  FIG. 4 is a diagram illustrating the configuration and operation of a conventional image information processing system including a transmission side system 210 and a reception side system 230. As illustrated in FIG. 4, the transmission-side system 210 corrects the acquired plurality of camera images 210a and the image acquisition unit 211 that acquires the plurality of camera images 210a by photographing the subject 101 with the plurality of cameras 211a. A correction unit 212 that performs encoding, an encoding unit (encoder) 213 that encodes the corrected camera image 210b, and a transmission unit 214 that transmits the encoded camera image 210c and camera parameters via a transmission unit 220 such as the Internet. And have. The reception-side system 230 also includes a reception unit 231 that receives the camera image 210c and camera parameters transmitted from the transmission-side system 210, a decoding unit (decoder) 232 that decodes the received camera image 210c, and a decoding Image generation unit 233 that generates a light space 230b using the converted camera image 230a, an extraction unit 234 that extracts a display image from the light space 230b, and a display unit 235 that displays the extracted display image 230c. It has.

  FIGS. 5A to 5C are block diagrams illustrating a specific example of a conventional image information processing system or 3D service system. In the download / package service using the Internet shown in FIG. 5A, the digital archives 241 of the transmission side system is transmitted to the systems of a plurality of users on the reception side via the transmission / package means 242, and each user's system is transmitted. The system performs decoding by the decoder 243, generation of a light space by the image generation unit 244, and display of a free viewpoint image (2D image or 3D image) by the display unit 245. In the download / package service, real-time performance is not so required. In the broadcasting service shown in FIG. 5B, a camera image is acquired by the image acquisition unit 251 on the transmission side, the camera image is corrected by the correction unit 252, and an image encoded by the encoder 253 is transmitted by the transmission unit 254. Then, in a system of a plurality of users on the receiving side, the decoder 255 decodes the received image, generates a light space by the image generation unit 256, and displays a free viewpoint image. In the broadcast service, real-time property is required rather than the download / package service, but not complete real-time property is required. In the communication service shown in FIG. 5C, an encoded image is transmitted from the transmission side system having the image acquisition unit 261 or 271, the correction unit 262 or 272, and the encoder 263 or 273 via the transmission unit 264. The receiving system including the decoder 265 or 275, the image generation unit 266 or 276, and the display unit 267 or 277 performs decoding, image generation, and display of a free viewpoint image. In communication services, high real-time performance is required.

Masayuki Tanimoto, “Free Viewpoint Television”, Nihon Kogyo Shuppan, Image Lab, February 2005, pages 23-28 Shinya Oka, Na Ban Chang Prim, Toshiaki Fujii, Masayuki Tanimoto, “Light Space Information Compression for Free Viewpoint Television”, IEICE Technical Report, CS 2003-141, pp. 7-12, December 2003

  In the system shown in FIG. 4 or FIGS. 5A to 5C, image generation processing for generating a free viewpoint image from a camera image (processing by the image generation unit 233 in FIG. 4, FIG. 5A). The processing by the image generation unit 244, the processing by the image generation unit 256 in FIG. 5B, and the processing by the image generation units 266 and 276 in FIG. The generation of a free viewpoint image includes a process of calculating a depth map (Depth Map), which is depth information of the scene, and a light space interpolation process using the depth map. For this reason, when only a plurality of camera images (luminance signals) are transmitted to the receiving side system, it is necessary to calculate a depth map from the plurality of camera images in the receiving side system. However, the calculation process of the depth map requires a device having a very high calculation capability, and the receiving side system 230 that is a user side device has a high calculation capability for generating a free viewpoint image. There is a problem that it is difficult in terms of cost.

  For this reason, in the process for generating a free viewpoint image from a plurality of camera images, the depth map calculation process and the interpolation process are performed in the transmission side system, and the reception side system performs a process of cutting out the display image from the light space. Conceivable. FIG. 6 is a diagram showing the configuration and operation of such an image information processing system. As illustrated in FIG. 6, the transmission-side system 310 corrects the captured plurality of camera images 310a and the image acquisition unit 311 that acquires the plurality of camera images 310a by capturing the subject 101 with the plurality of cameras 311a. A correction unit 312 that generates an interpolated image from the corrected camera image 310b, generates an interpolated ray space 310c, an encoding unit 314 that encodes information in the ray space 310c, A transmission unit 315 that transmits the converted light beam space information via the transmission unit 320. In addition, the reception-side system 330 uses the reception unit 331 that receives the information of the light space 310c transmitted from the transmission-side system 310, the decoding unit 332 that decodes the received information, and the decoded information. An image generation unit 332 that generates the light space 310c, an extraction unit 333 that extracts a display image from the light space 310c, and a display unit 334 that displays the extracted display image 310d.

  However, when the light space is created by the transmission side system 310 of the image information processing system shown in FIG. 6, in order to be able to respond to all the requests of a large number of users existing on the reception side, it is only discrete. It is necessary to form an extremely large number of interpolated images so as to fill in a space between a plurality of non-existing camera images. For this reason, there is a problem that the amount of information processing in the transmission-side system 310 becomes enormous, and a sender, for example, a broadcaster, is forced to cost a lot.

  Accordingly, the present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to perform information processing in a transmission-side system in image information processing that generates an image of an arbitrary viewpoint based on a plurality of camera images. An object of the present invention is to provide an image information processing method and an image information processing system capable of reducing the amount of information, using a device with low calculation capability as a receiving system, and reducing the information processing amount of the entire system.

  An image information processing method of the present invention is a method in an image information processing system having a transmission side system and a reception side system, and the transmission side system captures a plurality of camera images by photographing a subject with a plurality of cameras. The step of obtaining, and the transmission side system includes depth information of the subject including the direction and distance of the portion of the subject corresponding to the plurality of photographed camera images and emitting light incident on the camera Generating a depth map; the transmitting system encoding the camera image and the depth map; and the transmitting system transmitting the encoded camera image and the depth map. The camera image and the depth map transmitted from the transmission side system by the reception side system Receiving, the receiving system decoding the received camera image and the depth map, and the receiving system using the decoded camera image and the depth map, A step of calculating an interpolated image between camera images and generating light space information comprising the camera image and the interpolated image; and the receiving system extracting and outputting a display image from the light space information. And a step.

  The image information processing system of the present invention is an image information processing system having a transmission side system and a reception side system, and the transmission side system captures a plurality of camera images by photographing a subject with a plurality of cameras. Generates a depth map consisting of a plurality of cameras to be acquired and depth information of the subject including the direction and distance of the portion of the subject that emits light incident on the camera, corresponding to the captured camera images A depth search unit, an encoding unit that encodes the camera image and the depth map, and a transmission unit that transmits the encoded camera image and the depth map. A receiving unit that receives the camera image and the depth map transmitted from the transmission-side system; the received camera image and the depth map; A decoding unit that decodes a map, and using the decoded camera image and the depth map, an interpolation image between the camera images is calculated, and a light beam composed of the camera image and the interpolation image An interpolation unit that generates spatial information and an extraction unit that extracts and outputs a display image from the light space information are characterized.

  In the image information processing method and the image information processing system according to the present invention, processing for calculating a depth map corresponding to each camera from a plurality of camera images is performed in the transmission side system, and the plurality of camera images and the plurality of depth maps are used. The receiving side system performs the process of interpolating the image in the light space. As described above, when the transmission-side system performs the depth map generation processing with a large amount of information processing, the calculation capability required for the reception-side system can be reduced, and the cost burden on the receiver can be reduced. .

  In addition, when the process of interpolating the light space using a plurality of camera images and a plurality of depth maps is performed in the transmission side system, it is necessary to perform the entire region between the plurality of camera images. In this case, it is sufficient to perform only the portion related to the area designated by the user. Therefore, according to the image information processing method and the image information processing system of the present invention, the calculation capability required for the receiving system that performs the interpolation processing is low, and the cost burden on the receiver is not greatly increased.

  Furthermore, when the process of interpolating the light space using the plurality of camera images and the depth map is performed on the transmission side system, it is necessary to perform the process on the entire area between the plurality of camera images. In this case, it is only necessary to perform a portion related to the area designated by the user. For this reason, according to the image information processing method and the image information processing system of the present invention in which the interpolation processing is performed in the reception side system, the information processing amount of the entire system is reduced as compared with the case where all of the image generation processing is performed in the transmission side system. As a result, the calculation cost can be reduced.

<< 1. First Embodiment >>
<1-1. Configuration and Operation of Image Information Processing System According to First Embodiment>
FIG. 7 is a diagram showing the configuration and operation of the image information processing system according to the first embodiment of the present invention, which is composed of a sender system 10 and a receiver system 30. The image information processing system of FIG. 7 is an FTV system that displays a two-dimensional image viewed from an arbitrary viewpoint, but may be a 3D display system that displays a three-dimensional (3D) image.

  The image information processing system shown in FIG. 7 is a system having a transmission side system 10 and a reception side system 30. The transmission side system 10 transmits information to the reception side system 30 through transmission means 20. The transmission means 20 is, for example, the Internet, a broadcasting facility, a communication facility, or the like, but may be any means that can transmit a signal. Further, the transmission means 20 may include storage means (storage) for holding information as necessary.

  As shown in FIG. 7, the transmission-side system 10 corrects the captured plurality of camera images 10a and the image acquisition unit 11 that acquires the plurality of camera images 10a by capturing the subject 101 with the plurality of cameras 11a. The correction unit 12, the depth search unit 13 that generates a plurality of depth maps 10d corresponding to the corrected plurality of camera images 10b, and the plurality of camera images 10b and a plurality of depth maps corresponding to the camera images. And a transmission unit 15 that transmits the encoded camera image 10c, the depth map 10d, and camera parameters related to the plurality of cameras 11a via the transmission unit 20. The correction unit 12 corrects the camera image when, for example, the plurality of cameras 11a are not lined up at equal intervals in one row or facing the same direction, and the plurality of cameras are arranged at equal intervals in one row. The camera images 10a are corrected so that they can be regarded as being aligned and facing the same direction. The depth map 10d includes depth information of the subject 101 including the direction and distance of each part of the subject 101 that emits light incident on the camera.

  As shown in FIG. 7, the receiving system 30 includes a receiving unit 31 that receives the camera image 10c, the depth map 10d, and the camera parameters transmitted from the transmitting system 10 via the transmission unit 20, and a reception unit 31. The interpolated image between the camera images is calculated using the decoding unit 32 that decodes the camera image 10c and the depth map 10d, the decoded camera image 10e, the depth map 10f, and the camera parameters, An interpolation unit 33 that interpolates an interpolated image in a light space composed of camera images, an extraction unit 34 that extracts a display image from the interpolated light space 10g, and the cut display image 10h are displayed on one or a plurality of displays. It has the display part 35.

<Operation of Image Information Processing System of the Present Invention (Image Information Processing Method of the Present Invention)>
In the first embodiment of the present invention, as shown in FIG. 7, the transmission-side system 10 obtains a plurality of camera images 10a by photographing the subject 101 with a plurality of cameras 11a, and takes a plurality of photographed images. The camera image 10a is corrected as necessary, a plurality of depth maps corresponding to the plurality of camera images 10b are generated, the plurality of camera images 10b and the plurality of depth maps are encoded, and the camera image 10c, the depth map 10d, The camera parameters are transmitted via the transmission means 20.

  Also, as shown in FIG. 7, the receiving system 30 receives the camera image 10c, the depth map 10d, and the camera parameters transmitted from the transmitting system 10 via the transmission unit 20, and receives the received camera image. 10c and the depth map 10d are decoded, and the interpolated image between the camera images is calculated using the decoded camera image 10e, the depth map 10f, and the camera parameters, and the interpolated image is formed in the light space composed of the camera images. Interpolate, cut out the display image from the interpolated ray space 10g, and display the cut out display image 10h.

  FIG. 8 is a diagram showing information transmitted from the transmission-side system in the image information processing system (FIG. 7) and the conventional image information processing system (FIGS. 4 and 6) according to the first embodiment of the present invention. In the case of the conventional image information processing system (comparative example) shown in FIG. 4, images transmitted from the transmission-side system 210 are camera parameters and a plurality of camera images 210c. In the case of the conventional image information processing system shown in FIG. 6 (comparative example), the image transmitted from the transmission side system 310 is the light space information 310c. On the other hand, in the image information processing system of the present invention shown in FIG. 7, images transmitted from the transmission side system 10 are camera parameters, a plurality of camera images 10c, and a depth map 10d of each camera.

FIG. 9 is a diagram showing a plurality of camera images and a plurality of depth maps in the image information processing system (FIG. 7) according to the first embodiment of the present invention. In FIG. 9, N is a positive integer, and k is a positive integer equal to or less than N. 9, 1, 2,..., K-1, k, k + 1,..., N are camera numbers of a plurality of cameras of the image acquisition unit (reference numeral 11 in FIG. 7). _{Further, I 1, I 2, ...} , I k-1, I k, I k + 1, ..., I N represents a plurality of camera _{images, D 1, D 2, ...} , D k-1, D k, _{D k + 1, ..., D} N denotes the depth map corresponding to each camera. Each of the camera images I ₁ , I ₂ ,..., I _k−1 , I _k , I _{k + 1} ,..., I _N shown in FIG. 9 is a single image (for example, a rectangle). 9 shows a case where camera images are arranged vertically on the paper surface on which FIG. 9 is drawn. Further, the depth map _{D 1,} D ₂ shown in FIG. _{9, ..., D k-1} , D k, D k + 1, ..., each of _{D N,} like the camera images one image (e.g., rectangular) FIG. 9 shows a case where the depth map is arranged vertically on the paper on which FIG. 9 is drawn.

FIG. 10 is a diagram illustrating an example in which one depth map is displayed with a difference in brightness. In FIG. 10, a bright part (white part) is a close part, and a dark part (black) shows a far part. When the camera images I ₁ , I ₂ ,..., I _k−1 , I _k , I _{k + 1} ,..., _N transmitted from the transmission side system 10 are N, the transmission side system 10 transmits them. depth map _{D 1, D 2, ...,} D k-1, D k, D k + 1, ..., D N also becomes N sheets.

FIG. 11 is a diagram showing an information processing process in the transmission side system 210 and the reception side system 230 of the conventional image information processing system (comparative example) shown in FIG. 11, 1, 2,..., K-1, k, k + 1,..., N are camera numbers of a plurality of cameras of the image acquisition unit (reference numeral 211 in FIG. 4). _{Further, I 1, I 2, ...} , I k-1, I k, I k + 1, ..., I N represents a plurality of camera _{images, D 1, D 2, ...} , D k-1, D k, _{D k + 1, ..., D} N denotes the depth map corresponding to each camera. Each of the camera image and the depth map shown in FIG. 11 can be represented as one image (for example, a rectangle) like the camera image, but in FIG. 11, the depth map is a paper surface on which FIG. 11 is drawn. Shows the case of being arranged vertically. As shown in FIG. 11, in the image information processing system (comparative example) shown in FIG. 4, depth search and depth maps D ₁ , D ₂ ,..., D _k−1 , D _k , D _{k + 1} ,. , _DN is generated by the receiving system 230.

FIG. 12 is a diagram showing an information processing process in the transmission side system 310 and the reception side system 330 of the conventional image information processing system (comparative example) shown in FIG. 12, 1, 2,..., K-1, k, k + 1,..., N are camera numbers of a plurality of cameras of the image acquisition unit (reference numeral 311 in FIG. 6). _{Further, I 1, I 2, ...} , I k-1, I k, I k + 1, ..., I N represents a plurality of camera _{images, D 1, D 2, ...} , D k-1, D k, _{D k + 1, ..., D} N denotes the depth map corresponding to each camera. Each of the camera image and the depth map shown in FIG. 12 can be represented as one image similarly to the camera image, but in FIG. 12, the depth map is arranged vertically on the paper on which FIG. 12 is drawn. Shows the case. As shown in FIG. 12, in the image information processing system (comparative example) shown in FIG. 6, depth search and depth maps D ₁ , D ₂ ,..., D _k−1 , D _k , D _{k + 1} ,. , _DN , and ray space are generated by the transmission side system 310.

FIG. 13 is a diagram illustrating an information processing process in the transmission-side system 10 and the reception-side system 30 of the image information processing system according to the first embodiment of this invention. 13, 1, 2,..., K-1, k, k + 1,..., N are camera numbers of a plurality of cameras in the image acquisition unit (reference numeral 11 in FIG. 7). _{Further, I 1, I 2, ...} , I k-1, I k, I k + 1, ..., I N represents a plurality of camera _{images, D 1, D 2, ...} , D k-1, D k, _{D k + 1, ..., D} N denotes the depth map corresponding to each camera. Each of the camera image and the depth map shown in FIG. 13 can be represented as a single image (for example, a rectangle) like the camera image, but in FIG. 13, the depth map is a sheet of paper on which FIG. 13 is drawn. Shows the case of being arranged vertically. As shown in FIG. 13, in the image information processing system according to the first embodiment of the present invention, depth search and depth maps D ₁ , D ₂ ,..., D _k−1 , D _k , D _{k + 1} ,. , _DN is generated by the transmission side system 10, and the reception side system 30 performs generation of an interpolation image and extraction of a display image from the interpolated ray space.

<1-2. Image generation processing>
FIG. 14 is a diagram illustrating a process for generating an image of an arbitrary viewpoint from a camera image acquired by an actual camera. In FIG. 14, reference numerals 401 to 405 denote cameras arranged in the same direction in a line on a straight line, and reference numeral 411 denotes a virtual camera (viewpoint position specified by the user) that does not actually exist. The cameras 401 to 405 photograph the light rays passing through the cameras 401 to 405, but do not photograph the light rays passing between adjacent cameras. Therefore, the virtual camera 411 is placed where the user wants to see, that is, between adjacent cameras. From the light rays passing through the actual camera (cameras 402 and 403 in FIG. 14), the light rays passing through the virtual camera 411 are interpolated. However, although it can be seen from which direction the light rays that pass through the cameras 401 to 405 come from, it is not known from which position. Therefore, for example, the depth of the subject 412 is searched using the cameras 402 and 403 shown in FIG. In this depth search, assuming that light from a certain point on the object surface has the same intensity in all directions (Lambert reflection), the shooting point (portion on the subject) of the camera 402 and the shooting of the camera 403 A position where the luminance of the point (portion on the subject) matches is searched, and the matched position is estimated as the surface of the object. At this time, the distance between the camera 402 and the camera 403 is known, and the angle at which the light beam from the matching position enters the camera 402 and the angle at which the light beam from the matching position enters the camera 403 are also known. Can be calculated. After the depth of the object can be estimated in this way, an image viewed from the virtual camera 411 is estimated to generate an interpolation image.

FIG. 15 is a diagram illustrating the relationship between depth and parallax. An interpolated image in the light space can be generated if the depth of the scene at the camera position to be generated can be obtained. Here, the geometric relationship between depth and parallax will be described. For simplicity, consider a two-dimensional plane that does not consider the height direction. Let the left and right cameras be a camera L and a camera R. Assume that the depth of the light source M is D. In this case, light emitted by the light source M is incident on each point V _L and point V _R of the left and right cameras L and camera R. A point (in this case, V _L , V _R ) where a different camera captures a light beam emitted from the same light source M is called a corresponding point. Further, called corresponding points _V L between the cameras L and camera R, the difference between _{V R} and a parallax is described as d.

The relationship between the depth D and the parallax d is obtained. In FIG. 15, the focal length of the camera is f, the distance between the left and right cameras is J, the light source is M, and the distance of the X coordinate from the light source M to the camera L is m. d ₁ and d ₂ of the can obtained as follows.
d ₁ = f · (J−m) / D
d ₂ = f · m / D
Since the parallax d is given by d ₁ + d ₂ ,
d = d ₁ + d ₂ = f · (J−m) / D + f · m / D = f · J / D
It becomes. Distance _{D L} from the camera L to light source M is _D L = D / cos [theta]
The relationship between the parallax d and the distance D _L when viewed from the camera L is
d = (f · J · cos θ) / D _L
It becomes. That is, the parallax d is proportional to the camera interval J, it becomes inversely proportional to the depth D _L of light source M as viewed from the camera L. The depth estimation method of the present invention is depth estimation at the camera position to be interpolated. Therefore, depth estimation is performed using only peripheral image information from a state where there is no information, and an interpolation image is generated. According to this depth estimation method, an interpolation image can be generated more accurately by obtaining the depth at the viewpoint to be created. The relationship of such depth D _L and the parallax d, altering the parallax d is between the left and right images, found to be equivalent to changing the depth D _L of the ray source. That is, by changing the parallax d in various ways, a corresponding point between images is found, the light source is estimated based on the obtained parallax information, and the pixel value to be interpolated is specified.

<1-3. Depth map generation method considering occlusion>
FIG. 16 is a diagram illustrating a depth search method when occlusion exists. In FIG. 16, a point 421a on the surface of the object 421 is a position where both the camera position k and the camera position k + 1 can be seen. Therefore, depth estimation when viewing the point 421a from the viewpoint 431, which is the virtual camera position, can be performed using image information from the viewpoint k and image information from the viewpoint k + 1. However, the point 422a on the surface of the object 422 in the rear is visible from the camera position k + 1, but cannot be seen from the camera position k behind the object 421. The fact that it can be seen from one camera viewpoint but cannot be seen from the other camera viewpoint is called half occlusion. Also, the fact that it cannot be seen from both camera positions is called full occlusion.

  FIG. 17 is a diagram illustrating a half occlusion range and a full occlusion range when the object 422 is viewed from the camera position k and the camera position k + 1. As shown in FIG. 17, the range 440 of the object 422 is a range that can be seen from any camera position k and camera position k + 1, and the ranges 441 and 443 of the object 422 are seen from the camera viewpoint k + 1, but the camera viewpoint k. The range 442 of the object 422 is a range of full occlusion that cannot be seen from either the camera viewpoint k or the camera position k + 1.

FIG. 18 is a diagram illustrating a depth map calculation method according to the first embodiment of the present invention. In FIG. 18, 1, 2,..., N at the top are camera numbers of a plurality of cameras in the image acquisition unit (reference numeral 11 in FIG. 7). In addition, I ₁ , I ₂ ,..., I _N indicate a plurality of camera images, and 1, 2,..., N-1 near the middle stage indicate camera image pairs. Of the two lines included in one pair of camera images, a thick line indicates a depth map based on the left camera in the camera pair, and a thin line indicates a depth map based on the right camera.

FIG. 19 is a diagram illustrating a parallax calculation method according to the present invention. In Figure 19, _{I k} and _{I k + 1} denotes a pair of camera images (pair), _{I k} represents the left side of the camera the camera images of the pair of _{cameras, I k + 1} denotes a camera image of the right camera . Further, in FIG. 19, 500 is a diagram showing a parallax of camera image _{I k} and the camera image _{I k + 1.}

As shown in FIG. 19, first, a portion of brightness and shape that can be considered to be the same as (or similar to) the portion 501 _k in the image I _k taken with the left camera was taken with the right camera. Detection is performed from the camera image I _{k + 1} . The part of the camera image I _{k + 1} detected by this processing is 501 _{k + 1} , and the parallax at that time is R _k as shown in the parallax diagram 500. Here, being able to be regarded as being the same (or similar) means that the difference value of the luminance values is within a predetermined range.

Next, can be regarded as the same as the detected partial 501 k + ₁ in the image I _k taken by the right camera (or similar) the portion of the luminance and shape, camera image I _k taken by the left camera Detect from inside. The portion of the camera image I _k detected by this processing is 501 _k, and the parallax at that time is L _k as shown in the parallax diagram 500.

Therefore, the detection portion in the image I _{k + 1} taken by the right camera when based on the portion 501 _k in the image I _k taken by the left camera (corresponding portion) is a portion 501 k + _1, and The detected portion (corresponding portion) in the image I _k photographed by the left camera when the portion 501 _{k + 1} in the image I _{k + 1} photographed by the right camera is used as a reference is a portion 501 _k . Therefore, L _k = R _k for the part 501 _k and the part 501 _{k + 1} . In this case, the depth can be calculated based on the two camera images. In this case, it can be said that the reliability of the depth is high.

Next, another example will be described. As shown in FIG. 19, a camera image obtained by photographing a portion of brightness and shape that can be considered to be the same as (or similar to) a portion 502 _k in an image I _k photographed by the left camera with the right camera. to detect from the I _{k + 1.} The part of the camera image I _{k + 1} detected by this processing is set to 502 _{k + 1} , and the parallax at that time is set to R _k as shown in the parallax diagram 500.

Next, luminance and shape portions that can be considered to be the same as (or similar to) the portion 502 _{k + 1} in the image I _{k + 1} photographed by the right camera are extracted from the camera image I _k photographed by the left camera. To detect. The portion of the camera image I _k detected by this processing is 503 _k , which is a position different from the portion 502 _k . As shown in disparity map 500, the parallax at that time and L _k.

Therefore, the detection portion in the image I _{k + 1} taken by the right camera when based on the portion 502 _k in the image I _k taken by the left camera is a portion 502 k + _1, and the right side of the camera The detected portion in the image I _k photographed by the left camera when the portion 502 _{k + 1} in the image I _{k + 1} photographed in step ₁ is used as a reference is a portion 503 _k . Therefore, L _k ≠ R _k for the portion 501 _{k + 1} . Such a situation often occurs where an occlusion exists, and the depth is a result based on one or zero camera images, and it can be said that the reliability is low.

Figure 20 is a diagram illustrating a method for determining the depth D _k at the viewpoint k. As shown in FIG. 20, first, the parallax calculation shown in FIG. 19 is performed for a pair of camera images I _k-1 and I _k from adjacent cameras, and the left camera image I _k-1 part is calculated. Parallax L _k-1 when a similar part of the right camera image I _k is detected as a reference and parallax when a similar part of the left camera image I _k-1 is detected using the right camera image I _k as a reference From R _k−1 , the depth D _k− and its reliability are calculated.

Next, the camera image I _k from the adjacent _cameras, the pair of I k + _1, perform calculations of parallax shown in FIG. 19, like parts of the right camera image I _{k + 1} based on the portion of the left side of the camera image I _k The depth D _{k +} and its reliability are calculated from the parallax R _k when a similar part of the left camera image I _k is detected with reference to the parallax L _k when the camera detects I and the part of the right camera image I _{k + 1} .

Next, if L _k−1 = R _k−1 or L _k = R _k , the reliability is high, otherwise the reliability is low. Therefore, the depth D _k is calculated from the depths D _{k +} and D _k− in consideration of the obtained reliability.

For example, if L _k−1 = R _k−1 and L _k ≠ R _k , it is determined that there is half occlusion, and the depth D _k− is set as the depth D _k . For example, if L _k−1 ≠ R _k−1 and L _k = R _k , it is determined that there is half occlusion, and the depth D _{k +} is set as the depth D _k . For example, if L _k−1 = R _k−1 and L _k = R _k , it is determined that there is no occlusion, and the average value of the depth D _k− and the depth D _{k +} is set as the depth D _k . For example, if L _k−1 ≠ R _k−1 and L _k ≠ R _k , it is determined that there is full occlusion, and the value obtained by another method (for example, the depth of the adjacent portion) is determined as the depth D _k. Or a predetermined value is used. Note that the method of determining D _{k +} is not limited to the above example.

Further, the range where L _k−1 ≠ R _k−1 or L _k ≠ R _k is a range where occlusion exists, for example, a region indicated by hatching in FIG. In FIG. 21, the cross-hatched area is a half-occlusion area that is visible from the right camera but not from the left camera, and the hatched area is visible from the left camera but not from the right camera. It is a range of half occlusion. Thus, the presence or absence of occlusion based on the coincidence or mismatch between the detected parallaxes L _k-1 and R _{k-1 and} the coincidence or mismatch between the parallaxes L _k and R _k , that is, based on the coincidence or mismatch between the depths. Since the one-side interpolation based on the pair of camera images on the side where the light rays reach is performed, the error of the depth D _k due to the presence of half occlusion can be reduced, and as a result, the generated image Quality can be improved.

<1-4. Effect of First Embodiment>
As described above, in the image processing method and an image information processing system of the first embodiment, a plurality of camera images I _1, I 2, _..., the depth map D ₁ for each camera from the I _N, D _2, ..., performed at the sending system 10 the process of calculating the _{D N,} a plurality of camera images _{I 1, I 2, ...,} I N and a plurality of depth map _{D 1, D 2, ...,} with _{D N} Thus, the receiving side system 30 performs processing for interpolating an image in the light space. Thus, greater depth map D _1, D 2 of the information processing amount, _..., By is performed sending system 10 a process of production _of D N, computing power required for the receiving system 30 is a user side The cost may not be high, and the cost burden on the user can be reduced.

Further, a plurality of camera images _{I 1, I 2, ...,} I N and a plurality of depth map _{D 1,} D 2, ..., the processing of interpolating the light space by using a _{D N} is the case of the transmission side system 10 a plurality of camera images I _1, I 2, _..., it is necessary to perform the entire area between _the I N, in case of the receiving system 30, the portions related to the user-specified region only line Just do it. Therefore, according to the image information processing method and the image information processing system of the first embodiment, the calculation capability required for the reception-side system 30 that performs the interpolation process may not be high, and the cost burden on the user is increased. There is no.

Further, a plurality of camera images _{I 1, I 2, ...,} I N and a plurality of depth map _{D 1,} D 2, ..., the processing of interpolating the light space by using a _{D N} is the case of the transmission side system 10 a plurality of camera images I _1, I 2, _..., it is necessary to perform the entire area between _the I N, in case of the receiving system 30, the portions related to the user-specified region only line Just do it. For this reason, according to the image information processing method and the image information processing system of the first embodiment in which the interpolation processing is performed in the reception side system 30, the entire system is compared with the case where all of the image generation processing is performed in the transmission side system 10. The amount of information processing can be reduced, and as a result, the calculation cost can be reduced.

<< 2. Second Embodiment >>
<2-1. Configuration and Operation of Image Information Processing System According to Second Embodiment>
FIG. 22 is a diagram illustrating the configuration and operation of an FTV system that is an image information processing system according to the second embodiment of this invention. In FIG. 22, the same reference numerals are given to the same or corresponding components as those in FIG. 7 (first embodiment). In the image information processing system according to the second embodiment, unlike the image information processing system according to the first embodiment shown in FIG. 7, the transmission-side system 10 includes depth information obtained by actual measurement, occlusion information indicating an occlusion range, Additional information such as specular reflection information indicating a range in which the subject is specularly reflected is added to the camera image 10c, the depth map 10d, and the camera parameters and transmitted to the receiving system 30.

  In the image information processing system according to the first embodiment shown in FIG. 7, the reception-side system 30 performs interpolation using a plurality of camera images, a plurality of depth maps, and camera parameters transmitted from the transmission-side system 10. An image is generated. However, for example, when specular reflection occurs on the surface of the subject 101, occlusion occurs, the surface of the subject has a periodic structure, the surface of the subject has a structure without a pattern (flat plate, white plate, curved surface). In some cases, the matching determination shown in FIG. 19 becomes inaccurate, and the depth may not be estimated accurately. Here, unlike the Lambertian reflection shown in FIG. 23A, the specular reflection is a reflection having an elliptical reflected light intensity distribution as shown in FIG. 23B.

  Therefore, in the image information processing system according to the second embodiment, the transmission-side system 10 measures the depth of the subject by another means (not shown), or measures the specular reflection range, or a range where occlusion occurs. And the measurement result is sent to the receiving system 30 as additional information E. Here, the depth of the subject, the range of specular reflection, and the presence or absence of occlusion are known devices such as detecting the presence or absence of reflected light, the direction of reflected light, the intensity distribution of reflected light, etc. by irradiating the subject with light. Can be obtained by: Note that in normal interpolation, an interpolation image is generated from two cameras close to the virtual camera using linear interpolation, etc., but in the case of specular reflection, the directivity is high, so the reflection intensity approximates appropriately in linear interpolation. There are cases where it is not possible. For this reason, it is possible to appropriately approximate the reflection intensity by performing interpolation using not only two cameras but also a large number of three or more camera images.

<2-2. Effect of Second Embodiment>
As described above, in the image information processing method and the image information processing system according to the second embodiment, in addition to the camera image and depth information, additional information E that is information that can be known only on the transmission side is received on the reception side. Since the image is transmitted to the system 30, the depth error can be reduced as compared with the image information processing method and the image information processing system of the first embodiment, and as a result, a high-quality image can be generated.

  In the second embodiment, points other than those described above are the same as in the case of the first embodiment.

It is a figure which shows notionally the basic composition of a FTV system. (A) is a figure which shows the object in real space, the camera by which linear arrangement | positioning, the reference plane, and the light ray, (b) is a figure which shows light ray space. (A) is a figure which shows light beam space, (b) is a figure which shows the image cut out from light beam space, (c) is a figure which shows the other image cut out from light beam space. It is a figure which shows the structure and operation | movement of the conventional FTV system. (A)-(c) is a block diagram which shows the specific example of the conventional FTV / 3D service. It is a figure which shows the structure and operation | movement of another conventional FTV system. It is a figure which shows the structure and operation | movement of an image information processing system of the 1st Embodiment of this invention. It is a figure which shows the information transmitted from the transmission side system of this invention and the conventional image information processing system. It is a figure which shows notionally the several image information used in an image information processing system, and several depth map information. It is a figure which shows the example which displayed the depth map of 1 sheet by the difference in the brightness. It is a figure which shows an example of the depth map in this invention. It is a figure which shows the process of the image generation in the transmission side system of the conventional image information processing system shown by FIG. 4, and a receiving side system. It is a figure which shows the process of the image generation in the transmission side system of the conventional image information processing system shown by FIG. 6, and a receiving side system. It is a figure which shows the process of the image generation in the transmission side system of the image information processing system of the 1st Embodiment of this invention shown by FIG. 7, and a receiving side system. It is a figure which shows the process which produces | generates the image of arbitrary viewpoints (virtual camera) from an actual camera image. It is a figure which shows the geometrical relationship between depth and parallax. It is a figure which shows the method of the depth search in this invention. It is a figure which shows a half occlusion and a full occlusion. It is a figure which shows the calculation method of the depth map in this invention. It is a figure which shows the calculation method of the parallax in this invention. It is a figure which shows the determination method of the depth in this invention. It is the figure which showed the location of the occlusion. It is a figure which shows the structure and operation | movement of an image information processing system of the 2nd Embodiment of this invention. (A) shows Lambertian reflection and (b) shows specular reflection.

Explanation of symbols

10 Sender system,
10a Camera image (before correction),
10b Camera image (after correction),
10c Camera image (after encoding),
10d depth map (after encoding),
10e Camera image (after decoding),
10f Depth map (after decoding),
10g ray space,
10h cropped image,
11 Image acquisition unit,
11a camera,
12 Correction part,
13 Depth search part,
14 encoding unit,
15 Transmitter,
20 transmission means,
30 receiving system,
31 receiver,
32 decryption unit,
33 Interpolator,
34 Extraction unit,
35 display,
101 Subject,
_{I 1, I 2, ...,} I k-1, I k, I k + 1, ..., I N camera image,
_{D 1, D 2, ...,} D k-1, D k, D k + 1, ..., D N depth map,
E Additional information.

Claims (18)

An image information processing method in an image information processing system having a transmission side system and a reception side system,
The transmitting system acquiring a plurality of camera images by photographing a subject with a plurality of cameras;
The transmission-side system generates a depth map that includes the depth information of the subject including the direction and distance of the portion of the subject that emits light incident on the camera, corresponding to the plurality of captured camera images. Steps,
The transmitting system encoding the camera image and the depth map;
The transmitting system transmitting the encoded camera image and the depth map;
The receiving system receiving the camera image and the depth map transmitted from the transmitting system;
The receiving system decoding the received camera image and the depth map;
The receiving system calculates an interpolated image between the camera images using the decoded camera image and the depth map, and generates ray space information including the camera image and the interpolated image. Steps,
The receiving system includes a step of extracting and outputting a display image from the light space information.   The image information processing method according to claim 1, wherein the display image is one or a plurality of two-dimensional images viewed from an arbitrary position where no camera is present.   2. The image information processing method according to claim 1, wherein the display image is a three-dimensional image viewed from an arbitrary position where no camera is present. The step of generating the depth map includes:
When k is a positive integer less than or equal to the number of the plurality of cameras,
Image information of the portion of the subject included in the kth camera image captured by the kth camera of the plurality of cameras, and the k + 1th camera captured by the k + 1th camera of the plurality of cameras. The result of matching judgment with the image information of the part of the subject included in the image,
Image information of the portion of the subject included in the kth camera image captured by the kth camera of the plurality of cameras, and the kth image captured by the k−1th camera of the plurality of cameras. And determining the depth of the portion of the subject included in the k-th camera image based on the result of the matching determination with the image information of the portion of the subject included in the -1 camera image. The image information processing method according to claim 1. In the matching determination in the step of generating the depth map,
It is determined that the image information of the portion of the subject included in the kth camera image matches the image information of the portion of the subject included in the k + 1th camera image, and the kth camera image When it is determined that the image information of the portion of the included subject matches the image information of the portion of the subject included in the k−1th camera image,
Depth information calculated based on image information of the portion of the subject included in the k-th camera image and image information of the portion of the subject included in the k + 1-th camera image, and the k-th camera image Included in the kth camera image from the depth information calculated based on the image information of the portion of the subject included in the image and the image information of the portion of the subject included in the k−1th camera image. The image information processing method according to claim 4, wherein the depth of the portion of the subject is determined. In the matching determination in the step of generating the depth map,
It is determined that the image information of the portion of the subject included in the kth camera image matches the image information of the portion of the subject included in the k + 1th camera image, and the kth camera image When it is determined that the image information of the portion of the included subject and the image information of the portion of the subject included in the k−1th camera image do not match, the image of the subject included in the kth camera image is determined. The depth of the portion of the subject included in the kth camera image is determined from depth information calculated based on the image information of the portion and the image information of the portion of the subject included in the k + 1th camera image. ,
It is determined that the image information of the portion of the subject included in the k-th camera image and the image information of the portion of the subject included in the k + 1-th camera image do not match, and the k-th camera image When it is determined that the image information of the portion of the included subject matches the image information of the portion of the subject included in the k−1th camera image, the image of the subject included in the kth camera image is determined. The depth of the portion of the subject included in the k-th camera image is calculated from the depth information calculated based on the image information of the portion and the image information of the portion of the subject included in the k-1 camera image. The image information processing method according to claim 4, wherein the determination is performed.   The transmission-side system measures an actual measurement value of a depth of a subject viewed from each of the plurality of cameras, and transmits actual measurement depth information based on the actual measurement value together with the camera image and the depth map. Item 7. The image information processing method according to any one of Items 1 to 6.   The transmission-side system obtains an imageable range for each of the plurality of cameras, and transmits occlusion information indicating an occlusion range other than the imageable range together with the camera image and the depth map. 8. The image information processing method according to any one of 1 to 7.   The transmission side system obtains a range in which the surface of a subject is specularly reflected for each of the plurality of cameras, and transmits specular reflection information indicating the specularly reflected range together with the camera image and the depth map. The image information processing method according to any one of claims 1 to 8. An image information processing system having a transmission side system and a reception side system,
The sending system is
A plurality of cameras for acquiring a plurality of camera images by photographing a subject with a plurality of cameras;
A depth search unit that generates a depth map composed of depth information of the subject including the direction and distance of the portion of the subject that emits light incident on the camera, corresponding to the plurality of captured camera images;
An encoding unit for encoding the camera image and the depth map;
A transmitter that transmits the encoded camera image and the depth map;
The receiving system is
A receiving unit for receiving the camera image and the depth map transmitted from the transmitting side system;
A decoding unit that decodes the received camera image and the depth map;
An interpolation unit that calculates an interpolated image between the camera images using the decoded camera image and the depth map, and generates ray space information composed of the camera image and the interpolated image;
An image information processing system comprising: an extraction unit that extracts and outputs a display image from the light space information.   The image information processing system according to claim 10, wherein the display image is one or a plurality of two-dimensional images viewed from an arbitrary position where the camera does not exist.   The image information processing system according to claim 10, wherein the display image is a three-dimensional image viewed from an arbitrary position where the camera does not exist. The depth search unit
When k is a positive integer less than or equal to the number of the plurality of cameras,
Image information of the portion of the subject included in the kth camera image captured by the kth camera of the plurality of cameras, and the k + 1th camera captured by the k + 1th camera of the plurality of cameras. The result of matching judgment with the image information of the part of the subject included in the image,
Image information of the portion of the subject included in the kth camera image captured by the kth camera of the plurality of cameras, and the kth image captured by the k−1th camera of the plurality of cameras. The depth of the portion of the subject included in the kth camera image is determined based on the result of the matching determination with the image information of the portion of the subject included in the −1 camera image. 13. The image information processing system according to any one of items 1 to 12. The depth search unit, in the matching determination,
It is determined that the image information of the portion of the subject included in the kth camera image matches the image information of the portion of the subject included in the k + 1th camera image, and the kth camera image When it is determined that the image information of the portion of the included subject matches the image information of the portion of the subject included in the k−1th camera image,
Depth information calculated based on image information of the portion of the subject included in the k-th camera image and image information of the portion of the subject included in the k + 1-th camera image, and the k-th camera image Included in the kth camera image from the depth information calculated based on the image information of the portion of the subject included in the image and the image information of the portion of the subject included in the k−1th camera image. The image information processing system according to claim 13, wherein the depth of the portion of the subject is determined. The depth search unit, in the matching determination,
It is determined that the image information of the portion of the subject included in the kth camera image matches the image information of the portion of the subject included in the k + 1th camera image, and the kth camera image When it is determined that the image information of the portion of the included subject and the image information of the portion of the subject included in the k−1th camera image do not match, the image of the subject included in the kth camera image is determined. The depth of the portion of the subject included in the kth camera image is determined from depth information calculated based on the image information of the portion and the image information of the portion of the subject included in the k + 1th camera image. ,
It is determined that the image information of the portion of the subject included in the k-th camera image and the image information of the portion of the subject included in the k + 1-th camera image do not match, and the k-th camera image When it is determined that the image information of the portion of the included subject matches the image information of the portion of the subject included in the k−1th camera image, the image of the subject included in the kth camera image is determined. The depth of the portion of the subject included in the k-th camera image is calculated from depth information calculated based on the image information of the portion and the image information of the portion of the subject included in the k-1 camera image. It determines. The image information processing system of Claim 14 characterized by the above-mentioned.   The transmission-side system has distance measurement means for measuring an actual measurement value of the depth of a subject viewed from each of the plurality of cameras, and transmits actual measurement depth information based on the actual measurement value together with the camera image and the depth map. 16. The image information processing system according to claim 10, wherein the image information processing system is any one of claims 10 to 15.   The transmission-side system obtains an imageable range for each of the plurality of cameras, and transmits occlusion information indicating an occlusion range other than the imageable range together with the camera image and the depth map. The image information processing system according to any one of 10 to 16. The transmission side system obtains a range in which the surface of a subject is specularly reflected for each of the plurality of cameras, and transmits specular reflection information indicating the specularly reflected range together with the camera image and the depth map. The image information processing system according to any one of claims 10 to 17.

Images