JP2005004487A - Apparatus and method for processing surround image photographed on capture path - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and method capable of processing full view image data and effective sampling of surround image data that is obtained through the processing of the full view image data. <P>SOLUTION: An image processing apparatus includes a position detection unit for detecting a position at which the full view image data are captured, a spatial factor detection unit for calculating a spatial factor corresponding to the position detected, the spatial factor being related to a geometric structure of surround environment, of which the full view image is captured from the detected position; and a filter unit for causing reduction in processed image data, that is to be outputted from the image processing apparatus, according to the spatial factor calculated. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an apparatus and method for processing a surround image, and more specifically, annotated video captured with an omnidirectional multi-head camera moving along a path, spatial filtering and view. The present invention relates to an apparatus and a method for inning.
[0002]
[Prior art]
The pursuit of authenticity in computer graphics is an endless goal. Image-based rendering (IBR), where object space rendering algorithms provide a surprisingly sharp image that lacks a “real world” feel, while on the other hand, modeling and analysis of image sources using inverse rendering means ), And provides a three-dimensional environment that attracts public attention in image space, although it lacks extensibility and interactivity.
[0003]
S. E. Chen's introduction to photo quality background depiction (“Quicktime VR-An Image-Based Approach to Virtual Environment Navigation” ACM SIGGRAPH, pp. 29-38, 1995), Light Space (M. Le f. Rendering ”, ACM SIGGRAPH, pp. 31-42, 1996; , Compressed ray space plane (W. C. Chen, J. Y. Bougue t, M. H. Chu, R. Grzezzczuk, “Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Lights. 200 p.”, ACM Tras. Virtual passage of photographic quality in a dynamic environment (D.G. Aliaga, T. Funkhauser, D. Yanovsky, I. Carlbom, “Sea Of Images”, IEEE Visualization, pp. 331-338, Id. It has become. Although still in its infancy, in many cases the algorithm first performs a geo-reference of the panoramic frame, then distorts and combines the source images based on feature matching, and a new Proceed to synthesize the viewpoint.
[0004]
Japanese Patent Publication No. 2003-141562, filed by the same applicant as the present application, discloses an image processing apparatus and method applicable to compression, accumulation and reproduction of cylindrical or spherical omnidirectional images mapped to three-dimensional coordinates. Has been.
[0005]
[Problems to be solved by the invention]
Depending on the use of a virtual walkthrough system, it may be necessary to capture the entire area of an indoor environment, such as a building, in order to render a virtual surround environment. For example, if an omnidirectional video camera is used to image the indoor environment of a building, if the omnidirectional video camera captures 60 frames per second and it takes 60 minutes to move within the building, the total number of image frames The number of sheets reaches 216,000.
[0006]
Some captured surround images do not bring much new information as compared to others. For example, when moving in the middle of a corridor surrounded by walls, there is not much visible change in the surround environment. Rendering the hallway of a virtual walk-through system in such a place requires less omnidirectional image data than other places in a building where more environmental changes are seen, such as the corners of a hallway or the entrance of a room. Needed.
[0007]
The present invention has been conceived in view of the above situation. It is desirable to provide an apparatus and method that can process omnidirectional image data and that can efficiently reduce the data size of a surround image obtained by processing omnidirectional image data.
[0008]
Furthermore, it would be desirable to provide an apparatus and method that can sample / filter omnidirectional image data, or that can properly compress surround image data used in an application.
[0009]
[Means for Solving the Problems]
According to one embodiment of the present invention, an image processing apparatus for omnidirectional image data processing is provided. The image processing apparatus relates to a position detection unit that detects a position where an omnidirectional image is captured; and a geometric configuration of a surround environment, and the omnidirectional image is captured from the detection position with respect to the position detection unit. A spatial coefficient detector that calculates a spatial coefficient corresponding to the position, and a filter that reduces the processed image data to be output from the image processing apparatus based on the calculated spatial coefficient.
[0010]
The spatial coefficient may be a curve length of a path in which an omnidirectional image is captured, or is determined based on a plurality of visibility envelopes or cells calculated in the path. May be. Alternatively, the spatial coefficient may be determined according to a parameter expression of the texture of the surround image corresponding to a set of visibility polyhedrons calculated in the path (a set of polyhedra).
[0011]
The filter unit may determine the sampling rate of the omnidirectional image or select the omnidirectional image according to the spatial coefficient. Alternatively, the filter unit may compress the processed image data according to the spatial coefficient.
[0012]
Further, the processing in the image processing apparatus may include addition of an event / trigger area to be superimposed on the processed image. The event trigger area is arranged at a predetermined position of the surround image obtained by processing the omnidirectional image data, and is configured to trigger a predetermined event when the area is selected by a user operation. Further, this processing may include mapping of a moving image texture and / or a three-dimensional object onto a surround image obtained by processing omnidirectional image data.
[0013]
The image processing apparatus may be provided with an omnidirectional image capturing unit.
[0014]
According to another embodiment of the present invention, an image processing method or computer program for omnidirectional image data processing is provided. The image processing method or the computer program detects the position at which the omnidirectional image is captured; the detection is related to the geometric configuration of the surround environment, and the omnidirectional image is captured from the detected position with respect thereto. Calculating a spatial coefficient corresponding to the calculated position; and reducing the processed image data to be output from the image processing apparatus based on the calculated spatial coefficient.
[0015]
DETAILED DESCRIPTION OF THE INVENTION
An apparatus according to an embodiment of the present invention will be described with reference to the drawings.
[0016]
FIG. 1 shows an example of an omnidirectional camera (omnidirectional camera) 10. The omnidirectional camera 10 includes a substantially regular dodecahedron-shaped frame composed of 12 pentagonal surfaces and 11 cameras mounted separately on different surfaces. Each camera captures a corresponding area in the surround scene and outputs data as part of the surround image. An omnidirectional image of an omnidirectional type is obtained by pasting these image portions.
[0017]
FIG. 2 is a schematic diagram of a system including the omnidirectional camera 10 and the omnidirectional image processing apparatus 100. As shown in FIG. 2, the omnidirectional camera 10 includes a plurality of VTRs 12 and microphones 13 in addition to the plurality of cameras 11, and records a plurality of video streams and audio signals. The recorded video stream is video-captured by the switcher 14 and output to the image processing apparatus 100 as computer data (for example, a bitmap file). The image processing apparatus 100 performs image processing such as preparation of a captured image for next image processing, image pasting, and accumulation of processed images.
[0018]
Further, in the present invention, the device 100 performs filtering of the captured image or compression of the processed image according to the importance of the captured image. In the present embodiment, the degree of importance is measured by a spatial coefficient calculated along the path of the omnidirectional camera 10. Details of the filtering / compression method according to this embodiment will be described later.
[0019]
The image processing apparatus 100 can be realized by a computer system having the configuration shown in FIG. The apparatus 100 includes a CPU 101, a memory 102, a display controller 103, an input device interface 104, a network interface 105, an external device interface 107, a bus 108, a video capture card 109, a display device 111, and a keyboard. 112, a mouse 113, a hard disk drive 114, and a media drive 115.
[0020]
The image processing apparatus 100 downloads various applications for image processing, downloads omnidirectional images instead of receiving them via the video capture card 109, or distributes processed image data over a network. It may be connected to the LAN or the Internet 120 via the network interface 105.
[0021]
The CPU 101 executes various applications such as pasting of a plurality of images output from the omnidirectional camera 10 via the video capture card 109 and image filtering / compression according to a spatial coefficient.
[0022]
FIG. 4 shows an example of a functional schematic diagram of the CPU 101 in an operation mode in which a plurality of captured images are processed to calculate a surround image. In this operation mode, omnidirectional image data including a plurality of captured image data is provided from the omnidirectional camera 10 to the surround image calculation unit 401.
[0023]
The surround image calculation unit 401 calculates and outputs a panoramic image or a surround image from the omnidirectional image data. When a computer graphic (CG) script is used, the unit 401 directly calculates a surround image from the CG script.
[0024]
The position detection unit 402 performs egogo recovery using the panoramic image for positioning the camera 10. The egomotion recovery process can be considerably simplified by using a fixed-height camera. Detailed description of egomotion recovery will be described later.
[0025]
The position of the camera 10 is output to the spatial coefficient detection unit 403. The spatial coefficient detection unit 403 detects a spatial coefficient at the position detected by the position detection unit 402. In the present invention, the spatial coefficient is introduced to measure the level of importance (significance) of the surround environment where the panoramic image is taken. The spatial coefficient is calculated based on, for example, (a) curve length unit, (b) visibility cell, (c) visibility envelope, or (d) texture parameter expression. When using a visibility cell or envelope, the volume and area attributes for all positions in the path are calculated. Envelope combination changes occur only at the critical visibility vents defined by the visibility graph (see FIG. 8). A detailed description of the visibility cell and envelope will be given later in this specification.
[0026]
The filter processing unit 404 performs surround image filter processing based on the spatial coefficient calculated by the spatial coefficient detection unit 403. For example, a large number of surround images may be sampled first, and then reduced by the filter processing unit 404. Or you may make it control the timing which samples the omnidirectional image data provided from the omnidirectional camera 10 based on a space coefficient instead of a filter process. Furthermore, each omnidirectional image may be weighted according to a spatial coefficient and compressed together with other images according to the weighting.
[0027]
Next, a method according to another embodiment of the present invention will be described. The method includes the steps of imaging, spatially filtering and viewing an annotated video taken with an omnidirectional multihead camera moving along a path.
[0028]
The following sections describe in detail the egomotion recovery algorithm designed for calculating the path of an omnidirectional camera and the efficient sampling of the plenoptic path based on geometric visibility vents. To do. With proper sampling, panoramic images can be filtered and compressed, redundancy is avoided, and memory space in the image database can be saved. Also described are some applications and results of plenoptic paths obtained by indoor shooting or fully rendered by computer graphics.
[0029]
1. Plenoptic functions and paths
The concept of plenoptics (M. Levoy, P. Hanrahan. “Light field rendering”, ACM SIGGRAPH, pp. 31-42, 1996; S. J. Gartler, R. Grzezczuk, R. Gzezzczk. Cohen, “The Lumigraph”, ACM SIGGRAPH, pp. 43-54, 1996) is a three-dimensional space E in the spectral response at any time t, in any direction (θ, φ) and at any wavelength λ.³If the seventh-order function L (•) = L (X, Y, Z, θ, φ, t, λ) associated with each Cartesian coordinate point (X, Y, Z) in FIG. It is based on the observation that modeling can be bypassed and interactive rendering can be provided by “rendering” directly from the plenoptic function L (•).
[0030]
In practice, freeze the time (ie, consider a static environment), select one wavelength (one color channel, eg red), and camera the ray sampling (X, Y, Z, θ, φ) By limiting to the path, the high dimensionality of this function is relaxed, that is, the one-dimensional plenoptic path P = {P_i= (X, y, z)_i}.
[0031]
By moving the omnidirectional camera along the path, the sampling of P can be forcibly performed, so that P is a set of points p P = {P_i}_{i = 1, p}As a discretization. L_{｜ P}From (3) three-dimensional discontinuous sampling, consider inverse rendering problems (eg, elucidation of macro / meso geometry and texture attributes, lighting conditions, etc.), or view synthesis It is possible to either proceed with extrapolation of the function for Since a large amount of sampling photography is time-consuming and cumbersome, this work may be performed using a power robot that moves a panoramic head equipped with an omnidirectional camera. Examples of such systems are presented in the following literature (DG Aliaga, T. Funkhauser, D. Yanovsky, I. Carlbom, “Sea Of Images”, IEEE Visualization, pp. 331-338. D. G. Aliaga, D. Yanovsky, T. Funkhauser, I. Carlbom, “Interactive Image-Based Rendering Using Feature Globalization”, ACM Symp. 3 ACM Symp.
[0032]
2. Capture plenoptic path
2.1 Multi-head camera
As an omnidirectional camera, a multi-head camera as shown in FIG. 1 can be used for omnidirectional video photography. The panoramic head consists of 10 CCD NTSC block cameras that share approximately the same optical center point (ie, nodal point), and displays 60 omnidirectional images (4π steradian angle) per second that overlap each other. Shoot with interlaced frames.
[0033]
Although an omnidirectional camera is used in this embodiment, it is not always necessary to provide a complete omnidirectional camera that covers 4π steradian angles. Depending on the final application, the angle range covered by the camera or image data used in the present invention may be 4π steradian angle or less.
[0034]
A surround environment may be freely photographed by mounting a camera and a recording system on a transport vehicle and driving the battery with a battery. The transport vehicle may be provided with a motor and a drive mechanism, or may be pushed by a human hand. The algorithm for stitching, viewing and coding the surround environment map is It is described in detail in the following article by Nielsen ("High Resolution Full Spheroidal Videos", IEEE Intl. Conf. On Information Technology; Coding and Computing, pp. 260-267). High resolution panoramic frame I on the order of p∝10000_i, I∈ {1,..., P}. For example, a 2048 × 1024 image size may be used as an equirectangular panorama. FIGS. 5A and 5B show examples of a regular hexahedral surround environment map and a regular rectangular surround environment map generated from omnidirectional image data captured by a multi-head camera.
[0035]
2.2 Computer graphics
Alternatively, omnidirectional images and plenoptic paths may be calculated using ray tracing or radiosity software. For example, since the source code is disclosed and considerable CG scripts are available, the surround image may be calculated using POV-Ray (trademark). That is, to define a bijective function that maps the corresponding angular coordinates (θ, ψ) to each pixel (x, y) of the CG image and output our image format The process “create_ray” of the file “render. Cpp” is changed. The example of the surround image shown in FIG. 18 was calculated from a CG script obtained from the site Internet Ray Tracing Competition (IRTC).
[0036]
Conventional tools such as the Alias Wavefront Maya ™ or Discreet 3DMAX ™ used in the computer graphics industry can also be used to output surround images using their rendering device or API.
[0037]
A CG image is a complete virtual surround camera (ie, there is no physical interference between devices and no actual camera device is required) and an accurate image (with constant illumination, no parallax, no noise, no vibrations, etc.) The operation using the CG image is useful for the mutual performance evaluation (benchmark) or the like.
[0038]
3. Egomotion recovery
An algorithm for recovering the extrinsic position of the camera according to the present embodiment will be described. Since panoramic images have no intrinsic parameters, we end up with a spatially indexed Euclidean path (defined up to a magnification factor) of the panoramic image.
[0039]
A GPS system may be used to add general annotations to videos of large panoramic paths such as outdoor environments. However, since a general GPS system is given only a rough position, it cannot be used for visual field synthesis (M. Hirose, “Space Recording Usage Augmented Technology”, Int. Mixed Reality Symp., Pp105-110; D. Kimber, J. Foote, S. Lertisichai, “FlyAbout; Spatially Indexed Panoramic Video”, ACM Multimedia 2001, pp. 339-341). The vision algorithm has recently proved useful in an industry that requires moving matching where the virtual and real camera paths must be matched to produce a visual effect.
[0040]
Based on feature tracking, omnidirectional image position tags (x, y, θ)_i, Iε {1,..., P}. Hereinafter, a simple and high-speed global egomotion algorithm when the omnidirectional camera is limited to move on a fixed height surface will be described. According to the egomotion algorithm, the position on the path is registered in the surround image.
[0041]
The algorithm includes the following steps.
・ Rough rotation sequence θ_i(Pixel based method),
-Outline parallel movement sequence (x, y)_i(Feature based method),
-By performing high density global optimization based on initial estimates (x, y, θ)_iFine-tuning (a feature-based method that takes all parameters into account at the same time: relative path),
• Path fixation (absolute path) using two or more landmarks.
[0042]
We do not need to indicate the landmark to be tracked, nor do we need to pre-initialize the path, which makes the acquisition system flexible, scalable and easy to use.
[0043]
3.1 Coarse orientation measurement
Absolute orientation refers to “north” of the panoramic image. Although simple, the algorithm works well without feature matching. For example, when a plurality of images are in a regular rectangular format (also called latitude-longitude) with dimensions w × h, each panorama frame I_iIn contrast, each column pixel is averaged to correspond to the corresponding one-dimensional ring image R._iIt becomes.
[0044]
Next, ring image R_i, R_{i + 1}Is continuously registered and the azimuth shift (resulting in sub-pixel accuracy (BD Lucas, T. Kanade, “An iterative Image Registration Technology and Application to Stereo Vision,” 7th Int. Intelligence (IJCAI), pp. 676-679, 1981)). The pixel unit in the ring image corresponds to a shift of 2π / ω radians. By weighting the pixels according to the angle length inherent in the vertical direction corresponding to the pixels, the column can be averaged within a limited latitude range.
[0045]
3.2 Coarse parallel movement
Once the approximate orientation of the image is determined, the sequence (x, y)_iIs determined. Similar to most configurations in motion algorithms, the Euclidean 3D point set is reconstructed. First, the one-dimensional parallel movement is calculated as follows. Initially, where the orientation does not change much, the path is divided into segments (continuous image sequences) (using an adjusted threshold). Segment length λ_iIf is not yet calculated, a polyline is defined. Details of the calculation algorithm will be described later with reference to FIG.
[0046]
Image I at both ends of unity length k + 1_dAnd I_{d + k}About I_d... I_{d + k}From the features tracked in common (see FIG. 6), the translation parameter λ_dIs calculated using standard numerical analysis methods. By converting from pole to Cartesian (θ, λ)_iTo sequence (x, y)_iIs required. 6 (a) -6 (c) show examples of features marked in an image tracked from a plenoptic path sequence.
[0047]
3.3 Fine parameter adjustment
Sequence (x, y, θ)_iIs numerically improved by appropriately correlating rotation and translation in a manner similar to that described in the following document (CJ Taylor, “VideoPlus; A Method for Capturing the Structure of Appealance of Immrive Environments ", IEEE Trans. On Visualization and Computer Graphics, Vol. 8 (2), pp. 171-182, 2002; Journal of omputer Vision, Vol. 49 (2/3), pp. 143-174, 2002). Although important in visibility synthesis applications, here a rough analysis of combinatorial events occurring along the path is performed, and this final step does not greatly improve the filtering process.
[0048]
3.4 Absolute position determination
Use two or more landmarks set by the user (eg, using a reference floor map) to match the size and origin of the available geometric information provided by the floor map, rough reconstruction, etc. Thus, the path is fixed using the rotation and parallel movement parameters.
[0049]
In the present invention, egomotion recovery is not limited to the above method. If the method can define the reference angles (for example, the north pole) and (x, y) positions of all surround images, Other methods using physical devices, viewpoint tracking devices, fiducials, distance meters, and the like may be used. Note that this step is not necessary for computer graphics scripts. This is because each CG surround image is calculated for each predetermined position.
[0050]
4). Spatial filtering of plenoptic paths
Once a plenoptic path is sought, the plenoptic path can be properly sampled / annotated so that “redundant” images (images that do not bring much new information) or less important images can be removed. Done. The sampling is also useful for progressive coding such as plenoptic paths.
[0051]
Sampling of plenoptic functions was studied by Chai et al. (J.-X.Chai, H.-Y.Shum, XT “Plenoptic sampling”, ACM SIGGRAPH, pp. 307-318, 2000). Chai et al. Studied the sampling method of L (•) using spectral analysis to determine the minimum sampling rate for light field rendering. The present invention, on the other hand, is a subset of the plenoptic function and is particularly adapted for plenoptic paths that have been geometrically partitioned.
[0052]
One way to reduce the number of surround images is to view the viewpoint P along the path P in proportion to the curve length._iIs selected / distributed (parameter expression of path length). This method is efficient when geometric information is not available. If l indicates the path length P, l is calculated from the relative translation parameter of the surround image by Σ_i{(Tx_{i + 1}-Tx_i)²+ (Ty_{i + 1}-Ty_i)²}^1/2Is defined as having
[0053]
If geometric information is not available, it is preferable to sample according to l. For example, if m subset images need to be selected from n recorded surround images, the path P is divided into m equal length intervals, one surround within each interval. An image is selected. By doing so, it is possible to compensate for non-uniform artifacts that occur during acquisition. For example, increasing the speed of the omnidirectional camera transporter can provide coarse data, while decreasing the speed increases the sampling rate.
[0054]
Another way to reduce the number of surround images is to use a visibility cell. In the following, a geometric analysis method for combination events occurring along P will be introduced.
[0055]
F represents a “free” space, ie the space is a scene S = {S₁,. . . , S_n} Is not obstructed by any of the n objects. F = E³\ ∪ⁿ _{i = 1}S_i(E³Is Euclidean three-dimensional space). Let ν⊆F be a “visible” space, ie a part of the free space where the user can move while interacting. Given a position Pεν, ε (P) shall denote the lower envelope surrounding P (for geometric terms, see JD Boissonnat, M. Yvinec, “Algorithmic geometry Press”, Cambridge University Press, 1998). That is, ε (P) is the distance from the position P to the object S first hit by a ray emitted in the direction (θ, φ) with respect to a given (θ, φ) angular coordinate, and that object exists. In this case, it is defined as a radial function r (θ, φ) for obtaining the distance. Note that ε (·) is not necessarily continuous.
[0056]
FIG. 7 shows an example of the envelope ε (O), where O indicates the center. A thick solid line 700 is a polyline describing a scene. The dotted star-shaped polygon is an envelope ε (O) originating from the center O. The envelope ε (·) changes as the position in the plenoptic path moves. For example, FIGS. 9A and 9B show changes in the envelope ε (·). The shaded portion of the figure shows the envelope ε (•) at different positions of the plenoptic path in the building 900.
[0057]
If P is moved slightly, the envelope ε (P) changes smoothly until a critical combination event defined by a visibility event (occlusion / disocclusion) is reached. Let A (S) be the division of ν into visibility cell elements. FIG. 8 is a schematic diagram showing the visibility diagram and its cell decomposition. In the figure, numeral 800 indicates one of the visibility cells.
[0058]
S is E³A (S) is O (n⁹) With complexity. However, when limited to two-dimensional cells cut by a straight line, the complexity is O (n³) And become easy to treat. Zone theory (JD Boisssonnat, M. Yvinec,
According to “Algorithmic geometry”, Cambridge University Press, 1998), the size of all envelope combinations cut in a straight line is O (n³)become.
[0059]
It should be noted that when the visibility cell is constrained to have a minimum “width”, its complexity is linear, ie proportional to the path length. For indoor photography, a floor map of a building is often used in a drawing exchange format (DXF format), and a visibility diagram as shown in FIG. 8 can be calculated. Here, it should be noted that for a small n, it is possible to calculate a restriction on a polyline by applying a prime quadratic algorithm. For a visibility cell with volume v, length l and a plenoptic path length intersecting it, the path in this cell is the ratio l³Sampling may be performed according to / v.
[0060]
Further, in a portion where the visibility information changes greatly (that is, when moving from one large visibility cell to another), the transition may be made smoother by locally increasing the sampling rate. (For example, if a wall is tangent to the current viewpoint, we need more samples because one side of the wall becomes clear when the camera moves in parallel, even a little.)
In this way, the image database can be reduced on the order of one or two digits while maintaining its semantics.
[0061]
Even for a very large n complex CG script having a non-trivial visibility diagram, the current visibility cell volume v is roughly estimated as follows.
[0062]
Color and depth information d for each pixel_{i, j}Each pixel e of the RGBZ environment image_{i, j}, Is the corresponding solid angle a_{i, j}Hang. Since the solid angle subdivides all unit spheres, these divisions are simply added and v = (1 / 4π) Σ_{i, j}a_{i, j}d_{i, j}Get.
[0063]
If the volume changes greatly, this means that the viewpoint is moving from one visibility cell to another, but the first combination event is searched along a given plenoptic path, Sampled (ie, renders P). Sampling for the CG script may be performed online because the next best viewpoint is additionally determined sequentially along its plenoptic path P.
[0064]
The above algorithm using the visibility cell is realized by the steps shown in FIG. 10, for example. In this example step, first, an omnidirectional image I is taken on the plenoptic path (step 1001). Next, in step 1002, the position of the plenoptic path is determined using, for example, the above-described egomotion recovery technique. In step 1003, a visibility cell along the plenoptic path is calculated. In step 1004, an image is assigned to the corresponding visibility cell. Finally, an image is selected according to v and l. Where v is the volume of the visibility cell and l is the length of the plenoptic path that intersects the visibility cell. In the setting, the importance of the image is set according to the combination change occurring in the scene.
[0065]
Another possibility is to calculate the volume of all envelopes at the acquired image position (tx, ty) and select or weight the image according to the volume of the envelope or the derivative of the envelope. . It is desirable to obtain more samples or better quality images with a larger envelope volume, while it is preferred to reduce fewer images or per image quality with a smaller envelope volume. Furthermore, combinations of continuous and combinatorial sampling techniques may be combined.
[0066]
A simple and efficient way to calculate the visibility diagram of an object set is to first rasterize primitives (lines, arcs, etc.) into a color image. Here, each primitive has a different color number (step 1101 in FIG. 11). Next, a discrete envelope as shown in FIG. 12 is calculated for each pixel position (px, py) of the image at a location where the object (detected as the background color) is not drawn (step 1102). ). Each envelope is annotated with an object sequence. Since the sequence is a cyclic sequence, it is classified in increasing order. It can be seen that the envelope has a different annotation on the boundary of the two visibility cells. Therefore, the calculation is performed within the annotation line (line by line). Each time an annotation between successive annotations is different, a color pixel corresponding to the visibility diagram is written (step 1103). Once individual visibility diagrams are calculated, individual areas or volumes can be calculated for each cell (step 1104).
[0067]
Calculate the volume v of the visibility cell and calculate the ratio l³Instead of sampling in proportion to / v, the area a of the visibility cell is calculated and I²The plenoptic path may be sampled in proportion to / a.
[0068]
Furthermore, the texture parameter representation for the visibility polyhedron may be used to control the filtering / selection of the surround image. The parameter expression of the texture is performed as follows. For a given visibility polyhedron and accuracy δ, all images corresponding to that visibility polyhedron are mapped. Each visibility polyhedron has a topology genus of 0, and there is a parameter representation with an accuracy of approximately δ in the texture for it. For each surround image corresponding to the visibility polyhedron, the surround image is inverse-mapped to a parameter representation of the texture. Finally, all the reverse mappings are averaged.
[0069]
The use of a texture parametric representation is useful in plenoptic path regions (such as corridors) where a rough three-dimensional reconstruction can provide sufficient quality.
[0070]
Alternatively, instead of reducing the number of panoramic images, for each image i, a weighting factor w according to the curve length or envelope volume / area_iAssign all images then_iDepending on, compression may be performed irreversibly.
[0071]
5. Application and results
The experimental results for the captured / synthesized sequence are shown in FIG. FIG. 13 shows a plenoptic path 1300, several main frames 1302, and their corresponding positions 1301.
[0072]
The implementation was performed in C ++ using OpenGL ™ on a general PC of Windows ™ / Intel ™. Using a mouse in a viewing / map window as shown in FIG. 14 or using a head mounting display (HMD) equipped with a gyro, the user interacts with the player at a refresh rate of 60 fps. You can move on the noptic path.
[0073]
For example, as shown in FIG. 14, the viewing / map window 1400 may include a viewing window 1401, a map window 1402, and a navigation window 1403. A viewing window 1401 displays an image for the current viewpoint position and angle. A map window 1402 displays the plenoptic path and the current position superimposed on the floor map. The navigation window 1403 displays a surround image at the current position and a pseudo frame 1405 superimposed on the panorama image to indicate a corresponding area of the viewing window 1401. When using the HMD, the inclination of the HMD may be used to indicate whether to move the viewpoint forward or backward.
[0074]
By using multimedia add-ons such as movie textures (corrected and superimposed using homography) or image-based operations triggered in response to events, the virtual walk-through experience can be enriched. For example, it is possible to select to open and close the elevator door by pressing an elevator button.
[0075]
Examples of such multimedia add-ons are shown in FIGS. 15 (a) -15 (b). In FIG. 15A, a triggered region of a certain surround image is set by a polygon area 1501 (here, a rectangle). A polygon area 1501 indicates a trigger area of a virtual pinhole field of view synthesized from a surround image using homography. When the corresponding trigger areas intersect (that is, when the area 1501 and the area 1511 intersect as shown in FIG. 15B), a predetermined event occurs. These events may be performance of music or drawing of a three-dimensional object.
[0076]
FIG. 16 is a flowchart for explaining how an area 1501 is defined for a set of surround images. Initially, the user initially sets four points in the first surround image (step 1601). The four points are then tracked along the surround image sequence (step 1602). Finally, the tracking is performed more firmly by registering together the quadrilaterals obtained in each image (step 1603). This step avoids jitter effects. Finally, the number of surround images corresponding to the rectangular coordinates is stored in the XML file format (step 1604).
[0077]
Another example of such a multimedia add-on is shown in FIG. When the three-dimensional object 1702 is given to the viewpoint p and the position q on the plenoptic path, the following synthesis step is performed.
[0078]
Calculate the portion of the object 1702 that is visible from the viewpoint p. This step is performed on the basis of rough visibility information obtained from rough reconstruction, a floor map, or the like. I_OIs the object image. I_OIs the part of the object being drawn_OAn alpha channel (mask image) is provided for identifying the location within and separating the object from the background.
[0079]
Generate a virtual camera field of view from the surround image for viewpoint position p (eg, virtual pinhole camera field of view). I_CIs the image.
[0080]
・ First I_CAnd then using the alpha channel I_ODraw. By doing so, it becomes possible to obtain transparency effects, such as drawing of a window glass, for example.
[0081]
The technique described here can be extended to many objects. For example, it is possible to add several robots (according to CG images) that enter the hallway using this technique.
[0082]
The above technique is different from the function of tracking a surround image and mapping another image on it by homography. Here, the superimposed CG object is three-dimensional.
[0083]
A framework for capturing or synthesizing a plenoptic path in which sampling is controlled based on spatial coefficients, as described using the embodiments of the present invention. In the present invention, an image-based rendering browser is introduced so that the user can interact through these plenoptic paths. Although the above-described embodiments are mostly related to virtual walk-through systems, the present invention is applicable to any other application, such as games or telepresence.
[0084]
【The invention's effect】
According to the present invention, there is provided an apparatus and method capable of processing omnidirectional image data and effectively reducing the data size of a surround image obtained by processing the omnidirectional image data.
[0085]
Furthermore, according to the present invention, there is provided an apparatus and method capable of sampling / filtering omnidirectional image data or compressing surround image data used in an application.
[Brief description of the drawings]
FIG. 1 is an overview diagram showing an example of an omnidirectional camera.
FIG. 2 is a block diagram showing an omnidirectional image processing system.
FIG. 3 is a block diagram showing a configuration of an omnidirectional processing apparatus according to the present invention.
4 is a schematic functional diagram showing functional blocks according to an embodiment of the present invention in the omnidirectional processing apparatus of FIG. 3. FIG.
FIG. 5A is a diagram illustrating an example of a cube environment map image.
FIG. 5B is a diagram showing an example of a regular rectangular environment map image.
FIG. 6A is a diagram showing an example of features marked in an image and tracked from a plenoptic path sequence.
FIG. 6B is a diagram showing an example of the feature marked in the image and tracked from the plenoptic path sequence.
FIG. 6C is a diagram showing an example of the feature marked in the image and tracked from the plenoptic path sequence.
FIG. 7 is a schematic diagram illustrating an example of an envelope calculated at a certain position in a path.
FIG. 8 is a schematic diagram showing an example of a visibility diagram calculated along a path.
9A is a schematic diagram showing the envelope of FIG. 7 calculated at a certain position in the path.
FIG. 9B is a schematic diagram showing the envelope of FIG. 7 calculated at another position in the same path of FIG. 9A.
FIG. 10 is a flowchart showing an example of image filtering processing for each visibility cell calculated along a path.
FIG. 11 is a flowchart showing an example of steps for calculating a visibility diagram of an object set.
FIG. 12 is a schematic diagram showing an example of how annotation is made within an individual envelope.
FIG. 13 is a schematic diagram showing an example of a plenoptic path and some examples of main frame images at corresponding positions.
FIG. 14 is a schematic diagram illustrating an example of a viewing / mapping window according to an embodiment of the present invention.
FIG. 15A is a schematic diagram showing an example of an event-triggered area according to the embodiment of the present invention.
FIG. 15B is a schematic diagram showing the event-triggered region in FIG. 15A when a certain event is triggered.
FIG. 16 is a flowchart showing an example of an event triggered region setting step of FIG.
FIG. 17 is a schematic diagram illustrating an example of how to synthesize a three-dimensional object image in a surround image according to an embodiment of the present invention.
FIG. 18 is a schematic diagram illustrating an example of a surround image calculated from a CG script.
[Explanation of symbols]
401: Surround image synthesizing device, 402: Position detection unit, 403: Spatial coefficient detection unit, 404: Filter unit.

Claims (13)

In an image processing apparatus that processes omnidirectional image data,
A position detection unit that detects a position where the omnidirectional image is captured;
A spatial coefficient detector for calculating a spatial coefficient corresponding to the detected position, in which an omnidirectional image is captured from the detected position with respect to the geometric configuration of the surround environment;
An image processing apparatus comprising: a filter unit configured to reduce processed image data to be output from the image processing apparatus based on the calculated spatial coefficient. The image processing apparatus according to claim 1, wherein the spatial coefficient is a curve length of a path where an omnidirectional image is captured. The image processing apparatus according to claim 1, wherein the spatial coefficient is determined according to a plurality of visibility envelopes calculated in a path where an omnidirectional image is captured. The image processing apparatus according to claim 1, wherein the spatial coefficient is determined according to a plurality of visibility cells calculated in a path where an omnidirectional image is captured. The image processing apparatus according to claim 1, wherein the spatial coefficient is determined according to a texture parameter expression of a surround image corresponding to a set of polyhedrons calculated in a path where an omnidirectional image is captured. The image processing apparatus according to claim 1, wherein the filter unit determines a sampling rate of the omnidirectional image according to the spatial coefficient. The image processing apparatus according to claim 1, wherein the filter unit selects the omnidirectional image according to the spatial coefficient. The image processing apparatus according to claim 1, wherein the filter unit compresses the processed image data according to the spatial coefficient. The processing of the omnidirectional image data includes the addition of an event trigger region to be superimposed on the processed image,
The event trigger area is arranged at a predetermined position of the surround image obtained by processing the omnidirectional image data, and a predetermined event is triggered when the area is selected by a user operation. The image processing apparatus according to claim 1, wherein: The processing of the omnidirectional image data includes mapping of a moving image texture and / or a three-dimensional object on a surround image obtained by the processing of the omnidirectional image data. Image processing apparatus. In an apparatus including an omnidirectional image capturing unit and an image processing unit,
The image processing apparatus according to claim 1, wherein the image processing unit includes the image processing apparatus according to claim 1. In an image processing method for processing omnidirectional image data,
Detecting the position where the omnidirectional image is taken,
Calculating a spatial coefficient corresponding to the detected position, for which an omnidirectional image is taken from the detected position with respect to the geometric configuration of the surround environment;
An image processing method comprising a step of reducing processed image data to be output from the image processing apparatus based on the calculated spatial coefficient. In a computer program for executing an image processing method for processing omnidirectional image data in a computer, the image processing method detects a position where the omnidirectional image is captured,
Calculating a spatial coefficient corresponding to the detected position, for which an omnidirectional image is taken from the detected position with respect to the geometric configuration of the surround environment;
A computer program comprising causing reduction of processed image data to be output from the image processing device based on the calculated spatial coefficient.

Images