JP3660108B2 - Image storage method and machine-readable medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to processing of an electronically expressed image, and more particularly to the field of combining and combining images taken from different orientations with a digital camera or the like.
[0002]
[Prior art]
Since photography devices such as digital cameras have limited fields of view and resolution, in order to obtain higher resolution images, wide-angle images, panoramic images, etc., multiple images taken from different directions are combined and combined. The method to do is effective.
[0003]
Taking a panoramic image as an example, at present, only a panoramic image in a single direction (direction of drawing a circle on a celestial sphere when viewed from a certain point) can be obtained by pasting a plurality of images. An example of such a panoramic image creation method will be briefly described with reference to FIG.
[0004]
Each pixel (a, b, etc.) of the image G taken by the digital camera from a certain point is projected onto the cylindrical surface 300 with the lens center c of the digital camera as the central axis. a ′ and b ′ are points projected on the cylindrical surface 300 of the pixels a and b. While the lens center c of the digital camera is fixed, the 360 ° scene is moved while moving the optical axis of the digital camera parallel to the plane including Oa ′ (usually the horizontal plane), that is, swinging the digital camera in that direction. Divided shooting is performed, and pixels of an image shot in each direction are projected onto the cylindrical surface 300. By projecting a plurality of images onto the cylindrical surface 300 and pasting them together, a 360 ° panoramic image can be synthesized. When displaying this panoramic image on the screen of the display, a plane perpendicular to the line-of-sight direction to be observed is defined, and the pixels on the cylindrical surface 300 are reprojected on this plane. For storage and management of image data, a memory space corresponding to a rectangle or square obtained by developing a cylindrical surface 300 on which a plurality of images are projected is prepared, and information on all pixels projected on the cylindrical surface 300 is stored here. By storing the data, 360 ° panoramic image data can be collectively stored in the memory and efficiently managed.
[0005]
Unidirectional panoramic images can be synthesized by the method described above, and the image data can be efficiently stored and managed collectively. However, a plurality of images taken from any orientation around the camera position are pasted together to synthesize a three-dimensional panoramic image that covers the whole sky, for example, and the image data can be efficiently and collectively A practical technique for storing and managing has not been known so far.
[0006]
[Problems to be solved by the invention]
Regarding the composition of the above-described three-dimensional panoramic image, etc., the inventors project a plurality of images taken with a digital camera or the like from the same point onto a spherical surface centered on the lens center of the camera and paste them together. A method has already been proposed (Japanese Patent Application No. 9-166737).
[0007]
When this method is implemented, there remains a problem in storing and managing image data. In other words, since a spherical surface cannot be developed as easily as a cylindrical surface, in a simple method such as image data projected on a cylindrical surface, the data of images projected and pasted on the spherical surface are batched. It cannot be stored and managed efficiently.
[0008]
The data of each image before composition is stored separately in the memory, and when you want to observe the image in a certain direction, project and paste the images contained in the field of view of that direction onto the spherical surface, and re-display it on the display plane. It is also possible to employ a method of projecting. However, if image composition is performed each time it is displayed, it takes time to display, and it is difficult to observe images while continuously changing the orientation within a short time.
[0009]
In order to enable high-speed image display, image synthesis is performed in advance for many azimuths at a certain spatial angular interval, and data of each synthesized image is stored separately in a memory, and a synthesized image of the azimuth to be observed is stored. A display method is also conceivable. However, this method requires a large amount of memory for storing image data, and management of image data is troublesome. Furthermore, it is difficult to display an image having a direction that is not prepared in advance.
[0010]
Accordingly, an object of the present invention is to provide a method that enables efficient storage / management and high-speed image reproduction with a small memory amount such as the above-described three-dimensional panoramic image.
[0011]
[Means for Solving the Problems]
According to the present invention, an image projected on a spherical surface is projected onto a plane intersecting with the spherical surface, and the data of the projected projection image is collectively stored in a memory or a recording medium.
[0012]
According to this method, for example, data of a three-dimensional panoramic image obtained by projecting a plurality of images photographed in an arbitrary direction from the same position with a digital camera or the like onto a spherical surface and pasting them together is stored on a memory or a recording medium. Save and manage as one file. In this way, compared to the method of saving composite images of various orientations as separate files, the amount of memory required is much smaller, and the management of image data on the memory or recording medium becomes easier. . In addition, the image on the memory or the recording medium (the panoramic image that has already been synthesized) is re-projected onto the spherical surface, and the necessary field of view is further projected onto the display plane, so that the image can be displayed at high speed in any orientation. Can do.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
[0014]
FIG. 1 is a flowchart showing an outline of processing according to an embodiment of the present invention. 2, 3 and 4 are diagrams for explaining the processing.
[0015]
First, in the first step S1, for example, a plurality of images (referred to as elemental images) taken from different points from one point with a digital camera are equal to the distance between the lens of the digital camera and the imaging surface as shown in FIG. Projecting and pasting onto a spherical surface 100 having a radius R and having the center O as the center of the lens, a single composite image is created on the spherical surface 100. In FIG. 2, F1 is one element image projected onto the spherical surface 100, and p is one pixel thereof. N is a spherical pole directly above the center O and is called the north pole here. S is a spherical pole directly below the center O, and is called the south pole here. Reference numeral 102 denotes a vertical line connecting the north pole N and the south pole S, which is called a polar axis here. Reference numeral 104 denotes a horizontal plane passing through the center O, which is referred to herein as an equatorial plane. Q is an intersection of a longitude line passing through the projected point of the pixel p and the equator plane 104. The line segment Ox is a longitude reference line, and indicates, for example, the east direction. φ is an angle formed by the line segment OQ with respect to the reference line Ox, and is called the longitude of the pixel p. θ is an angle formed between the line of sight of the pixel p from the center O and the polar axis 102. In general, the elevation angle from the equator plane is called latitude, but here the angle θ with the polar axis 102 is called the latitude of the pixel p. That is, θ = 90 ° − [latitude of general meaning].
[0016]
Specifically, in this step S1, for example, the following coordinate calculation is performed. In general, the position of each pixel of an element image photographed by a digital camera is represented by an orthogonal coordinate system with the optical axis (usually the center) as the origin. Therefore, first, the coordinates (i, j) of each pixel of the element image are once converted into azimuth coordinates (φx ′, φy ′) by the equation (1). In equation (1), R is the distance between the lens of the digital camera that captured the element image and the imaging surface, that is, the radius of the spherical surface 100. Next, the azimuth coordinate of each pixel of the element image is converted into the azimuth coordinate (φ, θ) of the unified reference coordinate system by the equation (2). In the equation (2), φ and θ are the longitude and latitude described with reference to FIG. Further, A in the equation (2) is a conversion matrix between two coordinate systems. Such a coordinate transformation matrix is derived on the basis of the relative positional relationship between corresponding points of overlapping regions of adjacent element images, for example, as detailed in the specification of Japanese Patent Application No. 9-166737. can do. In order to obtain the coordinate transformation matrix A, the element images are photographed so that adjacent ones partially overlap each other.
[0017]
In the next step S <b> 2, the image projected and pasted on the spherical surface 100 is projected onto a plane 106 (FIG. 3) including the equator plane 104. Specifically, this orthogonal projection is performed by converting the coordinates (φ, θ) of each pixel on the spherical surface 100 into polar coordinates (r, Φ) according to the equation (3).
[0018]
FIG. 3 is a cross-sectional view of the spherical surface 100 cut along a longitude line passing through the projection point and the point Q of the pixel p in FIG. Reference numeral 106 denotes an orthogonal projection plane including the equator plane 104. O ′ is the center of the element image F1, and p ′ is the projection point of the pixel p on the element image F1 onto the spherical surface 100. The point p ′ is a point where the point p ′ is projected onto the plane 106 including the equator plane by plane projection.
[0019]
If only the scene above the horizon is targeted, only the image projected onto the hemisphere (north hemisphere) above the equator plane 104 of the sphere 100 needs to be projected by plane, and all the pixels are inside the equator plane 104. Projected on. However, as can be seen in FIG. 3, the element image F2 taken in the horizontal direction with a digital camera or the like also includes a portion below the equator plane 104, and the pixels in such a portion are projected from the equator plane 104 by plane projection. Projected outward. For example, the pixel q of the element image F2 is projected onto the point q ′ on the spherical surface 100, and further projected onto the point q ″ on the plane 106 by the projection projection. W is the lowest pixel on the element image F2. And projected onto a point w ′ on the spherical surface 100 and further projected onto a point w ″ on the plane 106. In this embodiment, by expanding the orthographic projection plane 106 from the equator plane 104, not only the scene above the horizontal direction but also the scene below the horizontal direction by a certain angle can store the image data as one file. Like that.
[0020]
FIG. 4 is a diagram for explaining the image projection area on the projection projection plane 106 in association with the image data storage area. Pixels on the northern hemisphere of the spherical surface 100 are projected in the range indicated by the circle 110. In order to project pixels up to a certain range below the equatorial plane 104 on the spherical surface 100, the projective projection area is expanded to an area indicated by a circle 112. Therefore, the area between the circle 110 and the circle 112 is an area onto which an image of a part of the southern hemisphere of the spherical surface 100 (continuous with the northern hemisphere) is projected.
[0021]
Returning to FIG. In the last step S3, the data of the images projected by the normal projection in the previous step S2 are collectively stored in the memory. Specifically, in this embodiment, the following operation is performed to store the image data in the memory.
[0022]
First, a memory area corresponding to the square ABCD area on the orthogonal projection plane shown in FIG. 4 is secured. In the previous step S2, each pixel position projected orthogonally is expressed by polar coordinates (r, Φ), but generally an integer address is assigned to the memory location. Therefore, the coordinates of each pixel projected by plane projection are converted into orthogonal coordinates (X, Y) by the equation (4). Next, the real number coordinates (X, Y) are converted into integer coordinates (I, J) by the equation (5). In the formula (5), [x] means the maximum integer not exceeding x.
[0023]
For example, the pixel value or luminance value f (I, J) calculated by the interpolation calculation of equation (6) is written in the memory location at the address corresponding to the integer coordinates (I, J) obtained in this way. In equation (6), f (Xi, Yj) is a calculated pixel value or luminance value in real number coordinates (Xi, Yj). Ip () is an interpolation function, for example, a linear interpolation function or a cubic interpolation function. This interpolation calculation is performed within a fixed distance around the focused integer coordinates (I, J), for example, within the range of the expression (7) (where Th is a constant).
[0024]
In this way, the data of the image projected and pasted onto the northern hemisphere of the spherical surface 100 and a part of the southern hemisphere continuous thereto can be stored in the memory in a lump and managed as one file.
[0025]
Generally, in order to efficiently manage data on a memory by a computer, it is desirable that the address range can be expressed by a simple symbol. In the case of a two-dimensional address (I, J), management by a section such as A <I <B and C <J <D is most common. The present embodiment satisfies such a requirement.
[0026]
In the case of handling image data, in terms of the uniformity of arithmetic processing and the accuracy of image representation, continuous pixels on the memory are almost equally spaced in the real space, that is, the pixels (I, J on the memory). ) And the pixel (I, J + 1 ) in the real space is desired to be substantially equal to the space in the real space between the pixel (I, J + 1) and the pixel (I, J + 2) in the memory. In this embodiment, the pixels arranged on the memory are distributed with a density of almost the same order as the pixels on the spherical surface (the difference in density between the vicinity of the center and the periphery is 1: 4 at the maximum). Satisfy the request.
[0027]
When reproducing the image stored in the memory as described above, the pixel on the memory is projected onto the spherical surface by a conversion process just opposite to the case of storing the image, and then re-projected onto the plane perpendicular to the direction to be observed. do it. Since the element images have been combined, an image in an arbitrary direction can be reproduced at high speed.
[0028]
When shooting a scene from the top of a high mountain or the roof of a high-rise building and obtaining a panoramic image thereof, there are cases where it is desired to include the scene considerably below the horizontal direction in the target range. If you want to capture a panoramic image of a scene from a balloon or helicopter, you want to include the scene just below. In this way, when it is difficult to uniformly determine how much processing should be performed from the horizontal direction, the image projected on the upper half (north hemisphere) of the sphere and the lower half (south hemisphere) It is preferable to project the images projected on the equatorial plane separately onto the equator plane and store each projected projection image in a memory and manage it as a single file. This can also be reproduced by the process described above. This is also one embodiment of the present invention.
[0029]
The processing of the present invention described above includes, for example, a CPU 200, a memory 201, a display device 202, a user input device (keyboard, mouse, etc.) 203, a hard disk device 204, a floppy disk device 205, and a PC card reader as shown in FIG. 206 and the like can be realized by a program on a computer having a configuration in which the system bus 207 is connected. For example, a series of image data shot from a certain point by a digital camera and recorded on the PC card 208 is read into a computer by the PC card reading device 206 and stored in the hard disk device 204. The program 210 for the processing of the present invention described with reference to FIG. 1 is read from, for example, the floppy disk 209 set in the floppy disk device 205, stored in the hard disk device 204, and loaded into the memory 201 when necessary. To be executed. When the program 210 is executed, image data to be processed is read from the hard disk device 204 into the memory 201. A memory area necessary for processing, for example, a memory area 211 in which the projection image data is written in step S3 of FIG. 1 is secured on the memory 201 at a necessary time. The projection image data written in the memory area 211 is stored in the hard disk device 204 or the floppy disk 209 as necessary, and is read into the memory 201 again at the time of image reproduction.
[0030]
As described above, according to the present invention, it is possible to reduce the amount of memory necessary for storing a three-dimensional panoramic image and the like and to reproduce the image at high speed. Mutual comparison and calculation become easy. For example, when measuring the perspective information of a subject by comparing a series of images taken of a surrounding scene at a certain point with a series of images taken of the surrounding scene by moving the place, It is desirable that a series of captured images be expressed in a unified coordinate system. In the present invention, a series of images taken from a single point are represented by projection projection. For example, if the vertical direction is taken as the center point of the projection coordinate system, the horizontal plane is taken as the circumference of the projection coordinate system, and a certain direction is taken as the reference axis. Since the orientation (parallax) of the corresponding subject in the image taken from a different point can be directly expressed as a difference in coordinates, the perspective information can be easily extracted.
[0031]
【The invention's effect】
As described above, according to the present invention, it is possible to efficiently store and manage data such as a three-dimensional panoramic image obtained by synthesizing a plurality of images taken from the same position and different directions, and to perform high-speed image reproduction. In addition, it is possible to obtain an effect of facilitating extraction of perspective information of a subject by comparing a series of images taken from different points.
[Brief description of the drawings]
FIG. 1 is a flowchart showing an outline of processing according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining spherical projection of an element image.
FIG. 3 is a cross-sectional view of FIG. 2 cut along a longitude line.
FIG. 4 is a diagram showing an orthographic projection area of an image and a memory area in association with each other.
FIG. 5 is a block diagram illustrating an example of a computer for executing processing according to the present invention.
FIG. 6 is an explanatory diagram of panorama image creation by projection onto a cylindrical surface.
[Explanation of symbols]
100 spherical surface 102 polar axis 104 equatorial plane 106 orthographic projection plane F1, F2 element image O center N north pole S south pole Φ longitude θ latitude p, q, w pixel on element image p ′, q ′, w ′ projection onto sphere Point p ", q", w "Plane projection point 200 CPU
201 Memory 202 Display Device 203 User Input Device 204 Hard Disk Device 205 Floppy Disk Device 206 PC Card Reading Device 207 System Bus 208 PC Card 209 Floppy Disk 210 Program 211 Memory Area

Claims (4)

Projecting and pasting a plurality of images obtained by photographing different orientations from the same position with a digital camera or the like onto a spherical surface centered on the lens center of the digital camera or the like, and combining the image on the spherical surface with the spherical surface a point on the projection center, image storage consisting of the steps of Stereographic projected on the projection plane that intersects the spherical surface, and the step of storing in bulk data Stereographic image projected on the projection plane in the memory or the recording medium Method. Projecting and pasting together a plurality of images taken from the same position with a digital camera or the like on a spherical surface centered on the lens center of the digital camera or the like; An image on one hemisphere and an image on a part of the second hemisphere on the other polar side continuous with the first hemisphere, with the other pole as the projection center, the equator of the spherical surface each non-overlapping regions of the same projection plane including the surface and Stereographic projection, image storage method characterized by comprising a step of storing the collectively memory or recording medium data Stereographic image projected on the projection plane . Projecting and pasting together a plurality of images taken from the same position with a digital camera or the like on a spherical surface centered on the lens center of the digital camera or the like; An image on one hemisphere is projected onto a first projection plane including the equatorial plane of the sphere, with the other pole of the sphere as the projection center, and a second hemisphere on the other pole side of the sphere A step of projecting an image on a plane onto the second projection plane including the equatorial plane with the one pole as a projection center ; and data of the images projected onto the first projection plane in a lump An image storage method comprising: storing data of images stored in a memory and projected onto the second projection plane at a time in a memory or a recording medium.   A machine-readable medium having recorded thereon a program for causing a computer to execute each step of the image storage method according to claim 1, 2 or 3.

Images