KR20180027379A - Method of obtaining stereoscopic panoramic images, playing the same and stereoscopic panoramic camera - Google Patents

Abstract

The present invention provides a stereoscopic panoramic image system including an image obtaining unit, an image processing unit, an image display unit, and an image selecting unit. The image obtaining unit includes a base portion in a lower portion and a rotation portion in an upper portion. The rotation portion is connected to the base portion to infinitely rotate with respect to the base portion around a rotation axis. Two line scan cameras having the same specification are installed in the rotation portion in parallel with each other in the same direction. View angles of a rotation axis direction of image forming lenses mounted in the cameras are 180° or larger. Sections are vertical to optical axes of the image forming lenses. Widths of the sections measured in a direction vertical to the rotation axis are shorter than a height measured in the rotation axis direction. Even though the image forming lenses move in optical axis directions, long axes of the image forming lenses maintain a rotation axis direction.

Description

METHOD OF OBTAINING STEREOSCOPIC PANORAMIC IMAGES, PLAYING THE SAME AND STEREOSCOPIC PANORAMIC CAMERA BACKGROUND OF THE INVENTION 1. Field of the Invention [0001]

The present invention relates to a stereoscopic image acquisition method and method, and a three-dimensional panoramic image camera, which allow a landscape of arbitrary direction to be viewed as a stereoscopic image with a head-mounted display device.

In 1788, British painter Robert Barker painted the sceneries of Edinburgh, Scotland in the interior of a cylindrical wall so that the audience at the center of the wall could see 360 ° all directions of the city. Figure 1 shows Robert Barker's all-round conversation at the University of Edinburgh library ([Non-Exercise 1]). He is also the person who made the word "panorama" by combining 'horama', which means 'pan' and 'view', which means 'all' in Greek, 3]).

As a method of obtaining an omnidirectional image as a photograph other than a painting, there is a method of using a panoramic lens having a function of capturing omni-directional images in the lens itself, and a method of using a omnidirectional camera. In this case, the two cases will be referred to as the omni-directional imaging system.

One of the easiest ways to obtain an omni-directional image is to use a catadioptric panoramic lens that combines a reflective lens and a refracting lens. It has been actively studied since the beginning of the 20th century and has been actively studied 1] to [8]), the theoretical studies on refractional lenses have not been very long ([Non-specific 4] to [Non-specific 11]).

FIG. 2 shows an example of an omnidirectional imaging system using a panoramic mirror having a rectilinear projection scheme included in Reference [10]. The vertical angle of view of the omnidirectional mirror is set to a human standard It was designed to be 46 ° to match the visual.

FIG. 3 shows an unfolded omnidirectional image captured by the omnidirectional imaging system of FIG. 2, FIG. 4 shows an expanded omnidirectional image obtained from the unfolded omnidirectional image of FIG. 3, Show some. In FIG. 3, it can be seen that although the simple trigonometric function conversion is used to obtain FIG. 4, the vertical direction is well proportioned as shown in FIG. This is because, from the beginning, the panoramic mirror shown in FIG. 2 is designed to have a linear aberration correcting projection method.

On the other hand, a panoramic camera that captures scenery in all directions in a 360-degree panoramic image on a scenic spot is an example of an omnidirectional imaging system. The omnidirectional imaging system refers to a vision system that captures all of the scenery that the observer sees when he or she turns around in a single image. In contrast, a system for capturing images of all directions in a single image from an observer's position is called an omnidirectional imaging system. In an omnidirectional imaging system, the observer not only rotates in place, but also includes all the scenes that can be seen by tilting or tilting the head. Mathematically, the solid angle of the region that can be captured by the imaging system is 4π steradian.

The omnidirectional imaging system or the omnidirectional imaging system can be used not only in a conventional field such as photographing a building, a natural landscape, a celestial object, etc., but also a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide- (Oxide film) semiconductors) Many researches and developments have been made to apply to security / surveillance systems using cameras, virtual tour of real estate, hotels, sightseeing spots, and the like, or mobile robots or unmanned aerial vehicles.

The omnidirectional camera has a long history and various kinds, but it can be classified into three kinds. The first is a rotating panoramic camera, such as the Lomography Spinner 360. When the photographer holds the camera in one hand and pulls the line on the handle with the other hand, the camera rotates 360 ° and the omnidirectional image is recorded on the 35mm film. The rotating omnidirectional camera in principle produces the most accurate omni - directional image. However, since the hand holding the camera can not be shaken at the time of shooting, accurate omnidirectional images can not be obtained in reality, and this product is close to a mysterious toy (gadget).

[0003] This patent application is related to a rotating scanning type omnidirectional camera filed on November 27, 1978, which is owned by Globuscope Inc., a Corp. of NY, It was sold as a product name, but it is now discontinued. Fig. 6 shows a state in which the cover of this omnidirectional camera is removed. The camera uses a 35mm film, which is rotated by 360 ° with manual power. The user must hold the Globusope by hand and remain stationary for more than a second when the camera head makes one revolution. This patent claims to solve banding problem, which is a chronic problem of scanning type omnidirectional camera. It uses thixotropic fluids such as corn starch to keep the rotation speed constant. Meanwhile, FIG. 7 shows a Seitz Round shot 35/35 Panoramic Film Camera of Seitz Corporation.

[Patent document 10] is a US patent application filed on June 28, 1983, which relates to a scanning panoramic television camera using a line sensor, not a 35 mm film.

This patent is a patent application filed on August 8, 1996, which includes a fish eye lens and a relay lens for transmitting the image of the fish eye lens to the lower end of the rotation stage. A camera is disclosed. The relay lens serves to derotate the image of the rotating fisheye lens in order to match the image of the rotating fisheye lens with a linear sensor array fixed at the bottom of the rotation stage and includes a mirror and a dove prism Which is a complex structure. Although it is possible in principle, it is expected that it is difficult to realize stable performance enough to be practically used outdoors, and it is presumed that it is not actually manufactured.

There is also a full-field camera that monitors a wide area, such as an airport or a port, that can reach several kilometers in width. Such a camera is usually a medium-infrared (MIR: Medium Wave Infrared, Mid Infrared), and it is characterized by extremely different angle of view in the horizontal and vertical directions. For example, the Spynel series radar in France's HGH Infrared System has 360 degrees of view in the horizontal direction, but 20 degrees in the vertical direction. These cameras update the omnidirectional image with the infrared lens and the linear image sensor rotating continuously in the horizontal direction to show the omni-directional image close to the moving image.

On the other hand, there are some omnidirectional cameras, such as Horizon, Noblex, and Widelux, which rotate the lens instead of the entire camera. Such an angle of view of the camera in the transverse direction is usually as small as 140 DEG, and the film is wound on the cylindrical cylinder on the opposite side of the lens. Therefore, even if the lens rotates, the distance from the lens to the film is always kept constant. Such a camera is suitable for taking landscape or group photographs, and should not be moved for 2 to 3 seconds when the photograph is taken.

Moultrie's omnidirectional camera, which is widely used for wildlife observation in national parks and the like, uses a lens rotation method, in which a lens rotates when the motion sensor senses motion, thereby capturing an omnidirectional image.

Such a rotating omnidirectional camera intuitively implements the concept of panorama and can obtain correct omni-directional images in principle, however, it is difficult to obtain a moving image. Anyone who accidentally moves a document while copying a document with a copy machine will be able to easily guess what will happen when you take a moving object with a moving camera.

The second method is to acquire multiple images and then connect them to obtain one omni-directional image. Stitching is said to be similar to stitching an image. You can rotate one camera to get multiple images, or you can use images from multiple cameras that are pointing in different directions from the beginning. The former is mainly used for obtaining a still image, and the latter is mainly used for obtaining a moving image.

In the heyday of 35mm film camera, I rotated the direction of the camera horizontally and obtained several photographs, then cut and superimposed the overlapping parts with scissors so that these pictures could be connected well. In order to obtain such an omnidirectional image in a more professional manner, a panoramic head which can mount the camera on a tripod and move the camera horizontally so that the nodal point of the camera lens coincides with the center of the tripod ) Is used. The nodal point of the camera is the point corresponding to the position of the eye of the needle when it is assumed that the camera is an ideal pinhole camera and the nodal point is located in the actual lens if the lens design is theoretically calculated , Or experimentally with a real camera. Such a method is called a nodal slide method.

With the advent of digital cameras, CCTV, and smartphones, it has become much easier to produce omni-directional images through stitching, and commercially available software that makes stitching easier can be found ([12] - [13], [12] ). Nowadays, software such as PTGui and AutoPano are popular.

8 is a DSC-HX1 camera of Sony having an omnidirectional imaging function using stitching technology, and Fig. 9 is a sample of omnidirectional images taken by this digital camera. As can be seen from Fig. 9, the angle of view in the horizontal direction is greatly different from 360 degrees. To take an omni-directional image with the DSC-HX1, select the panorama mode, and then panning horizontally at the proper speed while holding down the shutter ([Non-Spec 13]).

10 is a view of a Samsung Galaxy Camera EK-KC120L having an omnidirectional imaging function using a stitching technique, and its usage is similar to that of the DSC-HX1. However, if you shoot in panoramic mode, you can feel that the angle of view ends before it reaches 360 °. 11 is a panoramic image shot with this camera, but the left end and the right end of the image do not naturally connect. Also, it is impossible to maintain the level perfectly even when the camera is held horizontally while holding the camera in the raised position.

12 and 13 are examples of omnidirectional images taken with the Panorama 360, an Android app with omnidirectional imaging using stitching technology. As can be seen from FIG. 14, it can be seen that the left end and the right end of the image are naturally connected. On the other hand, FIG. 15 shows an enlarged view of a part of the omnidirectional image of FIG. 13, which shows a double view of a passenger car and a building. This is due to technical limitations of the omnidirectional image by the stitching technique.

When shooting an omnidirectional image using the stitching technique, approximately 8 ordinary photographs are taken. Of course, the direction of the camera when shooting the eight images is all different. However, when the direction of the camera changes according to the distance from the camera to the subject, the distance in which the subject moves from the image to the horizontal direction is different. Therefore, when stitching, it is necessary to stitch based on a main subject at a certain distance, and a subject at a distance different from the main subject is highly likely to be double captured.

This is a purely geometric problem and is not a problem that can be solved by the improvement of the lens or image processing algorithm. A realistic alternative is to erase one of the captured images, and some software provides such a function.

Two or more CCTV cameras are used to acquire omni-directional images from video ([14]). For example, all-purpose cameras such as Honeywell's ParaScan and Mobotix's D12 in Germany synthesize images obtained from two cameras with 90 ° angle of view in each direction to produce a security camera (CCTV )to be. Arecont Vision offers a high-resolution, all-in-one security camera with stitching using multiple CCTV cameras.

The biggest advantage of such a stitching omnidirectional camera is that it can produce an omni-directional image that is several times larger than the resolution of the currently available image sensor. GoPro's Odyssey uses 16 Hero 4 cameras so you can create a 360 ° omni-directional image with 16 times the resolution of one Hero 4 camera. Of course, the Stitching type omnidirectional camera is much more expensive than other types of cameras.

Internet portals such as Google, Naver, and Daum are collecting images by carrying these stitching type omnidirectional cameras in their cars or bicycles to provide Street View services linked to their maps.

Stitching type omnidirectional cameras have no moving parts, so they can obtain moving images, but they cause parallax errors in which the joints are inaccurate when the image and the image meet. This is due to the fact that the viewpoints of the two cameras are located at different positions, which is a physical limitation that can not be solved by software correction. Therefore, when the subject is close, it is possible to experience an unpleasant experience of shining the face with a broken mirror. However, it will not be a problem if it is intended to monitor only objects that are far from the beginning, such as a fence line, a coastline, a railroad track, or a highway.

The third method is an image processing-based panoramic camera that takes an image with a wide-angle lens and then produces an omni-directional image through image processing. The image processing based omnidirectional camera and fisheye lens are inseparable.

One way to obtain omni-directional images is to employ a fisheye lens with a wide angle of view. For example, when a fisheye lens with a 180 ° angle of view is directed vertically to the sky, it is possible to capture from the star constellation of the sky to the horizon in one image. For this reason, a fisheye lens is also referred to as an all-sky lens. In particular, Nikon's fisheye lens (6mm f / 5.6 Fisheye-Nikkor) has an angle of view of 220 °, so if you attach it to the camera, you can also include some scenery behind the camera. As described above, the image obtained by using the fisheye lens is subjected to image processing to obtain an omni-directional image.

In the reference documents [Nos. 15] to [Nos. 16], a core technology for extracting an image having a different viewpoint or projection method from an image having a given viewpoint and a projection scheme is proposed. Specifically, a cube panorama is presented in reference [non-feature 16]. Simply put, a cube panorama describes the landscape in all directions from the glass wall to the glass wall when the observer is at the center of a cube made of glass, which depicts all the scenes as the point of view from the center of the cube. However, there is a disadvantage that the virtual scenery is not a real scene obtained using an optical lens but an image captured by a hypothetical lens without distortion, that is, a camera with a pinhole.

Reference [Non-feature 17] shows an algorithm for projecting Omnimax images using a fish-eye lens on a semi-cylindrical screen. In particular, a method of finding the position of a water spot on a film surface forming a shop at a specific position of the screen is described in consideration of an error in the projection method and the projection method of the fisheye lens mounted on the movie projector. Therefore, in order to project a specific image on the screen, it is possible to know what type of image should be recorded on the film, and such image is produced using a computer. Particularly, since the distortion of the lens is reflected in the image processing algorithm, the viewer adjacent to the projector can appreciate the satisfactory panorama movie. However, in modeling the actual projection method of a fish-eye lens, it is inconvenient to use the zenith angle of the incident light as a dependent variable and the image size on the film plane as independent variables. In addition, we have unnecessarily limited the approximation to odd polynomials.

On the other side, all animals and animals, including humans, are bound to the surface of the earth by gravity, so most of the attention or attention needed events occur near the horizon. Therefore, even though 360 degrees around the horizon need to be monitored, there is little need to monitor it up to a height in the vertical direction, that is, the zenith or the nadir. However, in order to describe the 360 ° landscape in all directions in a two-dimensional plane, inevitably distortion is required. There is a similar difficulty in the mapping method for expressing the geography on the surface of the sphere on a two-dimensional plane map.

All inanimate objects such as plants, plants and buildings on Earth are all under the influence of gravity, and the direction of gravity is straight, ie, perpendicular. However, among all distortions, the distortion that the person feels most unnatural is the distortion that the vertical line shows as a curve. Therefore, even if there are other distortions, it is important to prevent such distortions. However, the ground is usually perpendicular to the direction of gravity, but of course it is not vertical in inclined places. Therefore, in a strict sense, it should be based on the horizontal plane, and the vertical direction is the direction perpendicular to the horizontal plane.

Equi-rectangular projection, Mercator projection, and cylindrical projection, which are widely known among various mapping methods, are described in Reference [Non-Specification 18] ] Summarizes the history of various projection methods. Among these, isochronous projection method is one of the most familiar mapping methods when we describe the geography on earth or the celestial figure to display constellations.

Examples of the fisheye lens having the angle of view of 190 degrees are described in Reference Documents [14] and [Non-Specification 20], and references [15] describe various fisheye lenses including a refraction type refraction type refraction type fisheye lens, An embodiment of the lens is proposed.

In addition, various embodiments of obtaining an omni-directional image according to a cylindrical projection method, an iso-orthogonal projection method, and a mechatotor projection method on an image acquired using a rotationally symmetric wide-angle lens including a fish-eye lens are described in Reference Document .

On the other hand, an example of another image system that can be attached to one wall of the room to monitor the entire room is a pan, tilt, and zoom camera. Such a camera is implemented by mounting a CCTV equipped with a lens having an optically zoom function on a pan / tilt stage.

The fan action refers to a function capable of rotating by a predetermined angle in the horizontal direction, and the tilt action refers to a function capable of rotating by a predetermined angle in the vertical direction. In other words, if the camera is at the center of the celestial sphere describing the celestial body, the pan means an operation to change the longitude, and the tilt means the action of changing the latitude. Therefore, the theoretical range of the fan action is 360 °, and the theoretical range of the tilt action is 180 °.

The disadvantage of such a pan, tilt and zoom camera is high price, large volume and weight. Lenses with optical zoom are bulky, heavy in weight, and expensive in design due to the complexity of the design and the complexity of the structure. In addition, the pan / tilt stage is as expensive as a camera. Therefore, a considerable amount of money is required to install a pan / tilt / zoom camera. In addition, the pan, tilt, zoom camera is bulky and heavy, which can be a considerable obstacle depending on the application. For example, if the weight of the payload is very important, such as an airplane, or if there is a space constraint to install the imaging system in a tight space. Furthermore, since the pan, tilt, and zoom operations are physical operations, it takes a long time to perform such operations. Thus, depending on the application, the mechanical response of such a camera may not be fast enough.

On the other hand, Reference Document [17] describes a video system capable of performing pan, tilt, rotate, and zoom functions without any physical moving parts. When the user inputs a principal direction of vision using an input device such as a joystick after acquiring an image using a camera equipped with a fisheye lens having an angle of view of 180 degrees or more, (I.e., a rectilinear image) is extracted. The difference between the present invention and the prior art is that a user generates a linear aberration corrected image in a direction selected by various input devices such as a joystick or a computer mouse. This technique is a key technology when trying to replace a virtual reality or mechanical pan / tilt / zoom camera. The keyword is "interactive picture". These techniques have the advantage of fast response speed of the system and less concern about mechanical failure because there is no physically moving part.

Generally, when installing a video system such as a surveillance camera, a vertical line perpendicular to the horizontal plane is displayed as a vertical line even in an acquired image. In such a state, the vertical line is continuously displayed as a vertical line in the image even if the pan, tilt, and zoom operations are performed mechanically. However, in the invention of Reference [17], a vertical line is not generally displayed as a vertical line in an image obtained by a software-based pan / tilt operation. In order to compensate for such an unnatural image, a rotation operation not performed in a mechanical pan / tilt camera should be additionally performed. However, in the above-described invention, what rotation angle is required to represent a vertical line as a vertical line is not shown. Therefore, in the above-described invention, there is a disadvantage that an accurate rotation angle must be found by a trial and error method in order to obtain an image in which a vertical line is displayed as a vertical line.

Further, in the above-described invention, the projection method of the fish-eye lens is assumed to be an ideal equi-distance projection scheme. However, the projection method of a real fish eye lens usually shows a considerable error with an ideal isometric projection method. Since the distortion characteristic of the actual lens is not reflected in the above-described invention, the image is distorted even in the image processed.

Reference [18] suggests an image processing method that complements the disadvantage that the actual projection method of the fisheye lens is not reflected in the reference document [17]. However, the disadvantage that the vertical line is not displayed as a vertical line in the image is not solved.

In the reference document [19], a video system is described in which an image captured by a fisheye lens is remotely transmitted to generate a linear aberration-corrected image without distortion at the receiving end. The most important advantages of the system are a mechanical pan / tilt camera It is not necessary to send a control signal from the receiving end to the transmitting end. Further, there is an additional advantage that a plurality of receiving ends can generate separate linear aberration corrected images for one transmitting end.

If a camera with a 180 ° fisheye lens or an omnidirectional camera with a 180 ° angle of view is attached to a wall in the room, there is virtually no blind spot in the room. This is because the area that the camera can not capture is a wall surface that does not need to be monitored. However, the image by the fisheye lens causes an aesthetic discomfort due to the cylindrical distortion, and the super-wide-angle straight-line aberration corrected image without distortion does not naturally capture the object in the direction far away from the optical axis even though most of the interior can be seen . In addition, the image by the omnidirectional camera can capture the entire room naturally, but the distant object may be captured too small, which may be difficult to identify. In this case, the most natural image is a straight-line aberration corrected image that is directed toward the camera in the direction of the subject and is viewed from the front.

As described above, a camera capable of physically performing this is a camera equipped with a linear aberration correcting lens without distortion and mounted on a pan-tilt stage. Since the camera can be rotated so that the direction requiring attention is the front, the most satisfactory image can be obtained. In addition, if there is a moving subject such as a cat or an illegal chip particle, the image can be captured while following the movement of the subject. Reference [Japanese Patent Application Laid-Open No. 11-241254] discloses a method of performing a tilt operation after performing a fan operation in a software manner and a method of performing a fan operation after a tilting operation in order to implement such a function in software. However, different images are obtained according to the posterior relationship between the fan action and the tilt action. Therefore, a preferable image processing method should be used according to the installation state of the camera.

Reference [20] provides a method of obtaining a linear aberration-corrected image without distortion by performing mathematically correct image processing on an image of a wide-angle lens which is rotationally symmetrical about an optical axis and various image systems using the method. In particular, an image processing algorithm is proposed in which a vertical line is displayed as a vertical line while implementing a digital pan / tilt effect. However, the invention of reference [20] relates to an algorithm for extracting an image obtained by a pan / tilt camera equipped with a linear aberration correcting lens in an image acquired by a camera equipped with a fisheye lens, It does not provide an algorithm that takes into account the various possibilities that can be encountered. For example, when the optical axis of the camera is parallel or perpendicular to the horizontal plane, or when the optical axis of the camera has a predetermined angle with respect to the horizontal plane, the linear aberration correction image to be acquired is a virtual straight line An image processing method considering only when the optical axis of the aberration correction camera is parallel to the horizontal plane is proposed.

A physical pan / tilt camera actually used is installed so that the optical axis of the camera is usually parallel to the horizontal plane when both the fan angle and the tilt angle are 0 °. Therefore, the invention of the reference [20] can realize the effect of a physical pan / tilt camera. However, when a digital pan / tilt effect is implemented in an image obtained by a camera equipped with a fisheye lens, the effect beyond the physical pan / tilt camera can be obtained. The invention of the reference [21] provides a mathematically rigorous image processing method and an image system that realize the effect beyond the limit of a physical pan / tilt camera. In particular, according to the invention of [21], the most general image processing method that can be used even when the horizontal direction of the image sensor surface is not parallel to the horizon line is also provided.

However, when viewing a TV or a movie, the images displayed on the left eye and the right eye are the same, so the screen feels flat rather than stereoscopic, and the immersion feeling is degraded. For that reason, the TV or movie screen does not feel as vivid as the naked eye.

On the other hand, a variety of stereoscopic image technologies have been developed to allow stereoscopic images to be viewed on TVs, movies, and computer screens. As shown in FIG. 16, two lenses of the same specifications are commonly used to acquire two images using a stereoscopic camera, which is directed in the same direction, and then two images are displayed on the left and right eyes, respectively. It shows. Of course, even in this case, the horizontal distance between the two lenses in the stereoscopic camera is set to be similar to the interval between the two eyes.

Virtual Reality (VR) is rapidly expanding due to explosive interest. Artificial reality (AR) and augmented reality (AR) or mixed reality (MR) are similar fields to virtual reality. However, both virtual reality, artificial reality, and augmented reality require images of different viewpoints on the left and right eyes in order to enjoy these images as stereoscopic images. The most representative of the devices that enable this is the left eye and right eye It is a head mounted display (HMD) device that can display different images to the eyes. FIG. 17 shows a Cardboard viewer made by Google, and FIG. 18 shows an example of a stereoscopic image corresponding to a left eye and a right eye in an HMD device.

On the other hand, there are other kinds of stereoscopic glasses or 3D glasses capable of viewing stereoscopic images, and the most inexpensive and effective glasses are anaglyph glasses. 19 shows cheap anaglyph glasses produced from paper and celluloid films. The left and right sides of anaglyph glasses have different colors such as red and blue, or red and green. Looking at the anaglyph-type stereoscopic images with anaglyph glasses, different images are seen in the left and right eyes In the brain, these images are synthesized and recognized as three-dimensional.

Alternatively, there are polarized glasses that use a polarizer opposite the left eye and the right eye, respectively. Anaglyph glasses or polarized glasses are passive 3D glasses because they do not require electric power in use.

In a more complicated manner, there is a shutter glass active 3D glass which alternately displays images corresponding to the left eye and the right eye on the monitor. In the shutter glass, the left eyeglasses And the right eyeglasses are each going through a transparent state and an opaque state. That is, in the active mode, the monitor and the 3D glasses are synchronized so that the left and right images alternate.

In order to actually generate a stereoscopic image, a stereoscopic camera of GoPro company using two fisheye lenses can be used as shown in Fig. Even in this case, a stereoscopic image of the front face of the camera can be generated, but a stereoscopic image of a diagonal direction other than the front is inferior in stereoscopic effect. The reason can be seen in FIG. 21 and FIG.

21 is a conceptual diagram of stereoscopic image acquisition by image processing in a stereoscopic omnidirectional camera using two fisheye lenses. The origin O of the World Coordinate System is located in the middle of the nodal points (N _L , N _R ) of the two fisheye lenses. Thus, the nodal points (N _L , N _R ) of the two fisheye lenses are at the same distance from the origin. Also, the optical axes 2101L and 2101R of the two fisheye lenses are directed in the same direction, the direction of which is perpendicular to the X-axis and parallel to the Z-axis. That is, the Z-axis of the world coordinate system passes through the origin and is parallel to the optical axis of the fisheye lens. In FIG. 21, the principal direction of vision 2103L of the left linear aberration corrected image and the viewing direction 2103R of the right linear aberration corrected image coincide with the optical axis directions 2101L and 2101R.

The nodal points (N _L , N _R ) of the two fisheye lenses are separated by a distance D ₀ . The distance D ₀ can be set to be similar to the distance between the average two eyes of a person, 6.35 cm. The coordinates of the nodal point of the left fish-eye lens in the world coordinate system _{N L (X, Y, Z} ) are the coordinates of _{(-D 0/2, 0,} 0) , and the nodal point of the right fish-eye lens is N _R (+ D _0/2 , 0, 0).

21, the angle of view of the left fisheye lens and the angle of view of the right fisheye lens are shown in a semicircle. Therefore, although it is assumed that the angle of view of the fish-eye lens is 180 degrees, the angle of view of the fish-eye lens may be arbitrary. The object plane 2131L and the object plane 2131R and the object plane 2135R corresponding to the image plane 2135L and the right fisheye lens of the linear aberration corrected image obtained by image processing in the left fisheye lens are also displayed .

There is also in a three-dimensional camera shown in 21 to the left fish-eye lens and a word point normal to the direction of the optical axis of the right fish-eye lens distance D ₀ enough apart, the two rectilinear image plane obtained through the image processing (2135L, 2135R) is distance D ₀ , and when the HMD device is displayed on the screen of the HMD device corresponding to the left eye and the right eye, the wearer of the HMD device can feel the stereoscopic feeling as if it is actually in the place.

Of course, not only HMD devices but also passive 3D glasses such as anaglyph glasses and polarized glasses, or active 3D glasses using shutter glasses are used. In this case, the process of generating the linear aberration corrected image corresponding to the left eye and the right eye is the same, except that the process of converting the pair of linear aberration corrected images into a three-dimensional image is different according to the 3D method.

22 shows that the stereoscopic effect is reduced or disappeared when a pair of linear aberration corrected images having a pan action is obtained in order to obtain a stereoscopic image in the direction different from the optical axis direction of the fisheye lens in such a stereoscopic omnidirectional camera Fig.

Referring to FIG. 22, when it is desired to obtain a linear aberration corrected image which has a fan action by the angle? With the optical axis directions 2201L and 2201R of the fish-eye lens, the interval between the nodal points N _L and N _R of the two fish- The measurement in the direction perpendicular to the viewing directions 2203L and 2203R of the aberration corrected image is given by Equation (1).

Therefore, as the fan angle becomes larger, the interval perpendicular to the viewing direction of the two nodal points gradually decreases. When the fan angle becomes 90 °, the interval becomes zero, and no stereoscopic effect can be expected. For this reason, a stereoscopic omnidirectional camera using two fisheye lenses shown in Fig. 20 can not provide a stereoscopic image in an arbitrary direction.

Fig. 23 shows GoPro Odyssey, another three-dimensional omnidirectional camera of GoPro, in which a plurality of cameras using fisheye lenses are arranged in a direction facing outward on a concentric circle. In such a case, it is possible to realize the effect of the stereoscopic image by generating a linear aberration-corrected image in each of two cameras arranged in the direction closest to the direction in which the user wants to view the HMD, and displaying them in the left eye and the right eye . However, since this method requires a large number of cameras, it is not only expensive to manufacture, but also has a disadvantage in that fisheye images from several cameras must be transmitted or stored at the same time.

FIG. 24 shows another GoPro 360 Hero, which is another stereoscopic omnidirectional camera, in which a plurality of cameras having the same specifications are integrated in various directions. At this time, it is presumed that the stereoscopic image is generated by two cameras facing the direction in which the observer desires to observe. However, it is apparent that it is impossible to provide accurate stereoscopic images.

In 1998, Professor Yi-Ping Hung and others presented a stereo omnidirectional camera using a pair of rotating cameras ([21]). As shown in FIG. 25, when two identical cameras facing the same direction are horizontally rotated and a plurality of photographs are acquired, the images obtained from the respective cameras are stitched to obtain an omnidirectional image corresponding to the left camera and the right camera, respectively. The camera used in the experiment is 30 ° in angle of view and it is said that the image was taken while rotating 15 ° horizontally. Microsoft's H. Shum et al. Also published similar findings ([22], [22]).

In principle, however, a better result is to rotate around the midpoint between the two cameras ([non-specification 23] to [non-specification 24]). 26 is a conceptual diagram of a stereoscopic 360 ° omnidirectional camera as shown in Reference [Non-Schematic 24]. In this structure, two cameras having the same specifications are mounted on a rotating stage. The midpoint between the nodal points of the two lenses is located on the rotational axis of the rotating stage. In addition, the distance between two lens node points is similar to the distance between two eyes of a person.

The camera disclosed in [Non-patent document 26] is described as using a 35 mm roll film, and a narrow slit is mounted in front of the lens in the longitudinal direction. Therefore, when the camera is pointing in one direction, a thin band-like image in the vertical direction corresponding to the direction is paired. When the rotation stage is rotated, a band-like image is obtained in all directions, and if all are obtained, a pair of omnidirectional images corresponding to the left eye and the right eye are obtained as shown in FIG.

28 is a conceptual diagram illustrating a process of obtaining a rectilinear aberration-corrected image in the obtained 360 ° omni-directional image. In such a case, since the nodal point of the camera is not rotated, the linearly aberration correcting image which is mathematically strictly accurate can not be obtained, but it is sufficient to obtain a satisfactory stereoscopic image with the naked eye. FIG. 27 shows a pair of omnidirectional images obtained by using a rotating camera, whereas FIG. 28 shows a pair of linear aberration corrected images generated by artificial omnidirectional images acquired by a virtual camera. Therefore, the technique disclosed in [Non-Patent Specification 24] fails to provide an image processing algorithm for generating a pair of wide-angle images without distortion in a pair of omniazimuth images obtained from a rotating camera.

29 is a conceptual diagram of a system for obtaining a stereoscopic 360 ° omni-directional image using a single rotating camera instead of a pair of cameras. In such a system, the nodal point of the camera lens is separated by a distance from the rotation axis of the rotation stage. In addition, if two thin band images captured by oblique lines are extracted from the image obtained when the camera is in one position and all of them are extracted, the same effect as the omnidirectional image obtained by using two cameras can be realized.

In the stereoscopic image system disclosed in [Non-Specified Nos. 21] to [Non-Specified No. 24], a strip-shaped image in a longitudinal direction is cut out from an image facing in one direction and used. However, when the stitching method is used as described above, it is impossible to fundamentally avoid an abnormal phenomenon as illustrated in Fig.

However, an omni-directional imaging system or a three-dimensional omni-directional imaging system may be implemented using a line scan sensor having only one line of image sensors in the vertical direction. As described above, an omni-directional imaging system using a line scan sensor is disclosed. In [11], a omnidirectional camera using a line scan sensor and a three-dimensional omnidirectional camera are disclosed. However, in the invention of [11], a derotator using a mirror and a dove prism is used, and it is judged that it is difficult to make a commercial product with such a structure.

In particular, a stereo omnidirectional camera using a fish-eye lens and a line-scan sensor is disclosed, but includes elements that are unnecessary for the elements of the invention and are practically impossible to implement. Although the applicant has carried out the stitching type omnidirectional camera business to the present, it is presumed that the invention disclosed in [23] has never been produced.

Although the method of obtaining a three-dimensional omni-directional image by rotating the camera horizontally has been disclosed in [Japanese Patent Application Laid-Open Publication No. Hei. For example, in panorama mode, panning horizontally while holding the shutter, detecting overlaps in a series of photographs, or performing various corrections is a technology that is already used in existing products. What is "rotation correction" There is no description of what it means and we can not know the process of obtaining a stereoscopic image rather than an omni-directional image in an image obtained by panning horizontally.

Although a method of obtaining a stereoscopic image using a single rotating camera has been disclosed, almost all of the examples of image stitching are excluded except for up-sampling of the image.

A method of obtaining a linear aberration correction image in a cylindrical omnidirectional image is disclosed in [Japanese Patent Application Laid-Open Nos. 26 and 27]. However, if a refraction type omnidirectional lens having a linear aberration correction projection method is used or a linear aberration correction lens is used If a scanning type omnidirectional camera is used, the linear aberration correction image shown in Fig. 30 can be obtained by a simple geometric transformation on the omni-directional image ([Non-specific 10]).

[1] R. Parodi, "Optical device for horizontal panoramic and zenithal view ", U.S. Patent No. 1,616,279, filed on February 1, 1927. [2] M. M. Kirtane, "Optical system for viewing tubes", US Pat. No. 1,653,575, issued December 20, 1927. [3] D. W. Rees, "Hyperbolic ellipsoidal real time display panoramic viewing installation for vehicles", US Pat. No. 3,229,576, filed January 18, 1966. [4] D. W. Rees, "Panoramic television viewing system", US Pat. No. 3,505,465, filed on April 7, 1970. [5] J. A. Pinzone, G. J. Troje and E. L. Schiavone, "Reflectors and mounts for Panoramic optical systems", US Pat. No. 3,846,809, November 5, 1974. [6] J. S. Chahl, M. G. Nagle, M. V. Srinivasan and P. J. Sobey, "Panoramic surveillance system", U.S. Patent No. 5,790,181, August 4, 1998. [7] Kwon Kyung Kill, "Linear aberration correction mirror and imaging system equipped with the same", Korean Patent No. 10-0552367, filed on February 8, 2006. [8] A. Kostrzewski, S. Ro, I. Agurok and M. Bennahmias, "Panoramic video system with real-time distortion-free imaging", U.S. Patent No. 7,336,299, February 26, 2008. R. P. Globus, R. D. Globus and S. E. Globus, "Panoramic camera", US Pat. No. 4,241,985, filed December 30, 1980. [10] G. Federau, "Line-scan panoramic camera", U.S. Pat. No. 4,532,544, filed on July 30, 1985. [0015] J. D. Keast and J. F. Buford, "Digital video panoramic image capture and display system ", U.S. Patent No. 5,721,585, filed on February 24, 1998. [12] P. C. Teo, "Method and system for creating panoramas ", U.S. Patent No. 6,246,413, filed on June 12, 2001. European patent EP 1,858,251, filed on May 16, 2007, entitled " Method of taking a panoramic photo and mobile communication terminal using it ", by K. Park and S. Cho. [Special Issue 14] Kwon Kyung Il, Milton Lykkin, "Fish eye lens", Korean Patent No. 10-0888922, Date of registration March 10, 2009. [Special 15] Kwon Kyung Il, Milton L. Keene, "Wide Angle Lens", Korean Patent No. 10-0826571, Date of Registration April 24, 2008. [16] Kwon Kyung-il, "A method and apparatus for obtaining omni-directional image using a rotationally symmetric wide-angle lens", Korean Patent No. 10-0882011, filed on January 29, 2009. [0004] S. D. Zimmermann, "Omniview motionless camera orientation system ", US Pat. No. 5,185,667, filed February 9, 1993. [0007] E. Gullichsen and S. Wyshynski, "Wide-angle image dewarping method and apparatus ", U.S. Patent No. 6,005,611, issued December 21, 1999. [0008] H. L. Martin, D. P. Kuban and S. D. Zimmerman, "System for Omnidirectional Image Viewing at a Remote Location Without Transmission of Control Signals to Select Viewing Parameters," U.S. Patent No. 5,877,801, issued March 2, 1999. [20] Kwon Kyung-il, "A method for obtaining omnidirectional image and linear aberration corrected image using a rotationally symmetric wide-angle lens and its imaging system", Korean Patent No. 10-0898824, filed on May 14, 2009. [21] Kwon, Kyung-il, "A method and system for obtaining a digital pan / tilt image using a rotationally symmetric wide-angle lens", Korean Patent No. 10-1233948, filed on February 8, 2013. Stereo reconstruction from multiperspective panoramas, U.S. Patent No. 6,639,596, Oct. 28, 2003, by H. Shum and R. S. Szeliski. [Special 23] D. J. McCutchen, "Rotating scan camera", US Pat. No. 7,791,638, filed September 7, 2010. [0004] H. Ooshima, "Stereoscopic panoramic image synthesis device, image capturing device, stereoscopic panoramic image synthesis method, recording medium, and computer program", U.S. Patent No. 9,210,408, December 8, [25] C. Richardt, H. Zimmer, Y. Pritch and A. S. Hornung, "Stereoscopic panoramas", US Pat. No. 9,398,215, July 19, 2016. [0031] S. E. Chen and G. S. P. Miller, "Cylindrical to planar image mapping using scanline coherence", U.S. Patent No. 5,396,583, filed March 7, 1995. K. F. Cheng, K. G. Kumar, J. S. Lipscomb, J. P. Menon, M. H. Willebeek-LeMair, "Method and Apparatus for Displaying Panoramas with Streaming Video," US Pat. No. 6,356,297, March 12, 2002.

[Univ. 1] University of Edinburgh, Main Library. Wikipedia, "Robert Barker (painter)". [Non-3] Kwon Kyung-il, "History and Prospect of Omnidirectional Camera", Vision System Design, 17-27 (May 2016). [Non-specific 4] P. Greguss, "Panoramic security", Proc. SPIE, 1509, 55-66 (1991). [Non-patent 5] J. S. Chahl and M. V. Srinivasan, "Reflective surfaces for panoramic imaging ", Appl. Opt. 36, 8275-8285 (1997). [Non-patent 6] H. Ishiguro, "Development of low-cost compact omnidirectional vision sensors and their applications ", in Panoramic Vision: Sensors, Theory, and Applications, R. Benosman and S. Kang, eds. (Springer, 2001), Chap. 3. [Ref. 7] R. A. Hicks and R. Bajcsy, "Reflective surfaces as computational sensors", Image and Vision Computing, 19, 773-777 (2001). [Non-patent document 8] G. Kweon, K. Kim, G. Kim, and H. Kim, "Folded catadioptric panoramic lens with an equidistance projection scheme", Appl. Opt. 44, 2759-2767 (2005). G. Kweon, K. Kim, Y. Choi, and S. Yang, "Catadioptric panoramic lens with a rectilinear projection scheme", J. Korean Phys. Soc. 48, 554-563 (2006). [10] G. Kweon, Y. Choi, G. Kim, and S. Yang, "Extraction of perspectively normal images from video sequences using a catadioptric panoramic lens with the rectilinear projection scheme", Technical Proceedings of the 10th World Multi -Conference on Systemics, Cybernetics, and Informatics, 67-75 (Olando, Florida, USA, June, 2006). G. Kweon, S. Hwang-bo, G. Kim, S. Yang and Y. Lee, "Wide-angle catadioptric lens with a rectilinear projection scheme", Appl. Opt. 45, 8659-8673 (2006). [Non-12] T. R. Halfhill, "See you around", BYTE, 85-90 (May 1995). [Unique 13] Sony Corporation, "Digital still camera instruction manual: DSC-HX1", 38-39 (2009) [Non-specific 14] A. N. de Jong and P. B. W. Schwering, "IR panoramic alerting sensor concepts and applications", Proc. SPIE, 5074, 658-668 (2003). [Fifteenth] J. F. Blinn and M. E. Newell, "Texture and reflection in computer generated images", Communications of the ACM, 19, 542-547 (1976). N. Greene, "Environment mapping and other applications of world projections", IEEE Computer Graphics and Applications, 6, 21-29 (1986). [Non-feature 17] N. L. Max, "Computer graphics distortion for IMAX and OMNIMAX projection ", Proc. NICOGRAPH, 137-159 (1983). [Non-specific 18] E. W. Weisstein, "Cylindrical Projection", http://mathworld.wolfram.com/CylindricalProjection.html. [Non-featured 19] W. D. G. Cox, "An introduction to the theory of perspective - part 1", The British Journal of Photography, 4, 628-634 (1969). G. Kweon, Y. Choi, and M. Laikin, "Fisheye lens for image processing applications", J. of the Optical Society of Korea, 12, 79-87 (2008). [Non-use 21] H. Huang and Y. Hung, "Panoramic stereo imaging system with automatic disparity warping and seaming", Graphical Models and Image Processing, 60, 196-208 (1998). [Non-Patent Literature 22] H. Shum and R. Szeliski, "Stereo reconstruction from multiper spectral panoramas", International Conference on Computer Vision, 14-21 (1999). [Pic 23] S. Peleg, Y. Pritch and M. Ben-Ezra, "Cameras for stereo panoramic imaging", IEEE Conference on Computer Vision and Pattern Recognition, 208-214 (2000). [Non-Profit 24] P. D. Bourke, "Synthetic stereoscopic panoramic images", Lecture Notes in Computer Graphics (LNCS), Springer, 4270, 147-155 (2006).

An object of the present invention is to provide a method and apparatus for obtaining stereoscopic 360 ° omni-directional images so that stereoscopic images in an arbitrary direction can be viewed using an HMD device or other 3D glasses. In particular, And a technique capable of viewing all directions 180 degrees in the vertical direction.

A pair of cameras arranged symmetrically on both sides of the rotating stage are used. The two cameras are line scan cameras having the same specifications and having a line scan sensor arranged in the longitudinal direction. The lens is preferably a fisheye lens having a vertical angle of view of 180 ° or more, but a horizontal direction is a slender shape having a width of 3 cm or less to be. Further, even if the fisheye lens is moved in the optical axis direction to focus, the fisheye lens does not rotate about the optical axis, and the long axis direction of the fisheye lens always maintains the vertical direction. Each time the camera makes one revolution, a pair of omni-directional images corresponding to the left eye and the right eye are obtained. This omnidirectional image is similar to a spherical panorama or an orthoimage projection omni-directional image, but there is a difference. The most significant difference is that the fisheye lens has an actual projection method in which the distortion of the fisheye lens is not corrected in the vertical direction, but it satisfies a perfect equidistance projection method in the horizontal direction due to the rotating stage. The size of the omnidirectional image in the vertical direction is the same as that of the line array sensor, but the size of the horizontal direction depends on the rotation speed of the rotation stage and the output speed of the line scan sensor. Therefore, there is no correlation between the horizontal size and the vertical size of the 360-degree omnidirectional image plane measured in pixel units.

After the omnidirectional image is obtained, a rectilinear aberration-corrected image having arbitrary fan angle and tilt angle can be generated using the developed image processing algorithm, and the 3D glasses can enjoy the stereoscopic image in an arbitrary direction .

By using this technique, a stereoscopic image in an arbitrary direction can be enjoyed.

Figure 1 is an all-round painting of Robert Barker.
2 is an embodiment of an image system using a panoramic mirror having a linear aberration correcting projection method.
Fig. 3 is an example of an indoor panoramic image captured by the omnidirectional imaging system of Fig. 2;
4 is an example of an unfolded omnidirectional image of FIG.
5 is a portion of the unfolded panoramic image of Fig.
Figure 6 is a view of the Globuscope.
FIG. 7 is a graphical illustration of the Seitz Roundshot 35/35.
FIG. 8 is a DSC-HX1 digital camera manufactured by Sony having an omnidirectional imaging function using stitching technology.
9 is a panoramic image taken with the DSC-HX1.
10 shows a Samsung Galaxy camera having an omni-directional imaging function using stitching technology.
Fig. 11 shows a panoramic image taken by a Galaxy camera.
12 is a panorama image captured by the Panorama 360, a smartphone application having an omnidirectional imaging function using a stitching technique.
13 is another example of the omni-directional image captured by Panorama 360. Fig.
14 is a photograph showing that the view angle in the horizontal direction of the omnidirectional image of Fig. 13 is 360 占 Fig.
15 is a photograph illustrating the technical limitations of the omni-directional image by the stitching technique.
16 is a three dimensional panoramic camera of Kodak Company.
17 shows a cardboard viewer of Google.
18 is an example of a stereoscopic image shown as a head-mounted display;
Fig. 19 shows an anaglyph glass as one of the 3D glasses. Fig.
20 is a 3D camera of GoPro equipped with two fisheye lenses.
FIG. 21 is a conceptual diagram of stereoscopic image acquisition by image processing in a 3D camera using two fisheye lenses. FIG.
FIG. 22 is a conceptual diagram illustrating that a 3D camera shown in FIG. 20 can not view a stereoscopic image in an arbitrary direction. FIG.
23 is an embodiment of another omni-directional camera of GoPro.
24 is an embodiment of another 3D omnidirectional camera of GoPro.
25 is a conceptual diagram of a stereoscopic omnidirectional camera using a rotating pair of cameras of Prof. Hung;
26 is a conceptual diagram of a three-dimensional 360 ° omnidirectional camera using a pair of rotating cameras.
27 is an example of an omni-directional image captured by a stereoscopic 360 DEG omnidirectional camera illustrated in Fig.
28 is a conceptual diagram for generating a stereoscopic image from a pair of 360 ° omni-directional images corresponding to the left eye and the right eye;
FIG. 29 is a conceptual diagram of a technique for generating a 360 ° stereoscopic omnidirectional image in a rotating camera. FIG.
30 is an example of a linear aberration correction image extracted from the omnidirectional image of Fig.
31 is a plan view of the omnidirectional camera of the first embodiment of the present invention.
32 is a side view of the omnidirectional camera of the first embodiment of the present invention;
33 is a conceptual diagram of a omni-directional image acquired by the omnidirectional camera of the first embodiment of the present invention;
34 is a photograph of a fish-eye lens according to an embodiment of the present invention;
35 is a three-dimensional incision view of a fish-eye lens of an embodiment of the prior art.
36 is a view showing an optical structure of a fish-eye lens and a path of a light ray according to an embodiment of the present invention;
37 is a conceptual diagram of an actual projection method of a generally rotationally symmetric imaging lens.
38 is a view for understanding the relationship between the coordinates in the horizontal direction and the incident angle in the horizontal direction on the 360 ° omnidirectional image plane.
FIG. 39 shows a virtual 360 ° omni-directional image plane.
40 is a virtual 360-degree omnidirectional image plane generated from the fisheye image of the conventional embodiment.
41 is a conceptual diagram of a linear aberration corrected image plane extracted from an omnidirectional image plane.
42 is a view of a linear aberration-corrected image plane generated from the 360-degree omnidirectional image plane of Fig.
43 is a view of a linear aberration-corrected image plane generated from the 360-degree omnidirectional image plane of Fig. 40;
44 is another linear aberration corrected image plane generated from the 360 DEG omni-directional image plane of Fig.
45 is a conceptual diagram illustrating a process of acquiring an omni-directional image using the omnidirectional camera of the first embodiment of the present invention;
46 is a conceptual diagram of the omnidirectional image obtained in the first embodiment of the present invention;
47 is a conceptual diagram illustrating a process of acquiring an omni-directional image using the omnidirectional camera of the first embodiment of the present invention when the subject moves;
48 is a conceptual diagram of the omnidirectional image obtained in the first embodiment of the present invention when the subject moves;
49 is a conceptual diagram illustrating a method of operating the omnidirectional camera of the second embodiment of the present invention;
50 is a plan view of the stereoscopic omniazimetic image acquisition apparatus according to the third embodiment of the present invention;
51 is a side view of a stereoscopic omniazoid image acquisition apparatus according to the third embodiment of the present invention;
FIG. 52 is a diagram for understanding characteristics of a linear aberration corrected image obtained by a rotating three-dimensional omniazimetic image acquisition apparatus; FIG.
53 is a view for understanding the camera structure of the third embodiment of the present invention;
54 is a conceptual diagram illustrating a camera structure of a third embodiment of the present invention;
FIG. 55 is a conceptual view showing a state in which the stereoscopic omnidirectional camera of the third embodiment of the present invention is focused on a subject at an infinite distance; FIG.
FIG. 56 is a conceptual diagram showing a state in which a stereoscopic omnidirectional camera according to a third embodiment of the present invention is focused on a close object. FIG.

[Example 1]

31 is a plan view of the omnidirectional camera of the first embodiment of the present invention. A camera body 3114 having an imaging lens 3112 and a line scan sensor 3113 therein is mounted on a rotating part 3122 which is a base part of the lower part, The rotation axis 3126 can rotate infinitely about a rotation axis 3126 with respect to the rotation axis 3124 and the rotation axis 3126 passes perpendicularly to the nodal point N of the image formation lens.

The preferred vertical angle of view of the omnidirectional image plane varies depending on the application. For example, if the purpose is to monitor the four sides in the middle of the sea, the vertical angle of view is very small. In fact, since only the horizontal line is monitored, the smaller the angle of view of the lens, the farther away you can see it. On the other hand, if you want to photograph a room in a palace filled with paintings and sculptures on walls and ceilings, it would be desirable to use a fisheye lens with a viewing angle of 180 ° or more.

By the way, the basic structure of the omnidirectional camera which uses a general aberration correcting lens with a small angle of view and a fisheye lens with an angle of view of 180 degrees or more is almost the same. In fact, it is desirable to design a omnidirectional camera with a lens interchangeable type and to use it while changing a telephoto lens, a standard lens, or a wide-angle lens according to the use. Therefore, even if an imaging lens is referred to as a fisheye lens for the sake of discussion, the scope of the present invention is not limited to a fisheye lens.

The World Coordinate System is fixed to the base 3124, and the origin of the world coordinate system coincides with the nodal point of the fisheye lens. The Y-axis of the world coordinate system coincides with the rotation axis. On the other hand, the Z-axis of the world coordinate system coincides with the optical axis 3101 of the fish-eye lens when the omnidirectional camera is in the initial zero state. The world coordinate system uses the Right Handed Coordinate System.

When the omnidirectional camera is in the initial stop state, the camera coordinate system coincides with the world coordinate system, but when the rotation unit rotates, it rotates as the rotation unit. The Y'-axis of the camera coordinate system always coincides with the Y-axis of the world coordinate system. On the other hand, the Z'-axis of the camera coordinate system coincides with the optical axis 3101 of the fisheye lens.

Since the fisheye lens 3112 has an angle of view of 180 ° or more, an image corresponding to at least a hemisphere is generated on the image plane 3132. The line scan sensor 3113 in the camera body 3114 is arranged in the Y'-axis direction. Therefore, the image of the subject in the longitudinal direction on the front face of the fish-eye lens is captured by the line-scan sensor. As the rotation unit rotates, images in different directions are successively captured by the line scan sensor. When the camera rotates one turn, an image of 360 degrees in all directions is obtained.

32 is a side view of the omnidirectional imaging apparatus. In FIG. 32, it can be seen that the Y-axis of the world coordinate system and the Y'-axis of the camera coordinate system pass the nodal point N of the fish-eye lens. Also, it can be seen that the Z'-axis of the camera coordinate system passes the nodal point of the fish-eye lens, and the line scan sensor 3213 in the camera body 3214 is parallel to the Y-axis and the Y'-axis.

The rotation part 3222 equipped with the camera is connected to the base part 3224 by a rotation shaft 3226 and rotates the rotation part at a high speed by a motor inside the base part. In addition, a position sensor 3242 is mounted on the base portion to measure the tilt in two directions with respect to the horizontal plane of the camera, that is, the tilt in the X-axis direction and the tilt in the Z- do. In addition, if GPS and digital compass are included, it is convenient to link the omni-directional image with the map.

The rotating part is connected to the base part and the rotating shaft, and can rotate infinitely. On the other hand, since the signal from the camera may have a plurality of signal lines, it may be difficult to transmit the video signal from the rotation unit to the base unit. Therefore, a transmitter 3244 may be installed in the rotation unit, and a receiver 3246 may be installed in the base unit 3224 to transmit a video signal to a wireless signal or a light signal. Of course, power and control signals must also be sent in the opposite direction. The power source may be transmitted via a slip ring or by using a near field wireless power transmission technique. Meanwhile, the control signal may be transmitted through a slip ring, or an optical signal or a radio signal may be used.

32 shows that the transmitting part 3244 and the receiving part 3246 are provided near the rotating shaft 3226. In reality, the rotating shaft 3226 is formed as an empty cylindrical inner part, wireless signal, or light. Using Direct Drive Motor (DDM) technology, it is possible to integrate the base of the omnidirectional camera with the motor, and the power, control signal and video signal can come and go through the center hole of the DDM.

FIG. 33 is a conceptual diagram of a 360-degree omnidirectional image plane acquired by the omnidirectional camera of the first embodiment of the present invention, and the unit of the coordinate system is a pixel. The horizontal axis of the 360 ° omnidirectional image plane 3334 corresponds to the X'-axis of the camera coordinate system, and the vertical axis corresponds to the Y'-axis of the camera coordinate system. The reference point C of this coordinate system corresponds to incident light incident along the optical axis when the Z'-axis of the camera coordinate system coincides with the Z-axis of the world coordinate system. Therefore, the elevation angle and the azimuth angle of the incident light forming the image point at the reference point C on the 360 ° omnidirectional image plane 3334 are all zero.

The size of the 360 ° omni-directional image plane in the vertical direction is K _max , which has a natural number value. The size of the vertical direction corresponds to the number of effective pixels of the line scan sensor. That is, if the line scan sensor has 2048 effective pixels arranged in one column, K _max becomes 2048. For the sake of discussion, a 360 ° omni-directional image plane, an omni-directional image plane, and an omni-directional image will be used in combination.

On the other hand, the horizontal size L _max of the omnidirectional image plane corresponds to the number of sampling during one rotation of the omnidirectional camera. Accordingly, as the rotation speed of the rotation unit is slower and the output speed of the line scan sensor is faster, the horizontal direction size of the omnidirectional image plane becomes larger.

The angle of view in the horizontal direction of the omnidirectional image plane is exactly 360 °. That is, every time the rotation unit rotates once, one 360 ° omni image plane 3334 one frame is generated. On the other hand, the vertical angle of view of the omnidirectional image plane corresponds to the angle of view of the fish-eye lens. That is, when the angle of view of the fish-eye lens is 190 °, the angle of view 2Γ in the vertical direction of the image plane in all directions becomes 190 °.

It is considered that 180 degrees in the longitudinal direction of the omnidirectional image plane is sufficient, but it is more preferable that the angle of view is larger. The reason for this is to obtain a satisfactory image even when the digital tilt angle is large.

One point Q on the omnidirectional image plane 3334 has a coordinate value (K, L), which has azimuth angle? In the transverse direction and elevation angle? In the vertical direction. That is, the incident light reaching the pixel having the coordinate value (K, L) has the azimuth angle? And the altitude angle?.

FIG. 34 is a photograph of a fish-eye lens (model name: FEED802) of a 190 ° angle of view described in Reference Documents 14 and 20, and FIG. 35 is a photograph of a fish- Dimensional perspective view. On the other hand, Fig. 36 shows the optical structure of the fish-eye lens and the path of the light ray.

37 is a conceptual diagram of a real projection scheme of a rotationally symmetric imaging lens 3712 including a fish-eye lens. The Z-axis of the world coordinate system that describes the object to be captured by the imaging lens coincides with the optical axis 3701 of the imaging lens 3712. [ The incident light 3705 having a zenith angle &thetas; with respect to the Z-axis is refracted by the imaging lens 3712 and then converged to an image point P on the focal plane 3732 as refracted light 3706 . The distance from the nodal point N of the imaging lens to the focal plane is approximately equal to the effective focal length of the lens. The portion where the actual shop is formed on the focal plane is an image plane 3733. The image surface 3733 and the image sensor surface 3713 inside the camera body 3714 must match to obtain a clear image. The focal plane and the image sensor plane are perpendicular to the optical axis. The distance from the intersection O - hereinafter referred to as the first intersection - of the optical axis 3701 to the image plane 3733 to the store P is r.

The image height r in a general image-forming lens which is rotationally symmetrical about the optical axis is given by Equation (2).

The unit of the incident angle? Is radian, and the function f (?) Is a monotonically increasing function passing through the origin passing through the origin for the zenith angle? Of the incident light.

The actual projection method of such a lens can be experimentally measured with an actual lens, or it can be calculated using a lens design program such as Code V or Zemax with a lens design. For example, using the REAY operator in Zemax, you can compute the y-axis coordinate y on the focal plane of the incident light with a given horizontal and vertical incidence angles, and the x-axis coordinate x is similar to the REAX operator . ≪ / RTI >

The fisheye lens (FEED802) having the angle of view of 190 degrees used in the embodiment of the present invention is well described as a polynomial passing through the origin as described in detail in [Reference 14].

However, in the omnidirectional image obtained by the rotating omnidirectional camera as in the first embodiment of the present invention, the image size r is replaced by the image size y in the vertical direction, and the incidence angle? Therefore, equation (4) holds.

If the distance between adjacent pixels in the line scan sensor is p, then the pixel coordinate K on the omnidirectional image plane and the elevation angle? Of incident light satisfy equation (5).

Here, the unit of image size is millimeter (mm). For example, if using the S11637-1024Q sensor from Hamamatsu, the sensor has a total of 1024 pixels, and each pixel has a size of 12.5 μm × 500 μm. Therefore, Kmax = 1024 and p = 0.0125 mm. Therefore, the gain g of the pixel is 80.0.

If there is no information about the pixel pitch of the imaging lens or the line scan sensor used for image acquisition, the user has no choice but to measure the projection method of the lens in the 360 ° omnidirectional image plane, useful.

Here, F ([delta]) is a vertical projection method of the imaging lens measured in pixel units.

38 is a view for understanding the imaginary radius S of the omnidirectional image plane of the embodiment of the present invention. Since the horizontal size of the omnidirectional image plane is L _max and the angle of view in the horizontal direction is 360 ° (that is, 2π), the omnidirectional image plane can be regarded as a cylindrical surface having an imaginary radius S given by Equation (7).

Therefore, the column number L of the pixel having the azimuth angle? Is given by Equation (8).

Conversely, the pixels having the column number L correspond to the incident light having the azimuthal angle of the horizontal direction given by Equation (9).

39 is a virtual 360 ° omnidirectional image plane, which is generated based on an artificial fisheye image produced by Professor Paul Bourke. 40 is a virtual 360 ° omni-directional image plane generated from a fish-eye image captured by a fish-eye lens of the embodiment of the present invention. It should be noted that L _max and K _max can have arbitrary ratios irrespective of the relationship between the viewing angle in the horizontal direction and the viewing angle in the vertical direction.

The ratio of the horizontal size to the vertical size of the omnidirectional image plane proposed by the prior art is almost unchanged at 2: 1 for the following reasons. In the polar coordinate system, the range of hardness is 360 ° from -180 ° to 180 ° and the range of latitude is 180 ° from -90 ° to 90 °. Therefore, if you draw a world map with an equirectangular projection scheme, the aspect ratio becomes 2: 1.

In the meantime, the size of the omnidirectional image plane in the present invention is equal to the number of effective pixels of the line scan sensor, and the size of the omnidirectional image plane is determined by the operation speed of the line scan sensor and the rotation speed of the rotation unit, And the size in the vertical direction may have an arbitrary ratio. Further, although the angle of view in the horizontal direction is 360 degrees, the angle of view in the vertical direction may have an arbitrary value depending on the angle of view of the image-forming lens.

Thirdly, the pixels adjacent in the horizontal direction in the omnidirectional image plane according to the embodiment of the present invention all correspond to the same azimuth increment. However, vertically adjacent pixels correspond to an increment of any elevation angle. This is because pixels corresponding to each row can correspond to an arbitrary altitude angle depending on whether the lens is a linear aberration correcting lens or a fisheye lens.

The present invention provides an image processing method capable of extracting a correct linear aberration-corrected image plane in an omni-directional image plane having an arbitrary horizontal-to-vertical ratio and an arbitrary vertical-direction projection method.

In the omnidirectional image plane illustrated in Fig. 33, the reference point C coincides with the direction of the camera optical axis when the omnidirectional camera is at rest. Therefore, although the coordinates are preferably given by the equations (10) to (11), they may have arbitrary values.

On the other hand, FIG. 41 is a conceptual diagram of the linear aberration corrected image plane 4131 extracted from the omnidirectional image plane of FIG. The coordinates of the center point? Of this linear aberration corrected image plane are given by the following equations (12) to (13).

If the horizontal angle of view of this linear aberration-corrected image plane is DELTAΨ, then the distance T from the origin of this linear aberration-corrected image plane is given by Equation (14).

Therefore, the coordinates of a point Q on the linear aberration-corrected image plane before digital pan / tilt is given as shown in equation (15).

In order to obtain a straight-line aberration-corrected image plane in which the angle of view in the horizontal direction is?, The azimuth angle of the principal direction of vision is? And the altitude angle is? In the omnidirectional image plane of the embodiment of the present invention, The aberration corrected image plane is panned about the Y-axis by?, And the coordinate values in the new coordinate system tilted by? Are centered on the X'-axis of the rotated coordinate system, and are given by the following equations (16) to (19).

The elevation angle and the azimuth angle of a point Q "on the pan / tilted linear aberration correction image plane are given by the equations (20) to (21).

The coordinates of a point on the omnidirectional image plane are given by equations (22) to (23) based on the information about the azimuth angle and elevation angle of the new coordinate system.

In this manner, all points on the linear aberration-corrected image plane 4131 are subjected to image processing to obtain a pan / tilted linear aberration-corrected image plane from the omnidirectional image plane.

FIG. 42 is a view of a linear aberration-corrected image plane generated in the omnidirectional image plane of FIG. 39, with a size of 240 pixels in the horizontal direction, a size of 180 pixels in the vertical direction, an angle of view of 90 degrees in the horizontal direction, And the tilt angle is 45 DEG. As can be seen in FIG. 42, a linear aberration corrected image in a desired direction is obtained, and all of the straight lines are captured as a straight line.

FIG. 43 shows a linear aberration correction image extracted from the omnidirectional image of FIG. 40, in which the size in the horizontal direction is 640 pixels, the size in the vertical direction is 480 pixels, the angle of view in the horizontal direction is 90 degrees, 0 deg. 44 is a linear aberration correction image obtained by using a fan angle of -90 DEG and a tilt angle of -45 DEG.

As can be seen from FIG. 42 to FIG. 44, by using such an algorithm, a rectilinear aberration-corrected image in an arbitrary direction can be obtained from an omni-directional image obtained from a rotating omnidirectional camera.

 The following is the MatLab code used to generate the linear aberration corrected image plane of FIG.

% Rectilinear image generation from a 360 deg. panoramic image.

% Programmed by Gyeong-il Kweon, the Super-Genius.

% D: \ S360VR 2017 / main.m

% format long

% 2017. 9. 2.

%

% *********** Real projection scheme **********************

% r as a function of theta

coeff = [-3.101406e-2, 7.612269e-2, -1.078742e-1, 3.054932e-2, 1.560778, 0.0];

%

% *** Reads the graphic image ***************************************

picture = imread ('image', 'jpg');

[Kmax, Lmax, Qmax] = size (picture);

CI = double (picture) + 1;

%

S = Lmax / (2 * pi); % Radius of the cylinder

%

Lo = (1 + Lmax) / 2;

Ko = (1 + Kmax) / 2;

gain = 312.5; 1/2-inch sensor

%

% Draw an empty canvas

Jmax = 640; % canvas width

Imax = 480; % canvas height

Jo = (1 + Jmax) / 2;

Io = (1 + Imax) / 2;

EI = zeros (Imax, Jmax, 3); % dark canvas

%

Dpsi = 90.0 / 180.0 * pi;

T = (Jmax - 1) / 2.0 / tan (Dpsi / 2.0);

%

ALPHA = -45.0;

alpha = ALPHA / 180.0 * pi;

sin_alpha = sin (alpha);

cos_alpha = cos (alpha);

%

BETA = -90.0;

beta = BETA / 180.0 * pi;

sin_beta = sin (beta);

cos_beta = cos (beta);

%

% Virtual screen

for I = 1: Imax

    for J = 1: Jmax

        X = J - Jo;

        Y = I - Io;

        Z = T;

        %

        Xp = X * cos_beta + Y * sin_beta * sin_alpha + Z * sin_beta * cos_alpha;

        Yp = Y * cos_alpha - Z * sin_alpha;

        Zp = -X * sin_beta + Y * cos_beta * sin_alpha + Z * cos_beta * cos_alpha;

        %

        rho_p = sqrt (Xp ^ 2 + Zp ^ 2);

        delta_p = attan2 (Yp, rho_p);

        psi_p = atan2 (Xp, Zp);

        y_p = Ko + gain * polyval (coeff, delta_p);

        x_p = Lo + S * psi_p;

        %

        Km = floor (y_p);

        Kp = Km + 1;

        dK = y_p - Km;

        Lm = floor (x_p);

        Lp = Lm + 1;

        dL = x_p - Lm;

        (Km &lt; = 1) & (Kp < = Kmax) & (Lm &

            EI (I, J,) = (1 - dK) * (1 - dL) * CI (Km, Lm,

                + dK * (1 - dL) * CI (Kp, Lm, ...)

                + (1 - dK) * dL * CI (Km, Lp, ...)

                + dK * dL * Cl (Kp, Lp, :);

        else

            EI (I, J, :) = zeros (1, 1, 3);

        end

    end

end

DI = uint8 (EI - 1);

imwrite (DI, 'result.jpg', 'jpg');

imagesC (DI);

axis equal;

[Example 2]

45 is a conceptual diagram illustrating a method of acquiring an omni-directional image using the omnidirectional camera of the first embodiment of the present invention. As shown in FIG. 45, if all of the subjects are in the stopped state, there is no technical difficulty to generate the omni-directional image with the rotating camera, and a natural omni-directional image as shown in FIG. 46 will be obtained.

However, as shown in Fig. 47, when the subject moves in addition to the rotating camera, distortion of an image called motion blur occurs. At this time, there occurs a phenomenon such as a distortion which occurs when a document to be copied is moved together while making a copy with a copying machine. If the scanner's built-in scanner is moving in the same direction as the document, the copied image will be stretched, while in the opposite direction, the copied image will be shortened.

The same phenomenon occurs in the situation of FIG. 47, the police patrol car is moving in the opposite direction with respect to the direction of camera rotation, and the truck is moving in the same direction. Then, as illustrated in FIG. 48, the police patrol car appears to be compressed in the longitudinal direction in the omnidirectional image, and the truck is captured to be elongated in the longitudinal direction. On the other hand, all non-moving trees are normally captured. Also, the faster the subject's speed is, the worse the motion blur will be.

Figure 49 shows a method for reducing such motion blur. A line scan sensor with a faster output speed is capable of outputting several million frames per second. A high-speed motor can rotate up to several hundred revolutions per second. Using such hardware capabilities can reduce motion blur to negligible levels.

Suppose you need 30 frames per second to get full-motion video. Then, the rotating part of the camera rotates in multiples of 30. That is, rotate at a multiple of 30, such as 60 revolutions per second, 90 revolutions per second, 300 revolutions per second.

For example, if the rotating part rotates 90 times per second, the camera generates a single omni-directional image by integrating the images output from the line scan sensor when the rotating part rotates first. When the second and third rotations are performed, idling is performed without acquiring the image. When the fourth rotation is performed again, the image output from the line scan sensor is integrated to generate one omni-directional image, the idle rotation which does not acquire the image is performed in the fifth and sixth rotation, and the omni-directional image is generated again in the seventh rotation .

Using this method, the speed of the camera is relatively higher than the speed of the subject, which reduces image distortion due to motion blur. If the rotation of the camera is 300 revolutions per second, the effect of motion blur is expected to be minimal, except in special cases.

[Example 3]

50 is a plan view of a stereoscopic omniazimensional image acquisition apparatus (stereoscopic omnidirectional camera) according to the third embodiment of the present invention. Two cameras having the same specifications and composed of the fisheye lenses 5012L and 5012R having a view angle of 180 degrees or more and the camera bodies 5014L and 5014R having the line scan sensors 5013L and 5013R inside thereof, And the rotary part 5022 is connected to the lower base part 5024 by a rotary shaft 5026. [ The rotary part horizontally rotates about the origin C on the rotary part. The origin C is located at the center point of the nodal points N _L and N _R of the two fisheye lenses 5012L and 5012R. The distance D between the nodal points of the two fisheye lenses is preferably 6.35 cm, which is the average distance between two eyes of a person. Further, the optical axes 5001L and 5001R of the two fisheye lenses rotate together when the rotating part rotates.

The direction parallel to the optical axes 5001L and 5001R of the two fisheye lenses passing through the origin C is the Z'-axis of the camera coordinate system, and the direction passing through the origin C and the nodal points N _L and N _R of the two fisheye lenses is the camera coordinate system Axis direction of the camera coordinate system, and the direction passing through the origin C and parallel to the rotation axis is the Y'-axis direction of the camera coordinate system. The camera coordinate system is the right hand coordinate system.

On the other hand, when the 3D omnidirectional camera is in the initial zero position, the direction in which the optical axes 5001L and 5001R of the two fisheye lenses are in the Z-axis direction of the world coordinate system, and the origin of the world coordinate system is the origin C . The Y-axis of the world coordinate system coincides with the Y'-axis of the camera coordinate system. The world coordinate system is also the right hand coordinate system.

In such a coordinate system, the nodal points of the two fisheye lenses are always located in the X-Z plane and the X'-Z 'plane. In addition, the line scan sensors 5013L and 5013R of the two cameras are always parallel to the Y-axis of the world coordinate system and the Y'-axis of the camera coordinate system. When the rotating part rotates at a constant speed and acquires an image with two cameras and splices the images, two omnidirectional image planes are generated each time the rotating part makes one rotation.

51 is a side view of a 3D omni-directional imaging system according to the third embodiment of the present invention, and includes an image acquisition device 5120, an image processing device 5130, and an image display device 5140. The image acquiring device 5120 includes a rotation unit and a base unit 5124 on which the two cameras are mounted. In the base unit, the two omnidirectional image planes 5126L and 5126R obtained by rotating the two cameras are combined, Plane 5126 is generated. That is, every time the rotation unit makes one rotation, one double-omnidirectional image plane 5126 is generated, and the double-omnidirectional image plane 5126 is generated by combining the omnidirectional image plane 5126L obtained from the left camera and the omnidirectional image plane 5126R ) Is an omnidirectional image plane obtained by one.

The dual omni image plane 5126 obtained from the image acquiring device is transmitted to one or more image processing devices 5130 in real time or posteriorly. The image processing apparatus may be a desktop computer, a notebook computer, a NVR (Network Video Recorder) which is a computer in fact, a smart phone or a dedicated terminal, and a CPU or AP (Application Processor) .

The image processing apparatus 5130 stores one or more frames of the dual omni image plane 5132 transmitted from the image capturing apparatus 5120. The image processing apparatus 5130 is connected to the image display apparatus 5140 by wire or wireless, and an HMD (Head Mounted Display) apparatus can be used as a typical image display apparatus. When a user wearing the HMD wears the HMD, a direction indicator incorporated in the HMD extracts direction information of the user and transmits the extracted direction information to the image processing apparatus. The image processing apparatus performs linear aberration correction Image pairs are extracted from the left omni-directional image 5134L and the right omni-directional image 5134R, respectively, and displayed on the image display device. Accordingly, the user wearing the HMD can appreciate the linear aberration-corrected image corresponding to the direction of the head of the user, and the image corresponding to the binocular vision is seen from the left omnidirectional image 5134L and the right omnidirectional image 5134R You can feel the stereoscopic feeling as if you are actually in the place.

The direction information sent from the image display apparatus 5140 to the image processing apparatus 5130 is preferably an azimuth angle ψ and an altitude angle δ toward which the viewing direction of the linear aberration corrected image is directed.

The image display device 5140 may be various devices as well as an HMD. For example, the image processing apparatus 5130 may be a main body of a computer, and the image display apparatus may be a 3D monitor. Alternatively, the image processing apparatus extracts a pair of linear aberration-corrected images from the dual omni-directional image plane, and then converts the pair of the linear aberration-corrected images into a computer screen, a TV screen, or a smart phone screen so as to view it as an anaglyph glasses. You can watch it with anaglyph glasses.

The 3D omni-directional imaging apparatus of the third embodiment of the present invention may be used to generate a still image or to generate a moving image. Particularly, if it is used for real-time broadcasting in a 3D omnidirectional image of a stadium or a venue in which scenic scenery or an exciting game is played, the dual omni image plane generated by the same image acquiring device 5120 may be used for a plurality of image processing apparatuses 5130 Lt; / RTI &gt; At this time, it is also possible to transmit only a part of the omnidirectional image plane corresponding to the linear aberration correction image required by each image processing apparatus in order to reduce the traffic amount of the network.

52 is a view for understanding characteristics of a linear aberration-corrected image plane that can be extracted from an omnidirectional image plane obtained from a rotating omnidirectional camera. Since the nodal points (N _L , N _R ) of the two fisheye lenses have an interval D, the two nodal points move along a circle having a diameter D as the rotation portion rotates about the origin C.

If the rotation unit rotates and the Z'-axis of the camera coordinate system forms an angle? With the Z-axis of the world coordinate system, the optical axes 5201L and 5201R of the two fisheye lenses are also directed in the same direction. When the position of the nodal point of the two fisheye lenses is the same as shown in FIG. 52, the desired field-of-view range 5205L, 5205R is obtained when the straight- It may be desirable to obtain an image having an image. However, since the line scan sensor acquires only one line of image in one direction when the rotation unit is in one position, in order to obtain an image having a horizontal angle of view other than 0, the rotation unit must continuously acquire images during rotation .

Since the nodal point of the fish-eye lens moves along the circle 5203 while the rotation unit rotates, it can not have a mathematically strict viewpoint. In FIG. 52, it is assumed that an image having an angle of view of 90 degrees in the horizontal direction is obtained. In this case, images obtained during 1/4 rotation of the rotating part should be used. However, if we look at the arc where the nodal point of the fisheye lens moves in the meantime, it can be seen that the fisheye lens is rather parallel rather than perpendicular to the direction of the optical axis of the fisheye lens. Therefore, even if an incomplete linear aberration correction image is extracted from the incomplete omni-directional image obtained while the nodal point of the fish-eye lens is moved, the error is not serious. Also, it can be seen that during the acquisition of the omni-directional image or the linear aberration corrected image, the time point is present in a circle of confusion of 6.35 cm in diameter.

If the diameter of the chaotic circle is D, it can be said that there is a certain degree of error in the omnidirectional image or the linear aberration corrected image of the object at a close distance based on this. However, if the subject's distance is more than 10 times the diameter of the chaotic circle, the error can be expected to be almost non-noticeable. By the way, if the diameter of the chaotic circle is 6.35cm, the 10 times is 63.5cm and it does not become 1m. Therefore, it is expected that a satisfactory image can be obtained even with a rotating omnidirectional camera, unless it is a close-up photographing.

53 is a view for understanding the camera structure of the third embodiment of the present invention. The pixels of the line scan sensor are much larger than the pixels of the area image sensor. The reason for this is as follows. An area image sensor capable of obtaining a full HD image would have a matrix of pixels with 1920 rows in the horizontal direction and 1080 rows in the vertical direction. If you want to take a video using this kind of image sensor, capture 30 frames per second is enough.

On the other hand, in order to capture a full HD image with a line scan sensor, it is necessary to capture an image of 1920 by a line scan sensor having 1080 pixels. Accordingly, in order to capture a moving image of 30 frames per second, the line scan sensor must capture an image of 1920 × 30 pixels per second, and a single image is captured within 20 microseconds (μs). I can not receive it.

The above-described Hamamatsu S11637-1124Q sensor has a pixel pitch of 12.5 mu m, a pixel height of 500 mu m, and an image sensor area of 12.8 mm, which is much larger than the area iamge sensor. In addition, the length of the image sensor area of S11637-2048Q is 25.6 mm.

When the lengths of the line scan sensors 5313L and 5313R are formed on the line scan sensor, the diameter of the image plane 5333L should be similar. However, in the wide-angle lens, the first lens element is always larger than the diameter of the image plane, and if the wide-angle lens is a fisheye lens, the diameter of the first lens element is much larger than the diameter of the image plane. Therefore, the interval between the imaging lenses 5312L and 5312R can not be maintained at the interval D between the two eyes of the average person. If the distance between the centers of the two lenses is significantly different from D, the image obtained from such a stereoscopic omnidirectional camera does not provide a natural stereoscopic effect.

54 shows the structure of a camera according to an embodiment of the present invention. Since the line scan sensor 5413 is long in the longitudinal direction and the distance between the two line scan sensors should be maintained at about 6.35 cm, each camera should be made in a slender shape having a width within 3 cm. Therefore, it is impossible to use lens elements having a round cross-sectional area such as a general image-forming lens, and both sides should be cut out. The lens 5412 made in such a slender shape can not be pinched or pulled out by the lens holder 5416 like a general lens. Therefore, the lens holder 5416 is fixed to the body of the line scan camera 5414, and the imaging lens 5412 should be made to be slidably inserted into the lens holder 5416. The insertion portion 5418 of the imaging lens 5412 has a size that fits into the hole of the lens holder 5416. [

Since the distance between the imaging lens 5412 and the line scan sensor 5413 must be adjustable in order to focus on a subject at a distance or a close range, a means 5442 for moving the imaging lens is needed. Although FIG. 54 shows a rack and a pinion gear, various means such as a linear motor can be actually used.

In addition, since the line scan sensor can only operate at a constant shutter speed in order to obtain an omnidirectional image, a separate means for adjusting the exposure is required. Thus, a motorized camera iris is preferred.

The imaging lens mounted on the camera has a viewing angle in the direction of the axis of rotation of 180 ° or more and a cross section perpendicular to the optical axis of the imaging lens is a slender shape measured in the direction perpendicular to the axis of rotation, , The long axis of the imaging lens is held in the direction of the rotation axis even if the imaging lens moves in the optical axis direction.

55 is a conceptual diagram of a stereoscopic omnidirectional camera according to the third embodiment of the present invention. The rotation unit 5522 can rotate infinitely with respect to the bottom base unit 5524, and two cameras 5512L and 5512R are mounted on the rotation unit.

56 shows a shape in which the optical axes 5601L and 5601R of the left camera 5612L and the right camera 5612R are inclined inward toward the Z'-axis of the camera coordinate system when focusing on a close object. Of course, without operating in this way, the optical axes 5601L and 5601R of the two cameras can always be kept parallel to the Z'-axis of the camera coordinate system regardless of the distance of the subject.

Such a stereoscopic omni-directional imaging system can be used to generate high resolution omnidirectional images or 3D omnidirectional images such as sightseeing, buildings, stadiums, and venues, and can be used as a 3D omnidirectional broadcasting apparatus for broadcasting exciting games or performances in real time . The user can extract a rectilinear aberration-corrected image in an arbitrary direction from the transmitted double-sided image and listen to the stereoscopic image with an apparatus such as an HMD.

5022:
5026:
5024:
5012L, 5012R: image forming lens
5014L, 5014R: Camera body
5013L, 5013R: Line scan sensor
5001L, 5001R: optical axis of the image forming lens
N _L , N _L : Nodal point

Claims (16)

An image processing method for generating a linear aberration corrected image plane on a 360 ° omni-directional image plane,
The size of the 360 ° omni-directional image plane in the vertical direction is K _max and the size in the horizontal direction is L _max ,
Where K _max and L _max are natural numbers greater than 1,
The view angle in the horizontal direction of the 360 ° omni-directional image plane is 360 °,
Let (X, Y, Z) be the coordinates of a virtual object point on the world coordinate system captured by a 360-degree omnidirectional image plane in a world coordinate system with one horizontal direction as a Z-axis and a vertical direction as a Y- ,
The elevation angle of the object point is given by the following equation,

The azimuth angle of the object point is given by the following equation,

The coordinates of the reference point, which is a shop on the 360-degree omnidirectional image plane corresponding to the Z-axis on the world coordinate system, is given as (K _o , L _o )
Where K _δ is a real number greater than 1 and less than K _max ,
L _o is a real number greater than 1 and less than L _max ,
The row number K of the shop corresponding to the elevation angle [delta] is given by the following equation,

Where F is the monotone increasing function across the origin,
The azimuth angle? _L of the object point corresponding to the shop having the column number L is given by the following equation,

The size of the linear aberration corrected image plane in the vertical direction is I _max and the size in the horizontal direction is J _max ,
Where I _max and J _max are natural numbers greater than 1,
The angle of view in the horizontal direction of the linear aberration-corrected image plane is &lt; RTI ID = 0.0 &gt;
The coordinates of the center of gravity on the linear aberration corrected image plane are given by (I _o , J _o )
Where I _o is given by the following equation,

J _o is given by the following equation,

When the azimuth angle of the viewing direction in which the linear aberration-corrected image plane faces is? And the altitude angle is?
The image signal of the shop having the coordinates (I, J) on the linear aberration corrected image plane is given as a video signal of a virtual shop having virtual coordinates (x _{I, J} , y _{I, J} ) But,
Wherein the coordinates (x _{I, J} , y _{I, J} ) are given by the following series of equations.

The method according to claim 1,
The 360 ° omni-directional image plane is obtained by a camera rotating at a uniform angular velocity around the Y-axis,
And a line scan sensor mounted in the body of the camera in a direction parallel to the Y-axis,
Wherein the nodal point of the imaging lens mounted on the camera is located at the origin of the world coordinate system,
Even if the camera rotates, the optical axis of the lens is always included in the XZ plane,
The actual projection method of the lens

However,
Where r is the image size on the image plane of the lens,
is an incident angle of incident light incident on the lens,
f is a monotone increasing function across the origin,
The line scan sensor has K _max effective pixels,
The spacing between adjacent pixels in the line scan sensor is p,
The function F is given by the following equation,

L _max sampling is performed during one rotation of the camera,
In each sampling, K _max video signals are obtained,
_Wherein the image signals of L _max sampling are all accumulated in the horizontal direction so that a 360 ° omni-directional image plane having a vertical size of K _max and a horizontal size of L _max is generated.
3. The method of claim 2,
Wherein the image-forming lens mounted on the camera is a fisheye lens having an angle of view of 180 degrees or more.
A 360 ° omni-directional image plane acquiring apparatus using a rotating camera,
The 360 ° omnidirectional image plane acquisition device includes a lower base portion and an upper rotation portion,
The rotary part is connected to the base part so as to rotate infinitely about the rotary part about the rotary part,
The base portion is provided with a means for rotating the rotating portion at a uniform angular velocity,
A camera equipped with an image-forming lens is mounted on the rotating portion,
Inside the body of the camera, a line scan sensor is installed in a direction parallel to the rotation axis,
The line scan sensor has K _max effective pixels,
Where K _max is a natural number greater than 1,
The nodal point of the imaging lens is located on the rotation axis,
The optical axis of the imaging lens is perpendicular to the rotation axis,
L _max sampling is performed at uniform time intervals while the rotation unit makes one rotation,
Where L _max is a natural number greater than 1,
Wherein the base unit integrates the image signals obtained from the camera in the lateral direction during one rotation of the rotation unit to generate a 360 ° omni-directional image plane having a vertical size of K _max and a horizontal size of L _max . Image plane acquisition device.
5. The method of claim 4,
Wherein the image-forming lens mounted on the camera is a fisheye lens having an angle of view of 180 degrees or more.
5. The method of claim 4,
The base unit is equipped with a tilt sensor, a digital compass, and a GPS receiver that can measure the position and tilt of a 360 ° omni-directional image acquisition device,
And the image signals obtained from the rotation unit are transmitted to the base unit as optical signals or radio signals.
A dual 360 ° omni-directional image plane acquiring apparatus using a rotating camera,
The 360 ° omni-directional image plane acquiring device includes a lower base portion and an upper rotation portion,
The rotary part is connected to the base part so as to rotate infinitely about the rotary part about the rotary part,
The base portion is provided with a means for rotating the rotating portion at a uniform angular velocity,
In the rotating part, two cameras having the same specifications are installed side by side in the same direction,
In each camera body, a line scan sensor is installed in a direction parallel to the rotation axis,
The line scan sensor has K _max effective pixels,
Where K _max is a natural number greater than 1,
The middle of the nodal points of the two imaging lenses mounted on each camera is located on the rotation axis,
The optical axes of the two imaging lenses are perpendicular to the rotation axis,
L _max sampling is performed at uniform time intervals while the rotation unit makes one rotation,
Where L _max is a natural number greater than 1,
Base is characterized in that in each integrated in the horizontal direction of the video signal obtained from the two cameras during a rotating part 1 rotates generates a pair of vertical direction is K _max is the horizontal direction is a 360 ° omni-directional image plane L _max size of the size of the Dual 360 ° omni-directional image plane acquisition device.
8. The method of claim 7,
Wherein the imaging lens mounted on the camera has an angle of view in the direction of the axis of rotation of 180 or more,
The cross section perpendicular to the optical axis of the imaging lens is smaller in width measured in the direction perpendicular to the rotation axis than in the direction of the rotation axis,
Wherein the long axis of the imaging lens maintains the rotation axis direction even if the imaging lens moves in the optical axis direction.
8. The method of claim 7,
The base unit is equipped with a tilt sensor, a digital compass and a GPS receiver that can measure the position and tilt of the dual 360 ° omni-directional image acquisition device,
And the image signals obtained in the rotation unit are transmitted to the base unit as optical signals or radio signals.
A stereoscopic omniazimensional imaging system including image acquiring means, image processing means, image displaying means, and image selecting means,
The image acquiring means includes a lower base portion and an upper rotation portion,
The rotary part is connected to the base part so as to rotate infinitely about the rotary part about the rotary part,
In the rotating part, two cameras having the same specifications are installed side by side in the same direction,
In each camera body, a line scan sensor is installed in the direction of the rotation axis,
The line scan sensor has K _max effective pixels,
Where K _max is a natural number greater than 1,
The middle of the nodal point of the imaging lens attached to each camera is located on the rotation axis,
L _max sampling is performed at uniform time intervals while the rotation unit rotates once
Come on,
Where L _max is a natural number greater than 1,
The base unit integrates the image signals obtained from the two cameras in the horizontal direction during one revolution of the rotation unit to generate a pair of 360 ° omni-directional image planes having a vertical size of K _max and a horizontal size of L _max ,
The image display means is provided with means for displaying a linear aberration-corrected image whose viewpoint is different from that of the user's left eye and right eye,
The image selecting means extracts the azimuth and altitude information of the direction desired by the user and transmits the information to the image processing means,
Wherein the image processing means generates a pair of linear aberration correction images in pairs of 360 ° omni-directional image planes based on the azimuth and altitude angles transmitted from the image selection means, and transmits the pair of linear aberration correction images to the image display means.
11. The method of claim 10,
Wherein the imaging lens mounted on the camera has an angle of view in the direction of the axis of rotation of 180 or more,
The cross section perpendicular to the optical axis of the imaging lens is smaller in width measured in the direction perpendicular to the rotation axis than in the direction of the rotation axis,
Wherein the long axis of the imaging lens maintains the rotation axis direction even if the imaging lens moves in the optical axis direction.
11. The method of claim 10,
The base of the image acquisition device is equipped with a tilt sensor, a digital compass and a GPS receiver that can measure the position and tilt of the image acquisition device,
And the image signals obtained in the rotation unit are transmitted to the base unit as optical signals or radio signals.
11. The method of claim 10,
The image display means is a head-mounted display device,
Wherein the image selecting means is a direction sensor mounted inside the head-mounted display device.
A stereoscopic omniazimensional imaging system including image acquiring means, image processing means, image displaying means, and image selecting means,
The image acquiring means includes a lower base portion and an upper rotation portion,
The rotary part is connected to the base part so as to rotate infinitely about the rotary part about the rotary part,
In the rotating part, two cameras having the same specifications are installed side by side in the same direction,
The two cameras are equipped with an imaging lens with the same specifications,
Inside the body of each camera, a line scan sensor is installed in a direction perpendicular to the optical axis of each imaging lens,
The line scan sensor has K _max effective pixels,
Where K _max is a natural number greater than 1,
The middle of the nodal points of the two imaging lenses is located on the rotation axis,
The rotating part continues to rotate at the same angular velocity,
The rotation of the rotating part has idle rotation in which image acquisition is performed and idle rotation in which image acquisition is not performed,
Jidoe in a uniform time interval during the activity made during the rotation of the rotational portion rotated 360 ° the sampling times L _max achieved,
Where L _max is a natural number greater than 1,
1 &quot; active rotation and (n-1) idling occur each time the rotation unit rotates n times,
Where n is a natural number greater than 1,
The base unit integrates the image signals obtained from the two cameras in the horizontal direction to generate a pair of 360 ° omni-directional image planes having a K _{max in} the vertical direction and L _max in the horizontal direction,
The image display means is provided with means for displaying a linear aberration-corrected image whose viewpoint is different from that of the user's left eye and right eye,
The image selecting means extracts the azimuth and altitude information of the direction desired by the user and transmits the information to the image processing means,
Wherein the image processing means generates a pair of linear aberration correction images in pairs of 360 ° omni-directional image planes based on the azimuth and altitude angles transmitted from the image selection means, and transmits the pair of linear aberration correction images to the image display means.
15. The method of claim 14,
Wherein the imaging lens mounted on the camera has an angle of view in the direction of the axis of rotation of 180 or more,
The cross section perpendicular to the optical axis of the imaging lens is smaller in width measured in the direction perpendicular to the rotation axis than in the direction of the rotation axis,
Wherein the long axis of the imaging lens maintains the rotation axis direction even if the imaging lens moves in the optical axis direction.
15. The method of claim 14,
Wherein the imaging lens mounted on the two cameras is a zoom lens capable of changing the angle of view of the lens.

Images