KR101882977B1

KR101882977B1 - Lens Module for Forming 360 Degree Image and Application for Forming 360 Degree Image

Info

Publication number: KR101882977B1
Application number: KR1020160060144A
Authority: KR
Inventors: 박철진
Original assignee: (주)유비크마이크로
Priority date: 2016-05-17
Filing date: 2016-05-17
Publication date: 2018-07-27
Also published as: KR20170129448A; WO2017200289A1

Abstract

The 360-degree lens module according to the present embodiment includes: a housing having a light outlet; a pair of fisheye lenses facing each other in the opposite directions; a pair of fisheye lenses And a fixing member formed on the housing and fixing the lens module to the smart terminal.

Description

[0001] The present invention relates to a lens module and a 360-degree image forming application capable of forming a 360-degree image,

The present invention relates to a lens module and a 360 degree image forming application capable of forming a 360 degree image.

Virtual reality is emerging. Virtual reality refers to the duplication of real or virtual environments. In other words, virtual reality refers to a technique of duplicating the relationship between a user's actual existence and a user or between a user and a specific object in order to form a mutual relationship. Virtual reality provides users with a sense of artificially created visual, tactile, auditory, and olfactory sense.

Among the virtual reality contents, the visual contents based on the reality are formed using the 360 degree camera. The 360 degree camera forms an external image using a plurality of lenses, performs an image processing process to form an image in a direction provided by the user, and provides the image to a user.

Conventional 360 degree cameras have a high price because optical parts, imaging parts and image processing parts are integrally formed. Therefore, there is a high cost that a consumer has to pay to experience a 360 degree image. In addition, according to the conventional 360-degree camera, when a user uploads a 360-degree image to a web or transmits it to another person, the user removes the memory from the camera, reads the memory from a terminal such as a computer, It is difficult to transmit the image immediately after shooting.

The lens module and the application according to the present embodiment are provided to solve the difficulties of the related art and provide a 360 degree lens module and an application which can be mounted on a portable smart terminal such as a mobile phone and can form an image of 360 degrees economically Is one of the main objects of the present embodiment.

It is also a primary object of the present invention to provide a 360 degree lens module and application that can be mounted on a portable smart terminal to instantly transmit an image stored in a portable smart terminal by forming and storing a 360 degree image.

(A) obtaining a pair of circular electronic images acquired by a lens module coupled to the smart terminal and including a pair of fish-eye lenses and provided to the smart terminal camera; and (b) converting electronic images into equirectangular image data, wherein the step of converting to equivalent square data comprises the steps of: (b1) converting a pair of Calculating coordinates of pixels forming circular electronic images; and (b2) writing pixel values of the calculated coordinates into pixel values of pixels forming an equivalent rectangular image.

According to the 360-degree lens module and the 360-degree image forming application according to the present embodiment, a 360-degree image can be formed using the computation capability of a smart terminal by mounting it on a widely deployed smart terminal, An advantage of being able to form a 360 degree image is provided. Also, since the photographed 360-degree image is stored in the smart terminal, it is advantageous that the smart terminal can be uploaded to the web after being photographed or transmitted.

1 is a diagram schematically showing an optical path of a lens module according to the present embodiment.
2 and 3 are views showing one side surface and the other side surface of the lens module according to the present embodiment.
4 is a diagram schematically showing an optical path of a fish-eye lens.
5 is a view showing a state in which the lens module is mounted on a smart terminal.
6 is a view showing that the light provided by the lens module according to the present embodiment is provided to the camera formed on the smart terminal through the light outlet.
7 is a flowchart schematically showing each step of the application process according to the present embodiment.
8 is a diagram showing an electronic image obtained by photographing an optical image provided by a lens module to a camera.
Fig. 9 is a diagram illustrating a state in which the optical axis of the fisheye lens and the optical axis of the sensor plane are aligned and unaligned.
Fig. 10 (a) is a diagram showing an outline of the equivalent rectangular image data, and Fig. 10 (b) is a diagram showing a spherical object plane when the fish-eye lens is at the center.
11 is a diagram showing a state in which the coordinate axes are rotated.
12 (a) is a fisheye lens image in which distortion correction is not performed, and FIG. 12 (b) is a fisheye lens image in which distortion correction is performed.
13 (a) is an equivalent quadrangle image formed by a fisheye lens image whose distortion is not corrected, and 13 (b) is an equivalent quadrangle image formed by a fisheye lens image whose distortion is corrected.

The description of the present invention is merely an example for structural or functional explanation, and the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, the embodiments are to be construed as being variously embodied and having various forms, so that the scope of the present invention should be understood to include equivalents capable of realizing technical ideas.

Meanwhile, the meaning of the terms described in the present application should be understood as follows.

The terms "first "," second ", and the like are used to distinguish one element from another and should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "on" or "on" another element, it may be directly on top of the other element, but other elements may be present in between. On the other hand, when an element is referred to as being "in contact" with another element, it should be understood that there are no other elements in between. On the other hand, other expressions that describe the relationship between components, such as "intervening" and "intervening", between "between" and "immediately" or "neighboring" Direct neighbors "should be interpreted similarly.

It should be understood that the singular " include "or" have "are to be construed as including a stated feature, number, step, operation, component, It is to be understood that the combination is intended to specify that it is present and not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Each step may take place differently from the stated order unless explicitly stated in a specific order in the context. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

The drawings referred to for explaining embodiments of the present disclosure are exaggerated in size, height, thickness, and the like intentionally for convenience of explanation and understanding, and are not enlarged or reduced in proportion. In addition, any of the components shown in the drawings may be intentionally reduced, and other components may be intentionally enlarged.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Terms such as those defined in commonly used dictionaries should be interpreted to be consistent with the meanings in the context of the relevant art and can not be interpreted as having ideal or overly formal meaning unless explicitly defined in the present application .

Hereinafter, a lens module and an application according to the present embodiment will be described with reference to the accompanying drawings. FIG. 1 is a view schematically showing an optical path of a lens module according to an embodiment of the present invention, and FIGS. 2 and 3 are views showing one side surface and another side surface of a lens module according to the present embodiment. 1 to 3, the lens module according to the present embodiment includes a housing H having a light outlet E and a pair of fisheye lenses 100a and 100b And a housing H for changing the paths of the lights provided by the pair of fisheye lenses 100a and 100b in the same direction and providing the light through the light outlet E to the outside of the housing H, And a fixing member 400 for fixing the lens module to the smart terminal.

The housing H accommodates the reflection member 200 therein and provides a light path through which the light condensed by the fisheye lenses 100a and 100b can travel. In one embodiment, the housing H is formed of a material having good processability such as synthetic resin and die cast aluminum.

Fig. 4 schematically shows an optical path of a fish-eye lens f. Referring to FIG. 4, the fish-eye lenses 100a and 100b distort the incident angle and have a wide field of view (FOV) as compared with other lenses. The relationship between the distance R, the focal distance f and the incident angle? According to the characteristics of the fisheye lens is shown in Table 1 below.

Fisheye lens type Relation Equidistance fisheye lens

Stereo graphic fisheye lens (stereo graphic)

Orthographic fisheye lens

Equisolid fisheye lens

Thoby fisheye lens

The distance f from the focal point of the fish-eye lens to the imaging surface is a value peculiar to the lens and fixed. The position R at which the image of the object is formed at the intersection of the sensor image pickup surface and the optical axis can be obtained irrespective of the type of the fisheye lens as long as the angle of incidence of the light entering the fisheye lens from the object P can be grasped . It can be seen that the position R is a function of the angle of incidence φ, and is expressed as R (φ).

1 to 3, the reflective member 200 includes a first reflective member (not shown) that reflects the light condensed by the fisheye lenses 100a and 100b in a direction perpendicular to the optical axis of the fisheye lenses And a second reflecting member 200b that reflects the light reflected by the first reflecting member 200b to the outside of the housing through the light exit.

The first reflecting member 200a and the second reflecting member 200b may be mirrors as illustrated in Fig. According to another embodiment not shown, the first reflective member 200a and the second reflective member 200b may be prisms.

In the lens module according to the present embodiment, a plurality of alignment marks 300 are formed on the side where the fisheye lenses 100a and 100b are located. The alignment marks 300 are located at predetermined positions, and the coordinates of the alignment marks 300 in an image photographed using the fisheye lenses 100a and 100b are stored in the application. Like the embodiment shown in FIGS. 2 and 3, the alignment marks 300 may be displayed differently depending on the plane on which the fisheye lens is located.

When the alignment of the fisheye lenses 100a and 100b and the reflection member 200 is changed in the lens module according to the present embodiment or when the lens module according to this embodiment is mounted on the smart terminal, E) and the camera (c) may be misaligned. An application running on the smart terminal s can correct the distortion of the misaligned image using the alignment marks 300.

The alignment marks 300 are formed on the same plane as the fisheye lenses 100a and 100b but the viewing angles of the fisheye lenses 100a and 100b are more than 180 degrees and the alignment marks 300 are formed on the fisheye lenses 100a and 100b, 100b. The alignment marks shown in FIGS. 2 and 3 are shown as patterns, but this is only an example, and may be formed as a protrusion in the plane in which the fisheye lenses 100a and 100b are located, or in a concave shape.

A fixing member 400, which can be mounted on the smart terminal s, may be formed in the housing of the lens module. The fixing member 400 has a clip shape for fixing the lens module to the smart terminal s as shown in FIG. 5, and fixes the lens module to the smart terminal s using an elastic member such as a spring. In the embodiment shown in FIG. 5 (b), the fixing member 400 may be a slot in which the smart terminal s can be fitted and fixed to the lens module, A friction member such as rubber, silicone or the like may be positioned to provide a frictional force. In one embodiment, the fixing member 500 may be integrally formed with the housing H, and in another example, may be coupled to the housing H.

6 is a view showing that the light provided by the lens module according to the present embodiment is provided to the camera c formed on the smart terminal s through the light outlet E. The light condensed by the fisheye lenses 100a and 100b is changed in the same direction by the reflecting member 200 and provided to the camera c. The smart terminal s refers to a terminal having a computing power capable of driving an application according to the present embodiment and including a camera capable of obtaining a static image and a dynamic image from a camera. For example, the smart terminal s may be a terminal such as a lap top computer, a mobile phone, a tablet, and the like.

In one embodiment of the lens module, the light exit E may be provided with a correction lens 500 that adjusts the path of light provided to the camera c or the image provided to the camera. The correction lens 500 may be a composite lens in which a concave lens and a convex lens are combined as shown, and may be a concave lens or a convex lens.

Hereinafter, a driving process of an application for forming a 360-degree image from a photographed image using the lens module according to the present embodiment will be described with reference to FIGS. 7 to 11. FIG. 7 is a flowchart schematically showing each step of an application driving process according to the present embodiment. Referring to FIGS. 7 and 8, a pair of optical images acquired by the lens module and provided to the camera c of the smart terminal s are acquired to acquire electronic images (SlOO). The optical image provided by the fisheye lens to the camera is circular as shown in Fig. 8, and the application controls the smart terminal (s) to convert the two circular optical images provided by the pair of fisheye lenses included in the lens module into electronic images Data.

9A is a diagram illustrating an example in which the optical axis of a fisheye lens and the optical axis of the sensor plane are aligned. In the state where the optical axis and the imaging plane of the sensor are aligned as shown in FIG. 9A, When the alignment mark 300 is photographed, a digital image that is symmetrical in the up, down, left, and right directions can be obtained as shown in the bottom view of FIG. However, if the optical axis of the fisheye lens and the sensor imaging plane are not aligned as shown in the upper part of Fig. 9 (b) and the fisheye lens is used, distortion as shown in the lower diagram of Fig. 9 Obtain the resulting image.

When an image in which a distortion occurs is obtained, a distortion of the image can be corrected by performing a perspective transformation. (X1 ', y1'), (x2 ', y2'), (x3 ', y3'), (x4 ', y4'), and the coordinates of the alignment marks in the image in which distortion occurs are (x1, y1), (x2, y2), (x3, y3), (x4, y4).

The distortion of the image due to misalignment can be corrected by a perspective transform as described above, and the perspective transformation can be performed by mapping the coordinates to a homography matrix expressed by Equation (1) below.

a, b, c, d, e, f, g, h: constant coefficient after the x,

The coordinate values x ', y' of the pixel where the perspective transformation has been performed are as shown in Equation 2 below.

In Equation (2), the formula for moving the coordinates of a point having coordinates (x, y) to a new point (x ', y') is eight in all, Y1 ', x2', y2 ', x3', y3 ', x4', y4 ', which are coordinates of the point at which the pair of alignment marks are located. Also, coordinate values x1, y1, x2, y2, x3, y3, x4, and y4 where the four pairs of alignment marks are located in the distorted image can be obtained. Thus, the unknowns are a, b, c, d, e, f, g, h, and they can be obtained by calculating the determinant of Equation 3 below.

The image distortion caused by the misalignment can be corrected by performing the perspective transformation process on the pixels included in the distorted image by obtaining eight unknown values in Equation (3) and calculating Equation (2) from the obtained result.

The distortion of the image is not limited to the case where the optical axis of the fisheye lens and the sensor plane are misaligned as well as the misalignment between the fisheye lens and the reflective member, the camera of the lens module and the smart terminal, And the fisheye lens of the other fisheye lens 100b. In this case, the influence due to misalignment can be corrected through the above process.

The application processes the two circular image data to form an equirectangular image. FIG. 10 (a) is a diagram showing an outline of the equivalent rectangular image data, and FIG. 10 (b) is a diagram showing a spherical object plane when the fish-eye lens is at the center. In Fig. 10 (b), the portion indicated by the light green color shows the object plane taken by one fish-eye lens, which corresponds to the pale green portion in the equivalent rectangular image in Fig. 10 (a). Similarly, in Fig. 10 (b), the portion indicated by yellow indicates the object plane taken by another fisheye lens, and corresponds to the yellow portion in the equivalent rectangular image in Fig. 10 (a).

In an equivalent rectangular image shown in FIG. 10 (a), a pixel P can be expressed by coordinates of? And? Indicating longitude and latitude, respectively. According to the general display method, the latitude ranges from + 90 ° to -90 °, but for convenience of explanation, the latitude of the zenith is set to 0 and the latitude of the lowest point (nadir) is set to 180 ° for convenience.

10 (b), the radius of the sphere is assumed to be 1, and the coordinates of the point P are expressed by an orthogonal coordinate system.

Axis to rotate the coordinate system along the y-axis as shown in Fig. 11 to facilitate calculation of the coordinates of the point corresponding to the point P in the circular image stored in the smart terminal. Let X ', Y', and Z 'be the new coordinate axes formed by the rotation, and obtain the coordinates of the P point in X', Y ', and Z'.

The spherical coordinate is calculated from the new coordinate pair obtained in Equation (5), as shown in Equation (6) below.

(acos: inverse function of cos, atan2 (x, y): function to find the angle between X axis and coordinate (x, y) in XY plane)

? 'Obtained from Equation (6) is an incident angle when light is incident on the fisheye lens from the point P of the object plane,?' Is the rotation angle from the X 'axis, and?' Does not change even when passing through the fisheye lens. Therefore, the coordinates of the point P in the circular electronic images stored in the smart terminal s are obtained as shown in Equation (7) below.

(R (?): A position function of an image of an object formed at an intersection of an image pickup surface and an optical axis in a fish-eye lens, see Table 1)

The coordinates of the X ', Y', and Z 'axes are converted into the coordinates of the X, Y, and Z axes again, and the coordinates of the X', Y ' And? 'Are replaced with the results of Equation (6), Equations (8) and (9) can be obtained. From the results of Equations (8) and (9), the coordinates of the image point provided by the fisheye lens to which the point having the coordinates? And? Of the equivalent rectangular image are mapped can be obtained (S200).

In Equation (5), if the z 'coordinate value is negative, it corresponds to an image obtained by another fisheye lens, and the x and y coordinates are calculated as shown in the following Equation 9 (S200).

Equations (8) and (9) are calculated with respect to the coordinates of the equivalent quadrangle to obtain the corresponding coordinates of the image obtained from the fish-eye lens, and then the pixel values of the obtained coordinates are read and used as pixel values of the corresponding coordinates of the equivalent quadrature image, Thereby forming an image (S300).

The application forms a virtual spherical image from the formed equivalent square image and performs a gnomonic projection on the spherical image to display the image on the smart terminal.

Experimental Example

Hereinafter, an experiment will be described with reference to Figs. 12 to 13, in which distortion is corrected using an application according to the present embodiment with respect to an image obtained in a state where the optical axis of the image sensor and the optical axis of the fisheye lens are misaligned.

12 is an image taken using two fisheye lenses in a misaligned state. The part shown in the circle is an alignment mark. Fig. 12 (a) shows the fisheye lens image without distortion correction, and Fig. 12 (b) shows the yellow solid line before distortion on the image. It can be seen from FIG. 12 (b) that the far-field transformation is performed by calculating the homography matrix accurately.

13 (a) is an equivalent quadrangle image formed by a fisheye lens image with no distortion corrected, and 13 (b) is an equivalent quadrangle image formed by a fisheye lens image whose distortion is corrected. It can be seen that in the equivalent quadrangle image formed by the fisheye lens image without distortion correction, a step is generated at the portion where the two images are combined and deviated, but it is confirmed that no step is generated in the equivalent square image formed by the distortion-corrected image .

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of illustration, It will be appreciated that other embodiments are possible. Accordingly, the true scope of the present invention should be determined by the appended claims.

100a, 100b: Fish eye lens 200: Reflecting member
200a, 200b: first reflecting member, second reflecting member 300: alignment mark
400: Fixing member 500: Correction lens

Claims

A 360 degree lens module comprising:
A housing having a light outlet formed therein;
A pair of fisheye lenses facing away from each other and located on different planes;
A reflecting member for changing a path of light provided by the pair of fisheye lenses in the same direction and providing the light to the outside of the housing through the light outlet;
A fixing member formed on the housing and fixing the lens module to the smart terminal; And
The fisheye lens and the smart terminal camera are arranged on the surfaces on which the fisheye lenses are disposed to perform a perspective transformation for correcting distortion caused by misalignment of at least one of the fisheye lens and the smart terminal camera, Each of which is arranged at a vertex of a quadrangle and includes an alignment mark displayed in a different form according to a plane on which the respective fisheye lenses are arranged,
Wherein the fixing member is capable of directly fixing the lens module to the smart terminal.

The method according to claim 1,
Wherein the reflective member comprises:
A first reflection member for reflecting the light condensed by the fish-eye lenses in a direction perpendicular to the optical axis of the fish-eye lenses,
And a second reflection member that reflects the light reflected by the first reflection member to the outside of the housing through the light exit.

3. The method of claim 2,
The lens module includes:
Disposed at the light exit
And an adjustment lens for adjusting the light reflected by the second reflection member.

The method according to claim 1,
Wherein:
A clip for picking up and fixing the smart terminal, and a slot in which the smart terminal is inserted and fixed.

The method according to claim 1,
Wherein the light outlet is formed at a position corresponding to a camera lens formed on the smart terminal.

delete

The method according to claim 1,
The smart terminal includes:
Wherein the camera is equipped with a laptop computer, a mobile phone, and a portable tablet.

36. A method of forming a 360 degree image, the method comprising:
(a) obtaining a pair of circular electronic images acquired by a lens module coupled to a smart terminal and provided to the smart terminal camera, including a pair of fisheye lenses facing away from each other and facing each other, ,
(b) converting the electronic images into equirectangular image data, the step of converting to equivalent square data comprises:
(b1) calculating coordinates of a pixel forming the pair of circular electronic images corresponding to image coordinates of a pixel forming the equivalent rectangular image;
(b2) writing the computed pixel value of the coordinate as the pixel value of the pixel forming the equivalent square image,
The lens module includes alignment marks which are respectively disposed on the faces on which the fisheye lenses are disposed and are arranged at vertexes of a quadrangle centered on the fisheye lens and are displayed in different shapes according to the plane on which the fisheye lenses are arranged ,
The 360 degree image forming method
(c) performing a perspective transform to correct for distortion caused by misalignment of one or more of the fisheye lens and the smart terminal camera using the alignment indications.

10. The method of claim 9,
The image forming method includes:
And performing a gnomonic projection on the equivalent rectangular image data to display a portion of the 360 degree image.

10. The method of claim 9,
The step (b1)
Computing the angular distance from any one of a zenith and a node of a pixel forming the circular electronic images,
And calculating coordinates of a pixel forming the circular electronic images using the obtained angular distance and the fisheye lens characteristic.

delete

10. The method of claim 9,
Wherein the step of correcting the perspective transition comprises:
Calculating a coefficient of the homography matrix using the coordinates of the plurality of alignment marks in the absence of misalignment and the coordinates of the plurality of alignment marks obtained in each photographed circular electronic image,
And performing perspective correction using the homography matrix in which the coefficients are obtained.

10. The method of claim 9,
Wherein the step of correcting the perspective transition comprises:
Equation

(X1, y1, x2, y2, x3, y3, x4) are performed using a homography matrix using the coefficients a, b, c, d, e, f, x4 ', y4', x4 ', y4' of the plurality of alignment marks in the absence of misalignment, x4 ', y2' Coordinates of multiple alignment marks).

10. The method of claim 9,
Wherein the 360 degree image forming method is performed by the smart terminal.