KR101915729B1 - Apparatus and Method for Generating 360 degree omni-directional view - Google Patents

Abstract

The present invention relates to a device for generating a 360-degree omnidirectional view image and a method thereof, which fit capturing synchronization of sensors between a plurality of cameras when capturing an omnidirectional region with the plurality of cameras, perform distortion correction of each camera lens and correction functions such as the entire camera position and posture and generate an omnidirectional view image. According to the present invention, the device for generating a 360-degree omnidirectional view image comprises: a sensor part including a plurality of image sensors capable of dividing and imaging a 360-degree omnidirectional region; an image process processor controlling the plurality of image sensors so that the plurality of image sensors control the same imaging time, the same exposure and the same color and receiving and processing a plurality of images imaged by the plurality of image sensors; and a multi-digital image processor matching the plurality of images received from the image process processor and generating the 360-degree omnidirectional view image shown at a set random point. According to the present invention, the 360-degree omnidirectional view image can be synthesized based on the image captured by the plurality of cameras.

Description

[0001] APPARATUS AND METHOD FOR GENERATING AN 360 DEG OMNI-DIRECTIONAL VIEW [0002]

The present invention relates to an apparatus and method for generating a 360-degree omnidirectional view image, and more particularly, to an apparatus and method for generating 360-degree omnidirectional view image by synchronizing shooting motions of sensors between a plurality of cameras, And more particularly, to a 360-degree omnidirectional view image generating apparatus and method for generating a omnidirectional view image by correcting a distortion of a lens and correcting functions of an entire camera position and posture.

In recent years, many security and surveillance cameras have been widely used for surveillance systems or for black boxes in vehicles. A widely used camera device is a PTZ camera equipped with a mechanical pan, tilt, and zoom function. However, the PTZ camera has a limited range of shooting at a specific point of time because the angle of view of the lens is not wide for the purpose of clearly capturing an object using the PTZ function.

On the other hand, a fish-eye lens or a wide-angle lens can be employed in order to capture a wide range of images.

In addition, a plurality of cameras can only be used to capture an omnidirectional image. As the number of cameras increases, the range of each camera becomes narrower. Therefore, a normal PTZ camera can be used without distortion. However, when the number of cameras is reduced by using a wide-angle lens, the captured image is distorted. Since each camera shoots independently, when a moving object appears at the overlapping part in the overlapping part, or when there is a backlight situation, there is a considerable sense of motion due to motion or color difference in the boundary part .

It is an object of the present invention to provide an image processing apparatus and a method for controlling a plurality of cameras so as to operate at the same time to minimize a degree of heterogeneity that may occur at a boundary portion and to reduce errors in view angle and distortion, And an object of the present invention is to provide an apparatus and method for generating a 360 degree omnidirectional view image based on a virtual camera capable of correcting distortion of an image.

According to another aspect of the present invention, there is provided a 360-degree omni-directional view image generating apparatus including a sensor unit including a plurality of image sensors capable of dividing a 360-degree omni-directional area, An image processing processor for controlling the plurality of image sensors to be in the same exposure and same color control, receiving and processing a plurality of images picked up by the plurality of image sensors, To generate a 360 degree omnidirectional view image that is visible at any set point in time.

In more detail, the image processing processor includes a plurality of image processing processors corresponding to each of the plurality of image sensors, one of the plurality of image processing processors is a main image processing processor, and the other is an auxiliary image processing processor The main image processing processor may provide a clock and an image synchronization signal to the plurality of image sensors and the auxiliary image processing processor so that the plurality of image sensors have the same imaging time. The main image processing processor detects a first optical characteristic of an image received from a corresponding image sensor and transmits the first optical characteristic of the image received from the corresponding image sensor to the main image processing processor, And determines the exposure value and the color control information by comparing the first optical characteristic and the second optical characteristic received from the auxiliary image processing processor and then transmits the control signal to the auxiliary image processing processor, The auxiliary image processing processor and the main image processing processor may adjust the shutter speed and automatic gain control value of the corresponding image sensor.

)로 변환하고, 크기 변환을 통해 원근감 보상을 수행하며, 영상 효과를 얻기 위해 투영 방식에 따라 장면 좌표계상의 광축 중심(optical center;

)을 설정하는 가상 카메라 모델부, 상기 가상 카메라 모델부에 의하여 생성된 장면 좌표계 상의 좌표(

)를 상기 복수 개의 영상의 각 영상의 대응하는 픽셀 좌표(

)로 매핑 시에 사용되고, 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)에 영상 효과를 삽입할 수 있는 2차 곡면을 모델링한 2차 곡면 모델(

)을 생성하는 2차 곡면 모델부, 상기 가상 카메라 모델부에서 획득한 장면 좌표계 상의 좌표(

), 장면 좌표계상의 광축 중심(

), 및 상기 2차 곡면 모델부에서 생성한 2차 곡면 모델(

)을 바탕으로 상기 장면 좌표계상의 좌표(

)를 상기 2차 곡면의 한 점으로 투영하고, 상기 2차 곡면에 투영된 한 점을 다시 상기 복수 개의 영상의 각 영상으로 투영하여 상기 장면 좌표계상의 좌표(

)를 상기 복수의 영상의 각 영상의 대응하는 픽셀 좌표(

)로 매핑하고, 상기 각 영상의 대응하는 픽셀 좌표(

)별로 주변 픽셀들의 픽셀값을 보간(interpolating)하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대한 상기 복수 개의 영상의 각 영상으로부터의 픽셀값을 획득하는 전방향 카메라 투영 모델부, 및 상기 전방향 카메라 투영 모델부에서 획득한 상기 복수 개의 영상의 각 영상으로부터의 픽셀값을 정합하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대한 픽셀값을 결정하는 정합부를 포함할 수 있다. 이에 더하여 상기 다중 디지털 이미지 프로세서의 각 부에서 사용될 파라미터를 사용자로부터 입력받아 설정하는 파라미터 설정부를 더 포함할 수 있다.The multi-digital image processor calculates a coordinate ( u ) of a point of a 360-degree forward view image ( J ) displayed at a viewpoint of the virtual camera on the basis of the set position and posture of the virtual camera,

), Performs perspective compensation through magnitude conversion, and calculates the center of the optical axis on the scene coordinate system according to the projection method to obtain a visual effect.

), Coordinates on the scene coordinate system generated by the virtual camera model unit (

) Corresponding pixel coordinates of each image of the plurality of images

), And a secondary curved surface model (hereinafter referred to as " secondary curved surface model ") modeled as a secondary curved surface that can insert a video effect into a 360-degree forward view image J

A second curved surface model unit for generating coordinates of a scene on the scene coordinate system acquired by the virtual camera model unit

), The center of the optical axis on the scene coordinate system (

), And a secondary curved surface model generated by the secondary curved surface model unit (

Coordinates on the scene coordinate system (

) Onto a point on the secondary curved surface, projecting a point projected on the secondary curved surface onto each of the images of the plurality of images,

) Corresponding pixel coordinates of each image of the plurality of images

), And the corresponding pixel coordinates (

) Interpolating pixel values of neighboring pixels for each of the plurality of images with respect to a coordinate u of a point of a 360-degree forward view image J viewed from the viewpoint of the virtual camera, Directional camera projection modeling unit for acquiring a 360-degree omnidirectional view image from the virtual camera, and acquiring a 360-degree omnidirectional view image ( U ) of a point of the point ( J ). In addition, the image processing apparatus may further include a parameter setting unit for receiving and setting parameters to be used in each unit of the multiple digital image processor.

Each part of the multiple digital image processor will be described in more detail.

)을 생성하는 위치설정 모듈, 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 가로축과 세로축의 크기 비율을 조정하고, 상기 조정에 따라 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)를 조정하는 크기 조정 모듈, 상기 위치설정 모듈에서 생성한 변환 행렬(

)를 바탕으로 상기 좌표 표시모듈 및 상기 크기 조정 모듈에서 생성하고 조정한 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)를 장면 좌표계의 좌표(

)로 변환하는 변환 모듈, 및 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 광축 중심(

)을 계산하는 광축 중심 설정모듈을 포함할 수 있다.The virtual camera model unit converts coordinates ( u ) of a point of a 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera into coordinates of a rectangular coordinate system of (x, y) and coordinates of a point of a 360-degree omnidirectional view image ( J ) displayed at the viewpoint of the virtual camera. The coordinates ( u ) Into coordinates on the scene coordinate system (

, A position setting module for adjusting the size ratio of the horizontal axis and the vertical axis of the 360 degrees forward view image J viewed from the viewpoint of the virtual camera, A size adjustment module for adjusting a coordinate u of a point of the forward view image J , a transformation matrix

( U ) of a point of a 360-degree omnidirectional view image ( J ) displayed at the viewpoint of the virtual camera created and adjusted by the coordinate display module and the size adjustment module is calculated based on the coordinate

) And a conversion module for converting the center of the optical axis of the 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera

) Of the optical axis centering module.

)을 이용하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)를

로 계산하여 직각 좌표 형식의 좌표(

)로 표시하고, 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)를

로 계산하여 극좌표 형식의 좌표(

)로 표시할 수 있다. 여기서The coordinate display module displays the center of the 360-degree forward view image J displayed at the viewpoint of the virtual camera

( U ) of a point of the 360-degree forward view image J shown at the viewpoint of the virtual camera using

To the coordinates in the rectangular coordinate form (

( U ) of a point of the 360-degree forward view image J shown at the viewpoint of the virtual camera is represented by

To the coordinates in polar coordinates (

). here

이고,

은 극좌표 형식에서 사용되는 반지름의 최소값,

는 극좌표 형식에서 사용되는 반지름의 최대값,

은 극좌표 형식에서 사용되는 반지름 방향의 픽셀 수,

은 극좌표 형식에서 사용되는 각도의 최소값,

는 극좌표 형식에서 사용되는 각도의 최대값,

는 극좌표 형식에서 사용되는 각도 방향의 픽셀 수이고,

이다.

ego,

Is the minimum value of the radius used in the polar coordinate format,

Is the maximum value of the radius used in polar form,

Is the number of radial pixels used in polar form,

Is the minimum value of the angle used in the polar coordinate format,

Is the maximum value of the angle used in the polar coordinate format,

Is the number of pixels in the angular direction used in the polar coordinate format,

to be.

In addition, the positioning module

식을 이용하여 변환행렬(

)을 구하되,

는 장면 좌표계상에서의 2차 곡면 모델의 중심이며, 상기 가상 카메라의 팬, 틸트 회전의 중심이고,

은 장면 좌표계상에서의 가상 카메라의 위치, φ_n 및 θ_n는 각각 상기 가상 카메라의 장면 좌표계의 z축에 대한 수평 회전각(azimuth) 및 수직 회전각(elevation)이고, trans()는 3차원 공간상의 이동 행렬을 나타내는 함수이고, rot_z()와 rot_x()는 3차원 공간상의 각각 z축과 x축에 대한 회전을 나타내는 함수이다.

The transformation matrix (

),

Is the center of the quadratic surface model on the scene coordinate system and is the center of the pan and tilt rotation of the virtual camera,

Is the position of the virtual camera on the scene coordinate system, phi _n And θ _n are the azimuth and elevation of the scene coordinate system of the virtual camera with respect to the z axis, trans () is a function representing a moving matrix on the three-dimensional space, and rot _z () And rot _x () are functions that represent the rotation about the z-axis and the x-axis on the three-dimensional space.

In addition, the optical axis centering module may determine whether the type of virtual camera is a perspective projection camera or a perpendicular projection camera

)을 계산하되,

은 초점 거리(focal length)이고,

는 상기 변환모듈에서 생성한 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대응되는 장면 좌표계의 좌표이고,

및

은 변환 행렬 (

) 을 단위 벡터 형태인

으로 표현할 때의 단위 벡터들이다.The center of the optical axis

),

Is the focal length,

Is a coordinate of a scene coordinate system corresponding to a coordinate ( u ) of a point of a 360-degree forward view image ( J ) viewed from the viewpoint of the virtual camera generated by the conversion module,

And

Lt; RTI ID = 0.0 > (

) As a unit vector form

Are unit vectors when expressed as a unit vector.

)을 생성하는 2차 곡면 모델 선정 모듈, 상기 2차 곡면상의 점들을 장면 좌표계상의 좌표들로 변환하기 위한 변환행렬(

)을 생성하는 좌표설정 모듈, 및 상기 변환행렬(

)을 이용하여 상기 선택된 2차 곡면 모델을 표현하는 행렬(

)을 장면 좌표계상에서 상기 선택된 2차 곡면 모델을 표현하는 행렬(

)로 변환하는 변환 모듈을 포함할 수 있는데, 상기 2차 곡면 모델 선정 모듈은 타원체(Ellipsoid) 모델, 실린더(Cylinder) 모델, 포물면(Paraboloid) 모델, 및 사용자 설정 2차 곡면 모델 중 하나의 모델을 선택하고, 상기 좌표설정 모듈은

식을 이용하여 변환행렬(

)을 구하되, φ_n 및 θ_n는 각각 상기 2차 곡면 모델의 장면 좌표계의 z축에 대한 수평 회전각(azimuth) 및 수직 회전각(elevation)이고, rot_z()와 rot_x()는 3차원 공간상의 각각 z축과 x축에 대한 회전을 나타내는 함수이고, 상기 변환 모듈은 상기 선택된 2차 곡면 모델을 표현하는 행렬(

) 및 변환행렬(

)을 바탕으로 장면 좌표계상에서 상기 선택된 2차 곡면 모델을 표현하는 행렬(

)을

식을 이용하여 계산할 수 있다.The quadric surface modeling unit may be configured to select one of the plurality of quadric surface models and to use the parameters of the selected quadric surface model to express a selected quadric surface model

), A transformation matrix for transforming the points on the quadratic surface into coordinates on the scene coordinate system ("

), And a transformation matrix ("

) To express the selected quadratic surface model

) Representing the selected curved surface model on the scene coordinate system (

The secondary curved surface model selection module may include a transformation module for transforming a model of an Ellipsoid model, a cylinder model, a paraboloid model, and a user- , And the coordinate setting module

The transformation matrix (

), Where φ _n And θ _n are a horizontal azimuth and a vertical rotation angle of the scene coordinate system of the quadratic curve model with respect to the z axis and rot _z () and rot _x () And a rotation about the x-axis, and the transformation module is a function for expressing the selected quadratic surface model

) And a transformation matrix (

) Representing the selected curved surface model on the scene coordinate system

)of

Can be calculated using an equation.

), 및 장면 좌표계 상의 광축 중심(

)을 연결하는 직선인 투영선과 상기 2차 곡면 모델(

)과의 교차점(

)을 계산하는 교차점 모듈; 및 상기 교차점(

)을 상기 복수 개의 영상의 각 영상의 대응하는 픽셀 좌표(

)로 매핑하고, 상기 각 영상의 대응하는 픽셀 좌표(

)의 주변 픽셀들의 픽셀값을 보간하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대한 상기 복수 개의 영상의 각 영상으로부터의 픽셀값을 획득하는 전방향 카메라 투영 모듈을 포함하고, 좀 더 상세히 보면, 상기 전방향 카메라 투영 모듈은 장면 좌표계로 표현된 상기 교차점(

)을 카메라 좌표계의 대응하는 좌표(

)로 변환하는 변환 모듈, 상기 변환 모듈에서 생성되는 상기 교차점의 카메라 좌표계 좌표(

)로부터 영상의 왜곡을 보정할 수 있는 상기 복수 개의 영상의 각 영상의 실제 픽셀 좌표(

)를 획득하는 좌표 계산 모듈, 및 상기 좌표 계산 모듈에서 획득한 상기 복수 개의 영상의 각 영상의 실제 픽셀 좌표(

)별로 주변 픽셀들의 픽셀값들을 획득하고, 상기 주변 픽셀들의 픽셀값들을 바탕으로 보간(interpolation)하여 출력하는 보간 모듈을 포함하고, 상기 좌표계산 모듈은 상기 복수 개의 영상의 각 영상에 존재하는 왜곡에 대하여 수치적으로 근사한 테일러 확장(Taylor expansion series) 기반의 근사 모델을 기반으로 하는 왜곡 보정 매핑 함수(

)를 이용하여 상기 교차점(

)의 카메라 좌표계 좌표(

)를 왜곡 없는 영상 평면상의 한 점(

)으로 매핑하고, 어파인(affine) 변환 행렬(

)을 이용하여 상기 왜곡 없는 영상 평면상의 한 점(

)에 대응되는 상기 복수의 영상의 각 영상의 픽셀 좌표(

)를 획득할 수 있다.The omnidirectional camera projection model unit may include a coordinate system

), And the center of the optical axis on the scene coordinate system (

) And the second curved surface model (

) ≪ / RTI >

); ≪ / RTI > And the intersection point

) Corresponding pixel coordinates of each image of the plurality of images

), And the corresponding pixel coordinates (

) To obtain a pixel value from each image of the plurality of images with respect to a coordinate u of a point of the 360-degree forward view image J viewed at the viewpoint of the virtual camera The omnidirectional camera projection module comprises an omnidirectional camera projection module for projecting the intersection point

) To the corresponding coordinates of the camera coordinate system (

), A transformation module for transforming the coordinates of the camera coordinate system of the intersection point generated by the transformation module

The actual pixel coordinates of each of the images of the plurality of images that can correct the distortion of the image

) Of the plurality of images obtained by the coordinate calculation module,

And an interpolation module for interpolating and outputting pixel values of neighboring pixels on the basis of the pixel values of the neighboring pixels and outputting the interpolated values, A distortion correction mapping function based on an approximate model based on a Taylor expansion series approximated numerically

) At the intersection (

) Coordinates of the camera coordinate system (

) To a point on the distortion-free image plane (

), And an affine transformation matrix (

), A point on the distortion-free image plane (

) Of the respective images of the plurality of images corresponding to the pixel coordinates

Can be obtained.

식을 이용하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대한 픽셀값(

)을 결정할 수 있다. 여기서, k는 상기 복수 개의 영상의 수이고,

는 상기 전방향 카메라 투영 모델부에서 i번째 영상으로부터 획득한 픽셀값이고,

는 i번째 영상에서 상기 좌표(u)에서 상기 i번째 영상의 경계까지의 거리에 따른 가중치이고, 상기

는The matching unit may multiply the pixel values from the respective images of the plurality of images acquired by the forward direction camera projection model unit by a coordinate u of a point of the 360 degrees forward view image J viewed from the viewpoint of the virtual camera (I) of the 360-degree forward view image J viewed from the viewpoint of the virtual camera, and determines the pixel value as a pixel value with respect to the coordinate u , Preferably

( U ) of a point of a 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera using the equation

Can be determined. Here, k is the number of the plurality of images,

Is a pixel value obtained from the i-th image in the forward camera projection model unit,

Is a weight according to the distance from the coordinate ( u ) to the boundary of the i-th image in the i-th image,

The

식을 이용하여 결정하되

Use the formula to determine

θ _max is the maximum angle of view of the image sensor, α is the overlap angle between adjacent image sensor, θ is the coordinates (u) of a point on the 360-degree omni-directional view image (J) shown in the timing of the virtual camera, This is the angle when displayed in polar coordinate format.

The 360 degree omnidirectional view image generation apparatus may further include a storage device for storing a plurality of images captured by the plurality of image sensors, wherein the plurality of images captured by the plurality of image sensors are synchronized with each other, And the voice information and GPS information acquired at the same time as the plurality of images can be stored together. The image processing processor may generate a 360-degree forward view image by reading and processing a plurality of images stored in a file synchronized with the storage device instead of a plurality of images captured by the plurality of image sensors.

According to another aspect of the present invention, there is provided a method of generating a 360-degree omnidirectional view image, the method comprising: controlling the plurality of image sensors so that a plurality of image sensors have the same imaging time, The method of claim 1, further comprising: acquiring a plurality of images captured by the image sensor; removing distortion included in each image based on the plurality of images; And generating a view image.

)로 변환하고, 크기 변환을 통해 원근감 보상을 수행하며, 특수 효과를 얻기 위해 투영 방식에 따라 장면 좌표계 상의 광축 중심(optical center;

)을 설정하는 가상 카메라 모델 생성 단계, 상기 가상 카메라 모델 생성 단계에서 생성된 장면 좌표계 상의 좌표(

)를 상기 복수 개의 영상의 각 영상의 대응하는 픽셀 좌표(

)로 매핑 시에 사용되고, 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)에 영상 효과를 삽입할 수 있는 2차 곡면을 모델링한 2차 곡면 모델(

)을 생성하는 2차 곡면 모델 생성 단계, 상기 가상 카메라 모델 생성 단계에서 생성한 장면 좌표계 상의 좌표(

), 장면 좌표계 상의 광축 중심(

), 및 상기 2차 곡면 모델 생성 단계에서 획득한 2차 곡면 모델(

)을 바탕으로 상기 장면 좌표계 상의 좌표(

)를 상기 2차 곡면의 한 점으로 투영하고, 상기 2차 곡면에 투영된 한 점을 다시 상기 복수 개의 영상의 각 영상으로 투영하여 상기 장면 좌표계 상의 좌표(

)를 대응하는 전방향 카메라 영상의 픽셀 좌표(

)로 매핑하고, 상기 각 영상의 대응하는 픽셀 좌표(

)별로 주변 픽셀들의 픽셀값을 보간(interpolating)하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대한 상기 복수 개의 영상의 각 영상으로부터의 픽셀값을 획득하는 전방향 카메라 투영 모델 생성 단계, 및 상기 전방향 카메라 투영 모델 생성 단계에서 획득한 상기 복수 개의 영상의 각 영상으로부터의 픽셀값을 정합하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대한 픽셀값을 결정함으로써 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)을 생성하는 영상 생성 단계를 포함할 수 있다. 그리고 설정된 임의 시점에서 보여지는 360도 전방향 뷰 영상을 생성하는 단계는 원근감 보정을 위하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 크기를 보상하는 단계를 더 포함할 수 있다.Controlling each of the plurality of image sensors so that the plurality of image sensors have the same image sensing time, same exposure, and same color control may be performed by the plurality of image sensors, Acquiring a plurality of images captured by the plurality of image sensors and detecting optical characteristics of the images; comparing the optical characteristics of the detected images to obtain exposure values and color correction information; And controlling the plurality of image sensors based on the determined exposure value and the color correction information. The method may further include removing distortion included in each image based on the plurality of images, Combine the removed images to generate a 360 degree omnidirectional view image that is displayed at a set point. Coordinates on the stage set and the virtual camera position based on the coordinates of the point (u) of 360 ° omni-directional view video (J) as shown from the point of view of the virtual camera, the scene coordinate positions (world coordinate) of (

), Performs perspective compensation through magnification conversion, and uses an optical center (XYZ) on the scene coordinate system according to the projection method to obtain a special effect.

A virtual camera model creation step of creating a virtual camera model,

) Corresponding pixel coordinates of each image of the plurality of images

And a secondary curved surface model ( J ) modeled as a secondary curved surface that can be used to insert a video effect into the 360-degree forward view image J , which is used at the time of the virtual camera

A second curved surface model generating step of generating a second curved surface model,

), The center of the optical axis on the scene coordinate system (

), And a secondary curved surface model obtained in the secondary curved surface model generation step (

Coordinates on the scene coordinate system (

) Onto a point on the secondary curved surface, projecting a point projected on the secondary curved surface onto each of the images of the plurality of images,

) To the pixel coordinate of the corresponding forward camera image (

), And the corresponding pixel coordinates (

) Interpolating pixel values of neighboring pixels for each of the plurality of images with respect to a coordinate u of a point of a 360-degree forward view image J viewed from the viewpoint of the virtual camera, And generating an omnidirectional camera projection model by matching the pixel values from the respective images of the plurality of images acquired in the omnidirectional camera projection model generation step and outputting the 360 degrees forward direction by determining the pixel value for the point coordinates (u) of the view image (J) 360 shown in the time point of the virtual camera may also include an image generation step of generating a forward view image (J). The step of generating the 360-degree omnidirectional view image at the set arbitrary time may further include compensating the size of the 360-degree omnidirectional view image J displayed at the viewpoint of the virtual camera for the perspective correction have.

Each step in the step of generating a 360-degree omnidirectional view image that is displayed at an arbitrary set point will be described in more detail.

) 및 광축 중심(

)을 계산하는 단계를 포함할 수 있는데, 상기 가상 카메라의 위치 및 자세를 설정하는 단계는 장면 좌표계의 x, y, z축의 방향 및 가상 좌표계의 x_n, y_n, z_n의 방향을 설정하는 단계, 장면 좌표계의 z축을 중심으로 수평 회전각(azimuth) 및 수직 회전각(elevation)을 설정하는 단계, 및 장면 좌표계의 중심으로부터의 거리를 설정하는 단계를 포함할 수 있고, 상기 가상 카메라의 유형 및 각 유형별 파라미터들을 설정하는 단계는 일반적인 원근 효과를 갖는 원근 투영 카메라(perspective projection camera), 원근 효과가 제거된 렌즈를 모방하는 직각 투영 카메라(orthographic projection camera), 및 전방향 파노라마 영상을 생성할 수 있는 파노라마 카메라(panorama camera) 중의 하나를 가상 카메라의 유형으로 선택하는 단계 및 선택된 상기 가상 카메라의 유형에 따라 파라미터를 설정하는 단계를 포함할 수 있다.The virtual camera model generation step may include generating a virtual camera image at a desired time point and generating a virtual camera camera position and attitude so as to emulate a digital pan, tilt, and zoom function of the mechanical camera step of setting, in the virtual camera, the method comprising setting the types and each type of parameters of, and set the virtual type of the camera, a position, and a 360-degree omni-directional view image (J) is shown the posture at the time point of the virtual camera on the basis Coordinates on the scene coordinate system for a point (

) And the center of the optical axis (

The step of setting the position and posture of the virtual camera may include setting directions of x, y, and z axes of the scene coordinate system and directions of x _n , y _n , and z _n of the virtual coordinate system Setting a horizontal rotation angle (azimuth) and a vertical rotation angle (elevation) about the z-axis of the scene coordinate system, and setting a distance from the center of the scene coordinate system, And setting the parameters for each type may be accomplished by a perspective projection camera having a general perspective effect, an orthographic projection camera mimicking a lens with a perspective effect removed, and an omni-directional panoramic image Selecting one of the existing panorama cameras as the type of the virtual camera and selecting the parameter according to the type of the selected virtual camera And a step of setting the step.

)을 계산하는 단계를 포함할 수 있는데, 나아가, 상기 2차 곡면 모델의 유형을 설정하는 단계는 가상 카메라의 팬, 틸트, 줌 효과를 얻기 위해 사용될 수 있는 타원체(Ellipsoid) 모델, 광각 영상 생성을 위한 실린더(Cylinder) 모델, 영상의 일정 부분만을 매핑하는 효과를 얻기 위한 포물면 모델, 및 사용자가 임의로 설정할 수 있는 사용자 2차 곡면 모델 중 하나를 선택하는 단계를 포함할 수 있고, 상기 설정된 상기 2차 곡면 모델의 자세 및 위치를 설정하는 단계는 장면 좌표계의 z축을 중심으로 수평 회전각(azimuth) 및 수직 회전각(elevation)을 설정하는 단계를 포함할 수 있다.The generating of the secondary curved surface model may include setting a type of the secondary curved surface model, setting the attitude and position of the set secondary curved surface model, setting the set secondary curved surface model based on the set attitude and position, A quadratic surface model on the scene coordinate system (

The step of setting the type of the secondary curved surface model may include an Ellipsoid model that can be used to obtain the pan, tilt, and zoom effects of the virtual camera, A curved surface model for obtaining an effect of mapping only a certain portion of an image, and a user curved surface model that can be arbitrarily set by the user, Setting the posture and position of the curved surface model may include setting a horizontal azimuth and an elevation about the z-axis of the scene coordinate system.

), 및 장면 좌표계 상의 광축 중심(

)을 연결하는 직선인 투영선과 상기 2차 곡면 모델(

)과의 교차점(

)을 계산하는 단계, 상기 교차점(

)을 상기 복수 개의 영상의 각 영상의 대응하는 픽셀 좌표(

)로 매핑(mapping)하는 단계, 및 상기 각 영상의 대응하는 픽셀 좌표(

)의 주변 픽셀들의 픽셀값을 보간하여 상기 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대한 상기 복수 개의 영상의 각 영상으로부터의 픽셀값을 획득하는 단계를 포함할 수 있는데, 특히, 상기 교차점(

)을 상기 복수 개의 영상의 각 영상의 대응하는 픽셀 좌표(

)로 매핑(mapping)하는 단계는 장면 좌표계로 표현된 상기 교차점(

)을 카메라 좌표계의 대응하는 좌표(

)로 변환하는 단계, 상기 복수 개의 영상의 각 영상에 존재하는 왜곡에 대하여 수치적으로 근사한 테일러 확장(Taylor expansion series) 기반의 근사 모델을 기반으로 하는 왜곡 보정 매핑 함수(

)를 이용하여 상기 교차점(

)의 카메라 좌표계 좌표(

)를 왜곡 없는 영상 평면상의 한 점(

)으로 매핑하는 단계, 및 어파인(affine) 변환 행렬(

)을 이용하여 상기 왜곡 없는 영상 평면 상의 한 점(

)에 대응되는 상기 복수의 영상의 각 영상의 픽셀 좌표(

)를 획득하는 단계를 포함할 수 있다.In addition, the omnidirectional camera projection model generation step may include a step of generating coordinates

), And the center of the optical axis on the scene coordinate system (

) And the second curved surface model (

) ≪ / RTI >

), Calculating the intersection point

) Corresponding pixel coordinates of each image of the plurality of images

), Mapping the corresponding pixel coordinates ("

) To obtain a pixel value from each image of the plurality of images with respect to a coordinate u of a point of the 360-degree forward view image J viewed at the viewpoint of the virtual camera , And in particular, the intersection (

) Corresponding pixel coordinates of each image of the plurality of images

) May include mapping the intersection points represented by the scene coordinate system

) To the corresponding coordinates of the camera coordinate system (

), A distortion correction mapping function based on an approximate model based on a Taylor expansion series that is numerically approximated to distortion existing in each image of the plurality of images

) At the intersection (

) Coordinates of the camera coordinate system (

) To a point on the distortion-free image plane (

), And a step of mapping an affine transformation matrix (

), A point on the distortion-free image plane (

) Of the respective images of the plurality of images corresponding to the pixel coordinates

). ≪ / RTI >

In addition, the step of compensating for the size of the 360-degree forward view image J displayed at the viewpoint of the virtual camera for correcting the perspective sensation may be performed by using the horizontal axis of the 360-degree forward view image J , And adjusting the size ratio of the vertical axis and the type and parameter setting of the secondary curved surface model to compensate for the depth and the size.

The 360 degree omnidirectional view image generation method may further include synchronizing a plurality of images captured by the plurality of image sensors and storing the images as a single file, And concurrently storing the acquired voice information and GPS information together with the image.

Another method of generating a 360-degree omni-directional view image is to select one of a real-time image captured by a plurality of image sensors and an image captured in the past and stored in a storage device as an original image, Generating a 360-degree omnidirectional view image that is displayed at an arbitrary point in time by synthesizing the distorted images, removing the distortion included in each image, generating a 360-degree omni- Changing a position and a posture of the camera, and generating a 360-degree omni-directional view image according to the changed viewpoint or the position and posture of the virtual camera.

It may also include a computer program stored on a computer readable recording medium that when executed on a computer or processor executes a method according to any of the methods described above.

According to the present invention, it is possible to synthesize a 360-degree omnidirectional view image based on an image captured by a plurality of cameras.

In addition, there is an effect that a sense of heterogeneity that may occur at the shooting boundary of a plurality of cameras of a 360-degree omnidirectional view image synthesized by synchronizing shooting times of a plurality of cameras can be minimized.

In addition, when synthesizing images photographed by a plurality of cameras, distortions contained in the images are corrected, thereby providing a feeling of heterogeneity with the actual space seen by the user and preventing the feeling of space from being deteriorated.

)의 일 예를 도시한 도면이다.
도 12는 본 발명의 일 실시 예에 따른 장면 좌표계상의 한 점(

)을 왜곡 없는 이상적인 영상 평면(

)의 대응하는 점으로 매핑하는 과정을 도시한다.
도 13은 3대의 카메라들 사이의 위치 및 자세를 나타내고 왜곡을 보정 하기 위한 파라미터를 획득하기 위한 방법의 일 실시 예를 도시한 도면이다.
도 14는 본 발명의 일 실시 예에 따른 가상 카메라의 투영선과 2차 곡면 모델과의 교차점(

)이 왜곡을 포함하는 이상적인 영상 평면상의 좌표(

)에 재투영되는 것을 도시한 도면이다.
도 15는 입력 영상 (a)로부터 왜곡이 보정된 영상 (b), (c)를 생성한 일 예를 보여준다.
도 16a 내지 도 16e는 본 발명의 일 실시 예에 따른 180도 이상의 화각을 갖는 2개의 전/후방 카메라를 이용하여 생성한 360도 전방향 뷰 영상을 도시한 것이다.
도 17a 내지 도 17g는 본 발명의 일 실시 예에 따른 가중치를 계산하는 예 및 가중치를 적용한 일 실시 예를 도시한 도면이다.
도 18은 본 발명의 일 실시 예에 따른 360도 전방향 뷰 영상 생성 방법을 도시한 도면이다.
도 19는 본 발명의 일 실시 예에 따른 복수의 영상 센서를 동기시키는 방법을 도시한 도면이다.
도 20은 본 발명의 일 실시 예에 따른 복수의 영상에 포함된 왜곡을 제거하고 복수의 영상을 정합하여 설정된 임의 시점에서 본 360도 전방향 뷰 영상을 생성하는 방법을 도시한 도면이다.
도 21은 본 발명의 일 실시 예에 따른 가상 카메라 모델 생성 단계(S310)를 구체적으로 도시한 도면이다.
도 22는 본 발명의 일 실시 예에 따른 2차 곡면 모델의 생성 단계(S320)를 구체적으로 도시한 도면이다.
도 23은 본 발명의 일 실시 예에 따른 전방향 카메라 투영 모델의 생성 단계(S330)를 구체적으로 도시한 도면이다.
도 24는 교차점(

)을 영상 센서에서 촬상한 영상의 대응하는 픽셀 좌표로 매핑하는 단계를 상세히 도시한 도면이다.
도 25는 본 발명의 일 실시 예에 따른 원근감 보정을 위한 크기 보상 단계를 구체적으로 도시한 도면이다.
도 26은 본 발명의 또 다른 일 실시 예에 따른 360도 전방향 뷰 영상을 생성하는 방법을 도시한다.FIG. 1 is a diagram illustrating a 360-degree omnidirectional view image generating apparatus according to an embodiment of the present invention.
2 is a diagram illustrating the structure of an image processing processor 200 according to an embodiment of the present invention.
3 is a diagram schematically illustrating the configuration of a multiple digital image processor 300 for generating a virtual camera-based random view image according to an embodiment of the present invention.
FIG. 4 is a diagram showing a schematic configuration of a virtual camera model unit 310 according to an embodiment of the present invention.
Figure 5 is a catadioptric type mirror (mirror) or by using a polar coordinate format image showing the virtual camera to generate an omnidirectional image from the panorama image pickup by using the lens image coordinates (u) and And the polar coordinates (r, [theta]).
FIG. 6 shows a virtual camera coordinate system {n} compared with a world coordinate.
7 is a view for explaining the center of the optical axis of the virtual camera.
8 is a view showing a schematic configuration of a quadric surface model unit according to an embodiment of the present invention.
9A and 9B illustrate parameters for the ellipsoidal model 324, the cylinder model 325, and the parabolic model 326 according to an embodiment of the present invention.
FIG. 10 is a diagram showing a schematic configuration of an omnidirectional camera projection modeling unit 330 according to an embodiment of the present invention.
Fig. 11 is a graph showing the relationship between the intersection of the projection plane of the secondary curved surface model and the virtual camera

).
FIG. 12 is a diagram illustrating a point on a scene coordinate system according to an exemplary embodiment of the present invention

) To the ideal image plane without distortion (

To the corresponding point of the second map.
13 is a diagram showing an embodiment of a method for indicating the position and attitude between three cameras and obtaining a parameter for correcting the distortion.
FIG. 14 is a graph showing the relationship between the intersection of the projection line of the virtual camera and the quadratic surface model according to the embodiment of the present invention

) Coordinates on the ideal image plane including this distortion

As shown in FIG.
FIG. 15 shows an example of generating images (b) and (c) in which the distortion is corrected from the input image (a).
FIGS. 16A to 16E illustrate a 360-degree omnidirectional view image generated using two front / rear cameras having an angle of view of 180 degrees or more according to an embodiment of the present invention.
FIGS. 17A to 17G are views illustrating an example of calculating a weight according to an embodiment of the present invention and an embodiment to which weights are applied.
18 is a diagram illustrating a 360-degree forward view image generation method according to an embodiment of the present invention.
19 is a diagram illustrating a method of synchronizing a plurality of image sensors according to an embodiment of the present invention.
20 is a view illustrating a method of generating a 360-degree omnidirectional view image at a predetermined time point by removing a distortion included in a plurality of images and matching a plurality of images according to an embodiment of the present invention.
FIG. 21 is a diagram specifically illustrating a virtual camera model generation step (S310) according to an embodiment of the present invention.
FIG. 22 is a diagram specifically showing a generation step (S320) of a quadric curved surface model according to an embodiment of the present invention.
FIG. 23 is a diagram specifically illustrating a generation step (S330) of an omnidirectional camera projection model according to an embodiment of the present invention.
Fig. 24 is a cross-

) To the corresponding pixel coordinates of the image captured by the image sensor.
25 is a diagram specifically illustrating a size compensation step for perspective image correction according to an embodiment of the present invention.
FIG. 26 illustrates a method of generating a 360-degree forward view image according to another embodiment of the present invention.

In order to clearly illustrate the present invention, parts not related to the description are omitted, and the same or similar components are denoted by the same reference numerals throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a part is referred to as " including " an element, it does not exclude other elements unless specifically stated otherwise.

If any part is referred to as being " on " another part, it may be directly on the other part or may be accompanied by another part therebetween. In contrast, when a section is referred to as being " directly above " another section, no other section is involved.

The terms first, second and third, etc. are used to describe various portions, components, regions, layers and / or sections, but are not limited thereto. These terms are only used to distinguish any moiety, element, region, layer or section from another moiety, moiety, region, layer or section. Thus, a first portion, component, region, layer or section described below may be referred to as a second portion, component, region, layer or section without departing from the scope of the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms as used herein include plural forms as long as the phrases do not expressly express the opposite meaning thereto. Means that a particular feature, region, integer, step, operation, element and / or component is specified and that the presence or absence of other features, regions, integers, steps, operations, elements, and / It does not exclude addition.

Terms indicating relative space such as " below ", " above ", and the like may be used to more easily describe the relationship to other portions of a portion shown in the figures. These terms are intended to include other meanings or acts of the apparatus in use, as well as intended meanings in the drawings. For example, when inverting a device in the figures, certain portions that are described as being " below " other portions are described as being " above " other portions. Thus, an exemplary term " below " includes both up and down directions. The device can be rotated by 90 degrees or rotated at different angles, and terms indicating relative space are interpreted accordingly.

Unless otherwise defined, all terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Commonly used predefined terms are further interpreted as having a meaning consistent with the relevant technical literature and the present disclosure, and are not to be construed as ideal or very formal meanings unless defined otherwise.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

The 360 degree omnidirectional view image generating apparatus proposed in the present invention can generate an omnidirectional view image that has the same effect as that taken by one camera using two or more cameras for photographing all the omnidirectional regions in the front and rear, And provides an apparatus and a method for generating and providing.

FIG. 1 is a diagram illustrating a 360-degree omnidirectional view image generating apparatus according to an embodiment of the present invention.

Two or more image sensors (cameras) for photographing all the omnidirectional regions are used to constitute the 360-degree omnidirectional view image generating apparatus. In the embodiment shown in FIG. 1, two image sensors are used. However, image sensors having two wide-angle lenses capable of capturing the forward and backward directions respectively, or images having two wide-angle lenses Sensors, or image sensors having four wide-angle lenses capable of photographing four sides, respectively.

Referring to FIG. 1, a 360-degree omnidirectional view image generating apparatus according to an embodiment of the present invention includes a sensor unit 100 including a plurality of image sensors capable of taking an image of an omnidirectional region, An image processing processor (200) for controlling the plurality of image sensors to control the image sensing time, the same exposure and the same color control, receiving and processing a plurality of images picked up by the plurality of image sensors, And a multi-digital image processor 300 for generating 360-degree omnidirectional view images, which are displayed at any time point set by the user, and outputting the 360-degree omnidirectional view images to the display 400.

The sensor unit 100 of the 360 degree omnidirectional view image generating apparatus according to an embodiment of the present invention includes a plurality of image sensors S1 and S2 capable of taking a 360-degree forward direction region. The image sensor may include a general camera, a wide-angle camera, and the like. As shown in FIG. 1, in order to photograph a 360-degree forward area with two image sensors, the field of view of each image sensor must be 180 degrees or more. If three image sensors are used, 120 degrees or more, and when using four image sensors, the angle of view of the lens should be 90 degrees or more.

In order to minimize the occurrence of motions and the occurrence of motion in overlapping portions of the images captured by the respective image sensors, it is necessary to set the 360 degrees forward direction All the image sensors controlled by the view image generating apparatus must be controlled to operate at the same time. For this, the image processing processor 200 has a function of capturing images at the same time in synchronism with the plurality of image sensors included in the sensor unit 100, a function of controlling the same exposure, And a white balance and color control function for maintaining the image quality. The same imaging, same exposure, and color control functions of the image processing processor 200 can be performed at the boundaries of the matched final 360 degree omnidirectional view images when moving objects appear at the boundary of the images acquired by each image sensor, It is possible to minimize the occurrence of motions and the heterogeneity due to the color difference.

The image processing processor 200 of the 360-degree omnidirectional view image generating apparatus according to an embodiment of the present invention performs the above-described functions. The image processing processor 200 includes one master image processing processor 210 and a plurality of (Slave ISP) 220, 230, and 240, respectively.

2 is a diagram illustrating the structure of an image processing processor 200 according to an embodiment of the present invention.

2, each of the main image processing processor 210 and the auxiliary image processing processors 220, 230 and 240 receives an image from each image sensor of the corresponding sensor unit 100, .

The main image processing processor 210 transmits the same clock signal and image synchronizing signal to each image sensor of the sensor unit 100 so that all the image sensors in the sensor unit 100 can image and output images at the same time do. The clock signal and the image synchronizing signal are also transmitted to the auxiliary image processing processors 220, 230 and 240 so that the auxiliary image processing processors 220, 230 and 240 can correctly receive the image from the corresponding image sensor .

The auxiliary image processing processors 220, 230 and 240 may detect the optical characteristics of the input image using an optical detection module and transmit the detected data (ODM information) to the main image processing processor 210 have. The optical characteristics may include an average value of colors to be detected and an average value of pixels corresponding to white, for each color channel R, G, B or Y, Cb, Cr. The main image processing processor 210 receives the ODM information 250 from each of the auxiliary image processing processors 220, 230 and 240 and compares all the ODM information including the ODM information generated by the main image processing processor 210, 230 and 240. The auxiliary image processing processors 220, 230 and 240 transmit the control signal 260 to the respective auxiliary image processing processors 220, 230 and 240. Based on the received control signals, And adjusts the shutter speed and the automatic gain control value of the connected image sensor 211, 221, 231, and 241, respectively, by determining the color correction information of the sensed image. Then, the main image processing processor 210 and the auxiliary image processing processors 220, 230, and 240 can use the same exposure value and color correction information, and the photographed images are set to the same exposure and the same color.

The same imaging, same exposure, and color control functions of the image processing processor 200 can be performed in the same manner as in the case of moving objects at the boundary between the images captured by the image sensor, Occurrence (the motion of the moving object appears in the captured image if the image is not captured at the same time), and the heterogeneity due to the color difference can be minimized.

The multi-digital image processor 300 of the 360-degree omnidirectional view image generating apparatus according to an embodiment of the present invention generates a 360-degree omnidirectional view image by using a plurality of images from the image processing processor 200, An image can be generated and output to the display 400.

The smaller the number of cameras used for photographing the panoramic view angle, the wider the field of view of each camera, and the more the lens distortion is caused thereby. There are various types of errors that occur in the process of fastening a plurality of image sensors. That is, images captured by a plurality of image sensors may include different distortions due to differences in view angle and distortion between the lenses, errors in camera tightening, errors in tightening each lens, and sensor position errors. The multiple digital image processor 300 may remove the above-mentioned distortion in a plurality of images, and smoothly perform image matching in the overlapped portions in a plurality of images to generate a 360-degree omni-directional view image.

For this purpose, the multiple digital image processor 300 generates a viewpoint image that is viewed from the viewpoint of the virtual camera based on the images input from the image processing processor 200.

3 is a diagram schematically illustrating the configuration of a multiple digital image processor 300 for generating a virtual camera-based random view image according to an embodiment of the present invention.

3, a multiple digital image processor 300 according to an exemplary embodiment of the present invention includes a virtual camera model unit 310, a quadric curved surface model unit 320, an omnidirectional camera projection model unit 330, A setting unit 340, and a matching unit 350.

The virtual camera modeling unit 310 may model the virtual camera as if it is at an arbitrary position set by the user so that the virtual camera model 310 displays the image on the display 400 at a certain time point designated by the user. In addition, digital PTZ (Pan, Tilt, Zoom) function and special lens effect can be created by using perspective projection camera with general perspective effect, right angle projection An orthographic projection camera, a panoramic camera model that generates a panoramic image using a polar coordinate image model in which the omnidirectional lens is imitated, and the like. The quadric surface modeling unit 320 may be used to add various image effects when mapping a plurality of original images captured by a plurality of image sensors to a 360 degree forward view image generated by a virtual camera And the forward direction camera projection modeling unit 330 provides a quadric surface providing a mirror function while the forward direction camera projection modeling unit 330 removes the distortion included in the image captured by the plurality of image sensors, And the parameter setting unit 340 may map the virtual camera model unit 310 and the quadratic surface model 340 to the virtual camera model unit 310 to minimize the loss of perspective caused by the distortion correction. The parameters of the unit 320 can be set. The matching unit 350 generates a 360-degree omnidirectional view image by matching the plurality of images from which the distortion of the plurality of original images generated by the omnidirectional camera projection model unit 330 is removed, Can be displayed.

Before explaining each part in more detail, some coordinate systems used in the present invention will be described. In the present invention, a virtual camera coordinate system {n} for representing a point or pixel of an image viewed by a virtual camera, a quadratic curve coordinate system {q} representing a coordinate by using the center of the quadratic curve as a zero point, We use camera coordinate systems {1}, {2},?, And {c} to represent the coordinates in each image captured by one image sensor. The coordinates displayed by this coordinate system can be converted into coordinates on the world coordinate coordinate system (Worldwide Coordinate) {0}.

The virtual camera modeling unit 310 is a virtual camera modeling unit that is similar to a PTZ (pan, tilt, zoom) camera having a pan, tilt, and zoom function using a mechanical driving unit and a motorized zoom lens. the position and attitude of the virtual camera are modeled so that the virtual camera can generate an image as seen at an arbitrary point of view, and the point (pixel) of the 360-degree forward view image J , ) Can be modeled. At this time, a virtual camera coordinate system can be used.

FIG. 4 is a diagram showing a schematic configuration of a virtual camera model unit 310 according to an embodiment of the present invention.

4, a virtual camera model unit 310 according to an exemplary embodiment of the present invention includes a rectangular coordinate display module 311, a polar coordinate display module 312, a size adjustment module 313, a position setting module 314, A transformation module 315, and an optical axis centering module 316. [

에 의해서 구해지는

의 직각 좌표 형식으로 좌표(u)를 표시할 수 있다. 여기서

는 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 중심을 직각 좌표 형식으로 나타낸 것이고, 설정 가능한 파라미터이다. 이처럼 영상 좌표를 직각 좌표 형식을 이용하여 표시하는 영상은 원근 투영 카메라 또는 직각 투영 카메라에서 사용될 수 있다.The rectangular coordinate display module 311 can display the coordinates ( u ) of a point of the 360-degree forward view image J shown in the viewpoint of the virtual camera in a rectangular coordinate format. The rectangular coordinate display module 311 can display the coordinate u as relative coordinates with respect to the center of the 360-degree forward view image J viewed from the viewpoint of the virtual camera. Therefore, the rectangular coordinate display module 311

Obtained by

( U ) can be displayed in the form of a rectangular coordinate of here

Represents the center of the 360-degree forward view image J viewed in the viewpoint of the virtual camera in the rectangular coordinate format, and is a settable parameter. An image displaying the image coordinates using a rectangular coordinate format can be used in a perspective projection camera or a rectangular projection camera.

)과 최대값(

), 반지름 방향 픽셀 수(

), 각도의 최소값(

) 과 최대값(

), 그리고 가상 카메라상에서의 극좌표 변환을 위한 극좌표 형식에서의 중심 (

)을 포함하며, 설정 가능한 파라미터들이다. 그러면 좌표(u)를 나타내는 극좌표 (r, ρ)는 다음과 같이 표현될 수 있다.The polar coordinate display module 312 can display the coordinates ( u ) of one point of the 360-degree forward view image J shown in the viewpoint of the virtual camera in a polar coordinate format. The parameter used here is the minimum value of the radius to be used in the polar coordinate format (

) And the maximum value (

), The number of pixels in the radial direction (

), The minimum value of the angle (

) And the maximum value (

), And the center of polar coordinates for polar transformation on the virtual camera (

), And are configurable parameters. Then, the polar coordinates (r, rho) representing the coordinate ( u ) can be expressed as follows.

이다. 그리고 극좌표 형식으로 표시되는 영상 좌표

는 다음과 같이 표현될 수 있다.here,

to be. And image coordinates displayed in the polar coordinate format

Can be expressed as follows.

의 극좌표 형식으로 좌표(u)를 표시할 수 있다. 이러한 극좌표 형식의 표시는 전방향 파노라마 영상을 생성하는데 이용될 수 있다.That is, the polar coordinate display module 312

( U ) can be displayed in polar coordinate form. This polar coordinate format display can be used to generate an omni-directional panoramic image.

Figure 5 is a catadioptric type mirror (mirror) or by using a polar coordinate format image showing the virtual camera to generate an omnidirectional image from the panorama image pickup by using the lens image coordinates (u) and And the polar coordinates (r, [theta]).

)에 의하여 어떤 표현 방식을 사용할 것인지를 결정할 수 있다.The virtual camera model unit 310 may represent the coordinates ( u ) of a point of a 360-degree forward view image J viewed from the viewpoint of the virtual camera in a rectangular coordinate format or a polar coordinate format,

) Can decide which representation to use.

)에 의해 직각 좌표 형식 또는 극좌표 형식으로 표현되는 가상 카메라 영상의 한 점의 좌표 (u')를 크기 보상 행렬과 초점거리를 이용하여 가상 카메라 좌표계 상의 좌표 (

)로 표현될 수 있다. 이때의 좌표(

)는 다음 식을 이용하여 구할 수 있다.The size adjustment module 313 is a process of compensating the size of the virtual camera image in a form that the user can easily view. The size adjustment module 313 can adjust the size ratio of the image to the horizontal axis and the vertical axis. Accordingly, the coordinate ( u ) of a point of the 360-degree forward view image J viewed from the viewpoint of the virtual camera can be adjusted. That is, when the size adjustment module generates a sense of heterogeneity between the image generated by the virtual camera and the actual space visible to the user, the user can feel dizziness while viewing the image, and the sense of distance is lost and the sense of space is lost. In order to minimize such a sense of heterogeneity, the size adjustment module 313 can be used. That is, the resizing module 313 may select

( U ' ) of a point of a virtual camera image expressed in a rectangular coordinate form or a polar coordinate form by using the magnification compensation matrix and the focal distance,

). ≪ / RTI > At this time,

) Can be obtained by using the following equation.

은 가로축과 세로축 크기 보상값들로 구성된 대각행렬(diagonal matrix)이고,

은 가상카메라의 초점 거리(focal length)다.here,

Is a diagonal matrix composed of the horizontal axis and vertical axis size compensation values,

Is the focal length of the virtual camera.

The position setting module 314 sets the position and posture of the virtual camera.

The virtual camera modeling unit 310 is a virtual camera modeling unit that is similar to a PTZ (pan, tilt, zoom) camera having a pan, tilt, and zoom function using a mechanical driving unit and a motorized zoom lens. the position and attitude of the virtual camera can be modeled so that the virtual camera can generate an image as seen at an arbitrary point in time. At this time, a virtual camera coordinate system can be used for setting the position and attitude of the virtual camera.

FIG. 6 shows a virtual camera coordinate system {n} compared with a world coordinate.

을 장면 좌표계 {0} 상에서의 가상 카메라의 위치,

를 장면 좌표계 {0} 상에서의 2차 곡면모델의 중심 위치이며 가상 카메라의 팬, 틸트 회전의 중심으로 두면, 장면 좌표계 {0} 상에서 2차 곡면모델로 매핑되는 가상 카메라 좌표계 {n}를 표현한 변환 행렬(

)은 다음과 같이 표현할 수 있다.The virtual camera coordinate system {n} can be rotated vertically and horizontally with respect to the center (10) of the scene coordinate system for the pan and tilt functions, and the position of the viewpoint can be moved in the _zn direction for the zoom function. 6, φ _n and θ _n can be respectively set to a horizontal rotation angle (azimuth) and a vertical rotation angle (elevation) with respect to the z axis 11 of the scene coordinate system of the virtual camera,

To the position of the virtual camera on the scene coordinate system {0}

Of the virtual camera coordinate system {n} mapped to the quadratic surface model on the scene coordinate system {0} if the central position of the quadratic surface model on the scene coordinate system {0} is the center of the pan and tilt rotation of the virtual camera procession(

) Can be expressed as follows.

)은

의 단위 벡터 형태로 표현될 수 있다. 여기서,

,

, 및은 각각 가상 카메라 좌표계 {n} 상에서의 각 단위 벡터({1,0,0}, {0,1,0}, 및 {0,0,1})를 장면 좌표계 {0} 상의 좌표로 변환했을 때의 단위 벡터 표현이고,

은 가상 카메라 좌표계 {n} 상에서의 원점({0,0,0})을 장면 좌표계 {0} 상의 좌표로 변환했을 때의 단위 벡터 표현이다. 상술한 것과 같이 수평 회전각과 수직 회전각을 사용한 가상 좌표계의 표현은 상하가 뒤집히지 않고, 어떤 중심을 기준으로 하여도 회전이 가능하며, 줌 기능을 효과적으로 표현할 수 있으므로 기계적인 구동부를 사용한 PTZ 기능을 거의 흡사하게 동작시킬 수 있는 장점이 있다. 이러한 가상 좌표계의 표현 이외에도 사용자의 설정에 따라 임의 방식으로 가상 좌표계를 표현하는 것도 가능하다.Here, trans (·) represents a 3-D translational matrix on a three-dimensional space, and rot _z (·) and rot _x (·) represent rotation matrices (3- D rotational matrices. And the transformation matrix (

)silver

Can be expressed in unit vector form. here,

,

, And Have transformed each unit vector ({1,0,0}, {0,1,0}, and {0,0,1}) on the virtual camera coordinate system {n} into coordinates on the scene coordinate system {0} , ≪ / RTI >

Is a unit vector representation when the origin ({0,0,0}) on the virtual camera coordinate system {n} is transformed into coordinates on the scene coordinate system {0}. As described above, the expression of the virtual coordinate system using the horizontal rotation angle and the vertical rotation angle can be rotated based on any center without upside down, and the zoom function can be effectively expressed. Therefore, the PTZ function using the mechanical driving unit There is an advantage that it can be operated almost like that. In addition to the expression of the virtual coordinate system, it is also possible to express the virtual coordinate system in an arbitrary manner according to the user's setting.

), 장면 좌표계상에서의 가상 카메라의 위치(

), 및 가상 카메라의 장면 좌표계의 z축(11)에 대한 수평 회전각(φ_n) 및 수직 회전각(θ_n)을 파라미터로 입력받아 장면 좌표계 상에서 가상 좌표계를 표현한 변환 행렬(

)를 계산한다. 이 변환 행렬에 의하여 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)를 장면 좌표계의 한 좌표(

)로 변환할 수 있다.That is, the position setting module 314 sets the center of the quadratic surface model on the scene coordinate system

), The position of the virtual camera on the scene coordinate system (

) And a horizontal rotation angle (? _N ) and a vertical rotation angle (? _N ) with respect to the z axis (11) of the scene coordinate system of the virtual camera as parameters,

). The coordinate ( u ) of a point of the 360-degree forward view image ( J ) viewed from the viewpoint of the virtual camera is transformed into a coordinate

). ≪ / RTI >

)을 이용하여 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대응되는 장면 좌표계의 한 좌표(

)를 계산한다.The transform module 315 transforms the transformation matrix (i. E.

( X ) of the scene coordinate system corresponding to the coordinate ( u ) of one point of the 360-degree forward view image J shown at the viewpoint of the virtual camera

).

The optical axis center setting module 316 is for setting the optical axis center of the virtual camera.

7 is a view for explaining the center of the optical axis of the virtual camera.

)은 다음과 같이 표현될 수 있다. 여기서

의 왼쪽 위에 있는 '0'은 좌표계를 나타내는 것으로 {0}은 장면 좌표계상의 좌표로 표시됨을 나타낸다.Referring to FIG. 7, the perspective projection camera and the right-angle projection camera may vary depending on whether the projection ray is gathered at the optical center. In the perspective projection camera, all the projection lines converge to one central point as shown in Fig. 7 (a), and in a rectangular projection camera, all projection lines are parallel to the principal axis as shown in Fig. 7 (b) do. Therefore,

) Can be expressed as follows. here

'0' in the upper left corner of the screen indicates that the coordinate system is represented by {0}, which is represented by coordinates on the scene coordinate system.

은 초점 거리(focal length)이고,

는 상술한 바와 같이 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 좌표(u)에 대응되는 장면 좌표계의 좌표이고,

및

은 상술한 것처럼 변환 행렬(

)을 단위 벡터 형태인

으로 표현할 때의 단위 벡터들이다. here,

Is the focal length,

Is a coordinate of a scene coordinate system corresponding to a coordinate ( u ) of a point of a 360-degree forward view image J viewed from the viewpoint of the virtual camera as described above,

And

Lt; RTI ID = 0.0 >("

) As a unit vector form

Are unit vectors when expressed as a unit vector.

) 및 가상 카메라의 광축(

) 중심에 대한 정보를 계산하여 출력할 수 있다.As described above, the virtual camera model unit 310 receives the parameters of whether to use the position of the virtual camera, the rectangular coordinate format, or the polar coordinate format, and outputs the 360-degree forward view image J , Of the scene coordinate system corresponding to the coordinate ( u ) of a point of the scene coordinate system

) And the optical axis of the virtual camera

) Information about the center can be calculated and output.

The secondary curved surface model generated by the quadric curved surface modeling unit 320 includes a plurality of images captured by a plurality of image sensors including distortion of a point of a 360 degree forward view image J viewed from the viewpoint of the virtual camera To a corresponding point (pixel) of the pixel. At this time, a secondary curved surface model that models various secondary curved surfaces according to a desired effect can be used. As an example, the Ellipsoid model can be used to obtain a PTZ effect of a virtual camera by being expressed as a spherical mirror shape, and a cylinder model is expressed as a cylinder columnar mirror shape, The paraboloid model can be used as a mirror image in the form of a hemispherical shape, and can be used to create an effect that maps only a part of an image.

8 is a view showing a schematic configuration of a quadric surface model unit according to an embodiment of the present invention.

8, the quadratic surface model 320 may include a quadratic surface model selection module 321, a coordinate setting module 322, and a transformation module 323 .

The secondary curved surface model selection module 321 may select one of a plurality of secondary curved surface models to obtain a desired effect by the user. That is, one of the ellipsoidal model 324, the cylinder model 325, the paraboloid model 326, and the user-set quadric surface model 327 can be selected using the parameter q _sel as shown in Fig. 8 .

The above quadratic surface models in the three-dimensional space can be expressed in the following common matrix form.

는

로 모델링되는 2차 곡면상의 한점이고, 2차 곡면 모델(

)은 다음과 같이 4x4 대칭 행렬로 표현될 수 있다.here,

The

, And a quadratic surface model (

) Can be expressed as a 4x4 symmetric matrix as follows.

) 행렬 내의 각 값은 2차 곡면별로 다를 수 있는데, 도 9a 및 도 9b는 본 발명의 일 실시 예에 따른 타원체 모델(324), 실린더 모델(325), 및 포물면 모델(326)에 대한 파라미터를 도시하고 있다. 여기서, 도 9a 및 도 9b의 x_c, y_c, z_c는 장면 좌표계 {0} 상에서의 각 모델의 중심 위치를 나타낸다.At this time, the secondary curved surface model (

) May be different for each quadratic surface. Figures 9a and 9b illustrate parameters for the ellipsoidal model 324, the cylinder model 325, and the parabolic model 326, according to an embodiment of the present invention, Respectively. Here, in Figures 9a and _{9b x c, y c, z} c denotes the center position of each model on the scene coordinate system {0}.

)을 생성할 수 있고, 이때의 변환행렬(

)는 다음 식을 이용하여 구할 수 있다. The coordinate setting module 322 is used to define a secondary curved surface on the scene coordinate system {0}, and a horizontal rotation angle? _Q and a vertical rotation angle? _Q , . That is, the coordinate setting module 322 includes a transformation matrix (e.g., a transformation matrix) for transforming points on the quadratic curve surface into points on the corresponding scene coordinate system

), And the transformation matrix (

) Can be obtained by using the following equation.

)을 장면 좌표계 상에서의 선택된 2차 곡면 모델을 표현하는 행렬(

)로 변환할 수 있다. 장면 좌표계에서의 2차 곡면 모델을 표현하는 행렬 (

)는 다음 식을 이용하여 구할 수 있다.The transformation module 323 is a matrix that represents a quadratic surface generated by the ellipsoidal model 324, the cylinder model 325, the paraboloid model 326, or the user-set quadratic surface model 327

) Representing a selected quadratic surface model on the scene coordinate system (

). ≪ / RTI > A matrix representing a quadratic surface model in the scene coordinate system (

) Can be obtained by using the following equation.

The omnidirectional camera projection modeling unit 330 re-projects a plurality of images captured by a plurality of image sensors including image distortion after mapping from the virtual camera to the quadratic curved surface model, and performs camera calibration ) And a camera projection model.

FIG. 10 is a diagram showing a schematic configuration of an omnidirectional camera projection modeling unit 330 according to an embodiment of the present invention.

10, an omnidirectional camera projection modeling unit 330 according to an embodiment of the present invention includes an intersection module 331, a transformation module 332, a coordinate calculation module 333, an interpolation module 334, And a memory 335.

The intersection module 331 can calculate the intersection point of the secondary curved surface model selected by the secondary curved surface modeling unit 320 and the projection line of the virtual camera.

)의 일 예를 도시한 도면이다.Fig. 11 is a graph showing the relationship between the intersection of the projection plane of the secondary curved surface model and the virtual camera

).

)과 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점(

)을 잇는 직선인 투영선(910)이 2차 곡면 모델과 만나기 위하여는 다음 식을 만족하여야 한다.Referring to FIG. 11, the center of the optical axis of the virtual camera

) And a point of a 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera

), The projection line 910, which is a straight line connecting the curved surface 910 to the curved surface model, must satisfy the following equation.

은 가상 카메라의 광축 중심이고,

으로 표현되는데

는 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점이다. 그리고 점

는 투영선(910) 상에 존재하는 점이면서, 2차 곡면 모델 상에 존재하는 교차점이 된다. 이 교차점의 위치는

에 의해서 결정되는데,

는 다음과 같이 계산될 수 있다.here,

Is the center of the optical axis of the virtual camera,

It is expressed as

Is a point of the 360-degree forward view image J viewed from the viewpoint of the virtual camera. And point

Is a point existing on the projection line 910 and an intersection existing on the quadratic curve model. The location of this intersection is

Lt; / RTI >

Can be calculated as follows.

이고,

이다.here,

ego,

to be.

)의 투영선(910)과 교차하는 2차 곡면상의 한 점(

)을 계산하여 출력할 수 있다.In other words, the intersection model is based on the above-mentioned formula, and a point of a 360-degree forward view image ( J )

A point on the secondary curved surface that intersects with the projection line 910

) Can be calculated and output.

)은 전방향 카메라 모델(336)에 의하여 센서부(100)의 복수의 영상 센서에 의해 촬상된 복수의 영상의 각 영상의 한 점으로 투영될 수 있다.Then, one point on the secondary curved surface (

Can be projected to a point of each image of a plurality of images captured by the plurality of image sensors of the sensor unit 100 by the forward camera model 336. [

)을 왜곡 없는 이상적인 영상 평면(

)의 대응하는 점으로 매핑하는 과정을 도시한다.FIG. 12 is a diagram illustrating a point on a scene coordinate system according to an exemplary embodiment of the present invention

) To the ideal image plane without distortion (

To the corresponding point of the second map.

에서 영상 센서의 광축 중심으로의 투영선(1010) 상에 벡터

가 존재하고, 이는 일반적인 원근 투영 기하를 따르고 다음 식의 관계가 성립된다.12, a point on the scene coordinate system

On the projection line 1010 to the center of the optical axis of the image sensor,

, Which follows the general perspective projection geometry and the relation of the following equation is established.

는 회전 행렬

과 이동 벡터

로 구성된 카메라 투영 행렬(projection matrix)이며,

는 장면 좌표계상의 한 점

의 동차 표현(homogenous representation)이다. here

The rotation matrix

And motion vectors

A camera projection matrix,

A point on the scene coordinate system

And is a homogenous representation of

와 카메라 좌표계 {c} 상의 한 점

를 서로 매핑한 것이다. 그런데 본 발명의 일 실시 예에 따른 360도 전방향 뷰 영상 생성 장치는 복수 개의 영상 센서로부터 영상을 획득하고, 각 영상 센서의 광축 중심이 다를 수 있고, 행렬

또한 다를 수 있기 때문에 장면 좌표계 {0} 상의 한 점

에 대하여 복수의 영상 센서의 각 카메라 좌표계 {1}, {2}, {...}, {c} 상의 한 점

을 매핑할 수 있다. 즉, 장면 좌표계 {0} 상의 한 점에 대하여 복수의 점이 매핑될 수 있다.The above-mentioned expression is a point on the scene coordinate system {0}

And a point on the camera coordinate system {c}

Are mapped to each other. However, the 360 degree omnidirectional view image generation apparatus according to an embodiment of the present invention acquires images from a plurality of image sensors, and the centers of the optical axes of the respective image sensors may be different,

Also, a point on the scene coordinate system {0}

A point on each camera coordinate system {1}, {2}, {...}, {c}

Can be mapped. That is, a plurality of points may be mapped to one point on the scene coordinate system {0}.

)와 카메라 좌표계 {c_i} 상의 한 점

를 서로 매핑할 수 있다. When the above description is applied to a plurality of image sensors, one point on the reference coordinate system {i}

) And a point on the camera coordinate system {c _i }

Can be mapped to each other.

는 회전 행렬(

)과 이동 백터(

)로 구성된 i번째 카메라의 투영 행렬(projection matrix)이며,

는 기준 좌표계 {i}상의 한 점

의 동차 표현(homogenous representation)이다. 그리고 절대 좌표계인 장면 좌표계를 {0}으로 표현하고, 장면 좌표계 {0}와 각 기준 좌표계 간의 변환 관계를 나타내는 변환 행렬(

)을 이미 알고 있다고 하면, 상기 식들은 모두 장면 좌표계 {0}상의 한 점

에 대한 식으로 다음과 같이 표현할 수 있다.Here, {c _i } and {i} represent the camera coordinate system and the reference coordinate system of the i-th image sensor, respectively,

Is a rotation matrix (

) And mobile vector

), Which is a projection matrix of the i-th camera,

Is a point on the reference coordinate system {i}

And is a homogenous representation of A scene coordinate system, which is an absolute coordinate system, is represented by {0}, and a transformation matrix ({0}) representing a transformation relation between a scene coordinate system {

) Are already known, all of the above equations are represented by a point on the scene coordinate system {0}

Can be expressed as follows.

으로 얻어질 수 있다. 즉, 장면 좌표계 {0} 상의 한 점

이 세 개의 카메라 좌표계의 서로 다른 세 점으로 매핑될 수 있다. 만약 카메라 수가 늘어나면 장면 좌표계 {0} 상의 한 점은 카메라 수만큼의 서로 다른 점들로 매핑될 수 있는 것이다.here,

. That is, one point on the scene coordinate system {0}

These three camera coordinate systems can be mapped to three different points. If the number of cameras increases, one point on the scene coordinate system {0} can be mapped to different points by the number of cameras.

의

는 장면 좌표계 상의 한 점

를 이상적인 영상 평면(

) 상의 한 점

으로 매핑하는 왜곡보정 매핑함수이며, 렌즈의 왜곡 현상을 이론적으로 분석한 모델(analytic model) 또는 수치적으로 근사한 모델(approximation model) 등을 사용하여 표현할 수 있다. 이에 의하여 이상적인 영상 평면(

)은 왜곡이 없는 영상 평면을 의미할 수 있다. 일 실시 예로서 Scaramuzza가 제안한 것과 같은 테일러 확장(Taylor expansion series) 기반의 근사 모델을 사용하여 표현할 수 있다.Referring again to Figure 12,

of

A point on the scene coordinate system

To an ideal image plane (

One point

And can be expressed using an analytic model or a numerically approximated model of the distortion phenomenon of the lens. Thus, the ideal image plane (

) Can mean an image plane without distortion. As an example, an approximate model based on the Taylor expansion series as proposed by Scaramuzza can be used.

는 카메라마다 서로 다른 영상 왜곡을 생성하기 때문에 카메라 좌표계 {c_i}별로 서로 다른 값을 가지게 된다.At this time

Have different values for each camera coordinate system {c _i } because they generate different image distortions for each camera.

을 복수의 영상 센서 중 하나의 영상 센서에서 촬상한 왜곡이 포함된 원본 영상(I)에서의 실제 픽셀 좌표라고 가정하면, 원본 영상(I)과 이상적인 영상 평면(

)의 관계는 어파인(affine) 변환 행렬로 다음과 같이 표현될 수 있다.And vector

Is assumed to be the actual pixel coordinates in the original image I including the distortion captured by one of the plurality of image sensors, the original image I and the ideal image plane ( I )

) Can be expressed as an affine transformation matrix as follows.

는 어파인 변환 행렬이고,

는 이동 벡터를 나타낸다. From here

Is an affine transformation matrix,

Represents a motion vector.

는 원본 영상(I)에서 픽셀 좌표

으로 매핑되므로, 만약 이 식들이 역변환 관계가 존재한다면, 원본 영상(I)에서 원래 장면 좌표계 상의 한 점

의 픽셀값을 구할 수 있다. As described above, one point on the scene coordinate system

In the original image I ,

, So that if there is an inverse transformation relationship between these equations, a point on the original scene coordinate system in the original image ( I )

Can be obtained.

을 카메라 좌표계 {c} 상의 한 점

로 왜곡 보정을 하면서 매핑시키기 위하여서는 상술한 바처럼 각 영상 센서의 위치와 자세 및 왜곡보정 매핑함수(

)의 계수들을 미리 알아야만 한다. One point on the scene coordinate system {0}

A point on the camera coordinate system {c}

In order to perform the distortion correction, the position, posture, and distortion correction mapping function of each image sensor

) In advance.

13 is a diagram showing an embodiment of a method for indicating the position and attitude between three cameras and obtaining a parameter for correcting the distortion.

), 이동 백터(

), 왜곡보정 매핑함수(

)의 계수, 및 어파인 변환 행렬(

)를 획득할 수 있다.Referring to FIG. 13, each of the image sensors can calculate a position, an attitude, and a distortion included in the sensed image by using a calibration chart in which a grid size in the horizontal and vertical directions is known in advance. For example, one horizontal and vertical size of a correction sheet for correction can be set to 5 cm x 5 cm, and the distortion included in the sensed image can be measured by measuring the positions of the intersections of the respective grids in the captured correction chart image . Then, based on the measured distortion, the rotation matrix (

), Mobile vector

), A distortion correction mapping function (

), And an affine transformation matrix (

Can be obtained.

)이 왜곡을 포함하는 이상적인 영상 평면상의 좌표(

)에 재투영되는 것을 도시한 도면이다.FIG. 14 is a graph showing the relationship between the intersection of the projection line of the virtual camera and the quadratic surface model according to the embodiment of the present invention

) Coordinates on the ideal image plane including this distortion

As shown in FIG.

)을 연결하는 선이 된다.Referring to FIG. 14, the secondary curved surface may serve as a mirror for reflecting the projection line 1110 of the virtual camera to the projection line 1120 of the image sensor. Herein, the projection line 1120 of the image sensor is located at an intersection of the center of the optical axis of the image sensor

).

)이 되고, 교차점(

)은 다음 식을 이용하여 카메라 좌표계의 좌표(

)로 변환될 수 있다.The point at which the projection line 1110 of the virtual camera meets the secondary curved surface is the intersection point

), And the intersection point

) Is calculated using the coordinates (

). ≪ / RTI >

는 장면 좌표계상에서 표현된 영상 센서의 위치와 자세를 나타내는 변환 행렬로써 카메라 투영행렬(

)의

과

로 구성될 수 있다. here,

Is a transformation matrix representing the position and orientation of the image sensor expressed on the scene coordinate system,

)of

and

≪ / RTI >

)은 도 12에 도시된 한 점(

)과 마찬가지로 전방향 카메라 광축 중심으로의 투영선(1120) 상에 존재할 수 있고, 이 투영선(1120)은 전방향 카메라의 광축을 중심으로 방사형으로 대칭이므로 이상적인 영상 평면(

)과 교차점(

)과의 각도로 표현할 수 있다. 즉,

의 관계식을 얻을 수 있는데 여기서,

는 이상적인 영상 평면(

)과 교차점(

)과의 각도로서,

로 표현된다.The intersection of the projection line 1110 of the virtual camera and the secondary curved surface (

) Corresponds to one point (

The camera may exist on the projection line 1120 to the center of the optical axis of the omnidirectional camera and the projection line 1120 may be radially symmetrical about the optical axis of the omnidirectional camera,

) And the intersection point

). ≪ / RTI > In other words,

Can be obtained. Here,

Is the ideal image plane (

) And the intersection point

),

Lt; / RTI >

)는 상술한 바처럼 모두

의 함수이므로, 이들의 해를 구하면 이상적인 영상 평면상의 좌표(

)를 구할 수 있고, 어파인 변환 행렬을 이용하여 원본 영상(I)에서의 픽셀값(

)을 구할 수 있다. 따라서, 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J) 내의 한 점(

)의 픽셀값은 영상 센서에서 촬상한 왜곡이 포함된 원본 영상(I)으로부터 얻어지고, 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J) 내의 각 점에 대하여 상술한 기능을 수행하면 최종 결과 영상인 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)을 획득할 수 있고, 이 영상은 임의 시점에서 생성한 렌즈 왜곡이 제거된 영상이 된다. 그리고 상술한 방식을 복수 개의 영상 센서별로 적용하면 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J) 내의 한 점(

)에 대하여 복수 개의 영상 센서의 각 영상 센서별로 획득한 픽셀값을 가지게 된다.The projection line 1120 to the center of the optical axis of the image sensor and the distortion correction mapping function

), As described above,

Therefore, when these solutions are obtained, the coordinates on the ideal image plane (

) Can be obtained, and the pixel value (?) In the original image ( I ) can be obtained using the affine transformation matrix

) Can be obtained. Therefore, a point within the 360-degree forward view image J shown at the viewpoint of the virtual camera (

) Is obtained from the original image I including the distortion imaged by the image sensor, and performs the above-described function on each point in the 360-degree forward view image J viewed at the viewpoint of the virtual camera It is possible to obtain a 360-degree forward view image J viewed from the viewpoint of the virtual camera, which is the final result image, and this image is an image from which the lens distortion generated at a certain point is removed. When the above-described method is applied to each of the plurality of image sensors, a point ( J ) in the 360-degree forward view image J

The pixel values of the respective image sensors of the plurality of image sensors are obtained.

)를 이용하여 교차점(

)을 카메라 좌표계의 좌표(

)로 변환한다. In order to perform the above-described operation, the conversion module 332 of the omnidirectional camera projection modeling unit 330 converts the transformation matrix (the transformation matrix) representing the position and the posture of the image sensor expressed on the scene coordinate system

) To determine the intersection

) To the coordinates of the camera coordinate system (

).

), 이동 벡터(

), 장면 좌표계상의 한 점

를 이상적인 영상 평면(

) 상의 한 점(

)으로 매핑하는 역할을 수행하는 테일러 확장 기반의 근사 모델의 계수들을 파라미터로 입력받아 가상 카메라의 투영선과 2차 곡면 모델과의 교차점(

)으로부터 영상 센서에서 획득한 왜곡이 포함된 원본 영상(I)에서의 실제 픽셀 좌표를 나타내는 벡터

를 획득한다. 좌표 계산 모듈(333)에서 사용하는 상술한 테일러 확장 기반의 근사 모델은 렌즈의 왜곡 현상을 이론적으로 분석하고, 이를 해소하기 위하여 수치적으로 근사화한 것으로 이에 의하여 원본 영상에 존재하는 왜곡 현상을 도 15와 같이 제거할 수 있다. 도 15는 입력 영상 (a)로부터 왜곡이 보정된 영상 (b), (c)를 생성한 일 예를 보여준다.Then, the coordinate calculation module 333 calculates an affine transformation matrix

), A motion vector (

), A point on the scene coordinate system

To an ideal image plane (

One point on

) Of the Taylor expansion based on the parameters of the approximate model based on the Taylor expansion, and the intersection of the projection line of the virtual camera and the second curved surface model

) Representing the actual pixel coordinates in the original image ( I ) containing the distortion obtained from the image sensor

. The Taylor expansion-based approximate model used in the coordinate calculation module 333 is a numerical approximation for analyzing the distortion phenomenon of the lens theoretically and solving the distortion phenomenon. Thus, the distortion phenomenon existing in the original image is shown in FIG. 15 Can be removed. FIG. 15 shows an example of generating images (b) and (c) in which the distortion is corrected from the input image (a).

를 이용하여 메모리(335)에 저장된 원본 영상(I)에서 해당 좌표의 주변 픽셀값을 읽어들이고 양선형 보간 (bilinear interpolation) 등의 방법 등으로 해당 좌표의 값을 계산 하여 출력할 수 있다.The interpolation module 334 interpolates the vectors representing the actual pixel coordinates

The neighboring pixel values of the corresponding coordinates are read from the original image I stored in the memory 335 and bilinear interpolation or the like is performed.

15 is a view showing an example of an image to be displayed on the display 400 by correcting the original image and the distortion included therein.

Referring to FIG. 15, (a) shows an original image including distortion obtained using a fish eye lens, and (b) shows an image in which distortion is corrected by a general method. (b), there is a large difference between the actual size and the distance feeling toward the end of the image. The reason for this is that there may be a lot of points on the space on the projection line coming into a point of the camera's distorted image and that there are more than two projection lines for the structure reconstruction, Because. However, by using the method proposed in the present application and setting the necessary parameters appropriately, the user can generate a comfortable image as shown in (c).

)에 대하여 단지 한 개의 영상 센서에서 촬상된 후 왜곡이 보정된 영상의 픽셀값만이 의미를 가질 수 있다. 즉, 정합부(350)에서 영상 정합 시에 대부분의 영역에서 겹쳐지는 부분(overlapping area)이 없다는 것이다. The omnidirectional camera model 336 of the omnidirectional camera projection modeling unit 330 may be independently performed on the camera images acquired by the respective image sensors S1 and S2. That is, an omnidirectional camera model corresponding to each of the plurality of image sensors may be provided, and an image from each image sensor may be processed to generate a distortion-free image at any time. That is, as shown in FIG. 3, the forward direction camera projection modeling unit 330 can generate a plurality of images from which distortion is removed for each camera image acquired by the image sensor, and transmit the generated images to the matching unit 350. At this time, since the cameras are arranged to image different regions, most of them are located at a point ( J ) in the 360-degree forward view image J

Only the pixel value of the image after the distortion of the image after being picked up by only one image sensor can have a meaning. That is, there is no overlapping area in most areas at the time of image matching in the matching unit 350.

)에 대하여 의미를 가질 수 있는 픽셀값이 복수 개가 될 수 있고, 최종 영상 합성시(image blending) 겹쳐지는 부분이 발생하게 된다.However, it is very difficult to find a precise specification lens without overlapping angles between cameras. Therefore, the angle of view of each camera has to overlap a certain amount. Then, in order to generate the 360-degree forward view image, the projection line of the virtual camera viewed at one view may be re-projected to the images of the plurality of cameras at the same time. That is, a point within the 360-degree forward view image J shown at the viewpoint of the virtual camera

) May be a plurality of pixel values which may have a meaning with respect to each other, and overlapping portions occur in image blending.

FIGS. 16A to 16E illustrate a 360-degree omnidirectional view image generated using two front / rear cameras having an angle of view of 180 degrees or more according to an embodiment of the present invention.

16A is a front camera input image, FIG. 16B is a rear camera input image, and a grid of curved lines displayed in each image represents an area used for generating a 360-degree forward view image.

이고, 이 점은 카메라 좌표계 {c}와 {c'}에 위치한 각각의 전방, 후방의 왜곡을 포함하는 카메라들 영상의

와

에 재투영되어 도 16d에 도시된 것과 같이 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)에는 밝게 표시된 겹쳐지는 부분 상의

에 위치하게 된다. 도 16d에 도시된 것과 같은 겹쳐지는 경계부분의 폭은 카메라 및 렌즈 체결 시 발생하는 오차 등으로 인해 서로 다르게 나타날 수 있다.16C is a diagram illustrating an example in which the above-described 360-degree forward view generation apparatus is applied. Referring to FIG. 16C, when two images are viewed from a virtual camera at a certain time point located in the virtual camera coordinate system {n}, the projection line of one point on the 360 degrees forward view image J and the 2 The intersection point with the car surface is

, Which is the difference between the camera coordinates {c} and {c}

Wow

As shown in FIG. 16D, the 360-degree omnidirectional view image J shown at the viewpoint of the virtual camera is displayed on the overlapped portion

. The width of the overlapping boundary portion as shown in FIG. 16D may be different from each other due to an error occurring when the camera and the lens are fastened.

The matching unit 350 of the omnidirectional camera projection modeling unit 330 performs a gradient-based image matching operation to smooth overlap portions in the 360-degree omni-directional view image J shown at the viewpoint of the virtual camera, (Poisson) -based image matching methods can be used. As an embodiment, a method of using a function indicating the distance to a boundary, which can be processed in real time, as a weight for combining two output images can be applied. The following equation can be used as an example of the weight applying method.

과

는 각각 전방과 후방 카메라 영상으로부터 상술한 방법으로 구한 임의 시점의 가상 카메라에 의해서 보여지는 왜곡이 보정된 360도 전방향 뷰 영상의 한 점(

)에서의 픽셀값들을 나타내고,

는 영상

에서 한 점(

)의 경계와의 거리에 따른 가중치,

는 영상

에서 동일 점(

)의 경계와의 거리에 따른 가중치를 나타내며, 가중치 함수는 경계와의 거리가 일정 임계치보다 클 때 1의 값을 갖고, 임계치에서 경계까지는 점점 줄어드는 값을 가지며, 경계를 벗어나면 0의 값을 갖도록 구현될 수 있다. 이러한 함수의 선택과, 임계치 값은 다양한 방식으로 구현될 수 있다.here,

and

Is a view of a 360-degree omnidirectional view image corrected by the distortion-corrected virtual camera viewed from the forward and backward camera images obtained by the above-

), ≪ / RTI >

The image

One point in (

), The weight according to the distance from the boundary,

The image

The same point (

), And the weight function has a value of 1 when the distance from the boundary is greater than a certain threshold value, a value that gradually decreases from the threshold to the boundary, and a value of 0 when the boundary is out of the boundary. Can be implemented. The selection of these functions and the threshold value can be implemented in various ways.

FIGS. 17A to 17G are views illustrating an example of calculating a weight according to an embodiment of the present invention and an embodiment to which weights are applied.

가 각 영상 센서에서 촬상한 영상에 재투영될 때, 첫번째 영상 센서(C1)에 투영되는 투영선(l)과의 각도(ρ)를 가지고 가중치를 계산하는 방법을 보여준다. 여기서 θ_max는 렌즈에서 보여지는 최대 화각을 나타내고, α는 이웃한 영상 센서와의 겹치는 각도를 나타낸다.

에 상응하는 가상 카메라 영상에서의 한 점

의 첫번째 카메라에서의 가중치

는 θ의 함수로 표현이 가능하다. 즉, 다음 식으로 표현된 것과 같이 겹치는 부분이 없으면 1, 화각을 벗어나면 0, 겹치는 부분은 각도에 따라 가중치를 가지게 구성할 수 있다.Fig. 17A is a view showing a point on the scene coordinate system

( 1 ) projected on the first image sensor (C1) when the projection image is projected back onto the image captured by each image sensor. Where _max denotes the maximum angle of view of the lens, and alpha denotes the angle of overlap with the neighboring image sensor.

A point in the virtual camera image corresponding to < RTI ID = 0.0 >

Of the first camera

Can be expressed as a function of θ. That is, as expressed by the following equation, it can be configured to have 1 if there is no overlapping portion, 0 if it deviates from the view angle, and a weight depending on the angle.

The weights in the second image sensor can also be configured in the same way.

과

를 도시하고 있고, 도 17d와 도17e는 상기 식으로 계산된 각각의 가중치인

와

를 나타내는데, 흰색은 1을 나타내고, 검정색을 0을 나타내며, 그 중간은 0과 1 사이의 값을 나타낸다. 도 17f와 도 17g는 영상

과

의 동일한 점 각각에 가중치가 곱해져서 생성된 영상이고, 도 17h는 최종 정합된 영상을 도시하고 있다. 17B and 17C show local images near the boundary surface generated by the front / rear cameras, respectively

and

17D and 17E show the respective weights calculated in the above equation

Wow

Where white represents 1, black represents 0, and the middle represents a value between 0 and 1. FIGS. 17F and 17G show the video

and

, And FIG. 17H shows the final matched image. In FIG.

Through this matching process, it is possible to eliminate the existing sense of heterogeneity in the overlapping portion of the images captured by the two image sensors.

By using the above-described apparatus, the present invention can generate a 360-omnidirectional view image without distortion, which is seen at a certain point in time, and can generate a 360-degree omnidirectional view smoothly matched without disturbance even in a boundary region between camera images .

The 360 degree omnidirectional view image generated by the 360 degree omnidirectional view generating apparatus according to the present invention can be played directly on the display 400 and the 360 degree omnidirection view image generated simultaneously can be stored in the storage device Lt; / RTI > In addition, a plurality of original images photographed by a plurality of cameras can be synchronously stored in one file.

At this time, when the original image is stored in a general recording apparatus, since the data size is large and the storage space size of the storage device may be a problem, the image data compressed through the video codec can be stored in consideration of the storage capacity and storage space . Various voice information and / or GPS information may be stored in the stored file.

The user can display the 360-degree omnidirectional view image created based on the 360-degree omnidirectional view image or the stored original image that has been created and stored according to the setting, or transmit the omnidirectional view image to another device. And changing the position and posture of the virtual camera, it is possible to display an image at a certain time in real time.

18 is a diagram illustrating a 360-degree forward view image generation method according to an embodiment of the present invention.

Referring to FIG. 18, a 360-degree omnidirectional view image generating method according to an exemplary embodiment of the present invention includes generating a 360-degree omnidirectional view image by combining images captured by a plurality of image sensors, A plurality of image sensors are controlled (S100) such that a plurality of image sensors have the same imaging time, same exposure, and same color control in order to minimize the occurrence of motion or a sense of heterogeneity in overlapping portions of images captured by the image sensor, The images captured by the image sensor are acquired (S200). Then, the distortion included in each image is removed based on the plurality of acquired images, the distorted images are synthesized, and a 360-degree omnidirectional view image viewed from a certain point of time set by the user is generated (S300).

In more detail, a plurality of image sensors are needed for a 360-degree omni-directional view. In the case where each image sensor has a field of view of 180 degrees, two image sensors are required. In the case where the image sensor has an angle of view of 120 degrees, three image sensors are required. When the image sensor has an angle of view of 90 degrees, . When a plurality of image sensors do not have the same imaging time, same exposure, and same color control, when the moving objects appear at the boundary of the image captured by each image sensor, or in the backlight situation, A motion may be generated at a boundary portion between a plurality of image sensors and a sense of heterogeneity due to a color difference may occur. In order to minimize the occurrence of such a sense of heterogeneity, the present invention performs a white balance and a color control function for controlling a plurality of image sensors to be synchronized and capturing images at the same exposure at the same time, and to maintain the same color at any illumination. As the angle of view increases and as it goes to the end, it contains severe distortion. Therefore, it is possible to generate a 360 degree omnidirectional view image by removing the distortion and synthesizing the distorted images.

19 is a diagram illustrating a method of synchronizing a plurality of image sensors according to an embodiment of the present invention.

In order to synchronize a plurality of image sensors, the same clock signal and image synchronization signal are transmitted to all image sensors (S110). If a plurality of image sensors are controlled by corresponding image processors, one of the plurality of image processors may be set as a main image processor, the other image processor may be set as an auxiliary image processor, The same clock signal and image synchronization signal can be transmitted to all image sensors by the main image processing processor. In addition, the same clock signal and image synchronization signal may be transmitted to the auxiliary image processing processor. Then, the plurality of image sensors and the auxiliary image processing processor can recognize the imaging time point determined by the main image processing processor, and can capture images at the same time.

Then, the image processing processor detects the optical characteristics of the images input from the plurality of image sensors using the optical detection device (S120). If there are a plurality of image processing processors corresponding to a plurality of image sensors, each image processing processor may detect the optical characteristics of the input image and transmit the optical characteristics to the main image processing processor. The optical characteristics include an average value of colors to be detected and an average value of pixels corresponding to white for each color channel R, G, B or Y, Cb, Cr. Then, the image processing processor compares the optical characteristics detected from the plurality of images captured by the plurality of image sensors to determine the exposure value and the color correction information (S130), and controls the plurality of image sensors based on the determination (S140) do. If there are a plurality of image processing processors, the main image processing processor determines the exposure value and the color correction information and transfers them to the auxiliary image processing processor, so that all the image processing processors can control the corresponding image sensor with the same exposure value and color correction information .

20 is a view illustrating a method of generating a 360-degree omnidirectional view image at a predetermined time point by removing a distortion included in a plurality of images and matching a plurality of images according to an embodiment of the present invention.

)로 변환하고, 크기 변환을 통해 원근감 보상을 수행하며, 특수 효과를 얻기 위해 투영 방식에 따라 장면 좌표계 상의 광축 중심(

)을 설정한다. Referring to FIG. 20, a virtual camera model is created (S310) as if a virtual camera existed at an arbitrary time set by the user to create a 360-degree forward view image that is displayed at a user-set arbitrary viewpoint. At this time, the virtual camera model includes coordinates of a point of a 360-degree forward view image ( J ) displayed at a viewpoint of a virtual camera at an arbitrary point set to generate a 360-degree forward view image, Coordinates on the scene coordinate system (

), Performs perspective compensation through size conversion, and calculates the center of the optical axis on the scene coordinate system according to the projection method

).

)를 영상 센서에서 촬상한 복수의 영상의 대응하는 픽셀 좌표(

)로 매핑 시에 사용되고, 가상 카메라의 시점에서 보여지는 영상(J)에 특수한 영상 효과를 삽입할 수 있는 2차 곡면을 모델링한 2차 곡면 모델(

)을 생성(S320)할 수 있다. 생성되는 2차 곡면 모델(

)은 장면 좌표계상의 4x4 행렬로 표현될 수 있다.And the coordinates on the scene coordinate system generated by the virtual camera model (

) Corresponding pixel coordinates of a plurality of images captured by the image sensor (

), And a secondary curved surface model (hereinafter referred to as " secondary curved surface model ") modeled as a secondary curved surface that can insert a special visual effect into an image ( J )

(S320). The generated secondary curved surface model (

) Can be represented by a 4x4 matrix on the scene coordinate system.

), 장면 좌표계상의 광축 중심(

), 및 2차 곡면 모델(

)을 바탕으로, 장면 좌표계상의 좌표(

)를 2차 곡면의 한 점으로 투여하고, 그리고 2차 곡면에 투영된 한 점을 다시 복수의 영상 센서에서 촬상한 복수의 영상으로 투영하여 장면 좌표계상의 좌표(

)를 대응하는 복수의 영상 각각에 존재할 수 있는 픽셀 좌표(

)로 매핑하고, 이 픽셀 좌표(

) 주변 픽셀들의 픽셀값을 보간하여 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점의 픽셀값으로 설정할 수 있다. 이때 영상 센서에서 촬상한 영상의 대응하는 픽셀 좌표(

)는 영상에 포함되어 있는 왜곡을 제거할 수 있는 좌표이다.Next, an omnidirectional camera projection model may be generated for each image captured by each image sensor (S330). The omnidirectional camera projection model is a coordinate system on the scene coordinate system (

), The center of the optical axis on the scene coordinate system (

), And a quadratic surface model (

), Coordinates on the scene coordinate system (

Is projected onto a plurality of images picked up by a plurality of image sensors, and the coordinates on the scene coordinate system (

(Pixel coordinates) that may exist in each of a plurality of corresponding images

), And the pixel coordinates (

And interpolates the pixel values of surrounding pixels to set a pixel value of one point of the 360-degree forward view image J to be viewed at the viewpoint of the virtual camera. At this time, the corresponding pixel coordinates of the image captured by the image sensor

) Is a coordinate that can remove the distortion contained in the image.

The pixel values obtained by generating the omnidirectional camera projection model for each image captured by each of the image sensors described above are limited to the portion of the angle of view of each image sensor of the 360-degree forward view image J viewed from the viewpoint of the virtual camera do. In one embodiment, when two image sensors having an angle of view of 180 degrees for photographing the front and the back are used, if an omnidirectional camera projection model is generated for the image taken in the foreground, It is possible to acquire pixel values for each point (pixel) of the front image of the view image J and generate an omnidirectional camera projection model for the backscattered image, Pixel values for each point (pixel) of the rear image of the image J can be obtained. In this way, it is possible to generate a plurality of images from which distortions have been removed by generating an omnidirectional camera projection model for each image captured by each image sensor, and to combine a plurality of images from which distortions have been eliminated, The direction view image J can be generated (S340).

Each step will be described in more detail.

FIG. 21 is a diagram specifically illustrating a virtual camera model generation step (S310) according to an embodiment of the present invention.

Referring to FIG. 21, the step of generating a virtual camera model according to an exemplary embodiment of the present invention includes generating a virtual camera image at a desired time point, and generating a digital pan, tilt, The virtual camera position and posture are set (S311) so that the function can be emulated. At this time, the azimuth and the vertical rotation angle can be set around the z axis of the scene coordinate system for the pan and tilt function, and the distance from the center of the scene coordinate system for the zoom function can be set In order to set the attitude of the virtual camera, the directions of the x, y, and z axes of the scene coordinate system and the directions of x _n , y _n , and z _n of the virtual coordinate system can be set accordingly. The user can set the position and the posture of the virtual camera using the above-described parameters in order to view an image at a desired time.

)을 파라미터로 설정할 수 있고, 파노라마 카메라는 극좌표계에서 사용할 반지름의 최소값(

)과 최대값(

), 극좌표계에서 사용할 각도의 최소값(

) 과 최대값(

), 그리고 가상카메라상에서의 중심 (

)을 파라미터로 설정할 수 있다.Then, the type of virtual camera and the parameters for each type can be set (S312) so as to have a special lens effect. The type of virtual camera is a perspective projection camera with a general perspective effect, an orthographic projection camera that mimics a lens with a perspective effect removed, or an image acquired using a refraction refracting mirror or lens A panorama camera capable of generating an omni-directional panoramic image using a polar coordinate system image, and the like. Then, the perspective projection camera or the right-angle projection camera displays the center point of the 360-degree forward view image J shown at the viewpoint of the set virtual camera

) Can be set as a parameter, and the panoramic camera can set the minimum value of the radius to be used in the polar coordinate system

) And the maximum value (

), The minimum value of the angle to be used in the polar coordinate system (

) And the maximum value (

), And center on virtual camera (

) Can be set as a parameter.

) 및 광축 중심(

)을 계산(S313)할 수 있다. When the setting of the type, position, and posture of the virtual camera is completed as described above, the coordinate (x, y ) of the scene coordinate system with respect to one point of the 360-

) And the center of the optical axis (

Can be calculated (S313).

FIG. 22 is a diagram specifically showing a generation step (S320) of a quadric curved surface model according to an embodiment of the present invention.

)로 표현될 수 있다.Referring to FIG. 22, a type of a quadric surface model may be selected (S321) for generating a quadric surface model. Secondary curved surface models include an ellipsoid model that can be used to obtain the pan, tilt, and zoom effects of a virtual camera, a cylinder model to generate a wide angle image, a parabolic model , And a user secondary surface model that the user can set arbitrarily. When selecting a secondary curved surface model, parameters suitable for each model can be set together. The second curved surface model thus selected is a 4x4 matrix (

). ≪ / RTI >

Next, the attitude and position of the secondary curved surface model can be set (S322). The attitude and position of the quadratic curve model can be set using the horizontal rotation angle? _Q and the vertical rotation angle? _Q , which can be easily set by the user, in the same manner as the attitude and position setting of the virtual camera. Alternatively, the user can set an arbitrary parameter to set the attitude and position of the quadratic curve model.

)은 설정된 자세 및 위치를 바탕으로 장면 좌표계상의 2차 곡면 모델(

)로 변환될 수 있다. The quadratic surface model expressed by the above-described 4x4 matrix (

) Is a quadratic surface model on the scene coordinate system (

). ≪ / RTI >

FIG. 23 is a diagram specifically illustrating a generation step (S330) of an omnidirectional camera projection model according to an embodiment of the present invention.

)을 계산(S331)한다. 이때의 투영선은 가상 카메라의 광축 중심(

)과 가상 카메라의 시점에서 보여지는 360도 전방향 뷰 영상(J)의 한 점(

)을 잇는 직선을 의미한다. Referring to FIG. 23, in order to generate an omnidirectional camera projection model, an intersection point of a projection line and a quadratic surface model with respect to a point of a 360-degree forward view image J ,

(Step S331). At this time, the projection line is the center of the optical axis of the virtual camera

) And a point of a 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera

Quot;).

)은 각 영상별로 해당하는 전방향 카메라 투영 모델을 이용하여 영상 센서에서 촬상한 영상의 대응하는 픽셀 좌표로 매핑(S332)된다. Computed intersection (

Is mapped to the corresponding pixel coordinates of the image captured by the image sensor using the corresponding omnidirectional camera projection model for each image (S332).

)을 영상 센서에서 촬상한 영상의 대응하는 픽셀 좌표로 매핑하는 단계를 상세히 도시한 도면이다.Fig. 24 is a cross-

) To the corresponding pixel coordinates of the image captured by the image sensor.

)로 변환(S341)하고, 왜곡 보정 매핑 함수를 이용하여 왜곡 없는 영상 평면상의 한 점으로 매핑(S342)하고, 어파인 변환 행렬을 이용하여 왜곡 없는 영상 평면상의 한 점에 대응하는 각 영상 센서에 의해 촬상된 원본 영상의 픽셀 좌표로 매핑(S343)할 수 있다. 이때 매핑된 픽셀 좌표는 상기 모델에 의하여 왜곡이 제거된 점일 수 있다.Referring to FIG. 24, a distortion correction mapping function obtained by a theoretically analyzed model and a numerically approximated model may be used to compensate for distortion of a lens of an image sensor. Preferably, the Taylor expansion series is used. Based approximate model can be used. The approximate model at this time may be different for each image captured by each image sensor because the distortion included in each image is different. For the mapping, the coordinates on the scene coordinate system of the intersection point are the coordinates (

(S341). Then, the distortion is mapped to a point on the image plane without distortion using the distortion correction mapping function (S342). Then, using the affine transformation matrix, (S343) to the pixel coordinates of the original image captured by the camera. At this time, the mapped pixel coordinates may be a point where distortion is removed by the model.

Then, the pixel value of the pixel coordinates of the mapped original image is read and interpolated to set the pixel value of one point of the 360-degree forward view image J displayed at the viewpoint of the virtual camera (S333). At this time, since each image sensor captures only one specific direction in 360 degrees forward direction, only one pixel value coming from one image sensor for one point of 360 degree forward view image J shown at the viewpoint of the virtual camera is mostly Lt; / RTI > However, since it is very difficult to find a lens having an exact specification that does not overlap the angle of view of the image sensor, the angle of view of the camera included in each image sensor is inevitably overlapped. Then, in order to generate a 360-degree omnidirectional view image, the projection line of the virtual camera viewed at one view may be re-projected to the original image of the plurality of image sensors at the same time. That is, there may be a plurality of pixel values for one point of the 360-degree forward view image viewed from the viewpoint of the virtual camera. At this time, in order to determine a pixel value for a specific point of the boundary portion of the 360 degree omnidirectional view image, according to the present invention, 360 degrees viewed from the viewpoint of the virtual camera generated based on the model generated in the step S310 to S330 A plurality of pixel values corresponding to one point of the forward view image are matched (S340), and the overlapped portions of the images captured by the image sensor are corrected so as to be smoothed.

25 is a diagram specifically illustrating a size compensation step for perspective image correction according to an embodiment of the present invention.

Referring to FIG. 25, if the perspective correction is not properly performed, the difference between the actual size and the distance may be generated toward the end of the image, and the user may experience inconvenience due to a difference from the actual size. In order to solve this problem, it is possible to adjust the size ratio of the horizontal axis and the vertical axis of the 360-degree forward view image J displayed at the viewpoint of the virtual camera (S410). In the step of converting the coordinates of one point of the 360-degree forward view image J shown in the virtual camera coordinate system represented by the virtual camera coordinate system into coordinates of the scene coordinate system in the virtual camera model creation step S310, And adjusting the ratio of the size of the horizontal axis and the vertical axis of the generated image.

For the perspective correction, the depth and size of the image can also be compensated through the type and parameter settings of the secondary surface model. That is, in the step of selecting the type of the quadratic surface model and setting the parameter of the selected type (S320), it is possible to solve the perspective problem by setting the parameters to compensate for the depth and the size.

The 360 degree omnidirectional view image generating apparatus and method can reproduce a 360 degree omnidirectional view image as seen from a user's desired viewpoint. Particularly, it is possible to eliminate distortions existing in an image and smoothly resolve overlapping portions between images captured by a plurality of image sensors, thereby minimizing a sense of heterogeneity in reproduced images.

However, the 360-degree omni-directional view image generated by the 360-degree omnidirectional view generating apparatus according to the present invention can be directly played on the display 400, and the 360- Lt; / RTI > In addition, a plurality of original images photographed by a plurality of cameras can be synchronously stored in one file. Here, the synchronization may mean setting a plurality of images captured at the same time to have mutual relevance. Also, a 360-degree omnidirectional view image at a completely new viewpoint may be newly generated based on the original image stored in synchronization with this. That is, the image displayed on the display can display the image currently captured by the image sensor in real time, but it can also generate the 360-degree omni-direction view image at various viewpoints based on the original image stored in the storage device and display it on the display.

FIG. 26 illustrates a method of generating a 360-degree forward view image according to another embodiment of the present invention.

Referring to FIG. 26, the user can select an original image (S510). Here, the original image may be a real-time image obtained from a plurality of image sensors in the 360-degree omnidirectional view image generating apparatus, or may be original images previously captured and stored in synchronization with a storage device. Based on the selected original image, the 360-degree forward view image can be generated (S520) by the above-described method. At this time, the user can change the viewpoint or the position and posture of the virtual camera in real time (S530), and the 360 degree omnidirectional view generating device can display a new 360 degree omnidirectional view image corresponding to the changed point or the position and posture of the virtual camera (S540). The generated image can be displayed on the display or transmitted to the device.

As described above, the 360-degree omnidirectional image generating apparatus of the present invention can generate a 360-degree omni-directional image at various viewpoints based on the same original image, The application field can be quite wide ranging from displaying a goal scene.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims and their equivalents. Only. It is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. .

100:
200: image processing processor (ISP)
210: Main Image Processing Processor (Master ISP)
220, 230, 240: A secondary image processing processor (Slave ISP)
300: Multiple Digital Image Processor (MDIP)
310: Virtual camera model part
320: Second curved surface model part
330: omnidirectional camera projection model unit
340: Parameter setting section
350:
400: Display

Claims (37)

A 360 degree omnidirectional view image generation apparatus,
A sensor unit including a plurality of image sensors capable of dividing and imaging a 360-degree omni-directional area;
An image processing processor for controlling the plurality of image sensors so that the plurality of image sensors have the same imaging time, same exposure, and same color control, and receiving and processing a plurality of images picked up by the plurality of image sensors; And
And a multi-digital image processor for matching the plurality of images received from the image processing processor and generating a 360-degree omni-
Wherein the image processing processor comprises:
And a plurality of image processing processors corresponding to each of the plurality of image sensors,
One of the plurality of image processing processors is a main image processing processor, and the other is an auxiliary image processing processor,
Wherein the main image processing processor provides clock and image synchronization signals to the plurality of image sensors and the auxiliary image processing processor so that the plurality of image sensors have the same imaging time,
Wherein the auxiliary image processing processor detects a first optical characteristic of an image received from a corresponding image sensor and transmits the first optical characteristic to the main image processing processor,
Wherein the main image processing processor detects a second optical characteristic of an image received from a corresponding image sensor and compares the first optical characteristic and the second optical characteristic received from the auxiliary image processing processor to obtain an exposure value and color control information And then transmits the control signal to the auxiliary image processing processor,
Wherein the auxiliary image processing processor and the main image processing processor adjust a shutter speed and an automatic gain control value of a corresponding image sensor,
360 degree omnidirectional view image generation device. delete delete The system of claim 1, wherein the multiple digital image processor comprises:
A coordinate ( u ) of a point of a 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera on the basis of the set virtual camera position and posture,

), Performs perspective compensation through magnitude conversion, and calculates the center of the optical axis on the scene coordinate system according to the projection method to obtain a visual effect.

A virtual camera model unit for setting the virtual camera model;
Coordinates on the scene coordinate system generated by the virtual camera model unit (

) Corresponding pixel coordinates of each image of the plurality of images

), And a secondary curved surface model (hereinafter referred to as " secondary curved surface model ") modeled as a secondary curved surface that can insert a video effect into a 360-degree forward view image J

A second curved surface model unit for generating a second curved surface model portion;
Coordinates on the scene coordinate system acquired by the virtual camera model unit (

), The center of the optical axis on the scene coordinate system (

), And a secondary curved surface model generated by the secondary curved surface model unit (

Coordinates on the scene coordinate system (

) Onto a point on the secondary curved surface, projecting a point projected on the secondary curved surface onto each of the images of the plurality of images,

) Corresponding pixel coordinates of each image of the plurality of images

), And the corresponding pixel coordinates (

) Interpolating pixel values of neighboring pixels for each of the plurality of images with respect to a coordinate u of a point of a 360-degree forward view image J viewed from the viewpoint of the virtual camera, An omnidirectional camera projection model unit for obtaining a value; And
( U ) of a point of a 360-degree omnidirectional view image ( J ) displayed at a viewpoint of the virtual camera by matching pixel values from respective images of the plurality of images acquired by the omnidirectional camera projection modeling unit And a matching unit for determining a pixel value for the pixel,
360 degree omnidirectional view image generation device. 5. The method of claim 4,
Further comprising: a parameter setting unit configured to receive and set parameters to be used in each unit of the multiple digital image processor from a user,
360 degree omnidirectional view image generation device. 5. The method of claim 4,
The virtual camera model unit includes:
The coordinate u of one point of the 360 degree forward view image J viewed from the viewpoint of the virtual camera is calculated as the coordinate of the rectangular coordinate form of (x, y) or the polar coordinate of (r, A coordinate display module for displaying coordinates in a format;
A transformation matrix (x, y , y , z ) for transforming coordinates ( u ) of a point of a 360-degree forward view image ( J ) into coordinates on a scene coordinate system

A location setting module for generating a location setting module;
Of 360 ° omnidirectional view image (J) to adjust the size ratio of the horizontal axis and a vertical axis and 360 ° omnidirectional view image (J) shown in the timing of the virtual camera in accordance with the adjustment shown in the time point of the virtual camera, A size adjustment module for adjusting coordinates ( u ) of a point;
The transformation matrix generated by the position setting module (

( U ) of a point of a 360-degree omnidirectional view image ( J ) displayed at the viewpoint of the virtual camera created and adjusted by the coordinate display module and the size adjustment module is calculated based on the coordinate

); And
The center of the optical axis of the 360-degree forward view image J shown at the viewpoint of the virtual camera

An optical axis center setting module for calculating an optical axis center; / RTI >
360 degree omnidirectional view image generation device. The method according to claim 6,
Wherein the coordinate display module comprises:
The center of the 360-degree forward view image J shown at the viewpoint of the virtual camera

( U ) of a point of the 360-degree forward view image J shown at the viewpoint of the virtual camera using

To the coordinates in the rectangular coordinate form (

),
360 degree omnidirectional view image generation device. The method according to claim 6,
Wherein the coordinate display module comprises:
( U ) of a point of the 360-degree forward view image J shown at the viewpoint of the virtual camera

To the coordinates in polar coordinates (

),

ego,

Is the minimum value of the radius used in the polar coordinate format,

Is the maximum value of the radius used in polar form,

Is the number of radial pixels used in polar form,

Is the minimum value of the angle used in the polar coordinate format,

Is the maximum value of the angle used in the polar coordinate format,

Is the number of pixels in the angular direction used in the polar coordinate format,

sign,
360 degree omnidirectional view image generation device. The method according to claim 6,
Wherein the positioning module comprises:

The transformation matrix (

),

Is the center of the quadratic surface model on the scene coordinate system and is the center of the pan and tilt rotation of the virtual camera,

Is the position of the virtual camera on the scene coordinate system, phi _n And θ _n are the azimuth and elevation of the scene coordinate system of the virtual camera with respect to the z axis, trans () is a function representing a moving matrix on the three-dimensional space, and rot _z () And rot _x () are functions representing the rotation about the z-axis and the x-axis on the three-dimensional space,
360 degree omnidirectional view image generation device. The method according to claim 6,
The optical axis center setting module includes:
Depending on whether the type of virtual camera is a perspective projection camera or a perpendicular projection camera

The center of the optical axis

),

Is the focal length,

Is a coordinate of a scene coordinate system corresponding to a coordinate ( u ) of a point of a 360-degree forward view image ( J ) viewed from the viewpoint of the virtual camera generated by the conversion module,

And

Lt; RTI ID = 0.0 > (

) As a unit vector form

Which are unit vectors when expressed as < RTI ID = 0.0 >
360 degree omnidirectional view image generation device. 5. The method of claim 4,
Wherein the quadric surface model unit comprises:
A matrix for selecting one of the plurality of secondary curved surface models and expressing the selected secondary curved surface model using the parameters for the selected secondary curved surface model (

A secondary curved surface model selection module for generating a curved surface model;
A transformation matrix for transforming the points on the quadratic surface into coordinates on the scene coordinate system (

A coordinate setting module for generating a coordinate value; And
The transformation matrix (

) To express the selected quadratic surface model

) Representing the selected curved surface model on the scene coordinate system (

), ≪ / RTI >
360 degree omnidirectional view image generation device. 12. The method of claim 11,
Wherein the secondary curved surface model selection module comprises:
An Ellipsoid model, a Cylinder model, a Paraboloid model, and a user-set quadric surface model,
360 degree omnidirectional view image generation device. 12. The method of claim 11,
Wherein the coordinate setting module of the quadric surface model unit comprises:

The transformation matrix (

),
φ _n And θ _n are a horizontal azimuth and a vertical rotation angle of the scene coordinate system of the quadratic curve model with respect to the z axis and rot _z () and rot _x () And the rotation about the x-axis,
360 degree omnidirectional view image generation device. 12. The method of claim 11,
Wherein the conversion module of the quadric surface model unit comprises:
A matrix expressing the selected curved surface model (

) And a transformation matrix (

) Representing the selected curved surface model on the scene coordinate system

)of
The formula used to calculate,
360 degree omnidirectional view image generation device. 5. The method of claim 4,
The omnidirectional camera projection modeling unit,
Coordinates on the scene coordinate system acquired by the virtual camera model unit (

), And the center of the optical axis on the scene coordinate system (

) And the second curved surface model (

) ≪ / RTI >

); ≪ / RTI > And
The intersection (

) Corresponding pixel coordinates of each image of the plurality of images

), And the corresponding pixel coordinates (

) To obtain a pixel value from each image of the plurality of images with respect to a coordinate u of a point of the 360-degree forward view image J viewed at the viewpoint of the virtual camera An omni-directional camera projection module,
360 degree omnidirectional view image generation device. 16. The method of claim 15,
The omnidirectional camera projection module comprises:
The intersection point represented by the scene coordinate system (

) To the corresponding coordinates of the camera coordinate system (

);
Coordinates of the camera coordinate system of the intersection point generated by the conversion module

The actual pixel coordinates of each of the images of the plurality of images that can correct the distortion of the image

); And
And an actual pixel coordinate of each image of the plurality of images acquired by the coordinate calculation module

And an interpolation module for interpolating and outputting pixel values of neighboring pixels on the basis of pixel values of the neighboring pixels,
360 degree omnidirectional view image generation device. 17. The method of claim 16,
Wherein the coordinate calculation module comprises:
A distortion correction mapping function based on an approximate model based on a Taylor expansion series which is numerically approximated to a distortion existing in each image of the plurality of images

) At the intersection (

) Coordinates of the camera coordinate system (

) To a point on the distortion-free image plane (

), And an affine transformation matrix (

), A point on the distortion-free image plane (

) Of the respective images of the plurality of images corresponding to the pixel coordinates

),
360 degree omnidirectional view image generation device. The connector according to claim 4,
From the coordinate ( u ) of a point of a 360-degree forward view image ( J ) displayed at a viewpoint of the virtual camera to a pixel value from each image of the plurality of images acquired by the forward direction camera projection modeling unit ( U ) of a 360-degree forward view image ( J ) displayed at a viewpoint of the virtual camera, and determining a pixel value for a coordinate ( u ) of a point of the 360-
360 degree omnidirectional view image generation device. 19. The apparatus of claim 18, wherein the mating portion

( U ) of a point of a 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera using the equation

),
k is the number of the plurality of images,

Is a pixel value obtained from the i-th image in the forward camera projection model unit,

Is a weight according to the distance from the coordinate ( u ) to the boundary of the i-th image in the i-th image,
remind

The

Use the formula to determine
θ _max is the maximum angle of view of the image sensor, α is the overlap angle between adjacent image sensor, θ is the coordinates (u) of a point on the 360-degree omni-directional view image (J) shown in the timing of the virtual camera, Which is an angle obtained by displaying in polar coordinate format,
360 degree omnidirectional view image generation device. The method according to claim 1,
Further comprising a storage device for storing a plurality of images captured by the plurality of image sensors,
A plurality of images captured by the plurality of image sensors are synchronized and stored as one file,
360 degree omnidirectional view image generation device. 21. The method of claim 20,
Wherein the image processing processor reads and processes a plurality of images stored in a file synchronized with the storage device instead of a plurality of images captured by the plurality of image sensors,
360 degree omnidirectional view image generation device. A 360-degree forward view image generation method,
Controlling the plurality of image sensors so that the plurality of image sensors have the same imaging time, same exposure, and same color control;
Acquiring a plurality of images captured by the plurality of image sensors; And
And generating a 360-degree omnidirectional view image that is displayed at an arbitrary point in time by synthesizing the distorted images, removing distortion included in each image based on the plurality of images,
Controlling the plurality of image sensors so that the plurality of image sensors have the same image sensing time, same exposure, and same color control,
Transmitting an identical clock signal and an image synchronization signal to the plurality of image sensors;
The method comprising: acquiring a plurality of images captured by the plurality of image sensors and detecting optical characteristics of the images;
Comparing the optical characteristics of each of the detected images to determine an exposure value and color correction information; And
And controlling the plurality of image sensors based on the determined exposure value and color correction information.
360 degree front view image generation method. delete 23. The method of claim 22,
Removing the distortion included in each image based on the plurality of images, and synthesizing the distorted images to generate a 360-degree omnidirectional view image that is displayed at a set point,
The coordinates of a point ( u ) of the 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera are plotted on the coordinates (

), Performs perspective compensation through magnification conversion, and uses an optical center (XYZ) on the scene coordinate system according to the projection method to obtain a special effect.

A virtual camera model creation step of setting a virtual camera model;
The coordinates on the scene coordinate system generated in the virtual camera model creation step (

) Corresponding pixel coordinates of each image of the plurality of images

And a secondary curved surface model ( J ) modeled as a secondary curved surface that can be used to insert a video effect into the 360-degree forward view image J , which is used at the time of the virtual camera

A second curved surface model generation step of generating a second curved surface model;
The coordinates on the scene coordinate system generated in the virtual camera model creation step (

), The center of the optical axis on the scene coordinate system (

), And a secondary curved surface model obtained in the secondary curved surface model generation step (

Coordinates on the scene coordinate system (

) Onto a point on the secondary curved surface, projecting a point projected on the secondary curved surface onto each of the images of the plurality of images,

) To the pixel coordinate of the corresponding forward camera image (

), And the corresponding pixel coordinates (

) Interpolating pixel values of neighboring pixels for each of the plurality of images with respect to a coordinate u of a point of a 360-degree forward view image J viewed from the viewpoint of the virtual camera, An omnidirectional camera projection model generation step of acquiring a value of the forward camera projection model; And
( U ) of a point of a 360-degree forward view image ( J ) displayed at the viewpoint of the virtual camera by matching pixel values from each image of the plurality of images acquired in the forward direction camera projection model generation step, And generating a 360-degree omnidirectional view image J viewed from the viewpoint of the virtual camera by determining a pixel value for the virtual camera.
360 degree front view image generation method. 25. The method of claim 24,
Compensating a size of a 360-degree forward view image ( J ) displayed at a viewpoint of the virtual camera for perspective correction;
360 degree front view image generation method. 25. The method of claim 24,
In the virtual camera model generation step,
Setting a position and a posture of a virtual camera so as to generate an image at a desired time and to emulate a digital pan, tilt, and zoom functions of a mechanical camera;
Setting a type of virtual camera and parameters for each type; And
The coordinates on the scene coordinate system for one point of the 360-degree forward view image J shown at the viewpoint of the virtual camera based on the type, position,

) And the center of the optical axis (

), ≪ / RTI >
360 degree front view image generation method. 27. The method of claim 26,
Wherein the setting of the position and posture of the virtual camera comprises:
Setting the directions of the x, y, and z axes of the scene coordinate system and the directions of x _n , y _n , and z _n of the virtual coordinate system;
Setting a horizontal rotation azimuth and a vertical rotation angle (elevation) about a z-axis of a scene coordinate system; And
And setting a distance from the center of the scene coordinate system.
360 degree front view image generation method. 27. The method of claim 26,
Wherein the setting of the type of the virtual camera and the parameters of each type includes:
One is a perspective projection camera with a general perspective effect, an orthographic projection camera that mimics a lens with a perspective effect removed, and a panorama camera capable of generating an omni-directional panoramic image As a type of virtual camera; And
And setting a parameter according to the type of the selected virtual camera.
360 degree front view image generation method. 25. The method of claim 24,
Wherein the secondary curved surface model generation step comprises:
Setting a type of the quadric surface model;
Setting a posture and a position of the set secondary curved surface model;
A second curved surface model on the scene coordinate system of the set second curved surface model based on the set attitude and position

), ≪ / RTI >
360 degree front view image generation method. 30. The method of claim 29,
Wherein the step of setting the type of the quadric surface model comprises:
An Ellipsoid model that can be used to obtain a pan, tilt, and zoom effect of a virtual camera, a cylinder model for generating a wide-angle image, a parabolic model for obtaining an effect of mapping only a certain portion of an image, And selecting one of the user's quadratic surface models that can be set.
360 degree front view image generation method. 30. The method of claim 29,
Wherein the step of setting the attitude and position of the set secondary curved surface model comprises:
And setting a horizontal azimuth and an elevation around the z-axis of the scene coordinate system.
360 degree front view image generation method. 25. The method of claim 24,
Wherein the forward camera projection model generation step comprises:
Coordinates (?) On the scene coordinate system acquired in the virtual camera model creation step

), And the center of the optical axis on the scene coordinate system (

) And the second curved surface model (

) ≪ / RTI >

≪ / RTI >
The intersection (

) Corresponding pixel coordinates of each image of the plurality of images

); And
The corresponding pixel coordinates (

) To obtain a pixel value from each image of the plurality of images with respect to a coordinate u of a point of the 360-degree forward view image J viewed at the viewpoint of the virtual camera ; ≪ / RTI >
360 degree front view image generation method. 33. The method of claim 32,
The intersection (

) Corresponding pixel coordinates of each image of the plurality of images

) Comprises the steps of:
The intersection point represented by the scene coordinate system (

) To the corresponding coordinates of the camera coordinate system (

);
A distortion correction mapping function based on an approximate model based on a Taylor expansion series which is numerically approximated to a distortion existing in each image of the plurality of images

) At the intersection (

) Coordinates of the camera coordinate system (

) To a point on the distortion-free image plane (

); And
Affine transformation matrix (

), A point on the distortion-free image plane (

) Of the respective images of the plurality of images corresponding to the pixel coordinates

), ≪ / RTI >
360 degree front view image generation method. 26. The method of claim 25,
The step of compensating the size of the 360-degree forward view image ( J ) shown at the viewpoint of the virtual camera for the perspective-
Adjusting a size ratio of a horizontal axis and a vertical axis of the 360-degree forward view image J viewed from the viewpoint of the virtual camera; And
And compensating depth and size through the type and parameter setting of the secondary curved surface model.
360 degree front view image generation method. 23. The method of claim 22,
Further comprising: synchronizing a plurality of images captured by the plurality of image sensors and storing the images as one file;
360 degree front view image generation method. delete A computer program stored on a computer readable medium for executing a method according to any one of claims 22, 24, and 35 when executed on a computer or a processor.

Images