KR100809049B1 - Method and apparatus for controlling zoom-convergence interlock in parallel axis typed stereo camera system - Google Patents

Abstract

The present invention relates to a zoom and viewing angle interlocking control method and apparatus for providing a stereoscopic image while maintaining a constant gaze point even when the zoom and focus is changed in a parallel axis stereo stereo camera. In the present invention, the relationship between zoom, focus and focal length is databased in advance, and when zoom and focus manipulation occurs by attaching a sensor to a lens or using a lens with a built-in lens, It calculates and maintains the distance to the viewpoint, and extracts only the necessary area from the left and right images in order to maintain the distance, and then expands and resizes it to the original image size.

Parallel Axis, Stereo Camera, Zoom-Visual Interlocking Control, Image Processing, Scaling

Description

TECHNICAL AND APPARATUS FOR CONTROLLING ZOOM-CONVERGENCE INTERLOCK IN PARALLEL AXIS TYPED STEREO CAMERA SYSTEM}

1 is a view for explaining three basic structures of a stereo camera,

2 is a part of a lens data table showing a focal length according to zoom and focus values of a zoom lens applied to the present invention;

3 is a structural diagram for explaining a principle of adjusting a gaze point by image processing and calculating a distance to the adjusted gaze point in a parallel axis stereo camera system;

Figure 4 is a block diagram for explaining the configuration of the zoom-periodic interlocking control device in a parallel axis stereo camera system applied to the present invention,

5 is a view showing the principle of the zoom-time visual interlocking control method and apparatus applied to the present invention,

FIG. 6 is a flowchart illustrating a method and apparatus for controlling zoom-periodic interlocking motion of a parallel axis stereo camera system according to the present invention.

The present invention relates to a zoom and vergence interlock control method and apparatus of a parallel axis stereo camera system.

In particular, the present invention relates to a zoom and viewing angle interlocking control method and apparatus for providing a stereoscopic image while maintaining a constant gaze point even when the zoom and focus are changed in a parallel axis stereo stereo camera.

In general, stereo cameras, unlike monocular cameras, require precise control of the camera's viewing angle in addition to zoom and focus to produce high quality stereoscopic images that minimize eye strain.

The difference between the left and right images caused by the left and right eyes of a person being about 6.5 cm apart is called binocular disparity (hereinafter, referred to as 'parallax'). Depends on the distance, and is known as the most important factor to make a three-dimensional feeling.

A point at which the parallax becomes '0' is called a 'main point of view', and controlling a viewpoint in the stereoscopic camera is called 'main point of view control' or 'main point of time control'. By controlling the gaze point, it is possible to prevent excessive parallax from occurring in the screen, which is one of the very important factors for producing high quality stereoscopic images.

1 is a view for explaining three basic structures of a stereo camera.

As shown in FIG. 1, a stereo camera can be largely divided into a parallel axis method, a cross axis method, and a parallel move axis method.

The parallel axis method has the advantage of being easy to manufacture a camera, but has the disadvantage of being unable to control the angle, while the cross-axis method is relatively easy to manufacture the camera and has the advantage of controlling the viewing angle, whereas three-dimensional The distortion of the image is excessive and it is difficult to manufacture a small camera. Lastly, the parallel movement axis has the advantage of being able to control high angle and control the stereoscopic image with less distortion of the stereoscopic image, while the camera sensor and the lens are separated and precisely control it. The disadvantage is that it is not easy to maintain.

Such a parallel axis or a parallel moving axis method is able to take a high-quality three-dimensional image, which is attracting more attention than the crossover method, but there is a disadvantage that the gaze point changes when zooming.

It is very common to use zoom when shooting high quality stereoscopic images, but it is almost impossible to manually control the changed gaze point each time you zoom. One of the very important features.

Since the preset gaze does not change despite the camera's zoom operation, the camera with interlocking control can maintain a proper parallax, so that high-quality stereoscopic images can be taken. Operation is efficient because you do not have to care about binocular parallax.

Zoom-Principal Interlocking Control method used in the parallel camera's stereo camera calculates the amount of parallax from a pre-made table whenever the zoom and focus changes, and based on this, the distance between the lens and the camera sensor is controlled by the motor. In other words, the angle of view set just before that is maintained.

The parallel axis method has the advantage of taking a high-definition, high-quality stereoscopic image by utilizing all the resolution of the camera sensor, but it is difficult to manufacture because the structure is complex and precise.

Therefore, there is a need for a method of applying zoom-peripheral coordinated control to a parallel camera having a simpler structure.

In addition, the parallel axis stereo camera does not have a function to adjust the viewing point unlike the parallel movement axis method, but it extracts a certain area of the image formed by the left and right camera sensors and linearly interpolates again to control the viewing point. You can get it.

The above method has a disadvantage in that the resolution of the image is deteriorated because some areas of the image sensor need to be devoted to the gaze point control. However, when the high resolution image sensor developed in recent years is used, the image quality exceeds a certain level and is sufficient for practical use. And, there is an advantage that it is easy to manufacture compared to the horizontal movable shaft method.

However, this method also requires the development of a technology that can automatically control the change of gaze point according to the zoom operation.

The present invention has been made in accordance with the necessity of the development of technology for automatically controlling the change of gaze point as described above, the object of the present invention is to database the relationship between zoom, focus and focal length in advance, and attach a sensor to the lens Or when zoom and focus manipulation is performed using a lens with a built-in sensor, the current zoom, focus value, and distance from parallax to the gaze point are calculated and maintained, and necessary areas in the left and right images are maintained to maintain the distance. The present invention provides a method and apparatus for controlling zoom and vergence of a parallel axis type stereo camera system which extracts only a magnification and then enlarges the image to an original image size.

,

); (5) 제어부는, 상기 초점거리(f0), 주시각 값(V0) 및 카메라 간의 간격(s0) 값을 연산하여 물체에서 렌즈까지의 거리(d0)(이하 '렌즈까지의 거리'라 함)를 산출하는 과정(

); (6) 제어부는, 상기 물체에서 렌즈까지의 거리(d0)와 초점 거리(f0)를 연산하여 물체에서 영상센서까지의 거리(D)(이하 '영상센서까지의 거리'라 함)를 산출하는 과정 (

); (7) 제어부는, 촬영자에 의해 줌 값 또는 포커스 값이 변경되었는지를 판단하는 과정; (8) 제어부는, 상기 줌 값 또는 포커스 값이 변경된 경우 변경된 줌 값 및 포커스 값을 이용하여 룩업 테이블로부터 변화된 초점 거리(f1)를 추출 및 보간하는 과정; (9) 제어부는, 영상센서까지의 거리(D) 및 초점 거리(f1)를 연산하여 렌즈까지의 거리(d1)를 산출하는 과정 (

); (10) 제어부는, 카메라 간격(s0), 상기 변화된 초점거리(f1)와 변경된 렌즈까지의 거리(d1)를 연산하여 주시각 값(V1)을 산출하는 과정 (

); (11) 제어부는, 변경된 주시각 값(V1)으로부터 영상처리량을 결정하는 좌, 우측 영상의 수평편이값(Vx1)을 산출하는 과정 (

); (12) 제어부는, 산출된 수평변이값(Vx1)을 사용하여 화면비율에 맞게 원영상을 발췌하고, 발췌한 영상을 다시 원래의 크기로 복원하는 과정; (13) 제어부는, 상기 초기 주시각 값(V0)에 변경된 주시각 값(V1)을 대입하고, 상기 초기 초점 거리(f0)에 변화된 초점 거리(f1)를 대입한 후 자동모드인 지 확인하는 초기과정부터 재 수행되도록 전체 시스템을 제어하는 과정으로 이루어진 것을 특징으로 한다.One embodiment of the present invention for achieving the above object, in the zoom-axis time-interlocking control method of a parallel-axis stereo camera, (1) the control unit, the current zoom value and focus when the automatic mode is selected by the photographer Extracting and interpolating a focal length f0 from the lookup table using the value; (2) the control unit calculating the excerpt region for adjusting the viewing angle by adjusting the horizontal shift value Vx0 of the image according to the control of the viewing angle of the user; (3) the control unit, extracting the calculated area in real time and restoring it back to the original image size; (4) The control unit calculates the viewing angle value V0 from the horizontal shift value Vx0 used for adjusting the viewing angle (

,

); (5) The control unit calculates the focal length f0, the vergence value V0, and the distance s0 between the cameras to calculate the distance d0 from the object to the lens (hereinafter referred to as a 'distance from the lens'). Process of calculating

); (6) The control unit calculates a distance D from the object to the lens and a focal length f0 to calculate a distance D from the object to the image sensor (hereinafter referred to as a 'distance to the image sensor'). process (

); (7) the controller determining whether the zoom value or the focus value has been changed by the photographer; (8) the controller extracting and interpolating the changed focal length f1 from the lookup table using the changed zoom value and the focus value when the zoom value or the focus value is changed; (9) the controller calculates the distance d1 to the lens by calculating the distance D to the image sensor and the focal length f1 (

); (10) the control unit calculates the viewing angle value V1 by calculating the camera interval s0, the changed focal length f1 and the distance d1 to the changed lens (

); (11) The control unit calculates a horizontal shift value Vx1 of the left and right images for determining the image processing amount from the changed viewing angle value V1 (

); (12) the control unit, extracting the original image according to the aspect ratio using the calculated horizontal shift value (Vx1), and restoring the extracted image back to its original size; (13) The control unit substitutes the changed viewing angle value V1 to the initial viewing angle value V0, substitutes the changed focal length f1 into the initial focal length f0, and checks whether the automatic mode is in the automatic mode. Characterized in that the process consisting of controlling the entire system to be re-run from the initial process.

Hereinafter, the present invention will be described with reference to the accompanying drawings.

First, the parallel-axis stereo camera system is composed of a sensor unit 200 for detecting the zoom, the focus position of the left and right lenses, and the zoom-main time interlocking control device 100.

As shown in FIG. 4, the zoom-time interlock device 100 includes a storage unit 110 that stores a look-up table that records a change in focal length of a lens according to zoom and focus, and a left of the CCD. ) An input video processor 120 that determines a video format of an input for an image input from a channel and a right channel, and zoom and focus position information of left and right lenses input from the sensor unit 200. Reads the reference zoom and focus values from the lookup table of the storage 110, calculates a horizontal shift value Vx0 of the image using the reference zoom and focus values, and controls the zoom and the viewing angle in conjunction with each other. 130, a vertical image processor 140 for controlling the vertical size and position of the left and right images input from the input video processor 120 under the control of the controller 130, and the controller 130. Above control The horizontal image processor 150 controls the horizontal position and size of the left and right images input from the input video processor 120, and the vertical image processor 140 and the horizontal image processor 150 control the vertical / horizontal size and And an output video processor 160 for determining a video format of the output with respect to the determined image.

The controller 130 varies the horizontal shift value Vx0 of the left and right images every time the photographer controls the gaze point, extracts the image in the vertical and horizontal directions according to the original aspect ratio, and then returns the image to its original size. After restoring the image, output the image to the viewfinder and other devices.

In addition, when the automatic mode is selected by the photographer, the controller 130 extracts and interpolates the focal length f0 from the lookup table of the storage unit 110 using the current zoom value and the focus value, and starts the extracted region. The vergence value V0 is obtained from (Vx0).

The controller 130 calculates the distance d0 to the lens by calculating the focal length f0, the viewing angle value V0, and the distance s0 between the cameras, and calculates the distance d0 to the lens. And the focal length f0 to calculate the distance D to the image sensor, determine whether the zoom value or the focus value has been changed by the photographer, and use the changed zoom value and the focus value of the storage unit 110. The extracted focal length f1 is extracted and interpolated from the lookup table, the distance D1 to the image sensor and the focal length f1 are calculated, and the distance d1 to the lens is calculated. The angle of view value V1 is calculated by calculating the changed focal length f1 and the distance d1 to the changed lens, and the horizontal shift value Vx1 of the image for the angle of view control from the calculated angle of view value V1. Calculate the required area on the left and right images based on the calculated horizontal shift value (Vx1). Taken into liver and controls the time by the main restored.

In this case, the lookup table of the storage unit 110 is generated by the focal length table according to the zoom value and the focus value precisely measured by the lens manufacturer. That is, a look-up table of the storage unit 110 for zoom-periodic interlocking control of a parallel-axis stereo camera is obtained by using a focal length table (see FIG. 2) according to zoom and focus values provided by a lens manufacturer. It is written and used. The look-up table of the storage unit 110 divides the zoom value into 50 steps of 2% from 0 to 100%, and the focus value of 2,500 lenses measured by dividing the focus value into 50 steps of 2% from 0 to 100%. Since it has data, it can be used to accurately control zoom, focus and visual angle. For zoom and focus values between 2% of sampled values, peripheral distances are interpolated in real time by linear interpolation to calculate the focal length.

Referring to the accompanying drawings, a method for interlocking control of the zoom-main time of the parallel-axis-type stereo camera system configured as described above is as follows.

First, Figure 3 is a geometric representation of a parallel-axis stereo camera image, d represents the distance from the lens to the gaze point, s is the distance between the two cameras, f is the focal length, W is the horizontal line occupied by the gaze point in the real image Indicates the length.

FIG. 5 is a diagram illustrating a camera state at an initial point in time and a camera state after a zoom-in operation, and schematically illustrate a method of maintaining a gaze point having binocular disparity of 0 regardless of zoom. In the CCD diagram of FIG. 5, the solid line indicates left and right camera images that are fully zoomed out before the gaze point is adjusted in the parallel axis camera, and the dotted line shows an image portion to be taken to control the gaze point. Shown. 5 also shows (a) a state before zoom-in and a state (b) zoomed in from that state.

In FIG. 5, the left and right images always show a difference as much as the camera interval s near the gaze point, and the difference s is represented as horizontal shifts Vx0 and Vx1 in the image. Since the gaze point is a part where the parallax becomes zero, in order to control the gaze point, the parallax of an object at a distance to be set as a gaze point should be removed. To this end, the viewing angle may be controlled by cutting the horizontal portion of the image by the horizontal shift (Vx0, Vx1), cutting the vertical portion back to the original aspect ratio, and restoring the original image size again. That is, the gaze angle value V0 is multiplied by the width of the CCD image to the pixel Vx0 of the region excluding the pixel of the portion to be extracted for gaze point control from the pixel in the fully zoomed-out state before adjusting the gaze point. The multiplied result is calculated by dividing the multiplied result by the pixel Hx in the zoomed out state before adjusting the gaze point.

Referring to the accompanying drawings, a method of determining the size (Vx1) of an image to be cropped in order to control the viewing angle in conjunction with zoom and focus will be described below.

6, the controller 130 determines whether the photographer inputs an end command after setting the gaze initial value V0 and the camera interval initial value s0 (S100). As a result of the determination, when the photographer inputs an end command (YES in S110), the controller 130 ends the operation of the camera, and when the photographer does not input an end command (NO in S110), the controller 130 zooms in and out. It is determined whether the control is in the automatic mode (S120). In the manual mode, the zoom and the viewing angle are controlled separately by the photographer's manual operation (S300), and when the automatic mode is selected by the photographer, the zoom-time interlocking control mode is started. The focal length f0 is extracted and interpolated from the lookup table 110 by using the and focus values (S130).

The controller 130 calculates a distance d0 to the lens by calculating the focal length f0, the viewing angle value V0, and the distance s0 between the cameras (S140). That is, when the focal length f0 is multiplied by the camera interval s0, the multiplication result value is divided by the vergence value V0, and the divided result value is the distance d0 to the lens. In this case, since the horizontal shift value Vx0 used for adjusting the viewing angle represents the number of pixels in the image, the horizontal shift value Vx0 should be converted to the viewing angle value V0 which is a geometric length.

The controller 130 calculates a distance D to the image sensor by calculating a distance d0 and a focal length f0 to the lens (S150). That is, when the focal length f0 is added to the distance d0 to the lens, the addition result is the distance D to the image sensor.

The controller 130 adjusts the zoom value or the focus value by the photographer (S160), and extracts and interpolates the focal length f1 changed using the adjusted zoom value and the focus value from the lookup table of the storage 110. (S170).

The controller 130 calculates the distance d1 to the lens by calculating the distance D to the image sensor and the focal length f1 (S180). That is, when the focal length is subtracted from the distance D to the image sensor, the result of the subtraction becomes the distance d1 to the lens after the zoom value or the focus value is changed.

The controller 130 calculates the viewing angle value V1 by calculating the changed focal length f1, the distance d1 to the changed lens, and the camera interval s1 (S190). That is, when the focal length is multiplied by the camera interval s1, and the multiplication result value is divided by the distance d1 to the lens, the dividing result value becomes the viewing angle value V1 after the zoom value or the focus value is changed. .

In addition, the controller 130 calculates a horizontal shift value Vx1 to be extracted from the left and right images from the viewing angle value V1 (S195), and calculates an image portion to be vertically extracted according to the aspect ratio. Based on the calculated result, the visual angle is controlled by extracting the image and restoring the size. In other words, when the horizontal angle value V1 is multiplied by the horizontal pixel number of the CCD and the multiplication result value is divided by the horizontal length of the CCD, the horizontal shift value Vx1 is obtained.

In addition, the controller 130 substitutes the modified viewing angle value V1 into the initial viewing angle value V0, and substitutes the changed focal length f1 into the initial focal length f0 (S200), and then, in the automatic mode. The whole system is controlled to be re-run from the initial stage of checking whether it is.

When the user manually adjusts the camera interval knob (S210), the controller 130 recognizes that the camera is adjusted, and first recognizes the newly generated camera interval s1 (S220), and recognizes the new camera. After adjusting to the interval s1 (S230), the newly created camera interval s1 is substituted into the initial camera interval s0 (S240) so that the camera interval is readjusted.

If the user manually adjusts the viewing angle knob (S250), first, the newly generated viewing angle V1 is recognized (S260), and the viewing angle is adjusted using the recognized new viewing angle V1 (S270). )do. Subsequently, the newly generated vertex V1 is substituted into the initial vertex V0 (S280) so that the vertex may be readjusted.

By using the basic geometric structure of the stereo camera as described above, each camera parameter is derived in a proportional relation, and when one parameter value is changed, another parameter value is directly substituted into the calculation formula and calculated. Since the zoom-time is interlocked so that it changes to a parameter value, it can be controlled smoothly without intermittent feeling to maintain a natural three-dimensional effect.

The present invention implements a zoom-perspective interlock control function using an image processing technique in a parallel-axis stereo camera, thereby making it possible to manufacture a stereo camera that is simple and practical to operate.

In addition, the present invention does not require a motor for watching point control, unlike the parallel moving axis method, which improves the stability and precision of the product, and is advantageous for maintenance. The effect is that it can be produced.

Claims (7)

In the zoom-vision time-linked control method of a parallel axis stereo camera, (1) extracting and interpolating a focal length f0 from the lookup table using the current zoom value and the focus value when the automatic mode is selected by the photographer; (2) calculating, by the controller, an excerpt region necessary for adjusting the gaze angle by adjusting the horizontal shift value Vx0 of the image according to the user's gaze control; (3) the control unit extracting the calculated area in real time and restoring the original image size again; (4) calculating, by the control unit, a viewing angle value V0 from the horizontal shift value Vx0 used for adjusting the viewing angle; (5) calculating, by the control unit, a distance d0 to a lens by calculating values of the focal length f0, the vergence value V0, and the distance s0 between the cameras; (6) calculating, by the controller, a distance D to the image sensor by calculating a distance d0 and a focal length f0 to the lens; (7) the controller determining whether the zoom value or the focus value has been changed by the photographer; (8) extracting and interpolating the changed focal length f1 from the lookup table using the changed zoom value and the focus value when the zoom value or the focus value is changed by the controller; (9) calculating, by the controller, a distance d1 to the lens by calculating a distance D to the image sensor and a focal length f1; Calculating, by the controller, a gaze value V1 by calculating a camera interval s0, the changed focal length f1, and a distance d1 to the changed lens; (11) calculating, by the controller, a horizontal shift value Vx1 of the left and right images for determining an image processing amount from the changed viewing angle value V1; (12) extracting the original image according to the aspect ratio by using the horizontal shift value Vx1 calculated by the controller, and restoring the extracted image back to its original size; And (13) an initial stage in which the controller substitutes the changed gaze value V1 into the initial gaze value V0 and substitutes the changed focal length f1 into the initial focal length f0 and then confirms whether the automatic mode is in the automatic mode. The process of controlling the entire system to be re-run Zoom-time visual interlocking control method in a stereo camera system of a parallel axis method, characterized in that consisting of. The method of claim 1, wherein the visual angle value V0 in the step (4) is: Before adjusting the gaze point, the width of the CCD image is multiplied by the pixel (Vx0) of the region excluding the pixel of the portion to be extracted for gaze point control from the pixel in the fully zoomed-out state, and the multiplied result value is taken as the gaze point. The method of claim 1, wherein the dividing is performed by dividing by the pixel Hx in the zoomed-out state before adjusting the zoom. The method of claim 1, wherein in the process (11), the horizontal shift value Vx1 is Zoom-major angle in a parallel-axis stereo camera system, characterized by multiplying the horizontal angle of the CCD by the vergence value V1 and dividing the multiplication result by the CCD image width Wccd. Interlocking control method. The method of claim 1, wherein the horizontal shift value Vx1, This is the reference value to crop the horizontal area of the image. In the step (12), the horizontal region of the image is cut out using the horizontal shift value Vx1, the vertical portion is cut out according to the original aspect ratio, and then the angle of view is controlled by restoring the cut image to the original image size. Zoom-periodic interlocking control method in a parallel camera system of a stereo axis system characterized in that the. A sensor unit 200 for detecting zoom and focus positions of the left and right lenses; And The storage unit 110 stores a look-up table that records changes in the focal length of the lens according to zoom and focus, and determines the video format of the input for images input from the left channel and the right channel of the CCD. The reference zoom and focus values are read from the look-up table of the storage unit 110 using the input video processor 120 and the zoom and focus position information of the left and right lenses input from the sensor unit 200. After calculating the horizontal shift value Vx0 of the image using the reference zoom and focus values, the controller 130 controls the zoom and the viewing angle in conjunction with each other, and the input video processor 120 under the control of the controller 130. A vertical image processor 140 for controlling the vertical size and position of the left and right images input from the first and second images, and the horizontal top of the left and right images input from the input video processor 120 under the control of the controller 130. And a horizontal image processing unit 150 for controlling the size, and an output video processing unit for determining an output video format for the image whose vertical / horizontal size and position are determined by the vertical image processing unit 140 and the horizontal image processing unit 150. A zoom-time visual interlocking control device 100 including 160; Zoom-periodic interlock control device in a stereo camera system of a parallel axis system comprising a. The method of claim 5, wherein the controller 130, Whenever the photographer controls the gaze point, the horizontal shift value (Vx0) of the left and right images is varied, the image is extracted in the vertical and horizontal directions according to the original aspect ratio, and the image is restored to its original size and output again. Zoom-peripheral interlocking control device in a parallel camera system of the stereoscopic axis system. The method of claim 6, wherein the controller 130, When the auto mode is selected by the photographer, the focal length f0 is extracted and interpolated from the lookup table of the storage unit 110 using the current zoom value and the focus value, and the gaze value from the start point Vx0 of the region to be extracted. And a zoom-periodic interlocking control device in a stereo camera system of a parallel axis type, wherein V 0 is obtained.

Images