KR20150033831A

KR20150033831A - The convergence control method using the depth map histogram characteristic

Info

Publication number: KR20150033831A
Application number: KR20130113570A
Authority: KR
Inventors: 김호영
Original assignee: 아이유미디어 주식회사
Priority date: 2013-09-25
Filing date: 2013-09-25
Publication date: 2015-04-02

Abstract

The present invention relates to a method and apparatus for controlling a natural main clock such as a left and a right eye of a person by controlling the intervals of the left and right cameras or the interval between the left and right images when the three- For histogram control to form a stable optimal optimal stereoscopic effect, a histogram representing characteristics such as a depth map representing a spatial distribution and a spatial distribution of a depth map is utilized. The histogram of a general image expresses how much each element of the brightness of the image is distributed. However, in the present invention, there is a difference between the brightness of the image and the spatial distance from the camera to each object or background . The depth map histogram representing the spatial distance can be used to grasp the overall spatial distribution and the judgment as to how to set the main time at a certain position of the space can be performed by using a cumulative histogram, It is possible to determine an optimal optimum time control position for each image based on the input value.

Description

[0001] The present invention relates to a method for controlling a main view using a depth map histogram characteristic,

The present invention relates to a method for automatically controlling a main time of an image in a stereoscopic camera system for stereoscopic photography.

More specifically, when taking a picture using a stereoscopic camera, the stereoscopic effect on the stereoscopic image of the captured image varies depending on the distance between the left and right cameras and the distance to the object to be photographed. In some cases, dizziness occurs, In order to eliminate such dizziness and to optimize the stereoscopic effect, it is desired to automatically set or control the interval of the left and right cameras or the interval of images according to the image to be photographed.

The stereoscopic effect forms the stereoscopic effect of the object according to the distance between the left and right cameras, the distance to the object, and the distance relative to the surrounding objects. That is, because each object of the image is at a different distance, the angle of the object seen right from the eyes of the person, that is, the main view of each object seen from the human eye, is different from object to object. Referring to FIG. 3 (a), in the case of the human eye, when the object approaches to the object, both eyes accumulate on the object and the main time becomes larger. As the main time increases as described above, it is recognized as a near object in the human brain, and the three-dimensional body is felt by the difference in the main time. Such a method is called width stock control. However, in the stereoscopic camera, when the object approaches as shown in FIG. 3 (b), the stereoscopic image is taken by reducing the interval of the camera in order to make a normal stereoscopic object. This method is called parallel main vision control, and the parallel main vision control method can be implemented by the horizontal movement of the photographed image rather than the physical method such as the left and right movement of the camera, Since there is no need for a physical hardware element such as an additional mechanism, recently, it is proceeding to implement main vision control through this method. However, in order to avoid the fatigue of eyes while watching for a long time, it is necessary to have a method for controlling the main time so as to prevent the dizziness through the stabilized main time control, that is, appropriate camera interval or horizontal movement of the left and right images according to the image Do.

As described above, the present invention controls the interval of the left and right cameras or the intervals of the left and right images when photographing a three-dimensional image, thereby controlling a natural main clock such as a person's left and right eyes In order to control the main time in the stereoscopic camera, the image of the human being is captured and the image is captured by the stereoscopic camera. Through the analysis, appropriate mechanical control of the camera or the interval of the image should be controlled and implemented. In general, since the control of the width of the common stock like the eyes of a human being is possible through the mechanical control to control the angle of the camera, the parallel stereoscopic control method of moving the image to the left and right in the fixed stereoscopic camera should be used. For this purpose, a stereoscopic camera that extracts the depth map of the left and right images, transforms it into a histogram, and automatically measures the left and right horizontal movement intervals, And to provide a main visual control method.

An object of the present invention is to control a stabilized optimum principal time, and it is an object of the present invention to provide a method and apparatus for controlling a stereoscopic camera using a stereoscopic camera, The analysis of whether it is appropriate is the most important factor.

There are various methods such as a method of controlling the main time with a certain object of the image or a method of controlling the specific position of the image at the main time, but this method is not applicable to the stereoscopic There is a limitation in controlling the main time at the position of the object or the fixed image. Since the three-dimensional effect depends on the relative spatial distribution of each object or background in the image, a method considering this is needed.

The present invention utilizes a histogram expressing characteristics such as a depth map expressing a spatial distribution and a spatial distribution of a depth map for controlling the main view for forming a stable optimum stereoscopic effect. The histogram of a typical image expresses how much each element of the brightness of the image is distributed. However, in the present invention, there is a difference in the spatial distance from the camera to each object or background, not the brightness component of the image. The determination of whether the spatial distribution can be grasped by using the depth map histogram representing the spatial distance and the main time corresponding to a certain position of the space or the size of the horizontal movement can be determined by using the cumulative histogram, It is judged based on the inputted value.

When proper stereoscopic control is not achieved when shooting with a stereoscopic camera, it causes dizziness due to stereoscopic visual confusion. In other words, if the stereoscopic effect on the stereoscopic image is different according to the distance between the left and right cameras and the distance to the object to be photographed, and if this is different from the stereoscopic characteristics normally seen by the eyes, it causes dizziness and headache Such visual inconvenience can be minimized. In addition, the most difficult point when shooting using a stereoscopic camera is to perform the same operation as manual operation of the camera through each operation of each scene every time. By eliminating the manual operation of such a camera, You can automate by analogizing the viewpoints.

Figure 1: Signal processing diagram for estimating the main time of representative road
2: (a) Left and right camera images
(b) Depth map with stereo matching
(c) Histogram plot according to distance of depth map
(d) Cumulative graph of the histogram
Drawing 3: Width Stock and Parallel Stereo Viewing Method
4: Filtering result at the main time extracted by frame unit

FIG. 1 is a signal processing diagram for estimating a main time point, and FIG. 2 is a specific example of each signal processing diagram.

2 (a) is an example of left and right images taken by a stereoscopic stereo camera. Because they are taken at different horizontal intervals, they are almost similar to each other visually, but if we look closely at the objects and backgrounds in the image, the position in the horizontal direction is slightly different depending on the distance. For example, in the left image, the human and the car are separated by 125 pixels and 630 pixels on the left side of the screen, respectively, but are 110 pixels and 620 pixels apart on the right side, respectively. In other words, there is a difference of 15 pixels in the case of a human being and a difference of 10 pixels in a case of an automobile. Such difference causes the relative stereoscopic effect of a person and a car to be felt.

The depth map as shown in FIG. 2 (b) should be generated using the photographed image. The depth map corresponds to a value representing a degree of deviation between objects, which is expressed relatively close to the distance and relatively far away using the brightness of black and white. In Figure 2 (a), when a person and a car have different horizontally shifted deviations and are represented by a magnitude corresponding to the brightness of the image, the depth map represented by the distance in space as shown in Figure 2 (b) do. In the case of a person, it has a deviation of 15 pixels, which is relatively larger than the difference of 10 pixels of the car, so it is represented as close and bright, and the automobile and other backgrounds are darker because they are relatively far away. The process of extracting the depth map on the left and right side is obtained through an image processing method called Stereo Matching, which is understood by people in the ordinary image processing field.

Using the depth map thus obtained, the reconstruction is performed using the histogram as shown in Fig. 2 (c). As described above, the histogram used in the present invention expresses the distribution of the space along the distance as frequency, and it can be inferred that human objects are distributed at a distance of about 5 m from the car at a distance of about 10 m. In this way, the depth map histogram shows how far objects are spaced, the composition and distribution of the space can be predicted, and an optimal stable main view control position visually can be derived through appropriate calculation.

In the case of 3D stereoscopic images, when people feel the most optimal stereoscopic effect in a 3D stereoscopic image, when the surface of the image screen of a display device such as 3DTV is defined as a reference of the main time, 70% It is analyzed that the most stable three-dimensional feeling is felt when the spatial distribution is about 30% in the future. The main time reference is a reference line for expressing whether an object is in front or behind when viewing a stereoscopic object. Normally, when a 3D television is viewed, the stereoscopic effect of the object is based on the screen surface. That is, 70% of the image is on the back of the screen, and 30% of the image is on the front of the screen. This value is a numerical value that can be different according to the person and should be set by the user in consideration of his / her three-dimensional feeling.

If the user sets 70%, the cumulative graph of the depth map histogram is required as the result of accumulating the frequency of the histogram from a distance far from the distribution of the spatial distance to 70% of the time corresponding to the surface of the screen.

If the total sum of the frequencies from the object at a distance from the histogram to the corresponding object element is 100%, the set value set by the user in the histogram accumulation graph of FIG. 2 (d) When the distance is reached, the spatial frequency distribution of the image is 70%: 30%, and the left and right movement proportional to this distance is calculated and the image is moved so that the optimal main visual control without visual dizziness is required. Since fConvergence_Point is the estimated distance in the drawing and the moving amount of the image for controlling the actual main time is different according to the shooting condition such as the size of the image and the interval of the camera, the conversion relation should be reflected by using the proportional formula as follows.

Left, the number of pixels of moving image = Function (fConvergence_Point);

The function Function corresponds to a simple proportional expression such as horizontally moving one image by 10 pixels when fConvergence_Point is 5m and horizontally shifting one image by 20 pixels when it is 10m.

Stereo Matching image processing for generating the depth map may cause errors depending on the image. When the image moves like a moving picture, each frame is very similar image, but the fConvergence Point corresponding to the main time control generated in each frame is changed by the error. That is, as shown in FIG. 4, since the main time control value is shaken every frame including errors, it is necessary to average a cumulative value of at least 4 to 8 frames to maintain a stable fConvergence_Point. However, when the scene changes, the previous scene and the subsequent scene are completely different from each other. Therefore, the main time control value of the frame corresponding to the new scene change point should be inputted through the initialization of the average value at the scene change point of the image.

Reference numeral 4, 5, 6, 7, 8 in FIG. 1 denotes f_num is a set value of a frame for averaging, and f_cnt is a counter that is incremented every frame. As shown in Block 5 in the specific frame, the point at which the cumulative value Sum of the depth map histogram becomes larger than the user set value (User%) set by the user is the distance i to be obtained at the time of the main time. If the current frame is a scene corresponding to the scene change point, the currently obtained main view distance i is set as the start point of the new scene, and this value is set to the fConvergence_Point corresponding to the last main time. If the current scene is not the scene change point, f_num , And finally averages f_num frames and sets them to fConvergence_Point at the last main time. The above operation is repeated for each frame.

Since the sensitivity is relatively low relative to the motion of the main visual control, the average time corresponding to 4 to 8 frames (in the case of 60 Hz video, 1 frame corresponds to 16.6 msec and 4 to 8 frames corresponds to 66 to 132 msec) If the appropriate control of the main time is stable in time, there is no difference in the 3D stereoscopic effect, so the frame number for maximum averaging is usually set within 8 frames.

1. A stereo matching block for generating a depth map using left and right images
2. Histogram transformation and generation block of depth map
3. Generate a histogram cumulative graph
4. Frame counter value to eliminate errors at the time of the week
5. Cumulative value for tracking set value of user's main time
6. Cumulative value calculation block of the histogram
7. A scene change point detection block for initialization at the time of day
8. Results of averaged and stable last-day time

Claims

A method for controlling a main image of a stereo image,
The depth map is estimated through image matching for left and right images,
A depth map histogram corresponding to the frequency according to the distance is generated using the depth map,
An accumulated graph is generated by using the generated histogram to determine a main time control position corresponding to a set value of the main time control defined by the user,
An average value in units of frames for minimizing the error of the main time control position is determined,
And in the case of the scene change point, setting the main time control position value obtained in the first frame after scene change to the initial control position.

The method according to claim 1,
And a depth map is generated by performing image matching on the received left and right images,

The method according to claim 1,
And generating a histogram corresponding to the distance using the depth map,

The method according to claim 1,
Generating an accumulated graph of the histogram, and using the calculated graph to determine a position of the main visual control amount desired by the user,

5. The method of claim 4,
In determining the main time control position, the main time control value for each frame is averaged to minimize the error, and in the case of the scene change point, the main time control of the first frame after scene change is initialized to the main time control value of the first frame Main time control method characterized by