KR20080014712A - System and method for automated calibrationand correction of display geometry and color - Google Patents

Abstract

A system and a method for automatically calibrating display geometry and display color are provided to correct geometric distortion and color distortion due to abnormal characteristics of several optical elements within the system and mis-alignment of several elements. A display calibration system is used for a display device having a screen. The display calibration system comprises a sensor(11) and a processor connected to the sensor. The sensor senses information of at least one of geometry, size, boundary, and direction of the screen. The processor calculates characteristics of the display device based on the sensed information. The sensor senses a test image displayed on the screen, and the processor calculates the display distortion based on the sensed test image and the calculated characteristics.

Description

SYSTEM AND METHOD FOR AUTOMATED CALIBRATIONAND CORRECTION OF DISPLAY GEOMETRY AND COLOR}

The present invention relates to a system and method for automatically correcting the shape and color of a display.

Most video display devices exhibit some form of geometric distortion or color distortion. The causes of this distortion are various, for example, geometrical settings, abnormal characteristics of the various optical elements in the system, misalignment of the various elements, geometric distortions due to the complexity of the screen and optical path, and panel incompleteness. Depending on the system, the amount of distortion can vary from almost undetectable to very unpleasant. The influence of the distortion is various and may cause a change in color or shape of the image.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems in the prior art, and it is an object of the present invention to provide a calibration system and a calibration method for displaying on a screen without distortion in an image applied to a display device with a screen.

In order to achieve the object of the present invention, according to the present invention, a display calibration system for use with a display device having a screen, comprising: a sensor for detecting at least one information of a shape, a size, a boundary and a direction of a screen; And a processor connected to the sensor and calculating a characteristic of the display device based on the information detected by the sensor.

In addition, according to another aspect of the invention, a display calibration system for use with a display device having a screen, comprising: a sensor for detecting information in a test image displayed on the screen; And a processor connected to the sensor and calculating a display distortion based on the information detected by the sensor to compensate for the display distortion by generating a precompensation map, when the precompensation map is applied to the input image data before display. A display calibration system is provided that implements a precompensation map with a surface function that eliminates distortion in the displayed image.

In addition, according to the present invention, a display calibration system for use with a display device having a screen, comprising: a sensor for detecting information in a test image displayed on a screen; And connected to the sensor, calculates the display distortion based on the information detected by the sensor, splits the screen into a number of patches according to the degree of display distortion, and generates a pre-compensation map for the display distortion for each patch, When the pre-compensation map is applied to the input image data before the display, a processor for removing distortion from the image displayed on the screen may provide a display calibration system including a.

On the other hand, a display calibration system for use with a display device having a screen, comprising: a sensor for detecting color information in a test image displayed on a screen; And a color correction map for color components calculated by calculating color unevenness based on the information detected by the sensor, and applying the color correction map to the input image data before display to display the image displayed on the screen. It provides a display calibration system comprising; a processor for removing the distortion.

On the other hand, a display calibration system for use with a screened display device comprising: a sensor for sensing information in an individual color component test image displayed on a screen; And independently calculate the geometric display distortions of the color components based on the information detected by the sensor and independently generate a precompensation map for the color components, thereby inputting the precompensation map before display. When applied to image data, there is also provided a display calibration system comprising a; processor that removes the color-based geometric distortion in the image displayed on the screen.

According to another aspect of the present invention, there is provided a display calibration method for a projection apparatus having a curved screen, the method comprising: projecting respective portions of an image to corresponding portions of the curved screen using a plurality of projectors; And forming the entire image on the screen with the optimal focus by matching the focus of each part of the image to the corresponding portion of the curved screen.

Meanwhile, according to the present invention, there is provided a display calibration method for a projection apparatus having a curved screen, comprising: measuring various distances from a curved screen to a focal plane of a projected image; And displacing the focal plane until the focal point is optimized by minimizing the function of the distance.

According to the method and system of the present invention configured as described above, it is possible to eliminate all distortion in the image generated on the screen.

Important distortions in display devices include distortions caused by lenses, distortions caused by reflections on mirror surfaces (curved or flat), projection types such as corners or rotational projections (keystones, rotations), and distortions from projection on curved screens, and multiple micro displays. Color-specific lateral chromatic aberrations and distortions, such as misconvergence, misalignment, and color / luminance unevenness, and distortions due to optical focus problems (spherical aberration, movie point aberration).

Most are geometric distortions in the final image, and the shape of the input image is not preserved. Chromatic aberration is also a kind of geometric distortion, but the distortion varies depending on the color. These distortions are very common in projection type (back or front) display devices and are collectively called geometric distortions. Unevenness of color and brightness affects all display devices, so that signals with fixed brightness or color also vary depending on the viewing angle or change across the screen of the display device. This type of distortion is due to uneven sensor response in light sources and panels (LCD, LCOS, plasma displays) with varying luminance and varying optical path lengths across the display. Distortion related to focusing is caused by blurring the image and focusing differently on the image plane for each point on the object plane. In some cases, the problem related to the focus and depth of focus will be described.

Here, a system and method for calibrating a display device to eliminate or mitigate the aforementioned distortions will be described. Create and correct calibration data. It also corrects for distortions that change over time. In the calibration phase (creating calibration data), the display is characterized, the sensor means such as a high resolution camera is captured to capture the special test patterns appearing on the display device, and the necessary data (eg calibration data) is extracted from these images. In the correction step, the image is pre-distorted through an electronic correction means such as a processor so that the image is displayed without distortion on the screen. A mechanism for optimizing the focus of the display and captured test patterns is also presented.

1 is a block diagram of an example of an automatic correction system for correcting an image appearing on screen 16 of a display device. The automatic correction system includes a test image generator 14, a sensor 11, a calibration data generator 12, a warp generator 13, and a digital warping unit 15. Display devices, such as TVs (back projection TVs, LCDs, plasmas, etc.) and front projection devices (eg, projectors with screens), display images and all have screens. The frame of the screen 16 surrounds the screen by separating the screen from the background. However, the screen frame does not necessarily have a physical form, but only needs to be able to distinguish the screen 16 from the background. For example, a rectangular shape projected on the screen from the outside into the frame may be referred to as a screen frame. Here, the screen 16 in the display device having the substance is taken as an example, but in some cases, the screen frame itself may be in the screen, and the screen frame surrounds the screen 16.

Curved screens adopt two viewpoints because their depth varies. Although the focal plane showing the modified form of the image may be a screen, this may be different from the actual screen 16 or may include only a portion of the actual screen 16. All points on the focal plane have the same depth of focus. In this case, a physical marker (observer) in the field of view of the sensor determines the focal plane boundary (see FIG. 2A). If there is a picture frame, the picture frame is used to determine the orientation of the camera relative to the screen 16.

On the other hand, the entire screen having a physical frame forming a boundary may be a curved surface (see FIG. 2B). In this case, the depth of focus of each point on the screen is different. The goal of correction is to match the final image to the surface boundaries.

Two viewpoints can be combined to identify display areas that need to be modified. For example, a combination of the outline of the captured image and the actual frame can be taken as a boundary at a particular focal plane. Projecting curved outlines allows you to create curved boundaries on flat displays. This can be seen in the special case where the boundary is curved but the screen itself is flat so that the radius of curvature is infinite.

If there is a distortion resulting in a change in shape, the entire image may not appear on the screen 16 (before correction) (see FIG. 3). In case of (a), the picture frame 18 is all contained in the image ABCD (overflow), in case of (b) the whole image is contained in the frame (underflow), and in case of (c), the screen ( 16) is an intermediate state (mismatch) over. All three come from either the front or rear projection unit and can be modified with the current system.

The test image generator 14 provides an image having a special pattern designed for calibration, which is also called a calibration test pattern. The most widely used calibration test patterns include square grid patterns, circles, squares, horizontal and vertical patterns, rods, linear, concentric patterns, rectangles, circles, and color levels. The color version is used for lateral color correction and color non-uniformity correction. The shape of the pattern is also referred to as a feature. Every pattern has its own characteristics and has factors such as number, position, size, border, color, and the like.

4 shows various calibration patterns. The guidelines show that features (such as center, radius, etc.) are not part of the test pattern. Changes in color and shape are also used to adjust black and white, replace black and white with color, use different colors for each feature in the pattern, combine shapes together in the pattern, and change gray and color levels.

Patterns using the primary colors are used to correct lateral chromatic aberration. In the typical color pattern (g), the colors of the horizontal / vertical bars and their intersections are different.

All patterns show some distinctive features, the most prominent of which is the center of the shape, and the boundaries are mathematically represented as points and lines.

The sensor 11 records the calibration test pattern appearing on the screen 16. The sensor 11 uses a camera for geometric distortion correction. The resolution and capture format of the camera are chosen to match the precision required for correction. When correcting color and luminance non-uniformity, the sensor 11 may be a color analyzer such as a photometer or a spectrometer.

Here, the sensor 11 is placed at an arbitrary position with respect to the display device for geometric error correction. This is because the sensor 11 can be placed in any position because the captured image is allowed to have a distortion component due to the position of the sensor 11. If the sensor 11 does not show the screen 16 directly, a keystone component due to the sensor 11 is generated. This distortion occurs along three axes, which are considered to be multi-axis keystone distortion components.

In addition, since the optical device of the sensor 11 such as a camera also has its own distortion, the optical distortion component must also be taken into account. It has inherent distortion depending on the type of sensor 11. The overall distortion by the camera or sensor 11 is called camera distortion. Camera distortion is determined and compensated for when generating calibration data.

To determine camera distortion, we use a physical reference marker that knows the direction / shape of distortion. These markers are captured by the camera and the camera distortion can be determined by comparing the direction / shape of the captured image with the distortion-free direction / shape. The frame itself, which knows its direction and shape (usually a rectangle without distortion), is a natural marker. Since the frame is also the basis for calibration, the corrected image should be straight with respect to the frame. Thus, when correcting for geometric distortions, the image captured by the camera should have a screen boundary (frame) 18.

On the other hand, when the boundary is not detected, the camera sensor for the screen 16 is determined by detecting a signal from the emitter of the screen using a sensor of the camera. This measurement produces a map of the screen 16 seen by the camera.

When correcting the lateral color, the camera captures K sets of images, where K is the number of color components and the number of three primary colors RGB. The test pattern of FIG. 4 is repeated for each color component.

The correction of brightness and color is made irrespective of the geometric correction. In the projection apparatus, such correction of brightness and color is made after correction of geometric distortion. In the flat panel display device, there is no geometric distortion, and the luminance and color are directly corrected. For example, a sensor such as a color analyzer is installed near the screen 16 to extract color information. In this case, positioning is unnecessary to modify the sensor. The sensor 11 captures the entire image or information of a specific point. In the latter case, we need to capture data about the point grid on the screen. If the sensor 11 is in the keystone position with respect to the screen 16, a correction due to this positioning must be made, similar to the modification of the camera.

In the case of display devices with geometric distortion, the luminance and color must be corrected after correcting the distortion. That is, first, the geometric distortion having the color related distortion is corrected. After color correction, color correction allows you to correct only the area (not the background) that contains the final image, taking into account any additional color distortion caused by the geometric correction.

In this case, the calibration data generator 12 analyzes the image and extracts calibration data in a format used by the warp generator 13, and the warp generator provides warp data to the digital warping unit 15.

Digital warping may be referred to as a process of using a pre-compensation map for mathematical conversion between input image coordinates and output image coordinates according to Equation (1) below.

는 입력화소의 컬러, (x _i , y _i )는 출력화소의 공간좌표로서 모두 출력공간에 매핑되는 것이고,

는 대응하는 화소의 출력 컬러이다. 3원색 장치의 경우,

는 단순히 RGB 값이다. (1)은 그리드형태의 수정을 표현한다. 프로세서에서 바로 그리드포맷을 이용하기는 어려우므로, 비디오의 60Hz 촬영속도처럼 실시간으로 수정을 적용해야 한다. 따라서, 워프생성기에서는 식 (1)을 하드웨어용 포맷으로 변환한다. 교정데이터 생성기(12)는 3가지 하위생성기로 이루어지는데, 이들은 각각 형상, 횡색수차(lateral color), 컬러불균일을 교정하기 위한 것이다. Where i is the range of input pixel coordinates, ( u _i , v _i ) is the spatial coordinate of the input pixel,

Is the color of the input pixel, ( x _i , y _i ) are the spatial coordinates of the output pixel, all mapped to the output space,

Is the output color of the corresponding pixel. For three primary colors device,

Is simply an RGB value. (1) represents the correction of grid form. It is difficult to use the grid format directly on the processor, so you need to apply corrections in real time, such as 60Hz video rates. Therefore, the warp generator converts equation (1) into a hardware format. The calibration data generator 12 consists of three sub-generators, each for correcting shape, lateral color and color unevenness.

Hereinafter, calibration data for correcting the shape will be described first. In the following description, the test patterns mainly analyzed are grid patterns such as (a) and (b) of FIG. 4. Patterns (e) to (g) of FIG. 4 can also be used because they form a grid at intersections.

The test image, such as the grid pattern, provides a shape centered on a known location in the input space. This center is represented by ( x ⁰ _i , y ⁰ _i ), where i is the range of the shape. Starting from the top left, there are a total of M x N shapes along the lines of the test pattern, with a resolution of W _T x H _T. The resolution of the test pattern does not need to match the original resolution of the display device. The center of the shape in the test pattern is distorted by the distortion when displayed ( x ⁰ _di , y ⁰ _di ). Formation is also distorted, for example, a circle is distorted into an ellipse. These coordinates are defined in space on the display about the origin of the upper left of the frame 18 of the screen 16. W _D x H _D is the resolution inside the frame 18 of the display device in any meter and its coordinates are ( x ⁰ _di , y ⁰ _di ). Since the display space is the same as the real or observer space, the modified image should appear without distortion in the display space.

The camera captures an image of the distorted grid pattern and sends it to the calibration data generator 12. The resolution of the camera is W _C x H _C. In this embodiment, the resolution of the camera does not need to match the resolution of the display, and the camera can be installed anywhere. The coordinates of the center of camera space are ( x ⁰ _ci , y ⁰ _ci ) and the origin is the upper left of the captured image.

The captured image is visible from the camera's viewpoint, but corrections must be made at the actual viewpoint, ie from the observer's point of view. Therefore, the calibration process must exclude the camera's viewpoint. As described above, the screen frame 18 is used as a marker. Therefore, the camera image must also capture the frame 18. In practice, define frame 18 with the following coordinates:

Upper left: (0,0)

Upper right: ( W _D , 0) (2)

Lower left: (0, H _D )

Lower right: ( W _D , H _D )

In the camera image, these coordinates are as follows:

Upper left: ( x ^d _cTL , y ^d _cTL )

Top right: ( x ^d _cTR , y ^d _cTR ) (3)

Bottom left: ( x ^d _cBL , y ^d _cBL )

Bottom right: ( x ^d _cBR , y ^d _cBR )

5 shows several spatial and coordinate systems. In the illustrated image, the circle is black and the background is white, but all test patterns may be different in color and shape (see FIG. 4). Three cases are shown: (a) an overflow state where the image completely covers the frame 18, (b) an underflow state where the image completely fits within the frame 18, and (c) The image and the frame 18 are mismatched and mismatched. As described above, the projection shapes are classified, and the input space and the camera space are defined based on pixels, but the display space may be defined in units of meters in addition to the pixels.

The pixel distortion represented by f _D is given by Equation 4 below:

는 식 (4)에서 주어진 값의 역수로서 아래와 같다:

Is the inverse of the value given in equation (4):

를 입력영상에 적용하여 디스플레이에 앞서 워프(사전왜곡)를 한다.The digital warping unit 15

Is applied to the input image to warp (predistortion) before the display.

이므로, 역방향 구조의 수정장치에 필요한 맵이나 워핑 데이터가 이런 디스플레이 왜곡맵일 뿐이다. 따라서, 교정데이터 생성기(12)에서 만들 그리드데이터는 아래와 같다:The above maps are all defined in the forward direction: the function domain is the input image and the range is the output image. As is well known, electronic correction circuits, such as processors, are more effective and accurate in making images using inverse structures. In the inverse warping structure, an output image of a processor is generated by mapping a pixel of an output unit to an input unit through a correction map and then filtering (allocating color values) in the input space. In this case, the correction map appears in the form of an inverse, which is denoted by fw . The modification of the inverse shape is the display distortion map itself.

Therefore, the map and warping data required for the reverse structure correcting device are only such display distortion maps. Thus, the grid data to be created in the calibration data generator 12 is as follows:

This information must be extracted from the captured image of the camera, and the camera is in camera space. The captured image corresponds to the mapping defined in (7) below:

This map is a full image map, which can be represented by a combination of the display distortion map f _D and the camera distortion map f _C , from which the required f _ㅉ is defined by (8):

Subtracting f _C from f _D is simply a concatenation (or combination) of two maps. Also, since the scale and origin of the display coordinate system cannot be applied, the coordinates ( x ⁰ _di , y ⁰ _di ) must be brought to the scale and origin of the correction pixel. This is described in detail later.

An example of a calibration data generator 12 is shown in FIG. 6. First analyzing the W _C x H _C camera image of the test pattern to extract the center of the shape ( x ⁰ _ci , y ⁰ _ci ); _F f is obtained from this. The shape center in camera space corresponds to the shape center in input space after mapping to display distortion and camera distortion. This shape cannot be used for an image that overflows the screen 16. These out-of-bounds shapes are generally invisible in rear projection TVs or front projection systems because they are on a background of different planes. Therefore, only the shape in the screen 16 determined as EFGH is analyzed (see FIG. 5).

Find shape center using various image processing algorithms. One way is to convert the captured image into binary (black and white) image using a threshold mechanism. The pixels of the shape in the binary image are displayed separately. The center of each classified pixel set is placed in the shape center. Analyzing the histogram of the image determines the threshold. The histogram may relate to a luminance or a specific color tone of the captured image.

The captured image is analyzed to find the coordinates and boundaries of the screen. Other images may be used at this stage. The screen coordinates are needed to determine the camera distortion f _C. If the camera has no optical distortion, the camera distortion is stereo distortion f _C ^P , and the coordinates defined in equation (3) of the four cameras are needed to determine f _C. If the camera has optical distortion, additional markers are needed. The frame boundary EFGH is sufficient as a marker that can be parameterized by the equation of the edge. The edge equation is also used to determine which shape to place in the screen 16 and the four corners. An actual rectangular grid whose coordinates in display space are known as ( x ^CC _di , y ^0C _di ) can also be used as a marker by attaching or projecting onto the screen 16, which is projected as ( x ^0C _ci , y ^0C _ci ) in camera space. do. This grid looks like a camera calibration (CC) grid. Determining screen frame coordinates and boundaries is called display characterization.

At the point of view of the sensor, the optical distortion in the camera lens and the curved screen cannot be distinguished. In either case, the markers and picture frames appear curved. Thus, the curved screen is placed within the frame of the camera grid associated with the camera distortion. When the camera distortion is corrected, the final image matches the frame. To modify the curved screen, a marker is attached to the frame 18 at regular intervals (measured on the screen) to form a CC grid. A marker may be attached inside the frame 18. Since curved screens are two-dimensional surfaces, they can be calibrated through a two-dimensional CC grid.

Edges or markers (of the attached CC grid or frame 18) are detected by standard image processing methods such as edge detection. Knowing the position of the edge allows the equation of the line to fit the edge, and the intersection of the line is the four corners and the CC grid coordinates. The edge and CC grid coordinates are defined as in Eq. (9), where Ncc is the number of points in the camera calibration grid.

(9)

(9)

For display devices with curved screens, it is not easy to get the CC grid from the actual markers. In this case, the CC grid is calculated mathematically using the Efat formula. How to place the point to edge and how to interpolate within the frame 18 is free. Either way, if the domain coordinates are chosen correctly, the final image will match the frame 18. One way is to place points along the edges at equal intervals and then linearly interpolate to the interior of the frame.

If the manufacturer specifies the specifications for the camera's optical distortion f ⁰ _C , these specifications can be combined with stereo-distortion or used instead to create a camera calibration grid, which is shown in Eq. (10).

(10)

10

The optical component of the camera distortion is determined before the display calibration because it is independent of the position / direction of the camera. The data of equations (3) and (9) are collectively called camera calibration data.

The coordinates obtained must be properly positioned. Mathematically, each range coordinate ( x ⁰ _ci , y ⁰ _ci ) is assigned to the corresponding domain coordinate ( x ⁰ _i , y ⁰ _i ). To create a full image map f _F, we need to determine the domain coordinates. This extraction process does not generate any information about domain coordinates. The centers do not necessarily need to be determined in an order that matches the shape order in the input test pattern.

Points are arranged using the test pattern as shown in Figs. 4 (c) and 4 (d). The pixels of the image captured by these test patterns are classified according to the bars to which the image belongs. The shape center is in this category. The horizontal and vertical bars to which the center belongs determine the domain coordinates ( x ⁰ _i , y ⁰ _i ), and i is defined in equation (11).

(11)

(11)

When arranging, it is important to determine which bars and shapes are in the screen frame 18. If the background part (outside the picture frame) does not provide high contrast in the image, only the shape and bars inside the picture frame 18 are measured when there is an appropriate threshold (in the extraction feature coordinate step). If the external shape is strongly projected, the shape and the bar inside can be determined compared to the edge of the frame. When numbering bars, consider all missing bars (outside the frame). The bars of a given numbering sequence blink one at a time to determine whether they are in the frame. You can also implicitly number using different colored bars.

Camera calibration data should also be arranged, where the domain coordinates are in the display space. However, this process is simpler than putting all the features in the frame 18. In most cases, a coordinate comparison is sufficient to determine the array. In the case of a CC grid, a grid ( x ^0C _di , y ^0C _di ), which is a domain coordinate (in the display space) of the CC grid, is allocated through an array, which is called a domain CC grid. The value of this domain CC grid depends on whether the grid corresponds to a real marker or is mathematically constructed. In the former case, the known coordinates of the marker provide the domain CC grid. In the latter case, there is some freedom to choose a domain CC grid. If the final image matches the frame 18, the points of the CC grid at the edge must match the corresponding edge of the square EFGH. That is, the edge should be mapped as follows.

Straight line through top edge ⇔ {(0,0), ( W _D , 0)}

Straight line through the right edge ⇔ {( W _D , 0), ( W _D , H _D )}

Straight line through the lower edge ⇔ {(0, H _D )), ( W _D , H _D )}

Straight line through left edge ⇔ {(0,0)), (0, H _D )}

Apart from this limitation, domain CC grid points can be selected appropriately. After extraction and arraying, we can find the mapping ( f _w ) using equation (8).

Camera calibration data is used to initially construct the inverse camera distortion map f ^- _1C . For most common scenarios of pure stereo camera distortion (i.e. f _C = f ^P _C ), four corner points are required.

(12)

(12)

(Inverse) stereo conversion is as below.

(13)

(13)

Where ( x _d , y _d ) are coordinates in display space and ( x _c , y _c ) are coordinates in camera space. Equation (12) is used to find eight linear equations and solve for the coefficients { a, b, c, d, e, f, g, h } that define three-dimensional transformations.

When the camera distortion includes the optical distortion component f ⁰ _C or is modified to fit the frame, use the local equation or CC grid to determine the inverse camera distortion map f ^-1 _C. Using a CC grid is one way, either because it provides information about distortion of internal points or only about edges. The CC grid is given by equation 10. This grid is interpolated (at least square) by a given set of basic functions. One option is to make spiral adjustments based on the spiral or to interpolate with the grid defined in equation (14).

에 의해 f _w 가 구해지는데, 여기서

는 식 (15)와 같이 주어진다.From f ⁻¹ _C and ( x ⁰ _ci , y ⁰ _ci ) calculated in the camera calibration data extraction step,

F _w is given by

Is given by equation (15).

(15)

(15)

This connection uses the full range of image maps for the domain to obtain the camera's inverse distortion map.

는 도 5의 중간 도표에 대응하고 디스플레이 왜곡을 수정하는데 필요한 맵을 (인버스 형태로) 제공한다. 전술한 바와 같이, 그리드는 화면틀(18) 안에 있는 포인트만 포함한다. 오버플로 왜곡의 경우(a, b의 경우), 도메인 공간(즉, 디스플레이 왜곡의 뷰포인트로부터의 입력영상)내의 (형상중심에 대응하는) 많은 화소가 이 그리드로 정의된 디스플레이 공간내에 좌표를 갖지 않는다. 본 실시예에서 디지털 워핑부(15)인 프로세서는 모든 도메인공간 화소를 처리하는데; 인버스 수정요소의 도메인공간은 실제로는 출력영상이다. 따라서, 미싱 그리드데이터를 계산해야 하는데, 이 계산은 외삽단계와 리샘플링 단계에서 한다. Obtained Grid

5 corresponds to the middle diagram of FIG. 5 and provides (in inverse form) the map needed to correct the display distortion. As mentioned above, the grid only includes points within the frame 18. In the case of overflow distortion (a and b), many pixels (corresponding to the shape center) in domain space (i.e., the input image from the viewpoint of display distortion) do not have coordinates in the display space defined by this grid. Do not. In this embodiment, the processor, which is the digital warping unit 15, processes all the domain space pixels; The domain space of the inverse correction element is actually the output image. Therefore, the missing grid data must be calculated, which is performed at the extrapolation and resampling stages.

을 외삽하여 미싱 데이터를 결정한다. 이 함수는 고속으로 리샘플링하여, 즉 도메인 포인트를 MxN에서 (nM -n+1)x(nN -n+1)으로 증가시켜(n=2,3..) 수정 그리드 덴서(denser)를 만드는데에도 사용된다. As when calculating camera distortion, the grid fw can be interpolated by a set of basic functions such as spirals. Interpolation

Extrapolate to determine missing data. This function is also used to resample at high speed, i.e. increase the domain point from MxN to ( nM -n + 1 ) x ( nN -n + 1 ) (n = 2,3 ..) to create a modified grid denser. Used.

가 되도록 할 수 있고, 입력공간상의 포인트 어레이에서 이 함수를 평가해 빠진 포인트를 포함한 수정 그리드를 구할 수 있다. 원래의 그리드를 유지하기 위해, 식 (16)과 같이 입력공간에 새로 규칙적으로 간격진 그리드 어레이를 형성하으로써

의 보간 형태를 이용한다. Now modify the map

We can evaluate this function on an array of points in the input space to get a correction grid containing missing points. To maintain the original grid, by forming a new regularly spaced grid array in the input space, as shown in equation (16)

Use the interpolation form of.

(16)

(16)

를 계산하면 식 (17)에 의한 수정 그리드 (x _di , y _di )가 생기는데, 물론 빠진 포인트를 포함하고 덴서일 수 있다. This array is a denser, where / M> M and / N> N. In this array

Calculations yield a correction grid ( x _di , y _di ) by equation (17), which may of course be a denser containing missing points.

화면틀내에서

일 경우 (17)

Within the frame

If (17)

에 보간점들을 조합하여 미싱 데이터의 외삽과 보간을 내부 데이터에 대해 할 수 있다.

Interpolation points can be combined to extrapolate and interpolate missing data to internal data.

The final step in generating calibration data is to fix the scale and origin. The correction grid is in display space and is defined relative to the upper right corner of the frame 18. The unit (scale) of the display space is arbitrary and may differ from the unit used for the input space. Before using this data in the warp generator 13, it is necessary to create an origin and scale that match the input space. This is seen by optimizing the origin and scale.

Considering the middle diagram of FIG. 5, when the correction is made, the final modified image should be square with respect to the frame 18. According to FIG. 7, a rectangle including the modified image is called an active rectangle A'B'C'D '. The active rectangle must be in the frame of the image (ABCD) and the frame of the image (EFGH). The origin and scale are selected so that the upper left corner of the active rectangle is (0,0) and the width multiplied by the height of the rectangle is W _T xH _T , and the product of the width and height is the resolution of the input image (see FIG. 7).

Once scaled and moved, the input space for calibration is actually the output space for electronic correction in an inverse structure, and the input space for correction is actually the same as the display space (ie, the calibration output space).

와

이면, 모든 그리드 격자의 축척을 조절해 식 (18)과 같이 변위시켜야 한다.The top left corner and size of the active rectangle

Wow

If so, all grid grids should be scaled and displaced as shown in Eq. (18).

(18)

(18)

The value of W _D xH _D to determine the square coordinates is selected as an integer to maintain the aspect ratio of the frame 18. Applying Eq. (18) changes the dimensions of the display space (bottom chart) to the input image dimensions (top chart) needed for correction.

를 입력영상의 종횡비(W _T / H _T )와 일치시켜야 한다. 여러 제한사항을 아래 C1~C4에 열거한다.Determination of the active rectangle is free, but there are natural limitations. To maximize the pixel resolution of the modified image, make the rectangle as large as possible. If the modified image should have the same aspect ratio as the input image, the aspect ratio of the rectangle

Must match the aspect ratio ( W _T / H _T ) of the input image. The various restrictions are listed in C1 through C4 below.

C1) The active rectangle falls into the light frame ABCD.

C2) The active rectangle falls within the frame EFGH.

C3) The area of the active rectangle is maximized.

C4) Make the aspect ratio of the act ratio rectangle the same as the aspect ratio of the input image

와

를 결 정하는 것)은 수치최적화의 문제이다. 위의 모든 제한을 수학적 형태로 두되, 이 문제를 해결하는데 여러 최적화 방법을 이용한다.Solving this limitation for active rectangles (i.e.

Wow

Is a matter of numerical optimization. All of the above limitations are in mathematical form, but several optimization methods are used to solve this problem.

Constrained minimization is one way, in which case you define a function that reorders the constraints into inequality and minimizes (or maximizes) them. Using the line equation for the frame edge (see Equation 9) and the outermost lattice point (see Eq. 17), the constraints C1 and C2 are inequality, with four square edges (≤). C4 is already an equal sign, but C3 is a function that will maximize the area of the active rectangle.

In the scenario (a) of FIG. 5, the image overflows to fill the screen 16, and the screen frame 18 automatically satisfies C1 to 3 as natural rectangles. When the scale of the display is fixed to the scale of the test image, the following equation (19) is obtained.

(19)

(19)

If the modified image exactly matches the screen frame 18, the entire screen frame 18 is used as an ideal state. Thus, in this case, the optimization step of FIG. 6 means using only equation (19), which means that there is no need to change the scale of the point or move the point.

The optimization step is also used to change the aspect ratio by changing the C4 condition as shown below (20).

Using equation (18), the aspect ratio of the corrected image is α . Since the aspect ratio can be freely selected, the image can be a mailbox or a cylinder having different aspect ratios in the display device. By adjusting the scale and displacement, the image can be easily enlarged (overscan) or reduced (underscan) with respect to the screen 16. Therefore, the surface function can be used to easily implement the overscan / underscan state.

The final calibration data produced by the calibration data generator 12 is the grid data / f'w given by equation (21).

(21)

(21)

The above explanation focuses on the distortion that corrects all the primary colors. In this case, the same grid data accounts for the correction of all colors, in which case it can be called a monochrome correction. However, in the case of lateral chromatic aberration distortion, multicolor correction is necessary because the grid data is different for all primary colors, and this case may be referred to as multicolor correction. Since the geometric distortion common to all the primary colors is included in the lateral correction, the correction data generator 12 can also be implemented in the special case of the multicolor correction described below.

An example of the correction data generator 12 for lateral chromatic aberration correction is shown in FIG. This generator is similar to repeating a single color correction K times, where K is the number of primary colors. The primary color is _denoted by I _i, where i = 1 ... K.

Correcting each of the primary colors is similar to the steps described in the Monochromatic modification.

Now colorize the test pattern to match the primary color you are calibrating. For example, when red is corrected, all the characteristics (circles, bars, etc.) of all the test patterns (refer to a to j of FIG. 4) are all red. The characteristics (such as the number of circles) of the features may vary depending on the color pattern.

The color image is used in all image processing steps, such as extracting the center and the etch. Adjust the threshold to handle the color being corrected. Once a binary image is obtained, image processing is independent of color.

In general, because of lateral chromatic aberration in the camera lens itself, camera calibration data differs depending on the primary colors, and it is necessary to calculate this data separately for all primary colors. The system is configured to correct lateral chromatic aberration in the camera itself. Similar to the test image pattern for calibrating the display device, camera calibration data is generated using a test image pattern different for each primary color. Creating (multicolor) calibration data in the camera is independent of display calibration and needs to be done once. When creating camera calibration data, use display devices that have no or minimal lateral chromatic aberration (much less than the camera). Without such a display device, color markers are used to create a physical grid of known coordinates. The final result of the multicolor camera calibration depends on the primary color obtained from equation (22) as reverse camera distortion.

(20)

20

After calculating the missing data, the K grids (similar to Eq. 17) are obtained from Eq. (23).

(23)

(23)

The number of points in each grid can vary with test patterns and resampling.

The primary test patterns belong to different projection shape classes (see Fig. 5). Some test patterns completely overflow the screen frame 18 as shown in a of FIG. 5, and some are entirely in the screen frame as shown in b of FIG. 5. Optimizing, the active rectangle must be within the image frame ABCDk of each color in the frame 18; At this time, the spatial crossing method of the image border is used. This means that one optimization is made and the edge ABCDk of all primary colors is taken into account in C1. Optimization determines the coordinates of the active rectangle common to all primary colors. These coordinates are used to scale and displace the grid according to equation (18).

The output of the optimization step is the K 'grid, which provides calibration data for all primary colors, as shown in equation (24).

These data sets are used in warp generator 13.

In this embodiment, non-uniform calibration data is generated after correcting the geometrical distortions (types 1 to 4) for both color and luminance or one color. There are various causes of color unevenness, for example, the distance of the screen 16RK paper due to the projection shape (keystone angle), the incompleteness of the micro-display panel, and the like.

In the case of a shape-modified display device, the test pattern image appears as a rectangle (active rectangle) in the screen frame in a size that matches the screen frame 18 as much as possible. The upper left corner of the active rectangle is used as the origin, not the upper left corner of the screen frame 18. The test pattern used is only a color version of the patterns used for shape correction of one color; The color of the feature (circle, bar) is k to correct the primary color k. This is the same as used to correct lateral chromatic aberration. For luminance, gray values (maximum white, half-maximum) can be used. The term color is used to identify all color components being modified; It may be luminance, a component of RGB or YCbCr, or a component in another color space that can be measured by the sensor 11.

As the sensor 11, a camera or a color analyzer (spectrometer, photometer, etc.) is used. To increase accuracy, a photometer or spectrometer must be used. These color analyzers capture the entire image (i.e. multiple points) or one point of data. The sensor 11 should be installed as close to the screen 16 as possible. If a single-point color analyzer is installed on the screen at the actual known coordinates (ie, the center of the shape), the data of these coordinates can be obtained. Although the multipoint color analyzer and camera can be placed in any position, it should be placed in the center of the screen as close to the screen 16 as possible to improve accuracy. 9 shows an example of a screen 91, a single point color analyzer 92, and a multipoint color analyzer 93. The calibration data generator for color unevenness is similar to that for correcting geometric distortion. 10 shows a calibration data generator 12 'of color non-uniformity.

와 대응 공간좌표(x ⁰ _i ,y ⁰ _i )로서, 측정을 한 모든 포인트에 대한 값이다. 여기서 k=1...K는 컬러를 분석하고 있음을 확인한다. 테스트패턴이 정의되어 있어 C ⁰ _kl 로 표시된 오리지널 컬러값도 알려졌다. 이렇게 해서 얻은 식 (25)는 컬러왜곡 맵이라 하는 컬러분균일 왜곡을 설명한다.The data captured by the single-point color analyzer 92 are the primary color values.

And the corresponding spatial coordinates ( x ⁰ _i , y ⁰ _i ), which are values for all points taken. Here k = 1 ... K confirms that we are analyzing the color. The test pattern was defined so that the original color values, denoted C ⁰ _kl , were known. Equation (25) thus obtained explains color uniform distortion called a color distortion map.

Spatial coordinates do not change with color non-uniform distortion. The original color value C ⁰ _kl is usually fixed C ⁰ _{kl =} C ⁰ _k in a given test pattern, meaning that all pixels other than the background have the same color. One or more sets of measurements s = 1 ... S can be made, each set corresponding to a test pattern with each fixed color value. For convenience, one subscript i covers each set of measurements, as shown in equation (26).

The spatial coordinates are the same for each set. The following description applies to each set (test pattern).

When the multipoint color analyzer 93 is a camera, the captured data corresponds to the entire image. In this case, some image processing is required before obtaining the grid. Calculate the center of the shape ( x ⁰ _ci , y ⁰ _ci ) and the domain coordinates ( x ⁰ _i , y ⁰ _i ). This step is the same as the extraction and arrangement step performed in shape correction. In addition to the center, the color value of the center of the shape is also calculated. The color values can be obtained by averaging or filtering the pixel color values near the identified center of the captured image according to Equation 27.

는 중심 부근에서의 캡처한 영상의 컬러값이다. 가장 가까운 4개 포인트를 평균내려면 필터계수가 α _j = 1/4, j=1..4이다.

Is the color value of the captured image near the center. To average the four nearest points, the filter coefficient is α _j = 1/4 and j = 1..4.

The final result is the grid data obtained from Equation 25. Here, (i) only the domain coordinates are needed because the color distortion does not change the spatial coordinates; (ii) there is no missing data because the image is in screen 16 and there is no geometric distortion; (iii) There is no need to calculate and correlate sensor distortions as there is no geometric correction.

Depending on the type of sensor used and the format of the captured data, color space conversion must be performed to bring the color data into the color space of the display. For example, spectrometers give data in terms of color values, but display devices and processors (processors) require RGB values. Color conversion is accomplished through matrix multiplication or more complex nonlinear equations. For color space conversion, grid data of the entire primary colors is used. In general, this transformation has the form

In the absence of color distortion, the color values across the coordinates ( x ⁰ _i , y ⁰ _i ) should be measured with the constant C ' ⁰ _k for a test pattern of a given color. The measured constant may be different from the original constant pixel value C ⁰ _k . In most displays, the measured and original values are proportional, and the proportional constant λ is constant in the absence of color distortion and spatially changes in the presence of color distortion. Therefore, the color distortion map of the display can be expressed as in Equation 29.

의 항으로 주어지는 공지의 디스플레이 컬러함수 f ₁ 으로 표현된다.In general, the input and measured color values are vectors such as

It is represented by the known display color function f ₁ given by the term

이 공간적으로 변한다. 식 31과 같이 데이터를 분석하면 주어진 좌표 (x ⁰ _i ,y ⁰ _i )를 결정할 수 있다.Without color distortion

This changes spatially. Analyzing the data as in Equation 31 allows us to determine the given coordinates ( x ⁰ _i , y ⁰ _i ).

You need a sufficient number of values per coordinate. Fit to the data, the analytical value approximates f ₁ . Likewise, if the same data is analyzed inversely as in Eq 32, f ₁ ^-1 is calculated.

에 따라 결정되고, 수정인자는 f ₁ 의 형태에서 결정되거나, 다항식과 같은 특정 기본함수를 이용해 역함수 데이터에 대입하여 계산된다. 선형 최소자승 고정법의 경우, 인버스 맵이 식 33의 형태를 취한다. The inverse function is also a color correction factor

The correction factor is determined in the form of f ₁ , or calculated by substituting inverse function data using a particular basic function, such as a polynomial. For linear least-squares fixation, the inverse map takes the form of equation 33.

r = 1 ... R is the number of parameters defining the inverse colormap, Br is the base function. Factors are different for central coordinates and primary colors. Usually, f ₁ ^-1 is determined by the representation used by the processor, which is usually represented by a polynomial. The above expression is sometimes used to adjust the final color level, as it may be necessary to reduce the original C ⁰ _ik value in the output. You can increase or decrease the inverse by simply adjusting the factor to scale.

Knowing the inverse function (of the center coordinates), a corrected colormap that corrects for color non-uniform distortion is given by Eq 34.

와

로 설명할 수 있다. 따라서, 수정을 위한 교정데이터 f _Wck 는 식 35에 따른 컬러수정인자와 관련된 그리드 데이터를 설명한다.The spatial variation of color distortion and correction

Wow

This can be explained as Therefore, the correction data f _Wck for correction describes the grid data related to the color correction factor according to equation (35).

For the most common case of equation 29, equation 36 is given.

The grid above can be denser by resampling with an appropriate interpolation function. Similar to the geometric correction, the new grid is given by equation 37.

This is the data output of the calibration data generator 12 '.

The complete data generated by the calibration data generator 12 'including all subgenerators is given by equation 38.

가 동일하여, 하나의 기하학적 교정그리드만이 계산되어 출력된다. 교정데이터는 워프생성기(13)에 입력된다. K grid without lateral chromatic aberration

Is the same, only one geometric correction grid is calculated and output. The calibration data is input to the warp generator 13.

As mentioned earlier, grid data is not directly used by the processor. Although grid representation is the most common format, it is inefficient to implement in hardware because it requires storing large amounts of data (coordinates of all pixels) and is difficult to manipulate (such as scale changes). Conventionally, lookup tables are used, which is not optimal. The warp generator 13 converts the grid representation obtained from Equation 38 into warp data, and the warp data may represent a modified form in an efficient form for application to hardware. If grid data is available directly from the processor, then a resampled grid can be used for all pixels and generation of warp data by the warp generator 13 is not necessary.

Warp data is generated according to the data conditions of the processor. The processor can apply geometric-color transformations using various architectures, most of which use geometrically modified inverse maps and the grid above is designed with an inverse architecture. The effective electron correction architecture introduced in US Patent Application 2006-0050074 A1, "System and method for representing a general two dimensional transformation", is based on linear function representations of grid data. Warp generator 13 converts the grid data into a function representation. 11 shows an example of the warp generator 13.

The general function expression ( x _i , y _i ) → u _i of a two-dimensional grid is expressed as shown in Equation 39.

Equation 39 is a two-dimensional surface function of the domain (x, y) , which is a linear combination of the basic function B _i ( x, y ) ( i = 1 ... L ) and its coefficient is the surface coefficient a _i . This coefficient is constant and does not change in the domain. The basic function does not have to be linear; the combination should be linear. In some cases, since the base function is a nonlinear function, the form represented by equation 39 is generally sufficient to represent the correction grid. Basic functions and their numbers are specified in the processor because they are implemented and evaluated in hardware. Warp generator 13 determines the necessary coefficients.

In one example, the basic function used in hardware is a polynomial. By deriving two exponents, the polynomial base function and the corresponding equation can be expressed as 40.

Since we know the basic function, the new data to be determined and stored is a set of surface coefficients a _i . To be a surface representation is to change from a grid value to a surface coefficient, as in Equation 41.

The efficiency of this representation can be said to be efficient in the fact that it is possible to calculate the grid value for a set of pixels due to the surface coefficient when grid values are to be stored for each pixel; Therefore, the number of surface coefficients to be stored is relatively much smaller.

The number of coefficients determines how accurately the original grid values are represented. Increasing the number of coefficients, that is, using more basic functions, improves accuracy. On the other hand, if you divide a domain into several patches and use different surface functions for each patch, you can use fewer basic functions. The structure of the patch is adapted to the degree of display distortion. This approach increases the flexibility to adapt the complexity of the combination surface to distortion. For example, the more complex the distortion, the more patches are used. The coefficient of the patch p = 1 ... P is a ^p _ij . The perfect surface takes the form of Equation 42.

A single surface corresponds to a single patch corresponding to the entire output image (domain). An example of division of a patch is shown in FIG.

The partitioning of a patch can be initiated with a starting configuration such as a 4x4 symmetric array of 16 patches. The array of patches (i.e. the number of patches and the boundaries of the patches) is called patch boundary D , which takes the form of equation 43.

Once the patch shape is established, the coefficients are calculated using the linear least squares fixation of the data according to equation 38. This fixation should ensure that the surface is continuous at the patch boundary. Once the surface is determined, error analysis is performed to compare the grid value with the calculated value (see Eq. 44).

Compare the error value with the allowable level E _max . If the maximum error is less than or equal to the allowable level, that is, max ( Error _i ) ≦ E _max , the surface coefficient is limited and output from the warp generator 13 as warp data. If the maximum error is greater, the patch is split further and the coefficients are further analyzed.

The surface of equation 38 is represented by equation 45.

The grid representation (i, j) is no longer needed because the function form is defined for the whole space and not for each of the coordinate sets. (i, j) now gives an exponential function or checks the base function. Surfaces are evaluated against patches with domain coordinates. The array of patches and the number of mood functions vary for each primary color. For example, changing the basic function of a patch can change the format above. The domain space for geometric correction is (x, y) and corresponds to the output image space (in the inverse structure), and the range space is (u, v) to correspond to the input image space.

를 구할 수 있다. For color correction, the domain space is fixed to (u, v). Color correction is done for geometrically correct images. This means that color correction must be performed before shape correction by warping the input image having the coordinate space (u, v). After warping the image and modifying it in the processor, you need to adjust the above coefficients for the new array to be corrected. In this case, color factors are defined in the (x, y) space. First the new grid in (x, y) space from the surface above as shown in equation 46

Can be obtained.

Fix this grid as described earlier and calculate the coefficients, but now the domain space is the output image space. The same expression is used for color correction surface coefficients. Error analysis is used for these rearranged grids.

The final output of warp generator 13 is a set of coefficients of equation 47 (arranged if necessary) that collectively form the warp data.

D ^k contains all the information that defines the patch shape of the primary color k . ( a, b ) The data is a geometric warp data or transformation that corrects distortions of types 1 to 4, and the vector c is a color warp or transformation that corrects distortions of type 5.

The digital warping unit 15 acts as a processor of the system as a processor. In this specification, the processor is used interchangeably with the digital warping unit. In practice, the digital warping unit 15 applies warp data to the digital input image (video) to prewarper or warp the input image in advance. The input image is warped both spatially and in color. Space warping is done to geometric warp and color warping to color warp. Pre-distortion cancels display distortion, and displays a distortion-free image on the screen 16.

를 만드는 표면평가요소와, 실제로 필요한 좌표를 사용해 화소 컬러값 C _i 를 계산하는 화소생성요소가 이것이다. 형상 수정을 위한 화소 생성은 필터링 단계인데, 여기서는 미리 계산된 계수 w _j (j=1...W)를 갖는 필터를 처리중인 현재 화소 (u _i , v _i )의부근화소에 적용한다. An example of the digital warping portion 15 correcting the shape-color unevenness is shown in FIG. The digital warping unit 15 includes a first block for geometric warping (ie, geometrically warping the input image) and a second block for warping only the input image of the color space to correct color unevenness. Here, color correction occurs after geometric correction, but this explanation also applies easily to inverse arrays. When no specific modification is needed, both blocks can be bypassed. Each block has two elements. For each primary color, the coordinates required by evaluating the surface polynomial defined by equation 45 at each pixel ( x _i , y _i )

These are the surface evaluation factor that produces, and the pixel generation factor that calculates the pixel color value C _i using the coordinates actually needed. The pixel generation for shape correction is a filtering step, in which a filter having a precomputed coefficient w _j ( j = 1 ... W ) is applied to the neighboring pixels of the current pixel ( u _i , v _i ) being processed.

In some cases, the filter coefficient may be calculated outside the system and applied to the digital warping unit 15. To correct the color non-uniformity, take the pixel value from the geometrically warped image and determine the new color value using Eq 33. The pixel generation step is summarized as 48.

는 형상수정을 한 뒤의 중간 컬러값을 나타낸다.Perform these steps for each primary color.

Represents the intermediate color value after shape correction.

가 필요하다. 마찬가지로, 간단한 컬러수정은 식 49와 같은 선형수정만 이용한다.The details of the filtering and color correction equations depend on the architecture of the hardware. A simple filter is to average four near points when w _j = 1/4. Complex filters use ellipses, the shape of which depends on the Jacobian determinants of the surface, and the filter coefficients can be obtained using filter generation algorithms. In this case, the neighborhood coordinates are used to estimate the Jacobian determinant.

Is needed. Similarly, simple color correction uses only linear correction as shown in equation 49.

On the other hand, complex color correction using a three-dimensional polynomial such as 50 is sometimes performed.

와 표면계수가 계산된다.Knowing the structure of the digital warping unit 15, the color factor

And surface coefficients are calculated.

The final result of the digital warping section 15 is a modification mathematically explained in equation (1), which is converted into the equation (51) below using a vector representation used to represent all the primary color components.

The warped or pre-compensated output image is input to the display device and is projected onto the screen 16 without distortion, thereby completing automatic correction and correction. Once you have completed the calibration and correction process, you can send normal (not test pattern) images and video to the display device.

Multicolor geometry corrections and corrections have been described together in Lateral Chromatic Aberration Correction. However, the main components can also be used to correct and correct all geometrically distorted distortions. Other cases include distortion due to misalignment of optical components due to the chassis or housing of the rear projection apparatus or due to the various micro-display devices arranged with respect to each other, and distortion due to different magnifications for each color component.

In the projection apparatus, color correction and correction are performed on geometrically corrected images. That is, color correction takes into account all the nonuniformities due to geometric warping itself. Geometrically warped images may differ in color or luminance from part to part due to the scaling / filtering process. In fact, the larger the scale factor, the more severe the change in luminance and color. This is automatically compensated for by color correction after geometric warping. Thus, the system automatically compensates for color irregularities due to geometric warping.

The system can also be installed in a single circuit for digital calibration / warping. Calibration data / warp generators 12, 13 are elements that run on any processor. The test image generator 14 may be replaced with a stored image set output from the processor. Processors embedded in hardware provide a single circuit solution for the entire calibration / correction process. The hardware can also be installed inside a display device like a camera to obtain a self-calibrating display device. In this case, the display distortion is calculated by receiving the information detected from the image sensor, and a warp map or color map, which is a precompensation map, is applied, and the precompensation map is applied to the input image data so that the image without distortion is displayed on the screen. Only a processor is required. However, in some cases it may be efficient to use more than one processor. Accordingly, one or more processors are required to implement the embodiments described above.

Various types of sensors used as the sensors 11 can be installed in the display device (used or used together with the camera). For example, as shown in FIG. 14, the sensor 143 is a distance sensor that is used together with or separately from the camera 142 to measure the distance of a point on the screen 141. This screen does not have to be flat. From the angle between the measured distance and these sensed distances, the relative angles of the camera 142 and the screen 141 are calculated. In addition, the shape of the screen can be calculated by this method (if not flat). In the case of FIG. 14, the denser lines on the right side of the screen indicate that the sensor 143 is closer to the right angle to the screen and the less dense lines on the left means farther from the right angle. Various sensors 143 can be used, including infrared sensors. In this embodiment, the physical structure is not required to draw the display screen (ie, the screen 141), and the camera 142 can be arranged arbitrarily.

In addition, there is a self-calibration display device that performs dynamic correction and correction, in which case the distortion can be corrected at any time by operating a correction-correction process without external resources. This allows correction of time-varying distortions such as keystone distortion of the projector or field calibration of rear projection devices such as RPTV. The correction system is placed in the housing or chassis of the RPTV for self-calibration. Other important distortions that change over time are optical shifts due to physical shifts, aging, and temperature. For example, in a rear projection device, the curvature of the mirror may change slightly due to the weight or temperature of the mirror, which requires dynamic correction and correction. The calibration-correction system is activated when the display device is activated or a change in distortion is detected.

Dynamic calibration and correction are particularly important in the field of fixed display devices such as TVs, where sensors may not be present. The distortion that occurs after the initial calibration and correction is due to small component changes over time. Manufacturers can use digital warping to simulate the various distortions ( i = 1 ... N ) expected in the field over time. This distortion can be corrected-corrected using the systems described in the above embodiments, but using two processors, one simulates the distortion and the other automatically tests the correction data. Warp data for modifying these N test cases can be stored on the display device. Over time, small distortions develop from the N warp correction, so choose the one that best corrects the distortion. Therefore, a complete system is not required, and since calibration is performed during the manufacturing process and N correction data sets are stored in the display device, only one digital warping unit needs to be installed in the display device. To automate the selection of the corrected data, a sensor in the display frame is used to detect specific test patterns. Thus, an image test pattern for optimal detection of distortion is loaded. This process can be carried out when the display device is operated for dynamic correction and calibration.

15 and 16, the calibration system finds the best projector focus on the screen. This is done by displaying a test pattern such as a certain number of parallel lines on the screen. The image is then captured and scanned by the processor to find contrast in the test pattern. Re-measure the contrast while displacing the projector focus, but continue until you find the maximum contrast. Maximum contrast corresponds to the best focus. The screen 151 has a poor focus state and the screen 161 has a good focus state. This technique can also be used to focus on sensors. Capture physical markers with sharp edges, such as the frame of a screen, and then perform maximum contrast analysis. If necessary, display a test pattern of appropriate color to reinforce the contrast between the marker and the background. Measure the contrast again while displacing the sensor's focus. Once the maximum contrast is established, it is the best focus. Focus the sensor before the display device.

On the other hand, the calibration system of Figs. 17 and 18 is used in a display device in which the screens 171 and 181 are curved and several projectors 1 to 3 are used. The projector covers the entire area of curved screens 171 and 181 and is controlled by the same electronic device. Geometric corrections are made to each of the projectors 1 to 3 and mapped to the respective areas of the screens 171 and 181. It also rotates and advances the projector image through geometric correction and combines it into adjacent images. In particular, the overlapping portions overlap the pixels. When the projectors 1 to 3 are mapped to the screens 171 and 181, the screens 171 and 181 are curved so that the incident angles are different and variable. An electronic device having or obtaining a map of curved screens 171 and 181 corrects the angular change across the screens 171 and 181 as represented by warp data.

In addition to geometric correction, color correction of the projectors 1 to 3 is performed so that the color features are visually identical in all projector areas. The electronic device divides the pixel color and the luminance between the projectors 1 to 3 to achieve uniform luminance-color mapping on the curved screens 171 and 181. Keep in mind, the number of projectors can be any number, and multiple projectors share the overlap, but the calibration is the same.

Focus is always a problem when projecting onto curved screens. This problem arises from the fact that projectors have a planar focal plane but the screen is curved, so that parts of the screen have different distances from the focal plane. In some parts of the screen the image is more focused than in others. In order to overcome this problem while using one projector, a technique of minimizing defocusing is used, an example of which is FIG. 19. In this case, the calibration system is arranged such that the sum of the squared distances of the vertical lines of the other focal plane 193RK paper on the curved screen 191 is minimized. For better focusing in the center of the screen, weight the section that connects the center of the screen to the focal plane.

In this case, the optimal focal plane can be calculated in advance based on the known shape of the screen. Intersecting the screen with the optimal focal plane results in the best point on the screen where the contrast is maximized. By calculating the optimal plane and the maximum contrast point, an image test pattern similar to that used in FIG. 16 is projected onto the screen, and then the image is captured to analyze the contrast. If the position where the contrast of the captured image is maximum coincides with the maximum contrast point determined previously, within the allowable limit, the projected image is brought to the optimum focal plane. If the maximum contrast point does not match the maximum contrast point determined previously, continue adjusting the projector focus until it matches. It should be noted that this technique can be applied to screens that are one-dimensionally curved (cylindrical, zero curvature) and screens that are two-dimensionally curved (spherical, non-zero curvature).

In FIG. 20, in addition to the above-described calibration problem, the problem of focusing is solved by projecting an image from several projectors each having an angle. As shown, different projectors are placed in each area of the curved screen 201 to significantly solve the defocusing problem. Position each projection axis nearly perpendicular to the screen portion and angle each focal plane to the center of this portion. In order to optimize the focus for each section, a technique as described in FIG. 19 is used. Meanwhile, the center of each section may maintain the tangent to the screen. In this case, the correction system is matched to the pixel shape, luminance and color as well as the focus of overlapping areas of the various projectors so that a smooth and seamless image is focused on the screen 201. As a result, the warping is much reduced since the angle between the focal plane and the screen tangent is reduced.

를 결정할 수 있는데, 컬러왜곡이 있으면 컬러수정인자가 공간적으로 변한다. 이 수정 모델은 예컨대 선형의 최소제곱 고정모델일 수 있다. 수정인자는 전적으로 카메라의 컬러왜곡에 대비한 교정데이터라는 특징을 갖는다.A system for calibrating sensors for multiple color shapes has been described. Likewise, this system can be used to correct (non-geometric) color distortion of the sensor. Using a calibrated / corrected display device, a fixed color pattern was displayed on the screen and recorded by the sensor; You can use the same pattern used to correct display color distortion. Knowing the original color values, the camera's color map can be obtained from Eq 25. The color correction factor of the camera in this colormap

If there is color distortion, the color correction factor changes spatially. This modified model may for example be a linear least square fixed model. Modifiers are characterized entirely by calibration data against camera color distortion.

Color correction is presented in terms of primary colors and luminance. The system can also correct or adjust any color. Using a test pattern or a variety of colors (not just primary or gray), a color map of the display can also be obtained, similar to Eq. 31, which is represented by Eq. 52.

는 컬러의 벡터로서 모든 성분을 갖는다. 사용된 컬러 집합은 전체 컬러공간에서 벡터를 정기적으로 샘플링하면서 선택할 수 있다. 인버스 맵은 식 53으로 표현된다.

Has all components as a vector of color. The color set used can be selected by regularly sampling the vector over the entire color space. The inverse map is represented by equation 53.

The color factor is a vector of length K (number of primary colors). In the previous display:

However, this is not simply a rearrangement of the color factors in one way, since the base function is now defined in the entire color space but not simply in the one-dimensional color space (ie one primary color). The basic function of the polynomial form is Eq 55.

By inducing the K- dimensional patch structure with the Q patch in the color space as shown in Equation 56, λ can be generalized.

This expression adds an index to the color factor, as shown in equation 57.

This equation provides a general transformation to the color space at all spatial grid points (center of the shape). This calibration color data is now defined by equation 58.

Without distortion, this grid would be the same at all coordinates. The warp generator converts this to a surface function with the shape defined in equation 59.

Finally, the polywarping unit calculates this polynomial and corrects the color using Equation 53.

If you have a normal colormap for each spatial coordinate, you can modify any color in any coordinate. Common color adjustments, such as white point adjustment, contrast adjustment, and hue adjustment, can also be made for each zone of the display. All of these adjustments are done in the color space, so a function approximation is possible in general form by equation 53. Selective color count is also possible through patch division of the color space, but it is limited to a specific color, and the rest of the colors are left as it is with the same grid except the color patch to be modified. Of course, selective hue correction is also possible, but it only modifies certain hue and leaves the rest. By using the general color correction and correction of the system, it is possible to increase the color accuracy of the display device.

를 통해 유저가 컬러를 조절케 할 수도 있는데, 이 컬러인자는 시스템 외부에서 계산되고 워프생성기(13)에 입력된다. 마찬가지로, 유저설정 기하학 인자

를 워프생성기(13)에 제공하면 유저설정 기하학적 효과(특수한 효과)를 얻을 수 있다.This system uses custom color parameters

The user may adjust the color through the color factor, which is calculated outside the system and input into the warp generator 13. Similarly, user-defined geometry parameters

Is provided to the warp generator 13 to obtain a user-defined geometric effect (special effect).

In the embodiment of Fig. 21, two cameras Cm1 and Cm2 are installed in one projector 213. When the input image is supplied to the projector 213, an image pattern is projected on the screen 211. Two cameras Cm1 and Cm2 are used to capture the image pattern projected on the screen 211. The system further includes a processor, which has not been shown but described above. The relative position of the two cameras relative to the processor is known. Two cameras can be arranged horizontally or vertically, or staggered with respect to the projector 213 both horizontally and vertically. Based on the comparison of the two images captured by the two cameras, the processor determines a distortion factor including the angle of the projector 213 relative to the screen 211. The processor then warps the input image to correct the distortion.

There is little distortion in the final projected image. This system and method can be applied to RPTV. In this case, the camera is installed in the RPTV in a fixed position and direction as shown in FIG. 22, but may be installed differently. The camera captures the pattern projected on the screen. If you look at the screen on the camera, you will see keystone distortion. However, the self-calibration can be performed as described above through the calibration system that is part of the display device.

In FIG. 23, three projectors P1 to 3 project images on the curved screen 231. Also, the images projected by the projectors are captured by the cameras Cm1 to 3. It is also possible to vary the number of cameras and projectors. For example, each of the cameras Cm1 to 3 may be used to project an image from the entire projectors P1 to 3. These cameras are alternately arranged horizontally as well as vertically with respect to each other. Each of the projectors P1 to 3 projects a known pattern or a test pattern on the curved screen 231 for calibration. Based on the images captured by the cameras Cm1 to 3, the processor calculates distortion factors including the shape and relative direction of the curved screen. The processor uses these factors to warp the input image provided to each projector during normal operation. Warping conversion of each projector precompensates display distortions caused by a particular projector. In addition, the luminance of each projector is analyzed to make the overall luminance of the projected image of the screen 231 uniform. In addition, the pixels of the superimposed regions are aligned by the processor, and the luminance of these superimposed pixels is dispersed in the projector to form a seamless image quality.

Using the system of FIG. 23, luminance data and color data can also be captured by the cameras Cm1 to 3. The processor uses these data to adjust the intensity for each pixel to match and blend the edges of each adjacent image. The overall brightness and color of all the projectors P1 to 3 can also be standardized by the processor.

In FIG. 24 a sensor (here a camera) is used to capture the projected image with or without the pattern. The camera may also detect the shape, size, relative direction, and boundary of the screen 241. The boundary edge may be the etch of the pull-down screen (ie, the retracted projector screen) or the corner of the room. Subsequently, the processor analyzes the direction of the edge of the image and the pattern of the test image to calculate screen characteristics such as shape, size, boundary, and relative direction. This calculation determines the display distortion. Based on the complexity of the pattern of the projected and captured image, the processor determines the distortion factor. If the pattern is simple, the projection angle can be determined compared to the normal screen. If the pattern is more complicated, it is possible to determine the shape of the screen, such as the shape of a curved or irregular screen. Distortion factors related to lens defects such as pincushion and convex distortion can also be determined. Once the distortion factors are collected, the distortion is corrected by applying a suitable precompensation warp map to the input image data to create a visually distortion free image.

24 illustrates a case of correcting an image projected onto a plane without any physical markers or edges. Distortions due to projection include both keystone distortion and lens distortion. In this system, the camera is attached to the projector by fixing its position and orientation. Calibration and correction are two steps. First, through a complete calibration process using the test image pattern, the image of the pattern captured by the camera is stored as a known keystone angle and lens distortion factor including a zoom level. It is also possible to store additional information necessary for correction, such as warp data. This step takes place at the factory, where the projector is assembled and calibrated at the factory. The second step takes place at the site using the projector. The projector projects the same pattern used in the first step, which is captured by the camera. The pattern captured in the field is compared with the pattern captured in the factory together with the distortion factor stored in the factory to determine the distortion factor of the projector in the field. Knowing these field distortion factors, the correction warp can be tracked (if stored) or established in real time to correct keystone and lens distortion of the projector. Since it compares with pre-stored information (images), no real edges or markers (such as picture frames) are needed at all. The data stored at the factory need not be complete images, but may be grid data, or other distortion levels may be factors that characterize different patterns.

On the other hand, a simple grid-like image pattern consisting of only four points is used to correct keystone distortion with the camera. In this case, the test pattern is composed of a 2x2 (4) grid as shown in FIGS. 2A-B. For keystone distortion with no lens distortion, four points are sufficient to determine the distortion. It is enough to know the point position of (before and after projection) to determine the keystone correction, so you can put 4 points anywhere. This method can also be applied to the lens displacement adjustment of the projector, in which case only four points need to be moved. For projectors with zoom lenses with or without lens distortion, first calibrate the axis for each zoom level and stored correction warp. Then, apply correction warps (for proper zoom level and lens distortion), and repeat keystone correction using only four points. Keystone distortion can be associated with zoom lens correction or functionally manipulated to obtain a final map of all projector distortions. In the factory calibration, the lens correction is calculated and stored only once. Subsequently, use the camera in the field to perform keystone correction with lens correction.

25 shows a case of projecting on the curved screen 251. In order to determine the map of the curved screen 241 including the shape and the distance, a two-dimensional image pattern such as a checkered image pattern is projected onto the screen. The projected image is captured by the camera. The contrast produced by each line of the check Mooney pattern is calculated at the processor (processor). By constantly changing the focus, the optimum contrast at each point in the pattern is obtained as a function of focus length. Thus, the surface map of the curved screen 251 is determined. The accuracy and detail of the map depends on the number of focal lengths used and the complexity of the projected pattern. This method can also be used to find the angle of the camera and eventually the angle of the projector to the screen at each point. The processor once calculates the distortion factors related to the shape, size, and angle of the screen at each point, then calculates the warp transform or uses the appropriate warp transform already stored. When such warp transform is applied to the input image data, an image that matches the characteristics of the screen without visual distortion is produced.

FIG. 26 illustrates a case where the screen is waved, and the shape and relative direction of the wavy screen may be determined at all points using the technique described with reference to FIG. 25. In this case, if the screen used for the display device is irregular, all of them are applied. When the map of the screen is ready, the processor uses this map to warp the input image. When warping the input image, the projected image matches the characteristics of the screen without visual distortion.

1 is a block diagram of an example of an automatic calibration-correction system;

2 is a view when the screen is curved;

3 shows a case where the geometric distortion is overflow, underflow and mismatch;

4 shows an example of calibration of an image test pattern;

5 shows various coordinate spaces associated with calibration shapes;

6 is a flowchart of a calibration data generator;

7 is an example of optimizing scale and origin;

8 is a flow chart of a multicolor calibration data generator;

9 is a setup diagram for color non-uniformity correction;

10 is a flowchart of a calibration data generator for color non-uniformity correction;

11 is a flowchart of a warp data generator;

12 shows a patch partition for display modification;

13 is a block diagram of a digital warping unit;

14 is a setup diagram for determining the shape and relative direction of the screen;

15 is a test pattern that is not in focus;

16 is a focused test pattern;

17 is a fragmented view of a calibration system with multiple projectors and one curved screen;

FIG. 18 shows a focal plane of the calibration system of FIG. 17; FIG.

19 shows an example of a focusing technique for minimizing a distance function;

20 shows an example of optimizing the focus by adjusting the projector position in another calibration system with multiple projectors and one curved screen;

21 is a schematic diagram of a calibration system using multiple cameras;

22 is an example of Real Projection TeleVision (RPTV) in which a calibration system is installed to perform self-calibration and dynamic distortion correction.

23 is a schematic diagram of a calibration system with multiple projectors and multiple sensors.

24 is a schematic diagram of a calibration system using actual edges and boundaries of the screen;

25 is a schematic diagram of a calibration system using focusing techniques to determine the shape of a curved screen;

26 is a schematic diagram of a calibration system using focusing technology to determine the shape of a wavy screen.

Claims (26)

In display calibration systems for display devices with screens: A sensor for sensing at least one of a shape, a size, a boundary, and a direction of a screen; And And a processor coupled to the sensor and calculating a characteristic of the display device based on the information detected by the sensor. The display calibration system of claim 1, wherein the sensor detects a test image displayed on a screen, and the processor calculates display distortion based on the detected test image and characteristics of the display device. The display calibration system of claim 2, wherein the processor generates a precompensation map based on display distortion, and when the precompensation map is applied to the input image data before display, the display image of the screen is free from distortion. . 4. A display calibration system according to claim 3, wherein the display distortion is distortion that changes over time and the display device is dynamically calibrated to precompensate such variable distortion. 4. The method of claim 3, wherein at least one of an overflow state in which the displayed image is larger than the screen, an underflow state in which the displayed image is smaller than the screen, and a mismatch state in which a portion of the displayed image is off the screen and the remainder is in the screen. Is modified by the processor. 4. A display calibration system according to claim 3, wherein the display device is a rear projection display device having a housing and a display calibration system is disposed within the housing. 4. The display calibration system of claim 3, wherein the sensor senses at least one of luminance information and color information, and the processor precompensates at least one of luminance nonuniformity and color nonuniformity. 4. A display calibration system according to claim 3, wherein the display device comprises an optical element with additional distortion, and the processor precompensates both the additional distortion and the display distortion in association with the additional distortion. The display calibration system according to claim 2, wherein the display distortion includes at least one of geometric distortion, optical distortion, misconvergence, misalignment, and lateral chromatic aberration. The display calibration system of claim 1, wherein the sensor senses a distance to various points on the screen, and the processor calculates a relative position and orientation of the screen based on the distance. The apparatus of claim 2, wherein the sensor detects each part of the test image on the screen at different focal lengths, and the processor determines the highest contrast of each part of the test image and then distances to each part of the screen based on the contrast. Display calibration system, characterized in that for calculating the shape and relative orientation of the screen. 3. A display calibration system according to claim 2, wherein the sensor has sensor distortion, and the processor calculates the sensor distortion and takes the sensor distortion into account when calculating the display distortion. 13. A display calibration system according to claim 12, wherein the sensor distortion is due to a sensor having an axis that is not parallel to the direction perpendicular to the screen. The sensor of claim 2, wherein the sensor is a plurality of image sensors arranged at various positions known by a processor, and the processor compares various test images sensed by each sensor and compares each detected image and position of each sensor. Display calibration system, characterized in that for calculating the display distortion based on. The display calibration system of claim 2, wherein the image sensor detects information about a test image having four markers on the screen, and the processor calculates keystone distortion based on the detected information. 4. The display calibration system of claim 3, wherein the sensor senses at least one of luminance information and color information, and the processor corrects at least one of luminance and color irregularities caused by the precompensation map. . In display calibration systems for display devices with screens: A sensor for detecting information in a test image displayed on the screen; And And a processor connected to the sensor and calculating a display distortion based on the information detected by the sensor to create a precompensation map to compensate for the display distortion. A display calibration system, characterized in that the precompensation map is implemented with a surface function that removes distortion from the image displayed on the screen when the precompensation map is applied to the input image data before display. 18. The display calibration system of claim 17 wherein the processor generates a surface function that associates several distortions and precompensates the associated distortions. 18. A display calibration system according to claim 17 wherein the surface function is polynomial. 18. The display calibration system of claim 17 wherein the processor adjusts a surface function to compensate for an overscan or underscan condition. In display calibration systems for display devices with screens: A sensor for detecting information in a test image displayed on the screen; And It is connected to the sensor, calculates the display distortion based on the information detected by the sensor, divides the screen into a number of patches according to the degree of display distortion, and generates a pre-compensation map for the display distortion for each patch, these dictionary And a processor for removing distortion from the image displayed on the screen when the compensation map is applied to the input image data before the display. In display calibration systems for display devices with screens: A sensor for detecting color information from a test image displayed on the screen; And Connected to the sensor, calculates color unevenness based on the information detected by the sensor and generates a color correction map for the color components. Display calibration system comprising a; processor to eliminate. In display calibration systems for display devices with screens: A sensor for detecting information in an individual color component test image displayed on a screen; And Connected to the sensor and the display device, independently calculating geometric display distortions of the color components based on the information detected by the sensor, and independently generating a precompensation map for the color components, so that the precompensation map is displayed before the display. And a processor that removes color-based geometric distortion from the image displayed on the screen when applied to the data. In the display calibration method used for projection devices with curved screens: Projecting respective portions of the image to corresponding portions of the curved screen using multiple projectors; And And focusing the respective portions of the image on the corresponding portions of the curved screen to form the entire image on the screen with optimal focus. The display calibration method according to claim 24, wherein the projection axes of each projector are perpendicular to the corresponding portions of the curved screen so that the projectors are independently placed and orientated so that the focus is optimized and the geometric distortion is minimized. In the display calibration method used for projection devices with curved screens: Measuring various distances from the curved screen to the focal plane of the projected image; And Displacing the focal plane until the focal point is optimized by minimizing the function of the distance.

Images