CN101482690B - Self-adapting template big-screen brightness emendation method with multi-projector combined display - Google Patents

Self-adapting template big-screen brightness emendation method with multi-projector combined display Download PDF

Info

Publication number
CN101482690B
CN101482690B CN2009100762061A CN200910076206A CN101482690B CN 101482690 B CN101482690 B CN 101482690B CN 2009100762061 A CN2009100762061 A CN 2009100762061A CN 200910076206 A CN200910076206 A CN 200910076206A CN 101482690 B CN101482690 B CN 101482690B
Authority
CN
China
Prior art keywords
brightness
projecting apparatus
value
image
screen
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN2009100762061A
Other languages
Chinese (zh)
Other versions
CN101482690A (en
Inventor
秦开怀
周艳霞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Priority to CN2009100762061A priority Critical patent/CN101482690B/en
Publication of CN101482690A publication Critical patent/CN101482690A/en
Application granted granted Critical
Publication of CN101482690B publication Critical patent/CN101482690B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Controls And Circuits For Display Device (AREA)
  • Transforming Electric Information Into Light Information (AREA)
  • Image Processing (AREA)
  • Picture Signal Circuits (AREA)

Abstract

A method for correcting self-adapting formwork large-screen brightness displayed in combination by multiple projectors belongs to the large-screen image display field, characterized in that, the method comprises: generating a formwork; obtaining the attenuated image by multiplying the brightness value of the original image in the computer and the value in the formwork; then eliminating the high bright band in a stacked area by the mode of outputting the image to the projector. The invention provides a plurality of generated formworks in advance for the images with different brightness to select a formwork with optimum effect. The invention can self-adapting select the appropriate formwork to perform the brightness correction according to the real-time allocated brightness values of the sub images of each projector, thereby the images with various brightness all can obtain better brightness correcting effects.

Description

多投影仪组合显示的自适应模板大屏幕亮度校正方法 Adaptive template large-screen brightness correction method for multi-projector combined display

技术领域technical field

本发明涉及多投影仪组合显示的亮度校正,实现多投影仪组合显示的大屏幕高分辨率显示。The invention relates to the brightness correction of multi-projector combination display, and realizes the large-screen high-resolution display of multi-projector combination display.

背景技术Background technique

现有的多投影仪大屏幕拼接显示的亮度校正,都是用多台投影仪在大屏幕上拼接显示一个大的画面,相邻的投影仪之间有交叠区域,也就是对应的投影图像有交叠区域,可以通过软件或者硬件的方法消除交叠区域的亮带,实现多投影仪大屏幕的无缝拼接显示。The brightness correction of the existing multi-projector large-screen splicing display uses multiple projectors to splicing and displaying a large picture on the large screen. There is an overlapping area between adjacent projectors, that is, the corresponding projected image There are overlapping areas, and the bright bands in the overlapping areas can be eliminated by software or hardware methods, so as to realize the seamless splicing display of multi-projector large screens.

硬件方法是利用遮光板遮住投影仪多余的光,使得在交叠区域只有一个投影仪的光能投影过去,其他投影仪的光都被挡住。这种方法的好处是被遮住的地方就一点光也漏不过去。缺点是要手动控制遮光板的位置,费时费力,对于很多个投影仪的组合不适用。The hardware method is to use a shading plate to cover the excess light of the projectors, so that only the light of one projector can be projected in the overlapping area, and the light of other projectors is blocked. The advantage of this method is that no light can leak through the covered place. The disadvantage is that it is time-consuming and labor-intensive to manually control the position of the visor, and it is not suitable for the combination of many projectors.

软件方法是通过在帧缓存中对投影仪图像的亮度进行改变,来达到亮度一致。现有的软件方法主要有边缘融合、亮度一致和亮度反馈三种方法。The software method is to achieve brightness consistency by changing the brightness of the projector image in the frame buffer. The existing software methods mainly include edge blending, luminance consistency and luminance feedback.

边缘融合采用一个函数作为模板对投影仪的重叠区域进行过渡处理。这个函数有的是线性函数,有的是余弦函数,有的是幂函数。这个方法的优点是简单,易于计算;缺点是无论采用哪种函数都不能使所有的图像的重叠区过渡均匀。Edge blending uses a function as a template to transition the overlapping areas of the projectors. Some of this function are linear functions, some are cosine functions, and some are power functions. The advantage of this method is that it is simple and easy to calculate; the disadvantage is that no matter which function is used, it cannot make the overlapping regions of all images transition uniformly.

亮度一致方法,是首先用相机分别获取每个投影仪显示白色图像的亮度分布,计算交叠区域的亮度,即每个投影仪在该位置的亮度之和;其次,每个投影仪生成一个亮度表LAM,亮度表中每一点的值为Lmin/L,其中,Lmin为所有投影仪的最小亮度,L为这一点的实际亮度;最后,对帧缓存中的每幅图像都乘以亮度表中的模板因子,再进行输出,从而达到亮度校正的目的。因为该方法中假设投影仪的亮度响应曲线是线性的,而实际上投影仪的亮度响应曲线是非线性的,所以又进行了非线性校正。非线性校正就是得到投影仪的亮度响应曲线,用其反函数作成亮度查找表,在投影仪输出之前,先查表,再输出。亮度响应曲线可以用亮度仪获取投影各个亮度的输出来生成,也可以用HDR方法,即用数码相机得到投影仪的亮度响应曲线。亮度一致方法的优点是在理论上得到整个屏幕的亮度一致;缺点一是要用到亮度仪比较昂贵,并且要对每个投影仪255个亮度进行测量,费时费力;缺点二是整个投影屏幕为了和最小的输出保持一致,使整个图像的亮度变小,浪费了很多亮度范围;缺点三是该方法假设交叠区域的亮度是每个投影仪的亮度之和、相机得到的亮度是真实的和均匀的,但这些都是不成立的。The brightness consistency method is to first use the camera to obtain the brightness distribution of the white image displayed by each projector, and calculate the brightness of the overlapping area, that is, the sum of the brightness of each projector at this position; secondly, each projector generates a brightness Table LAM, the value of each point in the brightness table is Lmin/L, where Lmin is the minimum brightness of all projectors, and L is the actual brightness at this point; finally, each image in the frame buffer is multiplied by the value in the brightness table The template factor, and then output, so as to achieve the purpose of brightness correction. Because the method assumes that the brightness response curve of the projector is linear, but actually the brightness response curve of the projector is nonlinear, so nonlinear correction is performed. Non-linear correction is to obtain the brightness response curve of the projector, and use its inverse function to make a brightness look-up table. Before the projector outputs, first look up the table and then output. The luminance response curve can be generated by obtaining the output of each luminance of the projection with a luminance meter, or the HDR method can be used, that is, the luminance response curve of the projector can be obtained with a digital camera. The advantage of the brightness consistent method is that the brightness of the entire screen can be obtained in theory; the disadvantage is that it is expensive to use a brightness meter, and it is time-consuming and laborious to measure 255 brightness of each projector; the disadvantage is that the entire projection screen is for Consistent with the minimum output, the brightness of the entire image is reduced, wasting a lot of brightness range; the third disadvantage is that the method assumes that the brightness of the overlapping area is the sum of the brightness of each projector, and the brightness obtained by the camera is real and Evenly, but these are not true.

亮度反馈的方法是根据相机获取的图像和参考图像的比较来修改图像的模板因子,过亮则减小,过黑则增大。参考图像的生成,非交叠区域直接用的相机图像,交叠区域通过非交叠区域线性插值获得。通过多次反馈得到最后亮度均匀过渡的模板。这种方法的优点是在理论上得到亮度的均匀过渡。缺点是受相机误差、环境光、屏幕的平滑度干净度的影响很大。The method of brightness feedback is to modify the template factor of the image according to the comparison between the image acquired by the camera and the reference image. If it is too bright, it will decrease, and if it is too dark, it will increase. For the generation of reference images, non-overlapping areas are directly used in camera images, and overlapping areas are obtained by linear interpolation in non-overlapping areas. The final template with uniform brightness transition is obtained through multiple feedbacks. The advantage of this method is that in theory a uniform transition of brightness is obtained. The disadvantage is that it is greatly affected by camera errors, ambient light, and the smoothness and cleanliness of the screen.

发明内容Contents of the invention

本发明的目的在于消除多投影仪组合显示交叠区域的亮带,实现多投影仪无缝拼接的大屏幕高分辨率显示。The purpose of the present invention is to eliminate the bright bands in the overlapping area displayed by the multi-projector combination, and realize the large-screen high-resolution display of the seamless splicing of the multi-projectors.

本发明的特征在于,依次含有以下的步骤:The present invention is characterized in that it contains the following steps in sequence:

步骤(1):初始化;Step (1): initialization;

步骤(1.1):为所述大屏幕建立一个直角坐标系,原点在该屏幕的左下角;Step (1.1): establishing a Cartesian coordinate system for the large screen, the origin is at the lower left corner of the screen;

步骤(1.2):用m×n个DLP投影仪组成一个投影仪矩阵,m为行,n为列;每两个所述DLP投影仪用一个客户端计算机进行控制;Step (1.2): form a projector matrix with m×n DLP projectors, m is a row, and n is a column; every two described DLP projectors are controlled by a client computer;

步骤(1.3):所述的每个客户端计算机把在每个投影仪空间均匀分布的8行8列等间距的特征点组成的图像分别输出到自己控制的DLP投影仪中,用数码相机在相同的位置对所述每个DLP投影仪的输出分别进行拍照,得到所述每个DLP投影仪在屏幕空间的特征点图像;Step (1.3): Each client computer outputs the image composed of 8 rows and 8 columns of equidistant feature points that are uniformly distributed in each projector space to the DLP projector controlled by itself, and uses a digital camera to Taking pictures of the output of each of the DLP projectors at the same position to obtain the feature point image of each of the DLP projectors in the screen space;

步骤(1.4):服务器对屏幕空间的所述每个特征点图像进行图像处理得到所述每个DLP投影仪在屏幕空间的特征点,根据特征点位置得到所述每个DLP投影仪在屏幕空间的初始显示区域,并且把每个投影仪空间的点和屏幕空间的点建立一一对应关系,作为每个投影仪空间和屏幕空间之间坐标转换的依据;Step (1.4): The server performs image processing on each feature point image of the screen space to obtain the feature points of each DLP projector in the screen space, and obtains the feature points of each DLP projector in the screen space according to the position of the feature points. , and establish a one-to-one correspondence between each projector space point and screen space point as the basis for coordinate transformation between each projector space and screen space;

步骤(1.5):所述服务器根据所述每个DLP投影仪在屏幕空间的初始显示区域计算一个最大的可显示矩形RECT,每个DLP投影仪在大屏幕上实际要显示的区域是通过该投影仪的初始显示区域与最大矩形RECT求交得到的,在每个DLP投影仪屏幕空间的实际显示区域等间距生成8行8列的点,把这些屏幕空间的点转换到投影仪空间中,并生成网格mesh-i;Step (1.5): The server calculates a maximum displayable rectangle RECT according to the initial display area of each DLP projector in the screen space, and the actual display area of each DLP projector on the large screen is through the projection It is obtained by intersecting the initial display area of the projector with the largest rectangle RECT, generating 8 rows and 8 columns of points at equal intervals in the actual display area of each DLP projector screen space, converting these screen space points into the projector space, and generate grid mesh-i;

步骤(1.6):所述客户端计算机把分配给自己控制的DLP投影仪的图像映射到所述网格mesh-i相对应的几何位置上,其余位置用黑色清除,计算屏幕空间上所述投影仪的重叠区域的位置信息,并映射到投影仪空间中;Step (1.6): The client computer maps the image assigned to the DLP projector controlled by itself to the geometric position corresponding to the grid mesh-i, the remaining positions are cleared with black, and the projection on the screen space is calculated The position information of the overlapping area of the projector is mapped to the projector space;

步骤(2):按以下步骤计算针对不同亮度值图像的模板值;Step (2): Calculate the template value for different brightness value images according to the following steps;

步骤(2.1):所述客户端计算机按下述线性函数计算线性过渡下的模板值m线性Step (2.1): The client computer calculates the template value m linear under the linear transition according to the following linear function:

m线性1=d1/(d1+d2),m linear 1 =d1/(d1+d2),

m线性2=d2/(d1+d2), mlinear2 =d2/(d1+d2),

其中,m线性1为左边DLP投影仪的模板值,m线性2为右边DLP投影仪的模板值,d1为所述重叠区部分中一点到所述左边DLP投影仪边界的距离,d2为所述重叠区部分中所述的一点到所述右边DLP投影仪边界的距离;Wherein, m linear 1 is the template value of the left DLP projector, m linear 2 is the template value of the right DLP projector, d1 is the distance from a point in the overlapping area to the boundary of the left DLP projector, and d2 is the the distance from the point described in the overlap area section to the border of the right DLP projector;

步骤(2.2):所述客户端计算机按下述余弦函数计算余弦过渡下的模板值m余弦Step (2.2): the client computer calculates the template value m cosine under the cosine transition according to the following cosine function,

m余弦1=cos(d1/(d1+d2)),m cosine 1 =cos(d1/(d1+d2)),

m余弦2=1-cos(d1/(d1+d2));m cosine 2 = 1-cos(d1/(d1+d2));

其中,m余弦1为左边DLP投影仪的模板值,m余弦2为右边DLP投影仪的模板值,d1为所述重叠区部分中一点到所述左边DLP投影仪边界的距离,d2为所述重叠区部分中所述的一点到所述右边DLP投影仪边界的距离;Wherein, m cosine 1 is the template value of the left DLP projector, m cosine 2 is the template value of the right DLP projector, d1 is the distance from a point in the overlapping area to the boundary of the left DLP projector, and d2 is the the distance from the point described in the overlap area section to the border of the right DLP projector;

步骤(2.3):对所述步骤(2.1)和所述步骤(2.2)得到的结果再用下述的非线性γ校正函数进行处理,Step (2.3): The results obtained in the step (2.1) and the step (2.2) are processed with the following nonlinear gamma correction function,

mγ1=(m1)1/γm γ1 = (m 1 ) 1/γ ;

mγ2=(m2)1/γm γ2 = (m 2 ) 1/γ ;

其中,m1对应所述线性过渡或所述余弦过渡下左边投影仪的模板值,m2对应所述线性过渡或所述余弦过渡下右边投影仪的模板值,γ从1.6到4.8,每隔0.4取一个值,共生成9个非线性γ函数,对应线性过渡和余弦过渡,共生成18个亮度校正模板;Wherein, m 1 corresponds to the template value of the left projector under the linear transition or the cosine transition, m 2 corresponds to the template value of the right projector under the linear transition or the cosine transition, and γ is from 1.6 to 4.8, every Take a value of 0.4 to generate 9 nonlinear gamma functions in total, corresponding to linear transition and cosine transition, and generate 18 brightness correction templates in total;

步骤(2.4):所述客户端计算机在亮度值140和255之间每隔20个亮度取一个亮度值L:140、160、180、200、220、240、255,再取一个亮度值100,共8个亮度值,用这些亮度值作为作为RGB值,构成8个不同亮度的纯色图像;Step (2.4): The client computer takes a brightness value L every 20 brightness values between brightness values 140 and 255: 140, 160, 180, 200, 220, 240, 255, and then takes a brightness value of 100, There are 8 brightness values in total, and these brightness values are used as RGB values to form 8 pure color images with different brightness;

步骤(2.5):所述服务器把所述步骤(2.4)中得到的一个图像,分配给所述客户端计算机;Step (2.5): The server distributes an image obtained in the step (2.4) to the client computer;

步骤(2.6):所述客户端计算机把传递过来的图像的重叠部分的亮度值分别乘以所述步骤(2.3)得到的一个模板中的亮度校正值,得到各点在亮度校正后的亮度值,以使所述重叠部分的原始图像进行衰减,再输出到的各个DLP投影仪中;Step (2.6): The client computer multiplies the brightness values of the overlapped parts of the transmitted image by the brightness correction value in a template obtained in the step (2.3), respectively, to obtain the brightness values of each point after brightness correction , so that the original image of the overlapping part is attenuated, and then output to each DLP projector;

步骤(2.7):对所述步骤(2.3)得到的18个模板中的每一个,重复所述步骤(2.6),通过人眼观察的方法选出针对所述步骤(2.5)的图像效果最好的一个模板;Step (2.7): For each of the 18 templates obtained in the step (2.3), repeat the step (2.6), and select the image for the step (2.5) by the method of human eye observation with the best effect a template for

步骤(2.8):对所述步骤(2.4)得到的8副图像中的每一副,重复所述步骤(2.5)、所述步骤(2.6)和所述步骤(2.7),从而得到针对8个不同亮度的8个最优模板;Step (2.8): for each pair in the 8 images that described step (2.4) obtains, repeat described step (2.5), described step (2.6) and described step (2.7), thereby obtain for 8 images 8 optimal templates with different brightness;

步骤(3):按以下步骤进行所述m×n个DLP投影仪组合显示的自适应模板大屏幕上所述重复区域的亮度校正;Step (3): Carry out the brightness correction of the repeated area on the large screen of the self-adaptive template displayed by the combination of the m×n DLP projectors in the following steps;

步骤(3.1):所述服务器采集一副图像,把要显示的图像,分配给所述客户端计算机;Step (3.1): the server collects an image, and distributes the image to be displayed to the client computer;

步骤(3.2):所述客户端计算机计算传过来的子图像中每一点的亮度值L1iStep (3.2): The client computer calculates the brightness value L1 i of each point in the sub-image passed over,

L1i=0.299*R+0.587*G+0.114*B,L1 i =0.299*R+0.587*G+0.114*B,

其中,R、G、B分别是所述点的R、G、B通道的像素值;Wherein, R, G, B are respectively the pixel values of the R, G, and B channels of the point;

步骤(3.3):所述客户端计算机计算所述子图像中所有点亮度的平均值L2,Step (3.3): the client computer calculates the average value L2 of the brightness of all points in the sub-image,

LL 22 == (( ΣΣ ii == 00 nno LL 11 ii )) // nno ,,

其中,L1i为所述步骤(3.2)中计算得到的所述子图像每一点的亮度值,n为所述子图像中点的个数;Wherein, L1 i is the brightness value of each point of the sub-image calculated in the step (3.2), and n is the number of points in the sub-image;

步骤(3.4):所述客户端计算机计算所述子图像的亮度分布,存放在数组num中,num[i]表示所述子图像中亮度值等于序号i的点的个数,序号i的取值是0到255;数组num中的最大值NL对应的序号L3,即为所述子图像的亮度峰值;Step (3.4): The client computer calculates the brightness distribution of the sub-image, and stores it in the array num, num[i] represents the number of points whose brightness value is equal to the serial number i in the sub-image, and the value of the serial number i is The value is 0 to 255; the sequence number L3 corresponding to the maximum value NL in the array num is the peak brightness of the sub-image;

步骤(3.5):所述客户端计算机计算所述子图像的亮度值L,当所述步骤(3.4)中得到的NL的值大于所述子图像中点的个数的60%时,所述子图像的亮度值L等于所述步骤(3.4)中得到的所述子图像的亮度峰值L3,否则所述子图像的亮度值L等于所述步骤(3.3)中得到的所述子图像的的亮度平均值L2;Step (3.5): the client computer calculates the brightness value L of the sub-image, and when the value of NL obtained in the step (3.4) is greater than 60% of the number of points in the sub-image, the The brightness value L of the sub-image is equal to the brightness peak value L3 of the described sub-image obtained in the step (3.4), otherwise the brightness value L of the sub-image is equal to that of the described sub-image obtained in the step (3.3). Brightness average L2;

步骤(3.6):所述客户端计算机自动选择一个与所述步骤(3.5)得到的所述子图像的亮度值L最接近的亮度的模板作为校正模板;Step (3.6): The client computer automatically selects a template with the brightness closest to the brightness value L of the sub-image obtained in the step (3.5) as a correction template;

步骤(3.7):所述客户端计算机把所述步骤(3.6)得到的自适应模板亮度校正值乘以所述重叠部分的亮度值,得到各点在亮度校正后的亮度值,以使所述重叠部分的原始图像进行衰减,再输出到所述子图像对应的DLP投影仪中;Step (3.7): The client computer multiplies the adaptive template brightness correction value obtained in the step (3.6) by the brightness value of the overlapping part to obtain the brightness value of each point after brightness correction, so that the The original image of the overlapping part is attenuated, and then output to the DLP projector corresponding to the sub-image;

步骤(3.8):对所述的各个DLP投影仪重复所述步骤(3.2)到所述步骤(3.7);Step (3.8): repeating the step (3.2) to the step (3.7) for each of the DLP projectors;

步骤(3.9):所述服务器在一定的时间间隔后,重复所述步骤(3.1)到所述步骤(3.8)。Step (3.9): the server repeats the step (3.1) to the step (3.8) after a certain time interval.

本发明的特征在于,把所述的DLP投影仪按行和列分成两个序列,在所述的行序列中,各自序号相邻的两个投影仪的投影存在水平方向的重叠,在所述的列序列中,各自序号相邻的两个投影仪的投影存在垂直方向的重叠。The present invention is characterized in that the DLP projectors are divided into two sequences by rows and columns, and in the row sequences, the projections of two projectors with adjacent serial numbers overlap in the horizontal direction. In the column sequence of , the projections of two projectors with adjacent serial numbers overlap vertically.

本发明的特征在于,所述的所有图像是实时显示的,其中包括视频图像。The present invention is characterized in that all the images are displayed in real time, including video images.

本发明的特征在于,m=2,n=2。The present invention is characterized in that m=2 and n=2.

本发明与现有的技术相比有以下的优点:Compared with the prior art, the present invention has the following advantages:

1.根据要显示的图像不同而自适应采用不同的模板进行亮度校正,使得每种图像都有一个较好的亮度校正效果,从而获得一个好的总体效果。1. Adaptively adopt different templates for brightness correction according to different images to be displayed, so that each image has a better brightness correction effect, thereby obtaining a good overall effect.

2.采用边缘融合的方法生成模板,简单易操作,避免了使用昂贵的亮度计,也避免了使用HDR方法产生很大的工作量。2. The edge fusion method is used to generate templates, which is simple and easy to operate, avoiding the use of expensive luminance meters and the heavy workload of using HDR methods.

3.自适应模板亮度校正的效果不受屏幕的污点、划痕影响,也不受环境光、特殊屏幕产生的光带的影响。3. The effect of adaptive template brightness correction is not affected by stains and scratches on the screen, nor is it affected by ambient light and light bands produced by special screens.

附图说明Description of drawings

图1是校正前投影仪显示区域的示意图;Fig. 1 is a schematic diagram of the display area of the projector before correction;

图2自适应模板亮度校正模块;Fig. 2 adaptive template brightness correction module;

图3是自适应模板亮度校正模块的流程图;Fig. 3 is the flowchart of adaptive template brightness correction module;

图4初始化模块的过程示意图;The schematic diagram of the process of the initialization module of Fig. 4;

图5初始化模块流程图;Fig. 5 initialization module flowchart;

图6是模板生成模块的流程图;Fig. 6 is the flowchart of template generating module;

图7是边缘融合的示意图;Fig. 7 is a schematic diagram of edge fusion;

图8是线性过渡的示意图;Figure 8 is a schematic diagram of a linear transition;

图9是余弦过渡的示意图。Figure 9 is a schematic diagram of a cosine transition.

具体实施方式Detailed ways

本发明通过以下技术方案予以实现:用m行n列个投影仪拼接显示一副大的画面,相邻投影仪的图像内容都是有重叠的,重叠的区域形成亮带。通过生成模板,在计算机中用模板中的值对原始图像进行衰减再输出到投影仪的方式消除交叠区的高亮带。首先,预先生成了18个模板以供不同亮度的图像来选择效果最好的模板;其次,给每个投影仪分别输出8个不同亮度的纯色图像,挑选出8个最优的模板;最后,根据要显示图像的内容不同而自适应采用不同的模板进行亮度校正,根据实时分配给每个投影仪的图像的亮度值,选择亮度最接近的模板进行亮度校正,从而使各种亮度的图像都得到一个较好的亮度校正效果。The present invention is realized through the following technical solutions: m rows and n columns of projectors are used to splice and display a large picture, the image contents of adjacent projectors are all overlapped, and the overlapped areas form bright bands. By generating a template, the value in the template is used to attenuate the original image in the computer and then output to the projector to eliminate the highlight band in the overlapping area. First, 18 templates are pre-generated for images with different brightness to select the template with the best effect; secondly, 8 solid-color images with different brightness are output to each projector, and 8 optimal templates are selected; finally, According to the content of the image to be displayed, different templates are adaptively used for brightness correction. According to the brightness value of the image assigned to each projector in real time, the template with the closest brightness is selected for brightness correction, so that images of various brightness are all the same. Get a better brightness correction effect.

本发明中自适应选择模板的根据是投影仪的子图像的亮度平均值或亮度峰值,当子图像中亮度等于亮度峰值的点的个数大于子图像中点的个数的60%时,用亮度峰值,否则用亮度平均值。In the present invention, the basis of self-adaptive selection template is the average value or peak value of the brightness of the sub-image of the projector. When the number of points whose brightness is equal to the peak value of the brightness in the sub-image is greater than 60% of the number of points in the sub-image, use Brightness peak, otherwise use brightness average.

本发明中自适应选择模板的规则是亮度值与投影仪的子图像的亮度平均值或亮度峰值最接近的模板。The rule for adaptively selecting templates in the present invention is the template whose brightness value is closest to the brightness average value or brightness peak value of the sub-image of the projector.

本发明中的模板是用边缘融合的方法生成的。不同的模板用不同的函数来生成。预先生成了18个模板,所有的模板都是用不同的γ函数组合线性函数或者余弦函数生成的。The template in the present invention is generated by edge fusion. Different templates are generated using different functions. 18 templates are pre-generated, and all the templates are generated by combining linear functions or cosine functions with different gamma functions.

本发明在亮度值140~255之间每隔20个亮度生成一个模板,亮度低于140的只取一个亮度值100的模板,因为亮度低的图像对模板不挑剔。最后针对8个亮度得到8个最优模板。The present invention generates a template every 20 brightness values between 140 and 255, and only takes a template with a brightness value of 100 if the brightness is lower than 140, because images with low brightness are not picky about the template. Finally, 8 optimal templates are obtained for 8 luminances.

图1是四个投影仪11、12、13、14在大屏幕上的显示区域15、16、17、18的示意图。为了让四个投影仪拼接显示一副大的画面,每个投影仪只显示大画面的一部分图像,子图像内容相邻的投影仪之间的投影是重叠的,这些交叠区域会出现高亮带。本发明的目的就是为了消除这些高亮带,实现多投影仪的无缝拼接大屏幕显示。FIG. 1 is a schematic diagram of display areas 15 , 16 , 17 , 18 of four projectors 11 , 12 , 13 , 14 on a large screen. In order for the four projectors to splice and display a large picture, each projector only displays a part of the image of the large picture, and the projections between the adjacent projectors of the sub-image content overlap, and these overlapping areas will be highlighted bring. The purpose of the present invention is to eliminate these highlight bands and realize the seamless splicing of multiple projectors for large-screen display.

本发明主要有3个模块:初始化模块、模板生成模块和自适应模板亮度校正模块。The present invention mainly has three modules: an initialization module, a template generation module and an adaptive template brightness correction module.

图2是自适应模板亮度校正模块的过程示意图。图中有一台服务器和两个计算机客户端,每个客户端计算机控制两台投影仪,共有四台投影仪21、22、23、24。图3是自适应模板亮度校正模块的流程图。服务器采集一副图像,要显示的子图像内容分配给两个客户端计算机,客户端计算机计算子图像的亮度值。Fig. 2 is a schematic diagram of the process of the adaptive template brightness correction module. There is a server and two computer clients in the figure, each client computer controls two projectors, and there are four projectors 21, 22, 23, 24 in total. Fig. 3 is a flowchart of an adaptive template brightness correction module. The server collects an image, and the content of the sub-image to be displayed is distributed to two client computers, and the client computer calculates the brightness value of the sub-image.

子图像中某一点的亮度值L1,The brightness value L1 of a certain point in the sub-image,

L1=0.299*R+0.587*G+0.114*BL1=0.299*R+0.587*G+0.114*B

其中,R、G、B分别是这一点R、G、B通道的像素值。Among them, R, G, and B are the pixel values of the R, G, and B channels at this point, respectively.

计算图像的亮度平均值,并且统计亮度分布。Calculate the average brightness of the image, and count the brightness distribution.

子图像的亮度值L等于子图像中所有点亮度的平均值或峰值,当子图像中亮度等于亮度峰值的点的个数大于子图像中点的个数的60%时,用亮度峰值,否则用亮度平均值;The brightness value L of the sub-image is equal to the average or peak value of the brightness of all points in the sub-image. When the number of points whose brightness is equal to the brightness peak value in the sub-image is greater than 60% of the number of points in the sub-image, use the brightness peak value, otherwise Use the brightness average;

客户端计算机自动选择一个与亮度值L最接近的亮度的模板作为校正模板;客户端计算机把得到的自适应模板亮度校正值乘以所述重叠部分的亮度值,得到各点在亮度校正后的亮度值,以使重叠部分的原始图像进行衰减,再输出到所述的各个DLP投影仪中。The client computer automatically selects a template with the brightness closest to the brightness value L as the correction template; the client computer multiplies the obtained adaptive template brightness correction value by the brightness value of the overlapping part to obtain the brightness correction value of each point The brightness value is used to attenuate the original image of the overlapped part, and then output to each of the DLP projectors.

亮度校正后四个投影仪拼接显示一个矩形区域,如图2中黑色粗框。每个投影仪在屏幕上的实际显示区域如图2中的25、26、27、28所示,是原来的显示区域与黑色粗框的相交区域。After brightness correction, the four projectors spliced to display a rectangular area, as shown in the thick black frame in Figure 2. The actual display area of each projector on the screen is shown as 25, 26, 27, 28 in Fig. 2, which is the intersection area between the original display area and the thick black frame.

图4是初始化模块的过程示意图。图5是初始化模块的流程图。用m×n个DLP投影仪组成一个投影仪矩阵,相邻的两个投影仪用一个客户端计算机进行控制。让每个投影仪分别输出8行8列的等间距的特征点,用数码相机进行拍照,得到每个投影仪的特征点图像,对获取的图像进行图像处理,把屏幕空间的点和投影仪空间的点建立一一对应关系,作为屏幕空间和投影仪空间转换的依据。通过与最大矩形RECT求交的方法得到投影仪在大屏幕上要显示的区域,把在大屏幕上要显示的区域映射到投影仪空间中,并在投影仪空间建立网格mesh-i。把分配给投影仪的图像映射到网格mesh-i对应的几何位置上,其他位置用黑色清除。在屏幕空间上得到投影仪重叠区域的位置信息,并映射到投影仪空间中。Fig. 4 is a schematic diagram of the process of initializing the module. Figure 5 is a flowchart of the initialization module. A projector matrix is composed of m×n DLP projectors, and two adjacent projectors are controlled by a client computer. Let each projector output 8 rows and 8 columns of equally spaced feature points, take pictures with a digital camera, get the feature point image of each projector, perform image processing on the acquired image, and compare the screen space points with the projector The points in the space establish a one-to-one correspondence, which is used as the basis for the conversion between the screen space and the projector space. Obtain the area to be displayed by the projector on the large screen by intersecting with the largest rectangle RECT, map the area to be displayed on the large screen to the projector space, and establish a grid mesh-i in the projector space. Map the image assigned to the projector to the geometric position corresponding to the grid mesh-i, and clear other positions with black. Get the location information of the overlapping area of the projector in the screen space, and map it to the projector space.

图6是模板生成模块的流程图。在每个投影仪的交叠区域,用18个不同的函数生成模板,后面对模板的计算方法具体阐述。最后,服务器给计算机客户端分别发送8个不同亮度的纯色图像,客户端分别采用18个不同的模板进行亮度衰减,通过人眼观察的方式选择出针对不同亮度的最优模板。Fig. 6 is a flow chart of the template generation module. In the overlapping area of each projector, 18 different functions are used to generate the template, and the calculation method of the template will be described in detail later. Finally, the server sends 8 solid-color images with different brightness to the computer client, and the client uses 18 different templates for brightness attenuation, and selects the optimal template for different brightness through human observation.

本发明用边缘融合的方法消除交叠区的高亮带。用不同的函数得到不同的模板,都是以距离为自变量,对于交叠区的任意一点,离哪个投影仪越近,模板中的值越大。图7中以两个投影仪为例,d1为交叠区一点到左边投影仪边界的距离,d2为同一点到右边投影仪边界的距离,那么线性过渡的话,左边投影仪的模板值为The present invention uses an edge fusion method to eliminate the highlight bands in the overlapping areas. Use different functions to get different templates, all of which use distance as an independent variable. For any point in the overlapping area, the closer it is to which projector, the greater the value in the template. In Figure 7, two projectors are taken as an example. d1 is the distance from a point in the overlapping area to the boundary of the left projector, and d2 is the distance from the same point to the boundary of the right projector. In a linear transition, the template value of the left projector is

m1=d1/(d1+d2),m 1 =d1/(d1+d2),

右边投影仪的模板值为The template value for the right beamer is

m2=d2/(d1+d2)。m 2 =d2/(d1+d2).

图8是两个投影仪线性过渡的示意图。余弦过渡的话,左边投影仪的模板值Figure 8 is a schematic diagram of a linear transition between two projectors. For cosine transition, the template value of the left projector

m1=cos(d1/(d1+d2)),m 1 =cos(d1/(d1+d2)),

右边投影仪的模板值为The template value for the right beamer is

m2=1-cos(d1/(d1+d2))。m 2 =1-cos(d1/(d1+d2)).

图9是余弦过渡的示意图,可以发现余弦函数没有突变,过渡更光滑。本发明中用的是DLP投影仪,投影仪的像素值与亮度的关系是非线性的γ函数关系,所以还要对它进行γ校正,Figure 9 is a schematic diagram of the cosine transition, and it can be found that the cosine function has no sudden change and the transition is smoother. What used in the present invention is a DLP projector, and the relation of the pixel value of projector and brightness is nonlinear gamma function relation, so also will carry out gamma correction to it,

mγ1=(m1)1/γ;mγ2=(m2)1/γm γ1 = (m 1 ) 1/γ ; m γ2 = (m 2 ) 1/γ ,

其中,m1是线性过渡或余弦过渡下左边投影仪的模板值,m2是线性过渡或余弦过渡下右边投影仪的模板值。Among them, m 1 is the template value of the left projector under linear transition or cosine transition, and m 2 is the template value of right projector under linear transition or cosine transition.

因为γ未知,我们取γ从1.6到4.8,每隔0.4取一个γ值,共生成9个γ函数。本发明分别对线性过渡和余弦过渡叠加9个不同的γ函数来生成模板,总共有模板9*2=18个。Because γ is unknown, we take γ from 1.6 to 4.8, take a γ value every 0.4, and generate 9 γ functions in total. The present invention superimposes 9 different γ functions on the linear transition and the cosine transition respectively to generate templates, and there are 9*2=18 templates in total.

在亮度值140和255之间每隔20个亮度取一个亮度值L:140、160、180、200、220、240、255,再取一个亮度值100,共8个亮度值,用这些亮度值作为作为RGB值,构成8个不同亮度的纯色图像,把这些图像分配给每个客户端计算机,分别用前面生成的18个模板进行亮度校正,取效果最好的模板作为这个亮度的模板。最后得到8个亮度的最优模板,每个亮度对应的模板生成函数如表1所示。Take a brightness value L every 20 brightness values between brightness values 140 and 255: 140, 160, 180, 200, 220, 240, 255, and then take a brightness value of 100, a total of 8 brightness values, use these brightness values As RGB values, 8 solid-color images with different brightness are formed, and these images are distributed to each client computer, and the brightness correction is performed with the 18 templates generated earlier, and the template with the best effect is taken as the brightness template. Finally, the optimal templates of 8 luminances are obtained, and the template generation functions corresponding to each luminance are shown in Table 1.

亮度brightness 255255 240240 220220 200200 180180 160160 140140 100100 γ值gamma value 2.02.0 2.82.8 3.23.2 3.63.6 4.84.8 4.04.0 2.52.5 2.82.8 线性过渡 余弦过渡Linear transition Cosine transition 线性linear 线性linear 线性linear 线性linear 余弦cosine 余弦cosine 线性linear 余弦cosine

表1各个亮度的最优模板Table 1 Optimal templates for each brightness

Claims (4)

1. the self-adapting template big-screen brightness emendation method of multi-projector combination demonstration is characterized in that, contains following step successively:
Step (1): initialization;
Step (1.1): set up a rectangular coordinate system for described large-screen, initial point is in the lower left corner of this screen;
Step (1.2): form a projecting apparatus matrix with m * n DLP projecting apparatus, m is row, and n is row; Per two described DLP projecting apparatus are controlled with a client computer;
Step (1.3): the image that described each client computer is formed the equally spaced characteristic point of equally distributed 8 row, 8 row in each projecting apparatus space outputs to respectively in the DLP projecting apparatus of oneself controlling, the output of described each DLP projecting apparatus is taken pictures respectively in identical position with digital camera, obtain the characteristic point image of described each DLP projecting apparatus at screen space;
Step (1.4): server carries out image processing to described each characteristic point image of screen space and obtains the characteristic point of described each DLP projecting apparatus at screen space, obtain the initial display of described each DLP projecting apparatus according to characteristic point position at screen space, and the point of the point in each projecting apparatus space and screen space is set up one-to-one relationship, as the foundation of Coordinate Conversion between each projecting apparatus space and the screen space;
Step (1.5): described server calculates the shown rectangle RECT of a maximum in the initial display of screen space according to described each DLP projecting apparatus, each DLP projecting apparatus actual zone that will show on large-screen is to ask to hand over by the initial display of this projecting apparatus and maximum rectangle RECT to obtain, equidistantly generate the point of 8 row, 8 row in the actual displayed zone in each DLP projecting apparatus screen space, the point of these screen spaces is transformed in the projecting apparatus space, and generating mesh mesh-i;
Step (1.6): described client computer handle is distributed to the image mapped of the DLP projecting apparatus of oneself controlling to the corresponding geometric position of described grid mesh-i, remove with black all the other positions, calculate the positional information of the overlapping region of the above projecting apparatus of screen space, and be mapped in the projecting apparatus space;
Step (2): calculate stencil value according to the following steps at different brightness value images;
Step (2.1): described client computer is pressed the stencil value m linearity under the following linear function calculating linear transitions:
m Linear 1=d1/ (d1+d2),
m Linear 2=d2/ (d1+d2),
Wherein, m Linear 1Be the stencil value of left side DLP projecting apparatus, m Linear 2Be the stencil value of the right DLP projecting apparatus, d1 is the distance that a bit arrives DLP projecting apparatus border, the described left side in the part of described overlay region, and d2 is the distance that a bit arrives DLP projecting apparatus border, described the right described in the part of described overlay region;
Step (2.2): described client computer is by the stencil value m under the transition of following cosine function calculating cosine Cosine,
m Cosine 1=cos (d1/ (d1+d2)),
m Cosine 2=1-cos (d1/ (d1+d2));
Wherein, m Cosine 1Be the stencil value of left side DLP projecting apparatus, m Cosine 2Be the stencil value of the right DLP projecting apparatus, d1 is the distance that a bit arrives DLP projecting apparatus border, the described left side in the part of described overlay region, and d2 is the distance that a bit arrives DLP projecting apparatus border, described the right described in the part of described overlay region;
Step (2.3): the result that described step (2.1) and described step (2.2) are obtained handles with following non-linear γ correction function again,
m γ1=(m 1) 1/γ
m γ2=(m 2) 1/γ
Wherein, m 1The stencil value of left side projecting apparatus under corresponding described linear transitions or the described cosine transition, m 2Corresponding described linear transitions or described cosine transition be the stencil value of the right projecting apparatus down, and γ from 1.6 to 4.8, gets a value every 0.4, and symbiosis becomes 9 non-linear gamma functions, and corresponding linear transitions and cosine transition, symbiosis become 18 gamma correction templates;
Step (2.4): described client computer is got a brightness value L every 20 brightness between brightness value 140 and 255: 140,160,180,200,220,240,255, get a brightness value 100 again, totally 8 brightness values, as as rgb value, constitute the pure color image of 8 different brightness with these brightness values;
Step (2.5): described server is distributed to described client computer to an image that obtains in the described step (2.4);
Step (2.6): described client computer multiply by intensity correction values in the template that described step (2.3) obtains to the brightness value of the lap of the image that passes over respectively, obtain the brightness value of each point behind gamma correction, so that the original image of described lap decays, in each DLP projecting apparatus that outputs to again;
Step (2.7): each in 18 templates that described step (2.3) is obtained, repeating said steps (2.6) is selected at a best template of the image effect of described step (2.5) by the method for eye-observation;
Step (2.8): each pair in 8 sub-pictures that described step (2.4) is obtained, repeating said steps (2.5), described step (2.6) and described step (2.7), thus obtain 8 optimum templates at 8 different brightness;
Step (3): the gamma correction that carries out the above repeat region of self-adapting template big-screen of described m * n DLP projecting apparatus combination demonstration according to the following steps;
Step (3.1): described collection of server one sub-picture, the image that will show, distribute to described client computer;
Step (3.2): described client computer calculates the brightness value L 1 that passes every bit in the subimage of coming i,
L1 i=0.299*R+0.587*G+0.114*B,
Wherein, R, G, B are respectively R, the G of described point, the pixel value of B passage;
Step (3.3): described client computer calculates the mean value L2 of all some brightness in the described subimage,
Figure F2009100762061C00021
Wherein, L1 iBe the brightness value of the described subimage every bit that calculates in the described step (3.2), n is the number of described subimage mid point;
Step (3.4): described client computer calculates the Luminance Distribution of described subimage, leaves among the array num num[i in] brightness value equals the number of the point of sequence number i in the described subimage of expression, and the value of sequence number i is 0 to 255; Maximum NL corresponding sequence number L3 among the array num is the brightness peak of described subimage;
Step (3.5): described client computer calculates the brightness value L of described subimage, when the value of the NL that obtains in the described step (3.4) greater than the number of described subimage mid point 60% the time, the brightness value L of described subimage equals the brightness peak L3 of the described subimage that obtains in the described step (3.4), otherwise the brightness value L of described subimage equal the described subimage that obtains in the described step (3.3) average brightness L2;
Step (3.6): described client computer automatically selects the template of the immediate brightness of brightness value L of a described subimage that obtains with described step (3.5) as calibration template;
Step (3.7): the adaptive template intensity correction values that described client computer obtains described step (3.6) multiply by the brightness value of described lap, obtain the brightness value of each point behind gamma correction, so that the original image of described lap is decayed, output to again in the DLP projecting apparatus of described subimage correspondence;
Step (3.8): described each DLP projecting apparatus repeating said steps (3.2) is arrived described step (3.7);
Step (3.9): described server is after certain time interval, and repeating said steps (3.1) is to described step (3.8).
2. the self-adapting template big-screen brightness emendation method that multi-projector combination according to claim 1 shows is characterized in that, described DLP projecting apparatus is divided into two sequences by row and column, in described capable sequence, there be the overlapping of horizontal direction in the projection of two projecting apparatus that sequence number is adjacent separately, in described row sequence, there be the overlapping of vertical direction in the projection of two projecting apparatus that sequence number is adjacent separately.
3. the self-adapting template big-screen brightness emendation method that multi-projector combination according to claim 1 shows is characterized in that described all images is to show in real time, comprising video image.
4. the self-adapting template big-screen brightness emendation method that multi-projector combination according to claim 1 shows is characterized in that m=2, n=2.
CN2009100762061A 2009-01-05 2009-01-05 Self-adapting template big-screen brightness emendation method with multi-projector combined display Expired - Fee Related CN101482690B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN2009100762061A CN101482690B (en) 2009-01-05 2009-01-05 Self-adapting template big-screen brightness emendation method with multi-projector combined display

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN2009100762061A CN101482690B (en) 2009-01-05 2009-01-05 Self-adapting template big-screen brightness emendation method with multi-projector combined display

Publications (2)

Publication Number Publication Date
CN101482690A CN101482690A (en) 2009-07-15
CN101482690B true CN101482690B (en) 2010-11-10

Family

ID=40879868

Family Applications (1)

Application Number Title Priority Date Filing Date
CN2009100762061A Expired - Fee Related CN101482690B (en) 2009-01-05 2009-01-05 Self-adapting template big-screen brightness emendation method with multi-projector combined display

Country Status (1)

Country Link
CN (1) CN101482690B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI818282B (en) * 2021-07-12 2023-10-11 明基電通股份有限公司 Projector device, projector system and method for calibrating projected image

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101964891B (en) * 2010-09-13 2012-07-25 广东威创视讯科技股份有限公司 Screen seamless mosaicing image processing method and device therefor
CN102053812B (en) * 2010-10-09 2012-09-05 清华大学 Multi-projector combined display feedback brightness correction method
JP6198605B2 (en) * 2010-11-15 2017-09-20 スケーラブル ディスプレイ テクノロジーズ インコーポレイテッド System and method for calibrating a display system using manual and semi-automatic techniques
CN102109972B (en) * 2011-02-14 2012-09-12 深圳雅图数字视频技术有限公司 Projector television wall display method and system
CN102231099B (en) * 2011-07-06 2014-06-18 清华大学 Method for correcting per-pixel response brightness in multi-projector auto-stereoscopic display
JP5950554B2 (en) * 2011-12-02 2016-07-13 キヤノン株式会社 Display device, information processing device, control method thereof, projection system, program, and storage medium
CN104601913B (en) * 2013-10-30 2017-11-03 北京莫高丝路文化发展有限公司 Ball curtain multi-channel edge fusion method
US10701375B2 (en) * 2016-03-23 2020-06-30 Dolby Laboratories Licensing Corporation Encoding and decoding reversible production-quality single-layer video signals
CN106791741B (en) * 2016-12-07 2018-09-21 重庆杰夫与友文化创意有限公司 Multi-screen marching method and device

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI818282B (en) * 2021-07-12 2023-10-11 明基電通股份有限公司 Projector device, projector system and method for calibrating projected image

Also Published As

Publication number Publication date
CN101482690A (en) 2009-07-15

Similar Documents

Publication Publication Date Title
CN101482690B (en) Self-adapting template big-screen brightness emendation method with multi-projector combined display
CN102053812B (en) Multi-projector combined display feedback brightness correction method
CN104954715B (en) Method based on the GPU special-shaped screen multi-projection system fusion video playbacks accelerated
JP5806269B2 (en) Method and apparatus for generating a scale-changed image by changing the scale of the image
CN1244228C (en) Electronic calibration of seamlessly tiled displays utilizing an optical function generator
JP3766672B2 (en) Image correction data calculation method
US7800628B2 (en) System and method for generating scale maps
KR101741638B1 (en) Display apparatus and image correction method of the same
US7907792B2 (en) Blend maps for rendering an image frame
CN111192552B (en) Multi-channel LED spherical screen geometric correction method
US20040085477A1 (en) Method to smooth photometric variations across multi-projector displays
US20070291184A1 (en) System and method for displaying images
CN101571663B (en) Distributed online regulating method for splicing multiple projectors
US20070291185A1 (en) System and method for projecting multiple image streams
US9052584B2 (en) Color seamlessness across tiled multi-projector displays
JP2011182079A (en) Correction information calculation device, image correction device, image display system, and correction information calculation method
CA2399678A1 (en) Digital correction module for video projector
CN102231099A (en) Method for correcting per-pixel response brightness in multi-projector auto-stereoscopic display
CN102542988B (en) Device and method for previewing correction effect of LED display screen
US20080260292A1 (en) Image generation apparatus, method, program, and recording medium
TW200917838A (en) Digital image tone remapping method and apparatus
CN105450945A (en) Video panorama generating method and parallel computing system
US20060012759A1 (en) Multiscreen display system, multiscreen display method, luminance correction method, and programs
CN103686029B (en) A kind of video pixel opposite-angle interpolation method for the many optical projection systems of giant-screen
Zoido et al. Optimized methods for multi-projector display correction

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20101110

Termination date: 20130105

CF01 Termination of patent right due to non-payment of annual fee