JP2011211276A - Image processing apparatus, image display system, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image display system, and an image processing method, capable of efficiently correcting variations in gradation of a plurality of partial images as a whole.SOLUTION: The image processing apparatus includes: a shape-correcting information calculator 38 for calculating shape-correcting information indicating the correspondence relation between each pixel position on an image forming element of a projector and each pixel position on a projected surface 9; shape-correcting processors 37a-37d for substantially matching each pixel position on the projected surface 9 to each pixel position defined in image data on the basis of the shape-correcting information; a gradation-correcting information calculator 35 for calculating gradation-correcting information indicating the correspondence relation between the luminance level on the projected surface 9 and a gradation value on each of plurality of evaluation regions defined in the image data for gradation measurement concerning the evaluation regions including an evaluation region selected from a partial image for gradation measurement projected by each of the projectors on the basis of the image data for gradation measurement after shape correcting processing; and a gradation-correcting processor 34 for causing the luminance level of each pixel on the projected surface 9 to approximate to the luminance level of each pixel defined in the image data on the basis of the gradation-correcting information.

Description

  The present invention relates to an image processing apparatus, an image display system, and an image processing method.

  Conventionally, a projector is known as one of image display apparatuses. The projector has features such as being easy to install and capable of displaying a large screen image. In recent years, an image display system that displays an image using a plurality of projectors has been proposed (for example, Patent Document 1). The multi-display device of Patent Document 1 projects a partial image (hereinafter referred to as a partial image) of a displayed image on each of a plurality of projectors, and displays one image as a whole of the plurality of partial images. This makes it possible to display an image with a large screen, high brightness, and high resolution.

  In order to display an image for viewers (hereinafter referred to as a content image) using this type of image display system, a plurality of projectors are arranged so that a plurality of partial images are displayed side by side on the projection surface. . In order to improve the display quality of content images, the distortion of each partial image and the relative position shift of multiple partial images are reduced, and the display conditions such as relative brightness and color tone of multiple partial images are managed with high accuracy. It is important to. In general, it is difficult to manually adjust the display conditions with high accuracy, and a technique for automating the adjustment of display conditions is expected from the viewpoint of reducing the burden on the user.

  In order to automate the adjustment of brightness and color tone, it is possible to apply a technique (for example, Patent Document 2) that automatically corrects luminance unevenness (gray gradation unevenness) and color unevenness (gradation unevenness for each color). Conceivable. Hereinafter, luminance unevenness and color unevenness are collectively referred to as gradation unevenness. The image display device of Patent Document 2 includes an image correction unit, a data storage unit, and an image forming unit. The image correction unit corrects the input image data based on the correction data stored in the data storage unit. The image forming unit forms an image to be displayed according to the corrected image data. The correction data is determined so that the output levels at a plurality of positions on the gray image displayed by the projector approach the input levels defined in the gray image data. The correction data is calculated based on input / output characteristic data of the image forming unit.

JP 2006-349791 A JP 2001-209358 A

  According to the technique of Patent Document 2, it seems that gradation unevenness for each partial image can be corrected, and gradation unevenness as a whole of a plurality of partial images can be reduced. However, if the technique of Patent Document 2 is actually applied to an image display system that performs display by a plurality of projectors, there are the following problems.

  In order to correct gradation unevenness for each partial image by the technique of Patent Document 2, it is necessary to display a gray image for each projector to correct gradation unevenness. Time will increase. In addition, in order to correct gradation unevenness in the entire image, it takes time and effort to adjust the mutual output of a plurality of projectors in order to align the luminance and color tone of a plurality of partial images.

  Further, when the distortion of the partial image and the shift of the relative positions of the plurality of partial images are corrected, the position of each pixel on the projection surface changes before and after the correction. Therefore, the position of each pixel on the projection surface does not correspond to the correction amount for each pixel, and as a result, gradation unevenness occurs.

  The present invention has been made in view of the above circumstances, and provides an image processing apparatus, an image display system, and an image processing method capable of efficiently correcting gradation unevenness in a plurality of partial images as a whole. One of the purposes is to provide it.

In the present invention, the following means are adopted in order to achieve the object.
The image processing apparatus of the present invention is an image processing apparatus that is used with a plurality of projectors each having an image forming element and projecting an image on a projection surface, and the position of each pixel on the image forming element and A shape correction information calculation unit that calculates shape correction information indicating a correspondence relationship with the position of each pixel on the projection surface; and the plurality of projectors based on image data after processing based on the shape correction information A shape correction processing unit that applies shape correction processing to the image data so that the shape of the image projected in cooperation with the shape to be projected on the projection surface substantially matches the shape defined by the image data before processing, and the shape Of the gradation measurement images that the plurality of projectors project in cooperation based on the gradation measurement image data that has been subjected to the shape correction processing by the correction processing unit, For a plurality of evaluation areas including an evaluation area selected from the partial image for gradation measurement projected by the brightness level in each evaluation area of the image for gradation measurement on the projection surface, and the floor A gradation correction information calculation unit for calculating gradation correction information indicating a correspondence relationship with a gradation value in each evaluation region defined in image data for gradation measurement indicating an image for gradation measurement; and the gradation Based on the correction information, the brightness level of each pixel on the projection surface of the image projected by the plurality of projectors in cooperation with the image data after processing is defined in the image data before processing. A gradation correction processing unit that performs gradation correction processing on the image data so as to approach a luminance level corresponding to the gradation value of each pixel.

  With this configuration, the shape correction information calculation unit calculates the shape correction information, and the shape correction processing unit performs shape correction processing on the image data based on the shape correction information, so that the image displayed on the projection surface is displayed. Can be made to substantially coincide with the shape of the image defined in the image data before correction. As a result, the image data can be corrected so as to reduce the distortion of each partial image and to reduce the shift of the relative position in the plurality of partial images.

  In addition, tone correction is performed using the luminance level in each evaluation area of the image for tone measurement on the projection surface projected based on the image data for tone measurement after the shape correction processing is performed. Since the information calculation unit calculates the gradation correction information, gradation correction information in which the movement of the pixels on the projection surface accompanying the shape correction process is taken into account is obtained. In addition, since the gradation correction information calculation unit calculates the gradation correction information using the luminance levels of a plurality of evaluation areas including the evaluation area selected from each partial image for gradation measurement on the projection surface, Gradation correction information in which gradation unevenness in each partial image and a difference in luminance level in a plurality of partial images are taken into consideration can be obtained. In other words, gradation correction information that takes into account the entire image when a plurality of partial images are viewed as one image is obtained, and the gradation correction processing unit adds image correction data that represents the entire image using the gradation correction information. Since gradation correction processing is performed, image data can be efficiently corrected so as to reduce gradation unevenness as a whole of a plurality of partial images.

  The image processing apparatus according to the present invention can take the following aspects as typical aspects.

  The shape correction information calculation unit defines the shape of an image in an effective projection area set as an area that fits in the entire projection area on the projection surface by the plurality of projectors by the image data for gradation measurement. The shape correction information is calculated so as to substantially match the shape to be obtained, and the gradation correction information calculation unit uses the shape correction information to perform the shape correction processing and then the gradation measurement image. The gradation correction information may be calculated using a luminance level of the evaluation area in the effective projection area of the image for gradation measurement on the projection surface projected based on the data.

  In this way, the calculation of gradation correction information and gradation correction processing for the area outside the effective projection area where an image is actually displayed out of the maximum areas that can be projected by a plurality of projectors are omitted. Therefore, it is possible to efficiently correct gradation unevenness in an actually displayed image.

  A data generation unit that generates image data indicating a partial image projected by each projector among images projected by the plurality of projectors in cooperation based on the image data after the gradation correction processing is performed. The shape correction processing unit may perform the shape correction processing on each image data generated by the data generation unit.

  According to this configuration, the data generation unit generates image data indicating each of the plurality of partial images (hereinafter referred to as partial image data) based on the image data corrected by the gradation correction processing unit. By projecting a plurality of partial images based on each partial image data, it is possible to display an image in which gradation unevenness is corrected in the whole of the plurality of partial images. In addition, since shape correction processing is performed on each partial image data, distortion of each partial image and displacement of relative positions of a plurality of partial images can be corrected with high accuracy.

  The data generation unit is an image showing the partial image such that each of a pair of partial images adjacent to each other among the plurality of partial images constituting the image includes an overlapping region overlapping the pair of partial images. It is good to generate data.

  In this way, when the entire image is displayed by overlapping the overlapping region between a pair of partial images adjacent to each other, the plurality of partial images are arranged without gaps on the projection surface. It is possible to display an image with an enhanced experience.

  Each of the overlapping regions when the overlapping regions are superimposed on each other in the pair of partial images indicated by the pair of image data after processing, with each of the pair of image data indicating the pair of partial images being processed. The luminance level of the pixel is set to a luminance level corresponding to the gradation value of each pixel in the overlapping area defined in the image data indicating the image before the gradation correction processing is performed by the gradation correction processing unit. It is preferable to provide an edge blend processing unit that performs an edge blend process for bringing them closer.

  In this way, with the overlapping areas of the pair of partial images superimposed on each other on the projection surface, the luminance level of each pixel in the overlapping area is higher than the luminance level corresponding to the gradation value defined in the image data. Can be avoided, and it is possible to make it difficult to identify the overlapping region, that is, the joint of the partial images, superimposed on the projection surface.

  The gradation correction information calculation unit performs the edge blend process on the image data indicating the partial image generated by the data generation unit based on the image data for gradation measurement by the edge blend processing unit. Gradation measurement in which the plurality of projectors collaborately project based on each image data indicating the partial image after, and the overlapping regions are superimposed on the projection surface in the pair of partial images The gradation correction information may be calculated using the luminance level in each evaluation area of the image for use.

  In this way, tone correction information is calculated using the brightness level in each evaluation region of the image for tone measurement in a state where a pair of partial images are superimposed on each other on the projection surface. Therefore, the calculation of the shape correction information and the shape correction processing for the overlapping region can be performed collectively with a pair of partial images. Therefore, it is possible to reduce a load such as calculation required for the gradation correction processing, and it is possible to efficiently correct gradation unevenness of an image.

  An image display system according to the present invention includes a plurality of the above-described image processing device according to the present invention and a plurality of images that cooperate to project an image based on the image data that has been subjected to the gradation correction processing by the image processing device. And a projector.

  In this way, since the image data is subjected to the shape correction process based on the shape correction information, it is possible to display an image in which the distortion of each partial image and the relative position shift of the plurality of partial images are reduced. Gradation correction information that takes into account the entire image composed of a plurality of partial images is obtained, and the gradation correction processing unit performs gradation correction processing on the image data indicating the entire image using the gradation correction information. Image data can be corrected efficiently, and an image with little gradation unevenness can be displayed efficiently.

According to the image processing method of the present invention, the position of each pixel on the image forming element in each of a plurality of projectors each having an image forming element and projecting an image on the projection surface, and each pixel on the projection surface A step of calculating shape correction information indicating a correspondence relationship with the position of the image, and the projected surface of an image projected by the plurality of projectors based on image data after processing based on the shape correction information And applying a shape correction process to the image data to substantially match the shape defined by the image data before processing with the image data for gradation measurement subjected to the shape correction process A plurality of evaluations including an evaluation area selected from the gradation measurement partial images projected by each projector among the gradation measurement images projected by the plurality of projectors in cooperation. For each area, the brightness level in each evaluation area of the image for gradation measurement on the projection surface and each evaluation area defined in the image data for gradation measurement indicating the image for gradation measurement. Calculating gradation correction information indicating a correspondence relationship with the gradation value of
Based on the gradation correction information, the luminance level of each pixel on the projection surface of an image projected by the plurality of projectors in cooperation with the image data after processing is set as the image data before processing. Applying to the image data a gradation correction process that approximates a luminance level corresponding to a prescribed gradation value of each pixel.

  In this way, since the image data is subjected to the shape correction process based on the shape correction information, the image data can be corrected so as to reduce the distortion of each partial image and the shift of the relative position among a plurality of partial images. . Correction information that takes into account the entire image composed of a plurality of partial images is obtained, and the gradation correction processing unit performs gradation correction processing on the image data indicating the entire image using this gradation correction information. Image data can be efficiently corrected so as to reduce gradation unevenness as a whole.

It is a conceptual diagram which shows the structural example of the image display system which concerns on this invention. It is explanatory drawing which shows the positional relationship of the projection area | region on a to-be-projected surface. It is a block diagram which shows the function structure of an image processing apparatus. It is a flowchart which shows the flow of a process when calculating shape correction information and gradation correction information. It is a flowchart which shows the processing flow when correct | amending image data. It is a flowchart which shows the processing flow of a projector installation operation | work. It is explanatory drawing of the guide image used for a projector installation operation | work. It is a flowchart which shows the processing flow of a shape correction information calculation process. It is explanatory drawing which shows the measuring method of the position of the feature point in a to-be-projected surface. It is explanatory drawing which shows the calculation method of shape correction information. It is a flowchart which shows the processing flow of a gradation correction information calculation process. (A), (b) is explanatory drawing which shows the processing content of an edge blend process. It is explanatory drawing which shows the measuring method of the luminance level on a to-be-projected surface. It is explanatory drawing which shows the calculation method of gradation correction information. (A)-(c) is explanatory drawing which shows the processing content when correcting image data in order of a process. (A)-(c) is explanatory drawing of the processing content which continues from FIG.13 (c).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used for explanation, in order to show characteristic parts in an easy-to-understand manner, dimensions and scales of structures in the drawings may be different from actual structures. In addition, in the embodiment, the same components are illustrated with the same reference numerals, and detailed description thereof may be omitted. The technical scope of the present invention is not limited to the following embodiment. Various modifications are possible without departing from the gist of the present invention.

  FIG. 1 is a conceptual diagram of an image display system according to an embodiment of the present invention, and FIG. 2 is an explanatory diagram showing a positional relationship of projection areas on a projection surface.

  As shown in FIG. 1, the image display system 1 includes a plurality of projectors 2, an image processing device 3, and a measurement device 4. For example, the image display system 1 projects a content image P indicated by image data input from a signal source 8 onto a projection surface 9 such as a screen or a wall in cooperation with a plurality of projectors 2. Here, a description will be given assuming that a first projector 21, a second projector 22, a third projector 23, and a fourth projector 24 are provided as the plurality of projectors 2. The application range of the present invention is not limited to the number and arrangement of projectors as long as a plurality of projectors are used. In the following description, an image projected from each projector may be referred to as “partial image”, and image data indicating the partial image may be referred to as “partial image data”.

  Each of the plurality of projectors 2 can display an image (here, a partial image) based on the supplied image data (here, partial image data). The scope of application of the present invention is not limited to the configuration of each projector constituting a plurality of projectors. For example, each projector may be a single-plate projector or may include a plurality of image forming elements such as a three-plate projector. Each projector may be a front projection type that projects from the same side as the viewer on the projection surface 9, or a rear projection that projects from the opposite side of the viewer across the projection surface 9. It may be of a type.

  Each of the plurality of projectors 2 of the present embodiment is a three-plate projector having a light source, three image forming elements, and a projection optical system. The light emitted from the light source is separated into red light, green light, and blue light, and enters the corresponding image forming element for each color light. Each image forming element has a plurality of pixels (hereinafter referred to as modulation elements) arranged two-dimensionally. In the present embodiment, the number of modulation elements of the image forming elements of each projector and the arrangement of the modulation elements are the same in the plurality of projectors 2. Examples of the image forming element include a transmissive or reflective liquid crystal light valve and a digital mirror device (DMD). The image forming element controls a plurality of modulation elements independently from each other based on data (hereinafter referred to as pixel data) indicating gradation values for each pixel of the partial image data input to the projector. The light incident on the modulation element is modulated for each modulation element and becomes light with a light amount corresponding to the gradation value defined in the pixel data. The light modulated by the plurality of modulation elements becomes an optical image (image) as a whole. The three image forming elements form optical images of different colors (red, green, blue). After the three color optical images are combined by a dichroic prism or the like, the combined optical image is projected onto the projection surface 9 by the projection optical system, and a full color image (here, a partial image) can be displayed.

  The projection area, which is the maximum area that each projector can project onto the projection surface 9, is set differently for the plurality of projectors 2. As shown in FIG. 2, the first to fourth projection areas A1 to A4 are set so that their peripheral edges overlap each other, and constitute the entire projection area A0 as a whole. Each of the first to fourth projection areas A1 to A4 includes an overlapping area A5 that is overlapped with another adjacent projection area.

  The projection area (first projection area A1) of the first projector 21 is aligned with the projection area (second projection area A2) of the second projector 22 in the horizontal direction on the projection surface 9. In the horizontal direction, the end of the first projection area A1 is overlapped with the end of the second projection area A2. Regarding the horizontal positional relationship between the projection area of the third projector 23 (third projection area A3) and the projection area of the fourth projector 24 (fourth projection area A4), the first and second projection areas A1. , A2 is the same as the horizontal positional relationship.

  The first projection area A1 is aligned with the third projection area A3 in the vertical direction orthogonal to the horizontal direction on the projection surface 9. In the vertical direction, the end of the first projection area A1 is overlapped with the end of the third projection area A3. The vertical positional relationship between the second projection region A2 and the fourth projection region A4 is the same as the vertical positional relationship between the first and third projection regions A1 and A3.

  By the way, normally, the outer shape of the entire projection area A0 is not rectangular. This is because the first to fourth projection areas A1 to A4 are distorted and the relative positions are shifted due to the arrangement of the first to fourth projectors 21 to 24. Here, a substantially rectangular area that falls within the entire projection area A0 is an area that is actually used for displaying an image (effective projection area A). The effective projection area A is set as, for example, the largest rectangular area that can be accommodated in the entire projection area A0, and the aspect ratio is set according to the aspect ratio of the content image P. The 1st-4th projectors 21-24 project each partial image with respect to the area | region which fits in the effective projection area | region A among each projection area | region.

  For example, the first partial image P <b> 1 is displayed in an area that falls within the effective projection area A in the first projection area A <b> 1 by the light projected from the first projector 21. Similarly, the second partial image P2, the third partial image P3, and the fourth partial image P4 are displayed by the second to fourth projectors 22 to 24. The first to fourth partial images P1 to P4 are displayed with the overlapping area P5 superimposed on each other in the overlapping area A5, and constitute the content image P as a whole.

  Returning to the description of FIG. 1, the measuring device 4 can measure the color tone distribution (XYZ or RGB three-axis two-dimensional information) of the luminance level of the image displayed on the projection surface 9. The measuring device 4 of the present embodiment is configured by an imaging device capable of imaging an area on the projection surface 9 including the entire projection area A0. When calculating shape correction information and gradation correction information, which will be described later, captured image data indicating a captured image captured by the imaging device is output to the image processing device 3.

  The imaging device is constituted by a two-dimensional image sensor such as a CCD camera. The imaging device may be composed of two or more two-dimensional image sensors. For example, different two-dimensional image sensors may be used for imaging when calculating shape correction information and for imaging when calculating gradation correction information.

  As an imaging device for imaging when calculating gradation correction information, an image is obtained using a filter having a spectral sensitivity approximate to xyz color matching function, and an XYZ tristimulus value is obtained by matrix correction calculation. And an image obtained by using an RGB filter different from the above to obtain an RGB image.

  By the way, if the whole projection area A0 is imaged by one imaging, the imaging between imaging is compared with the case where the whole projection area A0 is imaged by combining the captured images by several imaging. Errors due to movement of the device can be eliminated. Depending on the size and aspect ratio of the entire projection area A0, the entire projection area A0 may not fit within the angle of view of the imaging apparatus. In this case, by placing the imaging device on a tripod or the like, the entire projection area A0 is imaged for each part in a state where the relative position of the imaging device with respect to the projection surface 9 is substantially fixed, and the imaging result of two or more times In addition, a captured image obtained by capturing the entire projection area A0 may be acquired.

  The image processing device 3 receives image data indicating the content image P from, for example, the signal source 8. The image processing apparatus 3 performs a gradation correction process on the image data, and then generates a plurality of partial image data based on the image data after the gradation correction process. Each partial image data is data indicating one of the first to fourth partial images P1 to P4. The image processing apparatus 3 performs shape correction processing after performing edge blend processing on each partial image data. The outline of the gradation correction process, the shape correction process, and the edge blend process is as follows.

  In the gradation correction process, the content image P is displayed so that the gradation unevenness in the image projected based on the corrected image data is less than the gradation unevenness in the image projected based on the image data before correction. Correct the image data. In the gradation correction process, gradation unevenness in each partial image and relative luminance and color tone differences among the plurality of partial images P1 to P4 are corrected by the same process as gradation unevenness of the content image P. To do. The gradation unevenness in each partial image is, for example, local distortion of the projection surface 9 due to the deflection of the cloth-like screen, spatial nonuniformity of the light source light intensity in each projector, lenses, etc. It occurs due to chromatic aberration and the like in the optical component. Differences in relative luminance and color tone among the plurality of partial images P1 to P4 occur due to, for example, a difference in rated output light quantity among the plurality of projectors 2, a manufacturing error of the projector, and the like.

  In the edge blending process, as shown in FIG. 1, when the overlapping region P5 is superimposed on a pair of adjacent partial images, the luminance level value of each pixel in the overlapping region P5 indicates the content image P. Each partial image data is corrected so as to approach a luminance level corresponding to the gradation value of each pixel defined in the image data.

  In the shape correction process, the image distortion in each partial image is corrected and the relative position shift between the plurality of partial images P1 to P4 is corrected. As an example of image distortion, for example, image distortion (corresponding to the projection direction of each of the first to fourth projectors 21 to 24 with respect to the projection surface 9, that is, the vertical elevation angle, depression angle, horizontal inclination angle, etc.) For example, keystone distortion). As another example of the image distortion, there is an image distortion generated when the projection surface 9 is locally distorted due to, for example, a cloth-like screen deflection. The relative position shift of the first to fourth partial images P1 to P4 is, for example, that the projection directions do not match or the relative position is shifted in the first to fourth projectors 21 to 24. Due to etc.

  The image processing apparatus 3 inputs the partial image data after the shape correction processing to the projector that is in charge of this partial image. The first to fourth projectors 21 to 24 project the first to fourth partial images P1 to P4 on the projection surface 9 based on the partial image data input from the image processing device 3. The image processing apparatus 3 uses shape correction information and gradations used for shape correction processing at an appropriately selected timing such as when the image display system 1 is installed or when a predetermined period has elapsed since the image display system 1 is installed. Tone correction information used for correction processing is calculated.

Examples of modes for realizing the functions of the image processing apparatus according to the present invention include the following first to third modes.
As a first aspect, there is an image processing apparatus using one or more logic circuits such as an ASIC that performs various processes. A part or all of this image processing apparatus may be integrated with any of the first to fourth projectors 21 to 24, the measuring apparatus 4, and the signal source 8.
As a second aspect, there is a configuration by a computer in which a program is installed. That is, the functions of the image processing apparatus can be realized by causing a computer to execute various processes according to a program. For example, by performing a calculation in each process using a memory or a CPU, storing the calculation result in a storage unit such as a hard disk or a memory, and reading out the calculation result as necessary for other processes. Processing results similar to those obtained when a logic circuit or the like is used can be obtained. In the second aspect, various processes may be shared by a plurality of computers.
As a third aspect, there is a configuration in which a logic circuit that performs a part of various processes and a computer that performs other processes by a program are combined.

  As described above, in order to realize the functions of the image processing apparatus according to the present invention, in addition to an aspect as an independent apparatus that performs various processes, an aspect as a set of a plurality of functional units that perform a part of various processes can be taken. Yeah. That is, a plurality of functional units are separately provided in a plurality of separate devices, and the function of the image processing device can be realized even in a mode in which processing is performed in cooperation with each other.

  When the content image P is displayed by the plurality of projectors 2 as described above, the content image P can be displayed with a large screen, high resolution, and high luminance. For example, the content image P can be displayed on a large screen without reducing the resolution under the condition that the number of pixels per projector (the number of modulation elements) is the same as compared with the case of displaying with one projector. . Further, when the comparison is made under the condition that the content image P is displayed with the same screen size, the number of pixels can be increased according to the number of projectors, and the content image can be displayed with high resolution. When the comparison is made under the condition that the rated output light amount per projector is the same, the light amount contributing to the display can be increased according to the number of projectors, and the content image P can be displayed with high brightness.

  Compared with the method of increasing the number of pixels and output of the projector to obtain these effects, the cost of each projector can be drastically reduced, so the effect of reducing the equipment cost even with the number of projectors is expected Is done. In addition, for example, each projector is usually installed and used in another place (for example, a small meeting room), and an image display system is constructed in, for example, a large meeting room according to the application, and content images are displayed on a large screen, It is also possible to display with high resolution and high brightness. In this way, it is possible to select whether to use a projector alone or in combination with a plurality of projectors, depending on the application, so compared to the case where one projector tries to obtain the above effect , Increase convenience.

  In addition, since the content image P is displayed based on the image data that has been subjected to the gradation correction processing and the shape correction processing by the image processing device 3, gradation unevenness in the entire content image P and the image of each partial image are displayed. Distortion and relative position shift in a plurality of partial images are reduced. Therefore, the content image P can be displayed in a state where the sense of unity among the plurality of partial images is enhanced, and the content image P can be displayed as a high-quality image. In the present invention, it is possible to reduce a load such as computation required for the gradation correction processing, and it is possible to efficiently correct image data.

Next, the configuration of the image processing apparatus 3 will be described in detail with reference to FIG. FIG. 3 is a block diagram showing a functional configuration of the image processing apparatus according to the present invention.
As shown in FIG. 3, the image processing apparatus 3 includes a gradation correction unit 30, a data generation unit 31, first to fourth edge blend processing units 32 a to 32 d, and a shape correction unit 33.

  The gradation correction unit 30 includes a gradation correction processing unit 34, a gradation correction information calculation unit 35, and a storage unit 36. The storage unit 36 is configured by, for example, a hard disk or a nonvolatile memory. The storage unit 36 stores gradation correction information. The gradation correction information includes each pixel displayed on the projection surface 9 based on the gradation value of each pixel defined in the image data before the gradation correction process and the image data after the gradation correction process (hereinafter, referred to as the gradation correction information). , Which is a display pixel). That is, referring to the gradation correction information, when the image data is corrected so that the luminance level of the display pixel approaches a predetermined value, the value required as the correction amount of the gradation value of each pixel can be known. Yes.

  The gradation correction information of the present embodiment includes information indicating the correction amount of the gradation value of each basic color constituting the full color image for each pixel in the pixel array defined in the image data. Specifically, information indicating the correction amount of the red tone value, information indicating the correction amount of the green tone value, and the correction amount of the blue tone value are used as tone correction data for each pixel. This information is stored in the storage unit 36 in a table format.

  When displaying the content image P, the gradation correction processing unit 34 reads the gradation correction information from the storage unit 36, and performs gradation correction processing on the image data indicating the content image P using the gradation correction information. The gradation correction information calculation unit 35 calculates gradation correction information. When calculating gradation correction information, the plurality of projectors 2 cooperate to project an image for gradation measurement onto the projection surface 9. An image for gradation measurement on the projection surface 9 is captured by the measurement device 4, and captured image data indicating the obtained captured image is output to the gradation correction information calculation unit 35. The image for gradation measurement is appropriately selected from patterns that can measure the distribution of luminance levels in a state of being projected on the projection surface 9. As an image for gradation measurement, it is preferable to use an image (hereinafter referred to as a solid image) in which the gradation values of all the pixels are the same from the viewpoint of making it possible to measure gradation values with high accuracy. Image data for gradation measurement indicating an image for gradation measurement is stored inside (for example, the storage unit 36) or outside (for example, the signal source 8) of the image processing apparatus 3, and calculates gradation correction information. Sometimes output to each projector.

The gradation correction information calculation unit 35 uses a gradation measurement image (referred to as a gradation measurement captured image) reflected in the captured image for each of a plurality of evaluation regions in the gradation measurement image. The gradation correction information is calculated by comparing the gradation value with the gradation value defined in the image data for gradation measurement. The gradation value of each pixel of the captured image corresponds to the luminance level of the area on the projection surface corresponding to this pixel. The evaluation area is an area constituted by one pixel or two or more continuous pixels constituting an image for gradation measurement. When the area projected by each projector in the gradation measurement image on the projection surface 9 is a partial image for gradation measurement, each partial image for gradation measurement has one or more evaluation areas. If a plurality of evaluation areas are selected such that the evaluation area is included, the range, number, and arrangement of the evaluation areas can be freely set. The gradation correction information calculation unit 35 stores the calculated gradation correction information in the storage unit. 36, and the gradation correction information is updated.

  When displaying the content image P, the data generation unit 31 generates each partial image data based on the image data after the gradation correction processing. For example, the data generation unit 31 refers to the information indicating the overlap amount of each projection area and the information indicating the arrangement of each projection area on the projection surface 9, and determines the size of the overlap area A5 illustrated in FIG. In consideration, partial image data is generated. When the pair of projection areas are arranged in a predetermined direction (for example, the horizontal direction), the overlap amount is the width in a predetermined direction of one projection area and the predetermined direction of the area where the pair of projection areas overlap each other. It is expressed by the ratio with the width of For example, an image corresponding to a region represented by the sum of a plurality of regions obtained by equally dividing the content image P by the number of partial images and an overlapping region P5 determined by the overlap amount is used as a partial image, and the image data is converted into a partial image. Partial image data is obtained by extracting corresponding data.

  The data generation unit 31 outputs the first partial image data indicating the first partial image P1 to the first edge blend processing unit 32a. Similarly, the data generation unit 31 outputs the second to fourth partial image data indicating the second to fourth partial images P2 to P4 to the second to fourth edge blend processing units 32b to 32d.

  The first to fourth edge blend processing units 32a to 32d perform edge blend processing on the input partial image data. Arbitrary pixels in the overlapping region P5 are included in the plurality of partial images, and the gradation values of the pixels are defined in the plurality of partial image data. In the edge blending process, the integrated value of the values defined in the plurality of partial image data as the gradation value of the arbitrary pixel approaches the gradation value of this pixel defined in the image data before the gradation correction process. Thus, each partial image data is corrected. The first to fourth edge blend processing units 32 a to 32 d output the partial image data after the edge blend processing to the shape correction unit 33.

  The shape correction unit 33 includes first to fourth shape correction processing units 37 a to 37 d, a shape correction information calculation unit 38, and a storage unit 39. The storage unit 39 is the same as the storage unit 36 of the gradation correction unit 30 and may be configured by the same device as the storage unit 36. The storage unit 39 stores shape correction information. The shape correction information is information indicating a correspondence relationship between the position of each modulation element on the image forming element of each projector and the position of each display pixel on the projection surface 9. The shape correction information is stored in a table format as a value for each modulation element, for example. By referring to the shape correction information, it can be understood to which modulation element each pixel data should be input in order to display a predetermined pixel at a predetermined position in the effective projection area A.

  When displaying the content image P, the first to fourth shape correction processing units 37a to 37d perform shape correction processing on the first to fourth partial image data. For example, the first shape correction processing unit 37a receives first partial image data indicating the first partial image P1 projected by the first projector 21 from the first edge blend processing unit 32a. For example, the first shape correction processing unit 37a may display the partial image defined in the partial image data before the shape correction processing in the effective projection area A inside the first projection area A1, for example, as follows. Perform shape correction processing.

  The first shape correction processing unit 37a is associated with each non-display pixel by shape correction information for each pixel inside the entire projection area A0 and outside the effective projection area A (hereinafter referred to as a non-display pixel). The data supplied to each modulation element of the image forming element is made non-display mask data (for example, data indicating black). Further, for each pixel inside the effective projection area A, the data supplied to the modulation element associated with the display pixel by the shape correction information is changed to data defined in the partial image data as the pixel data for the display pixel. .

  When the position of the display pixel is deviated from the grid point position of the grid-like pixel array, the first shape correction processing unit 37a performs an interpolation process such as a rendering process as necessary. For example, pixel data corresponding to the position of the display pixel is obtained by interpolation using pixel data of pixels around the pixel on the image data corresponding to the display pixel. The positions of surrounding pixels to be referred to at the time of interpolation, weighting coefficients (interpolation coefficients) to be multiplied by the pixel data of the surrounding pixels at the time of interpolation, and the like may be obtained in advance as part of the shape correction information.

  The second to fourth shape correction processing units 37b to 37d perform the same shape correction processing as that of the first shape correction processing unit 37a. That is, the second shape correction processing unit 37 b receives the second partial image data from the second edge blend processing unit 32 b and outputs the second partial image data after the shape correction processing to the second projector 22. . The third shape correction processing unit 37 c receives the third partial image data from the third edge blend processing unit 32 c and outputs the third partial image data after the shape correction processing to the third projector 23. The fourth shape correction processing unit 37d receives the fourth partial image data from the fourth edge blend processing unit 32d, and outputs the fourth partial image data after the shape correction processing to the fourth projector 24.

  The shape correction information calculation unit 38 calculates shape correction information. When calculating the shape correction information, each projector projects a shape measurement image including a plurality of feature points onto the projection surface 9. The shape measurement image on the projection surface 9 is captured by the measurement device 4, and captured image data indicating the obtained captured image is output to the shape correction information calculation unit 38. Projection and imaging of an image for shape measurement may be performed in parallel by a plurality of projectors 2 or may be performed at different timings by a plurality of projectors 2.

  The shape correction information calculation unit 38, for each of a plurality of feature points in the shape measurement image, the position and shape in the shape measurement image (referred to as the shape measurement captured image) reflected in the captured image. The shape correction information is calculated by comparing the position defined in the image data for measurement. The shape correction information calculation unit 38 stores the calculated shape correction information in the storage unit 39 and updates the shape correction information.

  Next, the processing flow and processing contents of the image processing apparatus 3 will be described with reference to FIGS.

  FIG. 4 is a flowchart showing a processing flow when calculating shape correction information and gradation correction information, and FIG. 5 is a flowchart showing a processing flow when correcting image data. Here, an example in which the shape correction information and the gradation correction information are calculated when the image display system 1 is installed will be described. In addition, the one aspect | mode of the image processing method which concerns on this invention can be performed by performing the process of following step S2-S7. Further, the function of the image processing apparatus according to the present invention can be realized by using a program that causes a computer to perform the processes of steps S2 to S7.

  As shown in FIG. 4, prior to calculating the shape correction information and the gradation correction information, first, a plurality of projectors 2 are arranged on the projection surface 9, and the arrangement of each projector is roughly adjusted as necessary. (Step S1). Next, the shape correction information is calculated by the shape correction information calculation unit 38 of the image processing apparatus 3 using the image for shape measurement projected by each projector (step S2). Next, tone correction information is calculated by the tone correction information calculation unit 35 using the tone measurement images projected by the plurality of projectors 2 in cooperation (step S3).

  As shown in FIG. 5, in order to correct the image data indicating the content image P, the tone correction processing unit 34 executes tone correction processing on the image data using the tone correction information calculated in step S3. (Step S4). Next, each partial image data is generated by the data generation unit 31 based on the image data after the gradation correction processing (step S5). Next, the first to fourth edge blend processing units 32a to 32d are caused to execute edge blend processing on each partial image data generated by the data generation unit 31 (step S6). Next, using the shape correction information calculated in step S2, the first to fourth shape correction processing units 37a to 37d execute shape correction processing on each partial image data after the edge blend processing (step S7). ).

FIG. 6 is a flowchart showing a processing flow of the projector installation work, and FIG. 7 is an explanatory diagram showing processing contents of the projector installation work.
As shown in FIG. 6, in the projector installation work, for example, the user who installs each projector designates the size (overlap amount) of the overlapping area A5 (step S11).

  In the present embodiment, the image processing apparatus 3 is provided with a user interface (not shown) that receives input of information indicating the overlap amount from the user. Information input by the user is stored in an internal or external storage unit of the image processing apparatus 3. When executing a process that requires an overlap amount, the image processing device 3 reads out information indicating the overlap amount from the storage unit and supplies it to the process. Examples of the user interface include various input devices such as a touch panel, a keyboard, and a mouse. In addition, the user inputs the arrangement on the projection surface 9 of the projection area that each projector is in charge of to the image processing apparatus 3 using the various input devices described above.

  The data indicating the arrangement of the data projection area indicating the overlap amount can be set in advance as a fixed value or a variable value. In this case, input of information indicating the overlap amount is omitted. be able to. Further, when the computer executes the shape correction process and the gradation correction process by a program, the user may input information indicating the overlap amount to the computer using various input devices connected to the computer. In this case, the computer may store and manage the input value in its own storage unit, etc., and read the input value when necessary for processing, and use it for processing.

  Next, the image processing apparatus 3 receives an input value of the overlap amount from the user, and supplies image data indicating an image including a guide of the overlapping area A5 (hereinafter referred to as a guide image) to each projector. Each projector projects the guide image P6 on the projection surface 9 as shown in FIG. 7 based on the image data supplied from the image processing device 3 (step S12). In the present embodiment, the data generation unit 31 indicates the outline of the overlapping region A5 using the arrangement of the projection region of each projector on the projection surface 9, the overlap amount, and the number of pixels of the image forming element of each projector. Image data of a guide image P6 including a figure is generated.

  The data generation unit 31 generates image data of the guide image as follows, for example. Here, it is assumed that the number of pixels of the image forming element of each projector is 1920 × 1080, and the overlap amount is 20%. In the data generation unit 31, the first projection area A1 handled by the first projector 21 is arranged at the horizontal end and the vertical end in the arrangement of the first to fourth projection areas A1 to A4. The information indicating that is acquired from the information input by the user. The data generation unit 31 overlaps the position of 396 pixels (20% of 1920 pixels) from the end of the first projection area A1 on the side where the first projection area A1 is overlapped with the second projection area A2 in the horizontal direction. As the contour of A5, image data of the guide image P6 including the guide G1 indicating the contour is generated. The user moves the first projector 21 or the second projector 22 so that the outline of the second projection area A2 substantially overlaps the outline of the guide G1.

Similarly, the data generation unit 31 generates image data of guide images for the second to fourth projectors. The user installs the first to fourth projectors while referring to the guide image for each projector (step S13). By the way, the distortion of each projection area, the distortion of the partial image caused by the relative position shift in the plurality of projection areas, and the shift of the relative position are corrected by the subsequent shape correction processing. Therefore, it is not necessary to exactly match the outline of each projection area with the guide, and the burden on the user in the installation work of the projector can be reduced.
Note that when the shape correction information is calculated after the image display system 1 is installed, for example, during maintenance of the image display system 1, the above step S1 may be omitted.

FIG. 8 is a flowchart showing the processing flow of the shape correction information calculation process, and FIG. 9 is an explanatory diagram showing a method for measuring the position of the feature point on the projection surface.
As shown in FIG. 8, in the shape correction information calculation process, first, the partial image data correction operation by the edge blend process and the partial image data correction operation by the shape correction process are prohibited (step S21). Next, as shown in FIG. 9, an image P7 for shape measurement is displayed on the projection surface 9 by each projector (step S22). The shape measurement image P7 includes a feature graphic G2. A feature figure is a figure with a shape or luminance distribution that can be detected by various image processing techniques using edge detection processing, pattern recognition processing, and the like, and when a feature figure is detected, a point that represents the feature figure is obtained as a feature point. Is a possible figure.

  As long as the shape and position of a feature graphic can be detected and the position associated with the detected shape and position can be specified as the position of the feature point, the shape, number, and arrangement are not particularly limited. . The shape of the feature graphic may be, for example, a line (cross hatch, mesh), a dot shape, or a rectangular shape intersecting each other. When the feature graphic is a line that intersects with each other, the lines constituting the feature graphic are detected by Hough transform or the like, and the position of the intersection of the detected line segment can be specified as the position of the feature point. When the feature graphic has a dot shape, it is possible to detect the contour or luminance distribution of the dot shape by the above-described image processing technique and specify the position of the center point of the dot shape as the position of the feature point. FIG. 9 illustrates a mesh-like feature graphic G2 in which line segments are periodically arranged in two directions intersecting each other. Such a feature graphic G2 can specify the position of the intersection of each line segment as the position of the feature point.

  Note that the shape measurement image is selected so that a plurality of feature points can be defined from the captured image obtained by capturing the shape measurement image. The number of feature points included in the shape measurement image projected from each projector is preferably 4 or more from the viewpoint of correcting image distortion caused by the projection direction using projective transformation. The shape measurement images projected by the projectors may be the same or different for the plurality of projectors. Image data for shape measurement indicating an image for shape measurement is stored inside (for example, the storage unit 39) or outside (for example, the signal source 8) of the image processing apparatus 3, and when the image for shape measurement is displayed. Output to each projector.

  Next, the position of the feature point on the projection surface 9 is measured (step S23). Specifically, an image for shape measurement displayed on the projection surface 9 is picked up by the measuring device 4, and an image for shape measurement shown in the obtained picked-up image (a picked-up image for shape measurement) Is detected. Here, the shape correction information calculation unit 38 receives captured image data indicating the captured image and analyzes the captured image data to detect a captured image for shape measurement. In addition, the shape correction information calculation unit 38 detects the position of the feature point of the captured image for shape measurement using the above-described Hough transform or the like.

  Next, the shape correction information calculation unit 38 defines the position of the feature point in the captured image for shape measurement obtained in step S23 and the feature point defined in the shape measurement image data indicating the image for shape measurement. The shape correction information is calculated by comparing with the position (step S24). Hereinafter, the first and second methods will be described as examples of the method for calculating the shape correction information.

  FIG. 10 is an explanatory diagram showing a first method for calculating shape correction information using projective transformation. FIG. 10 conceptually illustrates a part of the image D for shape measurement defined in the image data for shape measurement and a part of the captured image T for shape measurement.

  In FIG. 10, symbols i and j indicate the pixel arrangement direction on the image data. In the case of image data indicating an image including 1920 × 1080 pixels, for example, 1920 pixels are arranged in the i direction and 1080 pixels are arranged in the j direction. The position of each pixel in this pixel array is represented by coordinates (i, j) with reference to the pixels arranged at the end in the i direction and the end in the j direction in the pixel array. Reference signs D1 to D4 indicate feature points included in the image D for shape measurement. The feature points D1 to D4 are selected so that a line connecting the feature points in order forms the outline of the quadrangular region D5. In FIG. 10, symbols T1 to T4 indicate feature points included in the captured image T for shape measurement. The feature points T1 to T4 are feature points corresponding to the feature points D1 to D4 in the projected image D for shape measurement. A line connecting the feature points T1 to T4 in order forms an outline of a quadrangular region T5.

The conversion formula of the projective transformation can be expressed by the following formula (1) and formula (2). In Expressions (1) and (2), (x, y) indicates the coordinates (i, j) of an arbitrary point before conversion, and (X, Y) indicates the coordinates (i, j) of the conversion destination of this point. ). a to h represent conversion coefficients, and one projective transformation is obtained by obtaining a to h.
X = (ax + by + c) / (gx + hy + 1) (1)
Y = (dx + ey + f) / (gx + hy + 1) (2)

  The coordinates of each of the feature points D1 to D4 are known because they are defined in the image data for shape measurement. The coordinates of each of the feature points T1 to T4 are known by detecting the position of the feature point from the captured image T for shape measurement. Substituting the coordinates of the feature point D1 into (x, y) of the expressions (1) and (2) and substituting the coordinates of the feature point T1 into (X, Y), two relational expressions a to h are obtained. It is done. Similarly, by substituting the coordinates of the set of feature points D2 and T2, the set of feature points D3 and T3, and the set of feature points D4 and T4, eight relational expressions are obtained for the eight unknowns a to h. can get. By solving this 8-element linear equation, projection transformations a to h for transforming the region D5 into the region T5 are obtained. For the obtained projective transformation, by substituting the coordinates of each pixel included on the circumference and inside of the region D5 into (x, y), the region T5 corresponding to each pixel in the region D5 has a one-to-one correspondence. The coordinates of each pixel are obtained.

  Here, four feature points D1 to D4 are selected from a plurality of feature points included in the image D for shape measurement, and projective transformation is obtained for a region D5 forming a part of the image D for shape measurement. Then, by selecting different feature points as the feature points D1 to D4, the projective transformation is obtained by changing the region D5. Using the obtained projective transformation, the coordinates of each pixel in the region T5 corresponding one-to-one with each pixel in the region D5 are obtained as described above. Similarly, projective transformation for each part of the image D for shape measurement is obtained, and the coordinates of each pixel of the image D for shape measurement and the coordinates of the pixel of the imaging pattern corresponding to each pixel are obtained.

  The coordinates of each pixel of the image D for shape measurement have a corresponding relationship with the coordinates of the modulation element in the image forming element of each projector. Further, the coordinates of each pixel in the captured image T have a corresponding relationship with the coordinates of each display pixel on the projection surface 9. As a result, shape correction information indicating the correspondence between the coordinates of the pixel (modulation element) on the image forming element and the position of the pixel (display pixel) on the projection surface is obtained.

  For example, by performing the above-described projective transformation on the coordinates of each modulation element of the image forming element, the coordinates of pixels on the projection surface 9 (hereinafter referred to as conversion pixels) can be obtained. The range of the effective projection area A is set automatically or manually by referring to the maximum value and the minimum value of the coordinates of the converted pixel. Then, in the effective projection area A, an array of pixels (display pixels) corresponding to the format of the content image and the number of pixels is arranged, and the coordinates of each display pixel on the projection surface 9 are set as the effective projection area. It is obtained from the set value in the range of A. When the coordinates of the display pixel deviate from the coordinates of the conversion pixel, the pixel data of the display pixel is displayed by interpolation using the pixel data supplied to the conversion pixel located around the display pixel. You may obtain | require the interpolation coefficient which shows the weighting of interpolation according to the distance with a pixel. Since this interpolation coefficient becomes a fixed value for each display pixel, it is preferable to store it as a part of the correction information.

  In the second method, a projective transformation for converting the captured image T into an image D for shape measurement (reverse transformation of the projective transformation in the first method) is obtained. Further, for example, the range of the entire projection area A0 is estimated using the coordinates of the feature points included in the captured image T, and the range of the effective projection area A is set automatically or manually. Then, in the effective projection area A, the arrangement of display pixels according to the format and the number of pixels of the content image P is arranged, and the coordinates of each display pixel on the projection surface 9 are set in the effective projection area A. Calculate from the set value of the range. By converting the obtained coordinates of each display pixel by projective transformation, the coordinates of the modulation element in the image forming element corresponding to each display pixel on the projection surface 9 can be obtained. When the coordinates of the modulation element thus determined do not match the coordinates of the actual modulation element, that is, when one modulation element of the image forming element corresponds to two or more display pixels, as necessary In order to obtain pixel data input to each modulation element by interpolation, an interpolation coefficient may be obtained. As described above, the obtained interpolation coefficient may be stored as a part of the correction information.

FIG. 11 is a flowchart showing the processing flow of the gradation correction information calculation processing.
As shown in FIG. 11, in order to calculate the gradation correction information, the image data correction operation by the gradation correction process is prohibited, and the partial image data correction by the edge blend process and the partial image data by the shape correction process are performed. This correction operation is permitted (step S31).

  FIG. 12A and FIG. 12B are explanatory diagrams showing the contents of the edge blend process. FIG. 12A illustrates a line segment B-B ′ extending in the horizontal direction across the overlapping region A5 over the first and second projection regions A1 and A2. FIG. 12B shows light incident on each position on the line segment BB ′ with respect to light emitted from the first and second projectors 21 and 22 based on the partial image data after the edge blending process. Is shown as a conceptual diagram. The horizontal axis in FIG. 12B indicates the position on the line segment BB ′, and the horizontal axis in FIG. 12B corresponds to the ranges of the first and second projection areas A1 and A2 and the overlapping area A5. The relationship is also shown. The vertical axis of FIG. 12A indicates the amount of light incident on each position by the partial image data after the edge blend process, which occupies the amount of light incident on each position by the partial image data before the edge blend process. Yes.

  The amount of light derived from the first projector based on the partial image data before the edge blending process is substantially 1 in the first projection area A1 and substantially 0 in the second projection area, if the light divergence angle is ignored. Similarly, the amount of light derived from the second projector is approximately 1 in the second projection area A2, and is approximately 0 in the first projection area. The amount of light that contributes to the display of the overlapping area of the first and second partial images when projected based on the partial image data that has not been subjected to the edge blending process is considered to be approximately 2.

  As shown in FIG. 12B, the amount of light derived from the first projector based on the partial image data after the edge blending process is substantially 1 in the first projection area excluding the overlapping area A5, and on the overlapping area A5. A smooth curve is drawn toward the second projection area A2 from the first projection area A1 and gradually approaches 0. The amount of light derived from the second projector is symmetrical to the amount of light derived from the first projector, with the center of the overlapping region A5 in the horizontal direction as the axis of symmetry. That is, as the amount of light derived from the first projector decreases in the overlapping area A5, the amount of light derived from the second projector increases, and the total amount of light derived from the first and second projectors is substantially reduced at each position in the overlapping area A5. The partial image data is corrected by edge blending so as to be 1.

  Returning to the description of FIG. 11, the partial images for gradation measurement are displayed on the projection surface 9 in parallel by the plurality of projectors 2 (step S <b> 32). Next, the luminance level of each color at each position is measured at a plurality of positions of the gradation measurement image on the projection surface (step S33).

  In this embodiment, it is planned to calculate gradation correction information indicating a correction amount when correcting image data using input / output characteristics of each projector. Prior to calculating the gradation correction information, an input / output characteristic acquisition process for acquiring input / output characteristic information indicating the input / output characteristics is performed. The input / output characteristic acquisition process may be, for example, a process of reading input / output characteristic information calculated in advance and stored in a storage unit or the like, or a process of calculating input / output characteristic information.

  Here, the input / output characteristic information is calculated and acquired, and the acquired input / output characteristic information is used for calculation of gradation correction information. In order to calculate the input / output characteristic information, the luminance level on the projection surface 9 of the output measurement image projected by each projector is measured, and the output measurement image data indicating the output measurement image is specified. What is necessary is just to compare with the gradation value made.

  The process of measuring the luminance level of the image for output measurement on the projection surface 9 is parallel to the process of measuring the luminance level at a plurality of positions of the image for gradation measurement, or a part of the process or All can be done in common. In addition, the output measurement image can be used without being distinguished from the gradation measurement image. In the following description, unless otherwise specified, an output measurement image is also referred to as a gradation measurement image.

  In the present embodiment, a solid image in which the gradation value of each pixel of the image indicated by the image data before the edge blend process is the same is used as the image for gradation measurement. As the color tone of the image for gradation measurement, each basic color (red, green, blue) and monochrome (white, gray, black) used for full-color display are selected. The measurement result of the luminance level using the image for measuring the gradation of each basic color is used to calculate input / output characteristic information for each color of each projector. The luminance measurement result using the monochrome gradation measurement image is used to measure color unevenness in the gradation measurement image on the projection surface 9.

  A plurality of images having different gradation values are prepared as images for measuring the gradation of each basic color, and the relationship of the luminance level on the projection surface 9 with respect to the gradation values is determined with respect to the gradation values of a plurality of stages. Find out. Here, for each color tone of red, green, and blue, 255, 224, 192, 160, 128, among 8-bit gradation values, that is, gradation values represented by integers from 0 to 255, Images (24 types) for gradation measurement of each gradation value of 96, 64, 32 are used. For monochrome color tone, gradation measurement images (9 types) having gradation values of 9 levels of 255, 224, 192, 160, 128, 96, 64, 32, and 0 are used. That is, a total of 33 types of gradation measurement images are used.

  In order to measure the luminance level of the gradation measurement image on the projection surface 9, each gradation measurement image data is input to the data generation unit 31. When the image data for each gradation measurement is stored in the storage unit 36 of the gradation correction unit 30, the gradation correction processing unit 34 reads out the image data for each gradation measurement and outputs it to the data generation unit 31. What should I do? When image data for each gradation measurement is supplied from the signal source 8, the gradation correction processing unit 34 receives the image data for each gradation measurement from the signal source 8 and outputs it to the data generation unit 31. That's fine.

  The data generation unit 31 generates partial image data for gradation measurement for each projector based on the input image data for gradation measurement. For example, the data generation unit 31 refers to the information indicating the overlap amount acquired when generating the image data indicating the above-described guide image and the information indicating the arrangement of each projection area on the projection surface 9. Then, partial image data for gradation measurement is generated. The data generation unit 31 outputs the generated partial image data for gradation measurement to the first to fourth edge blend processing units 32a to 32d.

  The first to fourth edge blend processing units 32a to 32d perform edge blend processing on the input partial image data for gradation measurement, and the first to fourth partial image data for gradation measurement after the edge blend processing are first to first. It outputs to the 4th shape correction process parts 37a-37d.

  The first to fourth shape correction processing units 37a to 37d perform shape correction processing on the input partial image data for gradation measurement. Here, the first to fourth shape correction processing units 37a to 37d are displayed so that the solid image of the target gradation value is displayed inside the effective projection area A and black is displayed outside the effective projection area. Performs shape correction processing. The first to fourth shape correction processing units 37a to 37d output the partial image data for gradation measurement after the shape correction processing to each projector.

FIG. 13 is an explanatory diagram showing a method for measuring the luminance level on the projection surface.
As shown in FIG. 13, the gradation measurement image P <b> 8 is projected based on the partial image data after the shape correction processing, and thus is displayed inside the effective projection area A on the projection surface 9. The gradation measurement image P8 includes a first partial image P81, a second partial image P82, a third partial image P83, and a fourth partial image P84. Since the first to fourth partial images P81 to P84 are projected based on the partial image data after the edge blending process, the luminance levels of the overlapping areas that are superimposed on each other in a pair of partial images adjacent to each other, It is substantially the same as the luminance level corresponding to the gradation value defined in the image data for gradation measurement.

In order to measure the luminance level on the projection surface, the measurement device 4 captures an area including the entire gradation measurement image P8 on the projection surface 9. The measuring device 4 outputs captured image data indicating the obtained captured image to the gradation correction information calculation unit 35. The gradation correction information calculation unit 35 extracts the gradation values of the pixels of the captured image corresponding to the luminance levels of the plurality of evaluation regions E1 to E4 in the image P8 for gradation measurement from the captured image data. As for the evaluation area, if one or more evaluation areas are selected from any of the first to fourth partial images P81 to P84, the range, number, and position of the evaluation areas can be appropriately selected.
Note that the function of extracting the image for shape measurement or the image for gradation measurement from the image captured by the measurement device 4 may be provided in any of the image processing device 3, the measurement device 4, and other devices.

  As the data for calculating the input / output characteristic information, for example, the gradation value of the pixel of the captured image corresponding to the region excluding the overlapping region in each partial image is extracted from the captured image data, and the average value is obtained. What is necessary is just to use as a luminance level (output) of the partial image on the projection surface 9.

  In this way, the gradation value corresponding to the luminance level of the evaluation area on the projection surface 9 and the gradation value corresponding to the luminance level of each partial image on the projection surface 9 are acquired, and one kind of The measurement of the gradation value for the image for gradation measurement is terminated. Depending on the amount of information indicating the obtained gradation value, information indicating the gradation value may be stored in the storage unit 36.

Returning to the description of FIG. 11, the gradation correction information calculation unit 35 determines whether or not the measurement of gradation values has been completed for all types of partial images for gradation measurement (step S <b> 34). When the measurement of the gradation value is not completed for any of the partial images for gradation measurement (Step S34; No),
The measurement target partial image for gradation measurement having a gradation value different from the previous one is changed, and the process returns to step S32 to start measuring the luminance level for the next partial image for gradation measurement to be measured. If the luminance level measurement has been completed for all types of partial images for gradation measurement (step S34; Yes), the process proceeds to step S35.

  The gradation correction information calculation unit 35 calculates the gradation correction information by comparing the luminance level of each evaluation area measured in step S33 with the gradation value defined in the image data for gradation measurement ( Step S35). Hereinafter, an example of a method for calculating the gradation correction information will be described in detail with reference to numerical examples.

  Here, a case where a gray solid image is displayed without performing the gradation correction processing, for example, a case where a gray gradation measurement image having a gradation value of N is displayed in step S32 is considered. At this time, the luminance level of the red component on the projection surface 9 is 100 × (Kr × N / 255), the luminance level of the green component is 100 × (Kg × N / 255), and the luminance level of the blue component is 100 ×. It is assumed that (Kb × N / 255). The above Kr, Kg, and Kb are coefficients indicating the contribution ratio of each color light to the luminance level of the combined light. Kr is about 0.299, Kg is about 0.587, and Kb is about 0.114. When attention is paid to the relative luminance level between pixels of the same color component, the coefficients Kr, Kg, and Kb can be ignored. Therefore, in the following explanation, the coefficients Kr, Kg, and Kb are omitted. There is also.

  The gray gradation measurement image has 128 gradation values of red, green, and blue. The luminance levels of the red component, green component, and blue component of the image to be displayed at this time are 50 Kr and 50 Kg. , 50 Kb. In the following description, these luminance levels are referred to as reference luminance levels. In addition, a set of luminance levels of red, green, and blue components may be indicated by (R, G, B).

  When such an image for grayscale measurement is projected, the measured value of the luminance level of each color in the first evaluation region E1 belonging to the first partial image P81 shown in FIG. 13 is (40Kr, 50Kg, Suppose that the measured value of the luminance level of each color in the second evaluation region E2 belonging to the second partial image P82 is (55 Kr, 50 Kg, 45 Kb). That is, the first and second evaluation areas E1 and E2 have color unevenness with respect to the image to be displayed.

  In the present embodiment, the whole of the plurality of partial images inside the effective projection area A is treated as one image, and the color unevenness in the first and second evaluation areas E1 and E2 is represented by the gradation of the whole of the plurality of partial images. Correct as unevenness. Specifically, first partial image data that defines the gradation value of the first evaluation region E1 so that the luminance level of each color component in the first and second evaluation regions E1 and E2 is equal to the reference luminance level; Then, the image data that is the basis of the second partial image data that defines the gradation value of the second evaluation area E2 is corrected.

  For example, when focusing on the first evaluation area E1, the luminance level of the red component is 40 Kr, which is lower than the reference luminance level 50 Kr. Therefore, the correction amount of the gradation value required to compensate for the insufficient luminance level 10 Kr Is calculated. Then, the red component in the first evaluation region E1 can be corrected by adding this correction amount to the gradation value on the image data. Further, the luminance level of the blue component is 55 Kb, which is 5 Kb higher than the reference luminance level 50 Kb. Therefore, the gradation value on the image data is reduced by the correction amount of the gradation value necessary to reduce the excessive luminance level 5 Kb. By doing so, the blue component of the first evaluation region E1 can be corrected. Similarly, the color unevenness on the projection surface 9 can be reduced by correcting the gradation value on the image data for the second evaluation region E2.

  FIG. 14 is an explanatory diagram illustrating a method of calculating gradation correction information using input / output characteristics. The correction amount can be easily obtained by using the input / output characteristics of each projector. In FIG. 14, the measurement value of the luminance level of the image for measuring the gradation of each basic color (red, green, blue) on the projection surface 9 is changed to the gradation value defined in the image data for each gradation measurement. On the other hand, the input / output characteristics are shown in the plotted graph.

When calculating the correction amount for the red component in the first evaluation region E1, first, as shown in the following equation (3), the correction amount ΔRL (E1 of the luminance level of the red component in the first evaluation region E1 ) Is calculated. In Expression (3), RL (PR) is the reference luminance level of the red component, and RL (E1) is the luminance level of the red component in the first evaluation region E1. The reference luminance level RL (PR) is a value common to the entire area of the gradation measurement image. The reference luminance level for one gradation measurement image is set to, for example, an average value of measurement values obtained by measuring the luminance levels on the projection surface 9 in a plurality of evaluation regions in the gradation measurement image. .
ΔRL (E1) = RL (PR) −RL (E1) (3)

In the above numerical example, since RL (PR) = 50 and RL (E1) = 40, ΔRL (E1) = + 10. That is, in the gradation correction process, the gradation value N on the image data may be corrected so that the luminance of the red component increases by the luminance level correction amount ΔRL (E1). The corrected luminance level CRL (E1) related to the red component in the first evaluation region E1 is given by the following equation (4). In the above numerical example, since RL (PR) = 50 and ΔRL (E1) = + 10, CRL (E1) = 60. CRL (E1) = RL (PR) + ΔRL (E1) (4)

  Next, referring to the input / output characteristics, the gradation value N (RL (PR)) corresponding to the reference luminance level RL (PR) and the corrected gradation corresponding to the corrected luminance level CRL (E1) The value N (CRL (E1)) is obtained. In the example shown in FIG. 14, N (RL (PR)) = 171 and N (CRL (E1)) = 177.

The gradation value correction amount ΔN (E1) for the red component in the first evaluation region E1 is given by the difference between the gradation value before and after the correction, as shown in the following equation (5).
ΔN (E1) = N (CRL (E1)) − N (RL (PR)) (5)
In the example shown in FIG. 14, ΔN (E1) = + 6.

  When data indicating a gradation value N (RL (PR)) = 171 corresponding to the reference luminance level RL (PR) = 50 is input to the projector, the luminance level on the projection surface 9 in the first evaluation area E1. RL (E1) becomes 40. On the other hand, if the corrected gradation value N (CRL (E1)) = 177 is input to the projector, the luminance level on the projection surface 9 increases by about 10 to about 50, and the reference luminance level RL (PR) Is almost equal to As described above, the gradation correction processing unit 34 corrects the image data so that the gradation value of the pixel corresponding to the first evaluation region E1 in the image data before the gradation correction process is increased by ΔN (E1). That's fine.

  In the present embodiment, the gradation correction information calculation unit 35 calculates the correction amount ΔN of the gradation value according to the position of the evaluation area as the gradation correction information. Actually, an effective projection area in a captured image obtained by capturing an image for gradation measurement is divided into a plurality of divided areas, and an area on the projection surface 9 corresponding to each divided area is used as an evaluation area. Each divided region is composed of one pixel of the captured image or two or more continuous pixels. The gradation correction information calculation unit 35 stores the correction value ΔN of the gradation value for each evaluation region in the storage unit 36 in a table format in association with the position on the image data of the pixels constituting the evaluation region. The gradation correction information calculation unit 35 similarly calculates the correction amount for the green component and blue component of each evaluation region, and stores them in the storage unit 36 in a table format.

  If the gradation correction processing unit 34 is configured by a multiplication circuit that amplifies the gradation value of the image data before correction, the multiplication coefficient of the multiplication circuit is set as the correction amount. Rather than using the input / output characteristic data corresponding to the image forming elements for each color, the light output characteristics for the monochrome gradation image data, that is, the light input / output characteristics synthesized by the three image forming elements One input / output characteristic data indicating the above may be used in common for each color.

  FIG. 15A to FIG. 15C and FIG. 16A to FIG. 16C are explanatory diagrams showing the processing contents when correcting the image data indicating the content image P in the order of processing.

  When correcting the image data indicating the content image P, the gradation correction processing unit 34 receives the image data from the signal source 8 or the like. The gradation correction processing unit 34 reads the gradation correction information from the storage unit 36, and performs gradation correction processing on the image data using the gradation correction information (step S4). The image P90 (see FIG. 15A) indicated by the image data after the gradation correction processing includes gradation unevenness that cancels out the gradation unevenness caused by the projection conditions and the like on the projection surface 9.

  Next, the data generation unit 31 refers to, for example, information indicating the overlap amount and information indicating the arrangement of each projection region, and sets the overlapping region P5 in the image P90 as illustrated in FIG. Then, the data generation unit 31 generates partial image data corresponding to the partial images P91a to P91d shown in FIG. 15C (step S5).

  Next, the first to fourth edge blend processing units 32a to 32d perform edge blend processing on the partial image data indicating the partial images P91a to P91d (step S6). As shown in FIG. 16A, in the partial images P92a to P92d indicated by the partial image data after the edge blend process, the gradation value of the overlapping region P5 is lower than that of the image P90.

  Next, the first to fourth shape correction processing units 37a to 37d read shape correction information from the storage unit 39, and perform shape correction processing using the shape correction information on the partial image data after the edge blend processing (step S7). ). As shown in FIG. 16B, the partial images P93a to P93d indicated by the partial image data after the shape correction processing are image distortions caused by the projection direction, distortion of the projection surface 9, and the like, and a plurality of partial images. Image distortion that cancels out the relative position shift on the projection surface 9 is included.

  Each partial image data after the shape correction processing is input to the projector in charge of each partial image. Each projector projects a partial image on the projection surface 9 based on the input partial image data. As shown in FIG. 16C, the partial images P94a to P94d on the projection surface 9 are displayed in a state where image distortion and gradation unevenness in each partial image are corrected. The partial images P94a to P94d are displayed in a state in which the relative position shift and the luminance level difference due to the relative position shift and the rated output difference of the plurality of projectors are corrected. In this way, the content image P is displayed as an image in which the sense of unity is enhanced by the partial images P94a to P94d.

  In the image processing apparatus according to the present embodiment as described above, the gradation correction information is calculated by using a plurality of partial images on the projection surface 9 as one image. The tone correction information taking into account can be obtained efficiently.

  Regardless of the number of projectors used to display the content image P, the time and effort required to project an image for gradation measurement and measure the brightness level are the same as when using one projector. Correction information can be obtained efficiently.

  Since the gradation correction information for the overlapping region P5 is calculated in a plurality of partial images, the calculation load required for calculating the gradation correction information can be reduced. For example, when calculating gradation correction information for each partial image for a pair of partial images adjacent to each other, calculation of gradation correction information for an overlapping region is performed twice, whereas in the present embodiment, 1 is calculated. The gradation correction information for the overlapping area can be obtained by performing the calculation once.

  Since the gradation correction information is calculated for the area inside the effective projection area A, the calculation of the gradation correction information for the area not used for displaying the content image P can be omitted, and the calculation required for calculating the gradation correction information can be omitted. The load can be reduced.

  Since the gradation correction information is calculated using the image for gradation measurement projected based on the partial image data after the shape correction processing, the area on the projection surface 9 of the area whose gradation value is to be corrected is calculated. The shape and the shape defined on the image data can be substantially matched. In other words, the gradation correction information is calculated by taking into account the effect that the position of the display pixel on the projection surface 9 substantially moves before and after the shape correction processing, so that gradation unevenness can be corrected with high accuracy. Become.

  Since the gradation correction information is calculated using the image for gradation measurement projected based on the partial image data after the edge blend process, the luminance level of the overlapping region P5 superimposed on the projection surface 9 is It becomes the same level as the luminance level corresponding to the gradation value defined in the image data for tone measurement. Therefore, the numerical value distribution range of the luminance level at each position in the image for gradation measurement shown in the picked-up image captured on the projection surface 9 becomes narrow, and the luminance level at each position can be evaluated with a fine step size. Therefore, gradation unevenness can be detected with high accuracy.

  In the image display system according to the present embodiment, the shape correction processing is performed on the partial image data based on the shape correction information, so that distortion of the partial image and relative position shift can be reduced, and the display quality of the content image P can be reduced. Can be increased. In addition, since gradation correction processing is performed on image data based on gradation correction information that takes the entire image into account when a plurality of partial images are made into one image, gradation unevenness in the content image P is highly accurate and efficient. Correction can be made automatically. As a result, the display quality of the content image P can be improved, and a highly convenient image display system can be realized.

  In the image processing method of the present embodiment, it becomes possible to correct partial image data so as to reduce the distortion of the partial image and the shift of the relative position, and the gradation unevenness of the content image P is highly accurate and efficient. Correction can be made automatically.

DESCRIPTION OF SYMBOLS 1 ... Image display system, 2 ... Projector, 3 ... Image processing apparatus,
4 ... Measuring device, 8 ... Signal source, 9 ... Projected surface,
21 ... 1st projector, 22 ... 2nd projector,
23 ... Third projector, 24 ... Fourth projector,
30 ... gradation correction unit, 31 ... data generation unit,
32a ... 1st edge blend process part, 32b ... 2nd edge blend process part,
32c: third edge blend processing unit, 32d: fourth edge blend processing unit,
33 ... shape correction unit, 34 ... gradation correction processing unit, 35 ... gradation correction information calculation unit,
36 ... storage unit, 37a ... first shape correction processing unit,
37b ... second shape correction processing unit, 37c ... third shape correction processing unit,
37d ... fourth shape correction processing unit, 38 ... shape correction information calculation unit,
39... Storage unit, E1 to E4... Evaluation area, G1.
G2: Characteristic figure,
S1-S7, S11-S13, S21-S24, S31-S35 ... steps,
A ... Effective projection area, A0 ... Whole projection area, A1 ... First projection area,
A2 ... second projection area, A3 ... third projection area, A4 ... fourth projection area,
A5 ... superimposition region, B ... line segment, D ... image for shape measurement,
D1 to D4 ... feature points, D5 ... area, P ... content image,
P1 ... 1st partial image, P2 ... 2nd partial image, P3 ... 3rd partial image,
P4 ... fourth partial image, P5 ... overlapping region, P6 ... guide image,
P7 ... shape measurement image, P8 ... gradation measurement image,
P81 to P84 ... partial image for gradation measurement, T ... captured image for shape measurement,
T1 to T4 ... Features

Claims (8)

An image processing apparatus that has an image forming element and is used with a plurality of projectors that project an image on a projection surface,
A shape correction information calculation unit that calculates shape correction information indicating a correspondence relationship between the position of each pixel on the image forming element and the position of each pixel on the projection surface;
Based on the shape correction information, the shape on the projection surface of the image that the plurality of projectors collaborately projects based on the processed image data is a shape defined by the image data before the processing. A shape correction processing unit that applies a shape correction process to substantially match the image data;
Of the gradation measurement images that the plurality of projectors project in cooperation based on the gradation measurement image data that has been subjected to the shape correction processing by the shape correction processing unit, the floor that each projector projects. For a plurality of evaluation areas including an evaluation area selected from the partial image for tone measurement, the luminance level in each evaluation area of the image for gradation measurement on the projection surface, and the image for gradation measurement A gradation correction information calculation unit for calculating gradation correction information indicating a correspondence relationship with the gradation value in each evaluation region defined in the image data for gradation measurement indicating
Based on the gradation correction information, the luminance level of each pixel on the projection surface of an image projected by the plurality of projectors in cooperation with the image data after processing is set as the image data before processing. A gradation correction processing unit that performs gradation correction processing on the image data so as to approach a luminance level corresponding to the gradation value of each defined pixel;
An image processing apparatus comprising: The shape correction information calculation unit defines the shape of an image in an effective projection area set as an area that fits in the entire projection area on the projection surface by the plurality of projectors by the image data for gradation measurement. The shape correction information is calculated so as to substantially match the shape to be obtained,
The gradation correction information calculation unit is for gradation measurement on the projection surface projected based on the image data for gradation measurement after being subjected to the shape correction processing using the shape correction information. The image processing apparatus according to claim 1, wherein the gradation correction information is calculated using a luminance level of the evaluation area in the effective projection area of the image of the image.   A data generation unit that generates image data indicating a partial image projected by each projector among images projected by the plurality of projectors in cooperation based on the image data after the gradation correction processing is performed. The image processing apparatus according to claim 1, wherein the shape correction processing unit performs the shape correction processing on each image data generated by the data generation unit.   The data generation unit is an image showing the partial image such that each of a pair of partial images adjacent to each other among the plurality of partial images constituting the image includes an overlapping region overlapping the pair of partial images. The image processing apparatus according to claim 3, wherein data is generated.   Each of the overlapping regions when the overlapping regions are superimposed on each other in the pair of partial images indicated by the pair of image data after processing, with each of the pair of image data indicating the pair of partial images being processed. The luminance level of the pixel is set to a luminance level corresponding to the gradation value of each pixel in the overlapping area defined in the image data indicating the image before the gradation correction processing is performed by the gradation correction processing unit. The image processing apparatus according to claim 4, further comprising an edge blending processing unit that performs an edge blending process to approach the image processing apparatus.   The gradation correction information calculation unit performs the edge blend process on the image data indicating the partial image generated by the data generation unit based on the image data for gradation measurement by the edge blend processing unit. Gradation measurement in which the plurality of projectors collaborately project based on each image data indicating the partial image after, and the overlapping regions are superimposed on the projection surface in the pair of partial images The image processing apparatus according to claim 5, wherein the gradation correction information is calculated using a luminance level in each evaluation region of the image for use. The image processing apparatus according to any one of claims 1 to 6,
A plurality of projectors for projecting images in cooperation with each other based on the image data after being subjected to the gradation correction processing by the image processing device;
An image display system comprising: Each of the plurality of projectors, each having an image forming element and projecting an image on a projection surface, shows the correspondence between the position of each pixel on the image forming element and the position of each pixel on the projection surface. Calculating shape correction information;
Based on the shape correction information, the shape on the projection surface of the image that the plurality of projectors collaborately projects based on the processed image data is a shape defined by the image data before the processing. Applying to the image data a shape correction process for substantially matching;
A gradation measurement partial image projected by each projector among the gradation measurement images projected by the plurality of projectors in cooperation based on the image data for gradation measurement subjected to the shape correction processing. For a plurality of evaluation areas including an evaluation area selected from the above, the luminance level in each evaluation area of the image for gradation measurement on the projection surface and the gradation measurement indicating the image for gradation measurement Calculating gradation correction information indicating a correspondence relationship with a gradation value in each evaluation area defined in the image data of
Based on the gradation correction information, the luminance level of each pixel on the projection surface of an image projected by the plurality of projectors in cooperation with the image data after processing is set as the image data before processing. Subjecting the image data to a gradation correction process that approximates a luminance level corresponding to the gradation value of each defined pixel;
An image processing method characterized by comprising:

Images