JP2013109117A - Image display apparatus, image display method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain good color reproducibility by performing a color tone correction based on accurate visual environment information on a displayed area.SOLUTION: An image display apparatus having display means for displaying an image in a displayed area includes: imaging means for imaging an observation image including the displayed area in which the display means displays an image; reference area determining means for determining a reference area for calculating visual environment information on the displayed area from the observation image; visual environment information calculation means for calculating the visual environment information on the displayed areal from the reference area, and image processing means for correcting an image signal based on the visual environment information on the displayed area which is calculated by the visual environment information calculation means and transmitting the image signal to the display means. Thereby, the image display apparatus is enabled to accurately calculate the visual environment information on the displayed area so as to perform color correction and obtain good color reproducibility.

Description

  The present invention relates to an image display device, an image display method, and a program.

  A projector is widely used as an image display device that projects and displays an image on a display area such as a screen. In many cases, a dedicated white screen is used for the image projection surface of the projector. However, a colored wall surface, a partition, or the like may be used, and the projection surface is not necessarily white. In some cases, the projection surface is used in a state where it is illuminated by light from the illumination or the environment.

  Therefore, for example, a technique is proposed in which a predetermined pattern image projected by a projector is photographed using an image sensor or the like, visual environment information such as wall color and illumination light is obtained from the photographed image, and image data actually projected is corrected. Has been.

  For example, Patent Document 1 includes a projection plane information detection unit that detects color information and luminance information of a projection plane on which an image is projected from a projector, and the projection plane color information detected by the projection plane information detection unit. And a projector with a correction circuit for performing white balance adjustment and luminance adjustment based on brightness information. For example, Patent Document 2 proposes an image display system that corrects and displays an image color based on visual environment information obtained by visual environment grasping means that grasps a visual environment in a display area of an image.

  According to the technique according to Patent Document 1 or Patent Document 2, it is possible to perform color adjustment by grasping the viewing environment of the projection surface, and thus it is possible to perform good color reproduction.

  Here, depending on the installation status of the projector and the state of the projection lens, the size and position of the projected area may change, and the image captured by the image sensor or the like may include an area other than the image display area by the projector. In such a case, accurate visual environment information of the display area cannot be obtained, and it is difficult to accurately correct the color tone of the projected image.

  Thus, for example, Patent Document 3 includes a distance detection unit that detects a distance from the image projection apparatus to the projection surface, and a zoom state detection unit that detects a zoom state of the projection optical system. An image projection apparatus is disclosed that calculates a projection area of an image based on the zoom state and performs color adjustment processing based on visual environment information of the calculated projection area.

  According to the image projecting apparatus according to Patent Document 3, since the visual environment information of the display area can be obtained by excluding the area where the image from the projector is not projected from the captured image of the image sensor or the like, the image is projected. It is possible to accurately perform color tone correction on image data.

  However, it is difficult to accurately extract only the image projected by the projector from the captured image of the image sensor or the like, and as a result, the visual environment information of the display area is obtained based on the image signal including other than the projected image of the projector. In some cases, it is difficult to perform color tone correction based on accurate visual environment information.

  Accordingly, an object of the present invention is to provide an image display device, an image display method, and a program capable of obtaining good color reproducibility by performing color tone correction based on accurate visual environment information of a display area. To do.

  In view of the above problems, the present invention is an image display device having display means for displaying an image on a display area, and an imaging means for capturing an observation image including the display area on which the display means displays an image; A reference area determining means for determining a reference area for calculating visual environment information of the display area from the observation image, and a visual environment information calculating means for calculating visual environment information of the display area from the reference area And image processing means for correcting an image signal based on the visual environment information of the display area calculated by the visual environment information calculating means and transmitting the corrected image signal to the display means.

  According to the embodiment of the present invention, good color reproducibility can be obtained by performing color tone correction based on accurate visual environment information of a display area.

It is a figure which shows the usage example of the image display apparatus which concerns on embodiment. 1 is a functional block diagram of an image display device according to an embodiment. It is a figure which shows the hardware structural example of the image display apparatus which concerns on embodiment. It is a flowchart of the color tone correction process in the image display apparatus according to the embodiment. It is a flowchart of the visual environment information calculation process in the image display apparatus according to the embodiment. It is a figure explaining the example of the display area of the image display apparatus which concerns on embodiment, an observation image, and a reference area. It is a figure which shows the example of the division | segmentation and weighting of a reference area in the image display apparatus which concerns on embodiment. It is a flowchart of the visual environment information calculation process which performs the smoothing process in the image display apparatus which concerns on embodiment. It is a figure explaining the distance with an image display area when an image display device inclines. It is a figure which shows the example of determination of the reference area | region when the image display apparatus which concerns on embodiment is inclined.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention (hereinafter referred to as “embodiments”) will be described in detail with reference to the drawings.

<Configuration of projector>
FIG. 1 is a diagram illustrating a usage example of the projector 10 according to the present embodiment.

  The projector 10 according to the embodiment of the present invention projects and displays an image input from, for example, a PC on a screen 20 or the like provided substantially in front of the display unit of the projector 10. The presenter 30 can make a presentation to a third party using an image displayed in the image display area 21 on the screen 20.

  Here, the image displayed on the screen 20 by the projector 10 is displayed in a color different from the color of the original image input from a PC or the like depending on the type of the screen 20 and the ambient light 40 such as illumination or natural light. May be visible.

  Therefore, the projector 10 according to the present embodiment includes an image sensor 50 as an imaging unit that captures an image including the image display area 21 of the screen 20 and outputs a signal corresponding to the captured image.

  The projector 10 obtains visual environment information including the color of the screen 20 and the ambient light 40 from the observation image captured by the image sensor 50, and corrects the color tone of the image to be displayed based on the visual environment information. The color tone of the original image can be reproduced on the screen 20.

  FIG. 2 shows a functional block diagram of the projector 10 according to the present embodiment.

  The projector 10 includes a control unit 11, a display unit 15, and an image sensor 50. The control unit 11 includes an image processing unit 12, a visual environment information calculation unit 13, and a reference area determination unit 14. In the projector 10, the built-in control unit 11 may be provided in, for example, a server connected to the projector 10, and the image sensor 50 may be provided outside the projector 10.

  The display unit 15 includes a light source, a lens, and the like, and projects and displays an image on the screen 20 based on image data sent from the image processing unit 11.

  The image sensor 50 includes, for example, a solid-state image sensor such as a CCD or a CMOS. The image sensor 50 captures an image projected on the screen 20 by the display unit 15 and forms an image on a light receiving plane of the image sensor using an optical system such as a lens. Photoelectrically converted and output as an electrical signal.

  The reference area determination unit 14 determines a reference area for obtaining visual environment information from the electrical signal output from the image sensor 50.

  The visual environment information calculation unit 13 calculates the visual environment information of the display region of the screen 20 from the reference region determined by the reference region determination unit 14.

  The image processing unit 12 receives an image signal from, for example, an external PC, and corrects the color tone of the input image signal based on the visual environment information of the image display area 21 calculated by the visual environment information calculation unit 13. The image data is transmitted to the display unit 15.

  FIG. 3 shows a hardware configuration example of the projector 10 according to the present embodiment.

  The projector 10 includes a control unit 11, a main storage unit 102, an operation unit 104, a recording medium I / F unit 105, an auxiliary storage unit 107, a network I / F unit 108, and the like, which are mutually connected by a bus. .

  The control unit 11 is a CPU (Central Processing Unit) that controls each device, calculates and processes data in a computer, and is a calculation device that executes a program stored in the main storage unit 102 or the like.

  The main storage unit 102 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like, and is a storage device that stores or temporarily stores programs executed by the control unit 11, data, and the like.

  The auxiliary storage unit 107 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The operation unit 104 includes, for example, a key switch (hard key), and is used when setting the functions of the projector 10.

  The recording medium I / F unit 105 is an interface between the recording medium 106 such as a flash memory and the projector 10 connected via a data transmission path such as USB (Universal Serial Bus).

  A predetermined program is stored in the recording medium 106, the program stored in the recording medium 106 is installed in the projector 10 via the recording medium I / F unit 105, and the installed predetermined program is controlled by the control unit 11. It becomes executable.

  The network I / F unit 108 has a communication function connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) constructed by a data transmission path such as a wired and / or wireless line. This is an interface between the device and the projector 10.

<About color correction processing>
Next, a basic tone correction algorithm executed based on the image signal output from the image sensor 50 will be described.

  The viewing environment information including the color of the screen 20 or the wall surface on which the projector 10 projects an image, the influence of lighting, and the like projects a plurality of image patterns on the display area, and the image sensor 50 includes the display area. It is obtained by determining the color tone correction parameter from the image.

  In general, the following relationship is established between the observation image (XYZ value) acquired by the image sensor 50 and the input image (RGB value) received by the projector 10.

ｃ ： イメージセンサ５０が撮影する観察画像のＸＹＺ値
ｉ ： プロジェクタ１０に入力される画像のＲＧＢ値
ｉチルダ ： 入力画像のＲＧＢ値に応答関数の逆関数をかけた値
ｔ ： 環境光成分
Ｍ ： ３×３の色変換行列
この様に、イメージセンサ５０が撮影する観察画像の色は、プロジェクタ１０の入力画像のＲＧＢ値を線形変換したものに、環境光成分を足し合わせたものとして表される。

c: XYZ value of observation image taken by image sensor 50 i: RGB value of image input to projector 10 i tilde: Value obtained by multiplying RGB value of input image by inverse function of response function t: Ambient light component
M: 3 × 3 color conversion matrix As described above, the color of the observation image captured by the image sensor 50 is expressed as a result of linear conversion of the RGB value of the input image of the projector 10 and the ambient light component. Is done.

  Since the output of the projector 10 is not linear with respect to the input image, a response function indicating input / output of the projector 10 is obtained in advance, and an RGB value (i tilde) obtained by multiplying the RGB value of the input image by the inverse function of the response function is obtained. , As an input to the projector 10.

  In order to determine the 3 × 3 color conversion matrix M and the parameter of the ambient light component t based on the above formula, the projector 10 projects a predetermined pattern image on the screen 20 and an observation image including each projected pattern image Is imaged by the image sensor 50.

  As the pattern image projected by the projector 10 to determine the above parameters, for example, black, red, green, and blue single color images can be used, and the RGB values of each input pattern image are expressed in 256 gradations as follows. It is as a formula.

黒一色の画像（入力ＲＧＢ値が全て０の画像）からは、環境光成分ｔを求めることができる。また、他の単色画像については独立なベクトルを３つ与えることで、色変換行列Ｍを求めることができる。ここでは、パターン画像の観察画像からＲＧＢ特性を分離し易いこと、また、使用するイメージセンサ５０のＷＢ（White Balance）の影響を受けにくいことから、単色画像をパターン画像として与えて色変換行列Ｍのパラメータを算出している。

The ambient light component t can be obtained from a black image (an image in which the input RGB values are all 0). In addition, the color conversion matrix M can be obtained by giving three independent vectors for other monochromatic images. Here, since it is easy to separate the RGB characteristics from the observation image of the pattern image and it is difficult to be influenced by the WB (White Balance) of the image sensor 50 to be used, a color conversion matrix M is given by giving a single color image as the pattern image. The parameters are calculated.

  By projecting a predetermined pattern image in this way, visual environment information such as the color conversion matrix M and the environmental light component t can be calculated.

  Accordingly, the RGB value of the input image signal is corrected so that the color information of the image observed by the image sensor 50 of the image projected by the projector 10 approximates the color information of the input image signal. Good color reproducibility can be obtained regardless of environmental information.

  FIG. 4 shows an example of a flowchart of color tone correction processing in the projector 10 according to the present embodiment.

  First, the projector 10 projects a predetermined pattern image on the screen 20 (S1). Next, the image sensor 50 acquires an observation image including the projected pattern image, and outputs a signal corresponding to the observation image (S2).

  A signal output from the image sensor 50 is sent to the reference region determination unit 14, and the reference region determination unit 14 and the visual environment information calculation unit 13 perform visual environment information calculation processing for calculating the color conversion matrix M and the environmental light component t. It is executed (S3).

  After the visual environment information is calculated, the input image signal is corrected based on the visual environment information, and the display unit 15 projects the color-corrected image (S4).

  Here, in order to accurately obtain the visual environment information of the non-display area, the image display area 21 on which the projector 10 projects the pattern image and the observation image acquired by the image sensor 50 are completely matched. desirable. However, since the installation state of the projector 10, the state of the lens, and the like are not always constant, it is difficult to configure the image display area 21 and the observation image to always match.

  Therefore, for example, as shown in FIG. 6, the observation image 51 acquired by the image sensor 50 is larger than the image display area 21 of the pattern image projected by the projector 10, and other than the pattern image projected by the projector 10 on the observation image 51. May be included.

  Even if the color conversion matrix M and the ambient light component t are obtained from the observation image 51 including a region other than the predetermined pattern image projected by the projector 10, it is difficult to calculate accurate visual environment information because it is influenced by other than the pattern image. become.

<About reference area>
Therefore, in the projector 10 according to the present embodiment, the reference region determination unit 14 determines the reference region 53 corresponding to the pattern image projected by the projector 10 from the observation image 51, and excludes the region other than the pattern image. Information is calculated.

  For example, when the projected pattern image is a single color image of R, G, and B, the reference region 53 extracts pixels having a color in a specific range from the observation image 51 and includes the extracted pixels. The smallest rectangle is determined as the reference area 53.

  Since the extracted pixels in the peripheral portion of the minimum rectangle are easily affected by resolution conversion or the like and have low reliability, the reference region 53 is excluded by excluding pixels located at a preset distance from the peripheral portion of the extracted minimum rectangle. It is preferable to determine as

  As illustrated in FIG. 6, the reference region determination unit 14 extracts the reference region 53 on which the pattern image is projected from the observation image 51, so that the visual environment information calculation unit 13 is affected by the region other than the pattern image. It is possible to calculate visual environment information without the need.

  FIG. 5 shows an example of a flowchart of visual environment information calculation processing in the projector 10 according to the present embodiment.

  First, the reference area determination unit 14 acquires an image signal corresponding to the observation image 51 acquired by the image sensor 50 (S31). Next, the image signal of the observation image 51 is analyzed (S32), and a reference area 53 corresponding to the image display area 21 projected by the projector 10 is determined (S33). The representative color of the determined reference area 53 is obtained, and visual environment information such as the color conversion matrix M and the environmental light component t is calculated (S34).

  When the entire display area where the projector 10 displays an image has a uniform color, the tone correction can be appropriately performed by obtaining one color conversion matrix M from the entire reference area 53. Further, when the visual environment of the image display region 21 is locally different, such as when projecting on a wall surface having a pattern or when the influence of illumination or the like is partially different, the reference region 53 is set to 64 × 64, for example. It is possible to obtain good color reproducibility by dividing the visual field information for each region and performing color tone correction for each divided region.

  As a method for obtaining the representative color of the reference region 53, for example, a uniform quantization method can be used. The uniform quantization method is a method in which the color distribution area is simply divided equally into N representative colors (N = 1 if the representative color is to be reduced to one), and the median value of each area is used as the representative color. is there. Normally, all pixel values of an image are projected onto an RGB space or the like, and the distribution area is sequentially divided equally for each axis until the number of divided areas becomes N, and the median value of each divided area is used as the representative color of the area. To do.

  Here, as the image projected by the projector 10, it is preferable that the color reproducibility of the image displayed at the central portion is better than the peripheral portion of the image display region 21 because of the size of the impression to be given. Therefore, after obtaining the representative color of the image display area 21 so as to be close to the color displayed near the center of the image display area 21, the visual environment information is obtained from the representative color of the image display area 21 to obtain the color tone. It is desirable to make corrections.

  Therefore, by dividing the reference area 53 and assigning a weight to each divided area, a representative color close to the color near the center of the image display area 21 can be obtained from the pixel values weighted for each area. For example, as shown in FIG. 7, the reference area 53 is divided into a plurality of areas from the central part to the peripheral part. The weight of the central part is “10”, and the distance from the center is “5”. “2” and “1” are set so that the weight becomes small.

  In this divided and weighted state, for example, all pixels are selected at the center of the weight “10”, and 50% of pixels are randomly selected from the area of the weight “5”. By adjusting the number of pixels to be plotted, the representative color can be obtained so as to be close to the display color near the center of the image display area 21. Therefore, color tone correction optimized near the center of the image display area 21 is performed, and a better impression can be given about the color reproducibility of the display image.

  Note that the division and weighting method is not limited to the example shown in FIG. 7, and the representative color of the reference region 53 can be obtained from only one point by setting the weight of the pixels other than one pixel near the center to “0”. It is possible and can be set as appropriate. As a method for obtaining the representative color of the reference region 53, a median value division method, a color space linear division method, a frequency method, or the like for obtaining a representative color based on the statistic of the color distribution of the original image can be used.

  In addition, noise (roughness) may occur in a single color portion of an image captured by the image sensor 50 depending on the characteristics of the image sensor and processing characteristics such as signal amplification processing and edge enhancement processing. Therefore, the observation image 51 including a solid area of R, G, B single colors used as an image pattern displayed by the projector 10 may have variations in pixel values.

  Therefore, it is effective to obtain visual environment information after smoothing the observation image 51 and reducing noise in a low-frequency region that is noticeable as roughness in the observation image 51. As the smoothing process, for example, by using a moving average method, instead of the density of a certain pixel, an average density of pixels in a predetermined area centered on the pixel is given, thereby removing noise included in the image. it can.

  FIG. 8 shows an example of a flowchart of visual environment information calculation processing for performing smoothing processing.

  First, the reference area determination unit 14 acquires an image signal corresponding to the observation image 51 from the image sensor 50 (S301), and analyzes the signal (S302). Next, the observation image 51 is smoothed (S302), and the reference region 53 is determined from the smoothed observation image 51 (S304). When the reference region 53 is determined, the visual environment information calculation unit 13 calculates the visual environment information by obtaining a representative color from the reference region 53 (S305).

  In this way, by performing the smoothing process on the observation image 51, the roughness in the observation image 51 is suppressed, and the visual environment information can be obtained more accurately. As the smoothing process, other than the moving average method, for example, a weighted average method, a median filter, a winner filter, or the like can be used.

  Further, when the projector 10 is installed in a state tilted from the horizontal, the upper side is caused by the difference between the distance from the display unit 15 of the projector 10 to the upper part in the image display area 21 of the screen 20 and the lower part. And may be projected at different brightness on the lower side.

  For example, as shown in FIG. 9, when the projector 10 is installed in a state where it is tilted backward by the inclination θ, the distance Lu from the projector 10 to the upper part of the image display area 21 of the screen 20 is the distance Ll to the lower part. Longer than. In this state, the image display area 21 of the screen 20 is displayed with different brightness so that the upper part is dark and the lower part is bright.

  As described above, when the visual environment information is obtained and the color tone correction is performed in a state where the brightness is partially different in the image display area 21, the image after the color tone correction may be too bright or conversely dark. .

  Therefore, the projector 10 is provided with an inclination sensor 55 as an inclination detecting means for detecting the inclination of the projector 10, and the reference region 53 is determined based on the inclination θ detected by the inclination sensor 55, thereby eliminating the above-described problems. .

  FIG. 10 shows an example of determining the reference area 53 when the projector 10 is installed at an angle.

  For example, when the projector 10 is tilted backward at an inclination angle θ as shown in FIG. 9, the area to be excluded is changed according to the inclination angle θ as shown in FIG. 21 is set so that the upper part of 21 becomes the reference area 53.

  When the projector 10 is tilted forward at the tilt angle θ, the area to be excluded is changed according to the tilt angle θ as shown in FIG. The reference area 53 is set.

  In this way, the projector 10 is provided with the tilt sensor 55, and the position of the reference region 53 is shifted according to the tilt angle θ, thereby calculating the visual environment information of the image display region 21 regardless of the installation state of the projector 10. Good color reproducibility can be obtained.

  In the embodiment described above, the projector 10 as the image display device realizes each function related to the present invention described above by executing a program provided in advance incorporated in a ROM or the like, for example. Can be configured. The program executed by the projector 10 has a module configuration including a program for realizing each means (the image processing unit 12, the visual environment information calculation unit 13, the reference area determination unit 14, and the like). As the CPU (processor) reads out the program from the ROM and executes it, a program for realizing the respective units (functional units) is loaded into the main storage unit 102 and the like, and the above-described means are realized. Yes.

  The program executed by the projector 10 according to the above-described embodiment is an installable or executable file, such as a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk), or the like. It may be configured to be recorded on a computer-readable recording medium.

  Furthermore, the program executed by the projector 10 according to the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the projector 10 according to the above embodiment may be configured to be provided or distributed via a network such as the Internet.

<Summary>
As described above, according to the embodiment of the present invention, the projector 10 projects and displays the pattern image on the screen 20 or the like, the image sensor 50 acquires the observation image including the displayed pattern image, and the pattern By excluding a region other than the image and extracting a reference region suitable for calculating visual environment information of the display region, accurate visual environment information of the display region can be obtained. Therefore, since it is possible to perform color tone correction on the input image based on accurate visual environment information of the display area and project and display the image, good color reproduction is always possible regardless of the installation environment of the projector 10. Sex can be obtained.

  It should be noted that the present invention is not limited to the configuration shown here, such as a combination with other elements in the configuration described in the above embodiment. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form.

10 Projector (image display device)
12 Image processing unit (image processing means)
13 color information detector (detection means)
14 Reference area determination unit (reference area determination means)
15 display unit 21 display area 50 image sensor (image signal output means)
51 Observation Image 53 Reference Area 55 Inclination Sensor (Inclination Detection Unit)

Japanese Patent No. 2973477 Japanese Patent No. 3707350 JP 2010-130481 A

Claims (8)

An image display device having display means for displaying an image in a display area,
Imaging means for capturing an observation image including the display area on which the display means displays an image;
A reference area determining means for determining a reference area for calculating visual environment information of the display area from the observation image;
Visual environment information calculating means for calculating visual environment information of the display area from the reference area;
Image processing means for correcting an image signal based on the visual environment information of the display area calculated by the visual environment information calculating means and transmitting the corrected image signal to the display means;
An image display device comprising: The reference area determining means extracts an image area corresponding to the display area from the observation image, and determines the reference area by excluding a predetermined range set in advance from a peripheral portion of the image area. The image display apparatus according to claim 1. The image display apparatus according to claim 2, wherein the visual environment information calculation unit performs weighting for each area obtained by dividing the reference area, and calculates the visual environment information of the display area.   The image display apparatus according to claim 3, wherein the weight given by the visual environment information calculation unit is set so as to become smaller for each region away from the center of the display unit. Inclination detecting means for detecting the inclination of the image display device,
5. The image display apparatus according to claim 2, wherein a range to be excluded from a peripheral portion of the image area is changed in accordance with the inclination detected by the inclination detection unit.   The visual environment information calculation unit calculates the visual environment information of the display area after performing smoothing processing on the reference area. Image display device. An image display method in an image display device having display means for displaying an image in a display area,
An imaging step of capturing an observation image including the display area on which the display means displays an image;
A reference area determining step for determining a reference area for calculating visual environment information of the display area from the observation image;
Visual environment information calculating step for calculating visual environment information of the display area from the reference area;
An image signal processing step of correcting the image signal based on the calculated visual environment information of the display area and transmitting the corrected image signal to the display means;
An image display method characterized by comprising: In an image display device comprising: display means for displaying an image in a display area; and imaging means for capturing an observation image including the display area in which the display means displays an image.
Computer
A reference area determining means for determining a reference area for calculating visual environment information of the display area from the observation image;
Visual environment information calculating means for calculating visual environment information of the display area from the reference area;
A program for functioning as an image signal processing unit that corrects an image signal based on the calculated visual environment information of the display area and transmits the corrected image signal to the display unit.

Images