JP2006311029A - Display image imaging method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a display image imaging method and apparatus capable of imaging at a free position a displayed image on an image display apparatus while being periodically updated by adopting a rolling shutter system so as to effectively suppress production of stripe noise without the need for using a synchronizing signal from hardware. <P>SOLUTION: In imaging a displayed image on the image display apparatus 110 for consecutively displaying images periodically by using the rolling shutter system, the display image imaging apparatus allows the image display apparatus 110 to display a prescribed image in advance and images the prescribed image, calculates a synchronizing exposure time at which the stripe noise caused by the display period of the image display apparatus 110 on the basis of the imaged prescribed image, and thereafter controls its exposure time on the basis of the calculated synchronizing exposure time to image the displayed image on the image display apparatus 110. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an imaging method and apparatus for imaging a display image displayed on an image display device.

  As an image display device, a multi-projection system is known in which a single image is synthesized and projected and displayed on a screen by a plurality of projectors. In such a multi-projection system, it is necessary to take measures such as making color differences and luminance differences between images projected from a plurality of projectors and the joints inconspicuous.

  Therefore, the present applicant projects an image for calibration on the screen, images it with an imaging means such as a digital camera, and performs various corrections based on the obtained imaging data. An apparatus has already been proposed (see, for example, Patent Document 1 and Patent Document 2).

  According to the image display devices disclosed in Patent Literatures 1 and 2, from the captured calibration image, the relative positional relationship between the screen and the plurality of projectors, the color difference and luminance difference between the projectors, the color unevenness and the luminance in the projector, and the like. Unevenness is measured, geometric correction parameters and color correction parameters are calculated based on the measurement data, and image correction is performed using the calculated parameters. Therefore, a large-screen seamless, high-resolution, high-quality image can be obtained. Can be projected.

  By the way, as a projector used for such an image display apparatus, for example, a single-plate projector using a single-plate display element and a three-plate projector using a three-plate display element are known.

  A single-plate projector, for example, has at least three primary color lights (red, green, red, green, etc.) between a white light source and a single-plate display element (for example, a light modulation element such as a digital micromirror device (DMD) or liquid crystal). A color wheel having a color filter that transmits blue) is arranged, and the degree of modulation of each pixel of the display element is controlled in synchronization with the rotation while rotating the color wheel at a predetermined frequency (for example, 240 Hz). Thus, each primary color image is displayed in the field sequential order. Here, since human vision recognizes an integral image of a predetermined time imaged on the retina, by setting a frame-sequential display period faster than the predetermined time (integration time), the observer The three primary color images are recognized as a full color display image.

  The three-plate projector has display elements for modulating the three primary color lights, and the three modulated lights modulated by the respective display elements are combined by, for example, a cross prism and then projected. Although this three-plate projector does not have a color wheel like a single-plate projector, a moving image can be displayed by switching the modulation image of the display element at a predetermined frequency (for example, 60 Hz).

  In addition, as a digital camera for capturing a calibration image projected on a screen at the time of calibration of an image display device, one using a CCD image sensor or one using a CMOS image sensor is known. Yes.

  Since a digital camera using a CCD imaging device generally employs a global shutter system in which an imaging start time is simultaneously within an imaging area as a shutter system, a captured image can be obtained even when an imaging target is moving. No distortion occurs.

  On the other hand, although a digital camera using a CMOS image sensor is inexpensive, it generally adopts a rolling shutter system as a shutter system. In this shutter method, for example, when imaging is performed with a predetermined exposure time, the imaging lines arranged in the vertical direction do not start exposure at the same time, but the exposure start time is changed from the uppermost imaging line to the lowermost imaging line. When the object to be imaged is stationary, there is no problem. However, when the object is moving, it is distorted due to the difference in the imaging start time of the imaging line depending on the moving speed. An image is captured.

  For this reason, if a periodically switched image displayed on the image display device is picked up by a digital camera using a CMOS image pickup device, a problem associated with the rolling shutter method occurs.

  Hereinafter, the problem associated with the rolling shutter system will be described with reference to FIGS. 10 and 11 by taking as an example the case where the image display apparatus uses a single-plate projector.

  Here, as shown in FIG. 10A, it is assumed that the color wheel 1001 of the single-plate projector rotates at a frequency α Hz and displays a uniform white image on the screen. In this state, the R (red), G (green), blue (B), and through (W) filters that make up the color wheel 1001 are only in the color light corresponding to each filter during the time that the filters are on the optical path of the white light source. Therefore, the relationship between the screen illuminance and time is as shown by a graph 1002 in FIG. 10B, and the screen illuminance is divided into R, G, B, and W areas in a time-sharing manner, and the cycle is β msec. Here, β = 1 / α.

  In order to further simplify the description, for example, a case where only a uniform red (R) image is displayed is considered. In this case, since the color light other than red is controlled to be blocked by the display element, the screen illuminance and time Is as shown by a graph 1003 in FIG. 10C, and the period of the red (R) light is β msec as in the case of the graph 1002.

  FIGS. 11A and 11B show the line positions of the CMOS image sensor superimposed on the vertical axis of the graph 1003 in FIG. In FIGS. 11A and 11B, the exposure time of the CMOS image sensor is different, and the rhombus regions 1101 and 1102 indicate the time during which the CMOS image sensor is receiving light. Note that the inclination amounts of the diamonds 1101 and 1102 differ depending on the specifications of the rolling shutter.

  FIG. 11A shows a case where the exposure time γmsec is not an integer N times βmsec. In this case, when each line position is accumulated in the rhombus 1101 in the time direction, an image with a horizontal stripe pattern having a contrast in the vertical direction as in the captured image 1103 is obtained. This occurs because the amount of received red (R) light displayed on the screen is different at each line position.

  FIG. 11B shows a case where the exposure time is β msec or an integer multiple of β msec. In this case, for each line position, if the inside of the diamond 1102 is integrated in the time direction, the received light amount of red (R) light becomes equal at all line positions, so that a uniform captured image 1104 is obtained.

  As described above, the case of capturing a red uniform display image has been described. Further, even when there are combinations of green, blue, white, etc., the condition that horizontal stripes do not occur is that the received light amount at each line position is equal. The exposure time remains the same as β × Nmsec (= integer N times the display period of the image display device). That is, when an image that is periodically displayed by an image display device is picked up by an inexpensive digital camera that employs a rolling shutter system with a CMOS image pickup device, it is necessary to select an exposure time that matches the display cycle.

  On the other hand, as a method of capturing a display image by the image display device, for example, the start and end timing of drawing of the image display device and the start of imaging of the image display device by a shutter control signal generated according to the vertical synchronization signal of the image display device In addition, it is known that image data in which noise such as horizontal stripes is not mixed is obtained by synchronizing with the end timing (see, for example, Patent Document 3).

  In addition, as a method of imaging a subject by a line sensor-driven imaging device, for example, a change in illumination light of the subject is detected, a frequency analysis is performed on the detection result, and an integral multiple of the cycle of the most frequent frequency component is exposed to the line sensor. It is also known that the influence of the illumination light change of the subject is reduced by setting the time (see, for example, Patent Document 4).

Further, as an image pickup apparatus including a rolling shutter type CMOS image pickup device, when an object is imaged under a known flicker light source (for example, a 50 Hz or 60 Hz fluorescent lamp), the shortest exposure time in which flicker does not occur and When the spatial frequency component of the horizontal stripes associated with the flicker light source excluding the spatial frequency component of the subject itself is extracted from two captured images with different shutter times and half exposure time, and this spatial frequency component exceeds a predetermined amount A device that determines that flicker has occurred is also known (see, for example, Patent Document 5).
JP 2002-72359 A JP 2002-116500 A Japanese Patent Laid-Open No. 11-184445 JP-A-7-336586 JP 2004-7402 A

  When a display image periodically updated by an image display device such as the multi-projection system described above is captured by an inexpensive digital camera that employs a rolling shutter system such as a CMOS image sensor, the display image switching cycle ( In addition to the need to accurately determine the exposure time suitable for the display cycle), the digital camera is installed at a free position sufficiently away from the screen to capture the entire image projected on the huge screen at once. There is a need.

  However, in the imaging method disclosed in Patent Document 3, it is possible to adapt the exposure time to the image display device with high accuracy, but the image display device to the imaging device is between the image display device and the imaging device. Since there is a need to connect a cable for transmitting a synchronization signal, there is a concern that it is difficult to install the imaging device in a free position sufficiently separated from the screen.

  Further, in the imaging method disclosed in Patent Document 4, low-frequency components such as shading are mixed in the captured image itself, or the size of the frequency analysis region is small and sufficient resolution cannot be obtained. In some cases, the period of the most frequent frequency component may not be the exposure time due to a change in illumination light. For this reason, in an imaging device used in a multi-projection system, there is a concern that it is impossible to stably calculate the exposure time synchronized with the image display device.

  Furthermore, the imaging device disclosed in Patent Document 5 detects flicker caused by imaging at an exposure time obtained on the assumption of known illumination light, and can handle any unknown illumination light. Can not.

  Accordingly, an object of the present invention made in view of the above points is to freely display a display image that is periodically updated and displayed on an image display device without using a hardware synchronization signal by a rolling shutter method. It is an object of the present invention to provide a display image capturing method and apparatus capable of capturing an image while effectively suppressing generation of stripe noise from a position.

The invention of the display image capturing method according to claim 1 that achieves the above object is to capture a display image of an image display device that continuously displays images periodically by a rolling shutter method.
A predetermined image is displayed and captured in advance on the image display device, and based on the captured image of the predetermined image, a synchronous exposure time is calculated that does not cause striped noise associated with the display cycle of the image display device, and then the calculation is performed. The display time of the image display device is picked up by controlling the exposure time based on the synchronized exposure time.

  According to a second aspect of the present invention, in the display image capturing method according to the first aspect, the synchronous exposure time includes a local peak frequency of a frequency component in a direction crossing the striped noise generated in the captured image of the predetermined image and its The evaluation time calculated based on the size is an exposure time that minimizes the evaluation value.

  According to a third aspect of the present invention, in the display image capturing method according to the first or second aspect, the predetermined image is a uniform gradation single primary color image.

Furthermore, the invention according to claim 4 that achieves the above object is a display image capturing device that captures a display image of an image display device that continuously displays images periodically by a rolling shutter method.
Synchronous exposure time calculation means for calculating an exposure time that does not cause striped noise accompanying the display cycle of the image display device based on a captured image of the predetermined image displayed on the image display device;
Exposure time control means for controlling the exposure time when the display image of the image display device is captured based on the synchronous exposure time calculated by the synchronous exposure time calculation means;
It is characterized by having.

  According to a fifth aspect of the present invention, in the display image capturing device according to the fourth aspect, the synchronous exposure time calculating means is a local peak frequency of a frequency component in a direction crossing the striped noise generated in the captured image of the predetermined image. And an exposure time at which an evaluation value calculated based on the size is minimized is calculated as the synchronous exposure time.

  According to a sixth aspect of the present invention, in the display image capturing device according to the fourth or fifth aspect, the synchronous exposure time calculating unit is configured to perform the synchronous exposure time based on a captured image of a predetermined image composed of a uniform primary gradation image. Is calculated.

  According to the first aspect of the present invention, since the exposure time synchronized with the display cycle of the image display device can be determined with high accuracy without using a hardware synchronization signal, a rolling shutter system such as a CMOS image sensor can be used. By using a low-cost imaging device to be adopted, it is possible to effectively suppress the stripe noise generated in the captured image of the display image while giving a degree of freedom to the installation position of the imaging device with respect to the image display device.

  According to the second aspect of the present invention, even when a non-uniform display pattern other than the stripe noise is mixed in the captured image of the predetermined image, the synchronous exposure time can be calculated with sufficient accuracy.

  According to the third aspect of the present invention, since it is possible to prevent high-frequency components other than striped noise from being mixed in the captured image of the predetermined image, the synchronous exposure time can be calculated with higher accuracy.

  According to the fourth aspect of the invention, the exposure time synchronized with the display cycle of the image display device can be determined with high accuracy without using a hardware synchronization signal. Can be configured at low cost, and stripe noise generated in the captured image of the display image can be effectively suppressed while providing a degree of freedom in the installation position of the imaging device with respect to the image display device.

  According to the fifth aspect of the present invention, the synchronized exposure time can be calculated with sufficient accuracy even if a non-uniform display pattern other than striped noise is mixed in the captured image of the predetermined image.

  According to the sixth aspect of the present invention, since it is possible to prevent high-frequency components other than striped noise from being mixed into the captured image of the predetermined image, the synchronous exposure time can be calculated with higher accuracy.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic diagram showing a schematic configuration of an image display imaging system using an imaging apparatus according to an embodiment of the present invention. In this embodiment, an image displayed by the image display device 110 is picked up by an image pickup device 111, and the image display device 110 constitutes a rear projector type multi-projection system including two projectors 107 and 108. A display image processing device 106 that controls a display image such as dividing one image into two projectors 107 and 108, and a screen 109 that displays a projection image emitted from the projectors 107 and 108 are provided.

  The imaging device 111 includes a camera 101 that captures a display image displayed on the screen 109, a monitor 104 that monitors the captured image, and a computer 102. The computer 102 determines the exposure time of the camera 101 and the like. It has a camera control unit 103 that controls, and a calibration data calculation unit 105 that calculates correction information of the color shift between the two projectors 107 and 108 and the geometric shift of the projected image using the captured image.

  Here, for example, as shown in FIG. 2, the camera 101 includes an imaging unit 201, an imaging lens 203, a turret 204, and a turret driving motor 202 including a CMOS imaging device and its driving circuit. The turret 204 holds tristimulus X, Y, and Z color filters 205, 206, and 207, ND filters 208 and 209 having different densities, a through hole 210, and a light shielding plate 211 in a concentric manner. Is driven to rotate, and each filter is moved to the position of the imaging lens 203.

  FIG. 3 is a functional block diagram showing the configuration of the camera control unit 103 shown in FIG. First, the calibration data calculation unit 105 first creates a table of exposure times corresponding to the image update frequency of the image display device 110 in order to capture a desired image with the camera 101. Therefore, when there is known information as an image update frequency for the image display device 110, the information is input to the exposure control unit 303.

  The exposure control unit 303 instructs the image display device 110 to display a uniform luminance image of a single primary color (for example, red) and displays it. Here, the reason why the uniform luminance image of a single primary color is displayed is to improve the calculation accuracy in the synchronous exposure time calculation method described later. That is, by displaying a uniform luminance image of a single primary color, it is possible to clearly capture horizontal stripe noise, thereby eliminating spatial high-frequency components that adversely affect calculation accuracy caused by other than horizontal stripe noise. .

Further, the exposure control unit 303 sets a search range of exposure time for imaging with respect to the camera 101. Here, the search range [T _short , T _long ] of the exposure time is T _short = (N−3 + 0.5) / α and T _long = (N + 0.5) / as the image update frequency αHz for the image display device 110. α (where N is an integer of 3 or more). If the image update frequency αHz is unknown, for example, the image update frequency α = 60 Hz is provisionally determined to calculate the exposure time range setting. The search exposure time within the synchronous exposure time search range [T _short , T _long ] is set to an interval that is an integer M times the minimum time of exposure control of the camera 101 itself, and the value of M is the detection of the synchronous exposure time to be calculated. It is determined in advance in consideration of the accuracy and the calculation processing time.

Next, the maximum value of the display area of the captured image obtained by capturing the displayed uniform luminance image of the single primary color with the camera 101 is a predetermined gradation level range (for example, 200 ± 10 for 256 gradations of 8 bits). Under these conditions, the turret 204 of the camera 101 is rotated so as to be in the vicinity of the exposure time T _long, and any one of the ND filters 208 and 209 or the through hole 210 is selected and imaged to determine the turret position. At this time, if the aperture of the imaging lens 203 can be automatically controlled, the aperture may be further adjusted. Further, when the adjustment cannot be made by these switching operations, the image display device 110 is requested to change the luminance level.

Specifically, the exposure control unit 303 obtains the exposure time in the default state of the camera 101 stored in the first exposure time table 312 and set to be selectable in the exposure control unit 303 via the table selection unit 313. Then, the acquired exposure time is set in the camera 101 to perform imaging, and it is determined whether or not the maximum value of the display area in the captured image is within a predetermined gradation level range with respect to the captured image. In this case, the filters such as the ND filters 208 and 209 of the camera 101 are switched and further the brightness level change control of the image display device 110 is performed. Thus, an initial state for measuring the synchronized exposure time, that is, a synchronized exposure time search range [T _short , T _long ] is determined.

Subsequently, the exposure controller 303 sets the determined synchronized exposure time search range [T _short , T _long ] to a predetermined interval D (D = M × minimum time during which exposure control is possible) from T _long to T _short ; The exposure time T _j (j = _{1 to} L; L is the number of samples, and T ₁ = T _long , T _L = T _short ) is set in the camera 101 and images are taken sequentially at each exposure time. Accordingly, the sequential captured images are input to the synchronous exposure time detection unit 304.

The synchronized exposure time detection unit 304 determines three synchronized exposure time candidates T _sync1 , T _sync2 , and T _sync3 from the synchronized exposure time search range based on the input captured image, and outputs them to the synchronized exposure time determination unit 310.

The synchronous exposure time determination unit 310 _obtains a regression line by the method of least squares from the three synchronous exposure time candidates T _sync1 , T _sync2 , T _sync3, and a condition of passing through the origin (T _sync = 0 and no horizontal stripes are generated). Thus, the minimum time interval β _m msec as the actual measurement value and the correlation coefficient are calculated, and it is determined that the synchronous exposure time has been determined only when the linearity is sufficiently satisfied, and the determination result is output to the exposure control unit 303. At the same time, an exposure time of β _m × N (where N is 1 or more and β _m × N is an integer not exceeding the longest exposure time as the hardware specification of the camera 101) is set in the second exposure time table 311.

  On the other hand, when it is determined that the synchronous exposure time cannot be determined, the determination result is similarly output to the exposure control unit 303, but the exposure time is not set for the second exposure time table 311. . The determination result that the synchronous exposure time cannot be determined is particularly when the display image switching in the image display device 110 is not clearly turned ON / OFF and the contrast of the horizontal stripe cannot be determined. Arise.

  Thus, the setting of the synchronized exposure time for capturing the display image of the image display device 110 by the image capturing device 111 is completed. Thereafter, when the exposure control unit 303 controls the table selection unit 313 to determine the synchronized exposure time, the selectable exposure time stored in the second exposure time table 311 as the first exposure control mode. Is used to determine the optimum exposure time. If the synchronized exposure time cannot be determined, the optimum exposure time is determined using the selectable exposure time stored in the first exposure time table 312 as the second exposure control mode. The display image is captured by the camera 101 with the determined optimum exposure time, and the captured image is sent to the calibration data calculation unit 105.

  The calibration data calculation unit 105 calculates calibration data based on the captured image from the camera control unit 103 so that the image display device 110 can display an optimal image, and uses the calculated calibration data as the image display device. 110 to the display image processing apparatus 106. As a result, the display image processing apparatus 106 performs image processing on the external video signal based on the calibration data, and outputs it to the projectors 107 and 108.

  Here, the synchronous exposure time detection unit 304 includes a detection area extraction unit 305, a discrete Fourier transform (DFT) calculation unit 306, a local peak extraction unit 307, a synchronous evaluation value calculation unit 308, and a minimum value detection unit 309. . Details will be described below.

The captured image of the exposure time T _j input to the synchronized exposure time detection unit 304 is input to the detection area extraction unit 305. The detection area extraction unit 305 displays a captured image on the monitor 104 and allows the operator to manually select a detection area of a predetermined size, and the display area of the display image from the input captured image has a gradation difference from the surroundings. The detection area is extracted by one of the modes for automatically extracting a detection area of a predetermined size (for example, 128 pixels × 128 pixels) in which the gradation level does not change suddenly in the display area. Then, the detection area is output to the DFT calculation unit 306. Here, switching between the two modes can be performed manually.

  The DFT calculation unit 306 extracts a plurality of n columns in the vertical direction of a predetermined detection area, and eliminates the discontinuity of the boundary portion by multiplying the extracted sequences by a window function such as a Hanning window. Processing is performed (in the case where the number of processing data is a power of two, fast Fourier transform (FFT) is also possible), and the spatial frequency component that is the processing result is output to the local peak extraction unit 307.

In the local peak extraction unit 307, the input n-sequence spatial frequency components for n columns are integrated at each frequency position, and an opening for calculating a minimum value within a predetermined local frequency range for the integrated spatial frequency components is performed. Apply processing to generate spatial frequency components without local peaks, and subtract the generated spatial frequency components without local peaks from the accumulated spatial frequency components to extract only the local peak waveform, and then extract the extracted local peaks A plurality of peak frequencies f _i and corresponding peak values p _i are extracted from the waveform and output to the synchronous evaluation value calculation unit 308.

The synchronization evaluation value calculation unit 308 calculates a synchronization evaluation value E _j = Σw (f _i ) f _i p _i for the captured image captured at the exposure time T _j and stores it in the internal memory. Here, w (f _i ) is a weighting coefficient, for example, a function having a predetermined frequency as a parameter to increase the weight on the high frequency side and decrease the weight in the low frequency example.

When the minimum value detection unit 309 receives from the exposure control unit 303 imaging end information indicating that imaging of the exposure time _TL , which is the last sample position in the synchronous exposure time search range, has been completed, the synchronization evaluation value calculation unit 308. The positions of at least three local minimum values (minimum values) are determined with higher accuracy than the sample time interval from the _L synchronization evaluation values E _{1 to} E _L stored in the memory. This local minimum value can be determined with high accuracy by interpolating the synchronization evaluation values at sub-sample positions finer than the sample interval using the synchronization evaluation values at a plurality of sample positions. The determined positions of the at least three minimum values are at least three synchronous exposure time candidates T _sync1 and T _{sync2 according} to the synchronous exposure time search range [T _short , T _long ] input from the exposure control unit 303 and the sample interval. , T _sync3 and output to the synchronous exposure time determination unit 310.

FIG. 4 schematically shows a part of the synchronous exposure time search range [T _short , T _long ] by the exposure control unit 303 and a detection area image in the captured image with respect to the exposure time T _j sampled within the range. It is a figure. A part of the sampled exposure time T _j and detection area image in the synchronous exposure time search range [T _short , T _long ] correspond to reference numerals 401 to 410. Further, a graph 411 with the exposure time on the horizontal axis and the horizontal stripe intensity on the vertical axis shows that no horizontal stripes are generated at the synchronized exposure times 404 and 410, and the dark portion of the horizontal stripes becomes thicker as the distance from the time increases. ing.

  FIG. 5 is a diagram for explaining the processing contents in the detection area extraction unit 305 and the DFT calculation unit 306. On the monitor 104, a captured image 504 obtained by capturing the display image displayed on the screen 109 of the image display device 110 with the camera 101 is displayed. In this captured image 504, reference numeral 501 indicates the projection area of the projector 107, and reference numeral 502 indicates the projection area of the projector 108.

  The detection area extraction unit 305 extracts, for example, the boundary portions of the three sides of the projection area 502 of the projector 108 and automatically extracts the detection area 503 at the center of the projection area 502 or by manual operation by the operator. A desired detection area 503 is extracted.

  The DFT calculation unit 306 first samples n columns in the vertical direction of the detection area 503 extracted by the detection area extraction unit 305. One column of the data is indicated by reference numeral 505, and the relationship between the line position of the image sensor and the gradation level is indicated by reference numeral 506. Next, a single row of data 506 is multiplied by a window function 507 such as a Hanning window, and DFT (FFT can also be used when the data size is a power of 2) to obtain a spatial frequency component 508.

  FIG. 6 is a diagram for explaining the processing contents in the local peak extraction unit 307 and the synchronization evaluation value calculation unit 308.

The n rows of spatial frequency components 508-1 to 508-n sampled from the detection area 503 are integrated for each same frequency to obtain one spatial frequency component 601. The spatial frequency component 601 is subjected to an opening process (Op) for calculating the trajectory of the circle 602 by tracing the circle 602 having a radius R from the lower side of the graph, for example. A frequency component 603 is obtained. Then, by subtracting the spatial frequency component 603 from the spatial frequency component 601, a spatial frequency component 604 obtained by extracting only the local peak component is obtained, and the local peak frequencies f ₁ ,..., F _N are obtained from the spatial frequency component 604. peak power value _p 1 corresponding to, extracts ..., and _{p N.}

The synchronization evaluation value calculation unit 308, the local peak frequency _f 1 is input, ..., peak power _p 1 corresponding to _{f N,} ..., using the _{p N} and the weight w _{(f 1),} The synchronous exposure evaluation value E _j is obtained from the following equation at the exposure time T _j . Here, the weight w (f _i ) is predetermined by a function of frequency.

  FIG. 7 is a graph of the synchronization evaluation values stored in the memory held by the synchronization evaluation value calculation unit 308. The horizontal axis indicates the exposure time, and the vertical axis indicates the synchronization evaluation value.

In FIG. 7, the search range 702 corresponds to the synchronous exposure time search range [T _short , T _long ] determined by the exposure control unit 303, and in this range, the synchronous evaluation value is at least three local minimums. The exposure time that becomes a waveform 701 that takes a value and becomes the local minimum value (minimum value) detected by the minimum value detection unit 309 is the synchronous exposure time candidates T _sync1 , T _sync2 , and T _sync3 . However, when the image update frequency αHz is unknown, a default value (for example, 60 Hz) is used. However, since the image update frequency of a normal image display device is larger than the default value, a local minimum of the waveform 701 is used. The value (minimum value) is 3 or more. In this case, three synchronized exposure time candidates are selected from the longer exposure time.

  FIG. 8 is a flowchart showing a processing procedure until the synchronous exposure time is determined in the multi-projection system of FIG.

  When the synchronized exposure time calculation process is started, first, a single primary color having a predetermined uniform luminance is displayed on the image display device 110 (step S801).

  Subsequently, the imaging device 111 obtains an exposure time in an asynchronous state that is a target gradation level (step S802).

  Thereafter, it is determined whether or not the obtained exposure time is equal to or longer than the predetermined exposure time (step S803). If it is less than the predetermined exposure time, it is determined whether or not the ND filter included in the imaging device 111 can be switched (step). (S804) If the switch is possible, the ND filter is switched (step S806), and the process returns to step S802.

  On the other hand, if the ND filter cannot be switched in step S804, a change process for lowering the luminance level of the image display apparatus 110 is performed (step S805), and the process returns to step S802.

  If the exposure time is equal to or longer than the predetermined exposure time in step S803, the synchronous exposure time search range and the search exposure time (number of samples) are determined (step S807), and one of the plurality of search exposure times is set in the camera 101. Then, the detection area in the captured image is extracted (step S808).

  Thereafter, it is determined whether or not the number of extracted predetermined column data in the vertical direction from the extracted detection area is less than a predetermined value (step S809). If the number of extracted columns is less than the predetermined value, the predetermined column of the detected image area is determined. Is subjected to DFT processing, integrated to the same frequency component of the column already subjected to DFT processing (step S810), and the process returns to step S809.

  On the other hand, if the number of extracted columns reaches a predetermined value in step S809, an opening process is performed on the integrated frequency component to extract a local peak (step S811).

  Thereafter, a synchronous exposure evaluation value E is calculated based on the extracted local peak information (peak frequency and peak power value) and stored in the memory (step S812).

  Next, it is determined whether or not the number of searched exposure samples is less than the number of samples determined in step S807 (step S813). If the number of samples is less than the determined number of samples, the search exposure time is changed (step S814), and step S808 is performed. Return to.

  On the other hand, if the number of searched exposure samples becomes the determined number of samples in step S813, a synchronized exposure time candidate is calculated for which the synchronized exposure evaluation value stored in the memory is a local minimum value (step S815).

  Thereafter, it is determined whether or not the calculated synchronized exposure time candidates are less than a predetermined number (for example, three) (step S816), and if it is less than the predetermined number, the process returns to step S815.

  On the other hand, when the calculated number of synchronized exposure time candidates reaches a predetermined number, it is determined whether or not the calculated number of synchronized exposure time candidates satisfies the predetermined condition (step S817), and the predetermined condition is satisfied. If it is, the minimum interval of the synchronized exposure times × integer multiple (1 to N) is determined as N synchronized exposure times and stored in the exposure time table (step S818), and the synchronized exposure time calculation process is terminated. To do.

  On the other hand, if the predetermined condition is not satisfied in step S817, the synchronous exposure time calculation process is terminated without updating the exposure time table without determining the synchronous exposure time.

  FIG. 9 is a flowchart showing a processing procedure until calibration data creation in the multi-projection system of FIG.

  First, before performing test pattern imaging, a usable synchronous exposure time of the imaging apparatus 111 with respect to the image display apparatus 110 is calculated (step S901).

  Next, the calculation state of the synchronized exposure time is determined (step S902). If it is determined that the calculation is possible, the usable synchronized exposure time is stored in the second exposure time table 311 (step S903), and the synchronized exposure is performed. The time use flag is set to ON (step S904).

  On the other hand, if it is determined in step S902 that the synchronous exposure time cannot be calculated, the use flag of the synchronous exposure time is set to OFF (step S905). Thus, the preparation for capturing the test pattern image for calibration data calculation is completed.

  Subsequently, as a test pattern image imaging process, first, a test pattern image is displayed (step S906). Thereafter, the use flag of the synchronous exposure time is determined (step S907), and when the use flag is ON, the second exposure time table 311 storing the calculated synchronous exposure time is displayed. When the use flag is OFF, the first exposure time table 312 in which all exposure times that can be used by the camera 101 are registered in advance is determined. The optimum exposure time of the displayed test pattern image is determined (step S909), and the test pattern image is captured with the optimum exposure time (step S910).

  Thereafter, it is determined whether or not all the test pattern images for calculating the calibration data have been captured (step S911). If there are any test pattern images to be captured, the process returns to step S906 and is not captured. When another test pattern image is displayed and imaging is repeated and imaging of all the test pattern images is completed, calibration data is calculated based on the captured image at that time, and the calculated calibration data is displayed as an image. It transmits to the display image processing apparatus 106 of the display apparatus 110 (step S912), and a calibration process is complete | finished.

  As described above, when the display image of the image display device 110 is imaged by the inexpensive image pickup device 111 using a rolling shutter method such as a CMOS image sensor, the synchronous exposure time in which no horizontal stripes are generated is expressed as the image display device 110. Thus, it is possible to calculate with high accuracy without inputting a synchronization signal. Therefore, the imaging device 111 that captures a test pattern for calculating calibration data used for a large-scale image display device 110 such as a multi-projection system is installed at a free position without being limited by the physical cable length. can do.

  In addition, this invention is not limited to the said embodiment, Many deformation | transformation or a change is possible. For example, in the above embodiment, the synchronous exposure time calculation process calculates three synchronous exposure time candidates at a time to determine the minimum interval of the synchronous exposure time, and an integral multiple of this minimum interval is determined as the synchronous exposure time. The synchronous exposure time may be calculated sequentially when the test pattern is captured. In this case, first, an exposure time at which a captured image has a predetermined luminance level is determined using an asynchronous exposure time table (first exposure time table), and thereafter, exposure time at a predetermined interval shorter than the exposure time. The image is captured and the synchronous exposure evaluation value is calculated in the same manner as in the above embodiment, and the process is repeated until the evaluation value takes the minimum value within a predetermined time. And it is sufficient. In this way, it takes time to determine the optimum synchronized exposure time, but since the synchronized exposure time is obtained one by one, the occurrence of striped noise can be suppressed more reliably.

  In the above embodiment, the display image of the multi-projection system is picked up. However, the present invention similarly eliminates the occurrence of striped noise when picking up the display image using a liquid crystal monitor of a personal computer or the like. be able to. Furthermore, the detection area of the striped noise is not limited to a part of the captured image, but may be the entire captured image corresponding to the display image.

It is the figure which showed typically the structure of the multi-projection system which concerns on one embodiment of this invention. It is a figure which shows schematic structure of the camera shown in FIG. It is a functional block diagram which shows the structure of the camera control part shown in FIG. It is the figure which showed typically a part of synchronous exposure time search range by the exposure control part shown in FIG. 3, and the detection area image in the picked-up image with respect to the exposure time sampled within the range. It is a figure for demonstrating the processing content in the detection area extraction part and DFT calculation part which are shown in FIG. It is a figure for demonstrating the processing content in the local peak extraction part and synchronous evaluation value calculation part which are shown in FIG. FIG. 4 is a graph showing the synchronization evaluation values stored in the memory held by the synchronization evaluation value calculation unit shown in FIG. 3. It is a flowchart which shows the process sequence until it determines the synchronous exposure time in the multi-projection system of FIG. It is a flowchart which shows the process sequence until calibration data creation in the multi-projection system of FIG. It is the figure which showed typically the surface sequential display of the single plate type projector. It is the figure which showed typically the noise which generate | occur | produces in the relationship between the surface sequential display of a single-panel projector, and the imaging time of a rolling shutter system, and a captured image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 102 Computer 103 Camera control part 104 Monitor 105 Calibration data calculation part 106 Display image processing apparatus 107,108 Projector 109 Screen 110 Image display apparatus 111 Imaging apparatus 303 Exposure control part 304 Synchronous exposure time detection part 305 Detection area extraction part 306 DFT calculation unit 307 Local peak extraction unit 308 Synchronization evaluation value calculation unit 309 Minimum value detection unit 310 Synchronization exposure time determination unit 311 Second exposure time table 312 First exposure time table 313 Table selection unit

Claims (6)

In capturing a display image of an image display device that continuously displays images periodically using a rolling shutter method,
A predetermined image is displayed and captured in advance on the image display device, and based on the captured image of the predetermined image, a synchronous exposure time is calculated that does not cause striped noise associated with the display cycle of the image display device, and then the calculation is performed. A display image capturing method for capturing a display image of the image display device by controlling the exposure time based on the synchronized exposure time.   The synchronous exposure time is an exposure time at which the evaluation value calculated based on the local peak frequency of the frequency component in the direction crossing the striped noise generated in the captured image of the predetermined image and its magnitude is minimized. The display image capturing method according to claim 1, wherein:   The display image capturing method according to claim 1, wherein the predetermined image is a uniform gradation single primary color image. A display image capturing device that captures a display image of an image display device that continuously displays images periodically by a rolling shutter method,
Synchronous exposure time calculation means for calculating an exposure time that does not cause striped noise accompanying the display cycle of the image display device based on a captured image of the predetermined image displayed on the image display device;
Exposure time control means for controlling the exposure time when the display image of the image display device is captured based on the synchronous exposure time calculated by the synchronous exposure time calculation means;
A display image imaging device comprising:   The synchronous exposure time calculation means, the exposure time when the evaluation value calculated based on the local peak frequency and the magnitude of the frequency component in the direction across the striped noise generated in the captured image of the predetermined image is minimized, The display image capturing apparatus according to claim 4, wherein the display image capturing apparatus calculates the synchronous exposure time.   6. The display image capturing apparatus according to claim 4, wherein the synchronous exposure time calculating unit calculates the synchronous exposure time based on a captured image of a predetermined image made up of a uniform gradation single primary color image.

Images