JP5955003B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image projection system that projects a single image by a plurality of image projection apparatuses.

  Conventionally, there has been proposed a method of projecting images from a plurality of projectors to different projection areas, and displaying these images by connecting the projected images (for example, Patent Documents 1 and 2). In Patent Document 1, calibration patterns are projected from a plurality of projectors arranged so that the projection areas overlap, and the projection images are subjected to correction processing such as geometric correction based on them to easily connect the projection images. It is matched. In Patent Literature 2, the luminance of the overlapping area is corrected according to the spatial frequency, thereby suppressing degradation of the resolution due to the positional deviation of the overlapping area.

JP 2005-500756 gazette JP 2009-260932 A

  However, in Patent Document 1, when the number of sensor pixels of the camera is small with respect to the screen resolution of the display area constituted by a plurality of projectors, the positional deviation between the projected images increases and the resolution of the superimposed area is impaired. There was a problem. Further, in Patent Document 2, there is a problem that the displacement itself cannot be reduced in the first place, and particularly when an image including a high frequency component is displayed, the resolution is impaired.

An image processing apparatus according to the present invention is an image processing apparatus connected to a plurality of projection means in a projection system that displays a single image by simultaneously projecting a plurality of images from a plurality of projection means, Input means for inputting a plurality of captured images obtained by capturing a plurality of projection images projected from the plurality of projection means from different viewpoint positions using an imaging device having a plurality of imaging means; In the plurality of captured images, a plurality of projection areas, each of which corresponds to one of the plurality of projection images, includes a projection area having a smaller number of pixels than the corresponding projection image to the plurality of using the captured image, and generating means for generating a high-resolution image resolution of the projection region pixel number is smaller than the projected image is improved to at least the corresponding, pre In one image displayed by projection from a plurality of projection means, an acquisition means for acquiring area information indicating an area corresponding to each image projected from the plurality of projection means, and each area indicated by the area information A setting means for setting a reference point for alignment; a detection means for detecting a point corresponding to the reference point in the projection area; and the plurality of projection means based on the high-resolution image A deformation parameter for deforming the image projected from the plurality of projection means, and the deformation parameter is determined such that each image projected from the plurality of projection means is projected onto a corresponding region indicated by the region information. and determination means for possess, said determining means, said a point detected by the detecting means, wherein as said reference point set by said setting means overlaps deformation parameters And determining the.

  According to the present invention, when a single image is displayed using a plurality of projectors, it is possible to improve the resolution while suppressing positional deviation between the projected images.

1 is a diagram illustrating an example of a system configuration of an image processing apparatus according to a first embodiment. 1 is a diagram illustrating an example of a multi-eye imaging device including a plurality of imaging units according to Embodiment 1. FIG. In Example 1, it is a figure which shows a mode that the single image which connected the projection area | region of the some projector was displayed based on the image data imaged with the imaging device. 1 is a functional configuration diagram of an image processing apparatus according to Embodiment 1. FIG. 3 is a flowchart illustrating a flow of image processing in the image processing apparatus according to the first embodiment. It is a figure which shows an example of the image for geometric correction, (a) is a plain white image, (b) is a lattice-pattern lattice image. It is a figure which shows an example of captured image data. It is a flowchart which shows the flow of the image generation process for correction coefficient calculation. It is a figure which shows the process of calculating the pixel count of each projection area | region. It is a figure explaining the resolution increasing process in Example 1. FIG. It is a flowchart which shows the flow of a correction coefficient calculation process. It is a figure which shows the correspondence of a display area and the image for a display. It is a figure which shows the correspondence of a high resolution image and a partial image. It is a flowchart which shows the flow of an image correction process. It is a figure which shows the correspondence of the image for a display, and a partial image. FIG. 6 is a diagram illustrating an example of an imaging apparatus using a multi-view system including a plurality of imaging units according to a second embodiment. In Example 2, it is a figure which shows a mode that the single image which connected the projection area | region of the some projector was displayed based on the image data imaged with the imaging device. It is a figure which shows an example of captured image data. It is a figure explaining the resolution increasing process in Example 2. FIG.

Example 1
First, a system configuration example of the image processing apparatus according to the present embodiment will be described.

  FIG. 1 is a diagram illustrating an example of a system configuration of an image processing apparatus 100 according to the present embodiment. In FIG. 1, a CPU 101 uses a RAM 102 as a work memory, executes a program stored in a ROM 103 and a hard disk drive (HDD) 105, and controls each component to be described later via a system bus 118. Thereby, various processes described later are executed.

  The HDD interface (I / F) 104 is an interface such as serial ATA (SATA) for connecting a secondary storage device such as the HDD 105 or an optical disk drive.

  The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD I / F 104. Further, the CPU 101 can expand the data stored in the HDD 105 in the RAM 102 and similarly store the data expanded in the RAM 102 in the HDD 105. The CPU 101 can execute the data expanded in the RAM 102 as a program.

  An imaging interface (I / F) 106 is a serial bus interface such as USB or IEEE 1394 for connecting an imaging device 107 having a plurality of imaging units (camera units). The CPU 101 can control the imaging device 107 via the imaging I / F 106 to perform imaging. Further, the CPU 101 can read data captured from the imaging device 107 via the imaging I / F 106.

  An input interface (I / F) 108 is a serial bus interface such as USB or IEEE 1394 for connecting an input device 109 such as a keyboard and a mouse. The CPU 101 can read data from the input device 109 via the input I / F 108.

  The output interfaces (I / F) 110 to 113 are video output interfaces such as DVI and HDMI that connect image projection apparatuses (projectors) 114 to 117 that display images by projecting them onto a screen or the like. The CPU 101 can send data to the projectors 114 to 117 via the output I / Fs 110 to 113 to execute display.

  FIG. 2 is a diagram illustrating an example of a multi-eye imaging device 107 including a plurality of imaging units according to the first embodiment. The imaging device 107 includes nine imaging units 201 to 209 that capture an image. Nine imaging units are equally arranged on a square lattice.

  When receiving an imaging instruction, the imaging units 201 to 209 receive light information of the subject with a sensor (imaging device), A / D conversion is performed on the received signal, and a plurality of images (digital data) are simultaneously acquired. .

  With such a multi-eye imaging device, image data (multi-view image data) obtained by imaging the same subject from a plurality of viewpoint positions can be obtained.

  Although the number of imaging units is nine here, the number of imaging units is not limited to nine. In addition, although an example in which nine image capturing units are uniformly arranged on a square lattice has been described here, the arrangement of each image capturing unit is arbitrary. For example, they may be arranged radially or linearly, or may be arranged at random.

  Furthermore, instead of using an imaging device having a plurality of imaging units as described above, multi-viewpoint image data is obtained by imaging multiple times while changing the imaging position using an imaging device having one imaging unit. It may be.

  In this embodiment, based on the image data simultaneously captured by the imaging device 107 shown in FIG. 2, the projection areas of the four projectors 114 to 117 are connected to form a display area to be described later to display a single image. The case where it does is demonstrated. FIG. 3 is a diagram illustrating a state in which a single image obtained by connecting projection areas of a plurality of projectors is displayed based on image data simultaneously captured by the imaging device 107. Three rectangular areas 301 to 303 indicated by three different types of broken lines respectively indicate the imaging ranges of three imaging units among a plurality of imaging units in the imaging device 107. That is, the imaging range 301 corresponds to the imaging unit 204, the imaging range 302 corresponds to the imaging unit 205, and the imaging range 303 corresponds to the imaging unit 206. Four projection areas 305 to 308 by the projectors 114 to 117 are projected onto the screen 304 so as to partially overlap. An area 309 indicated by diagonal lines is a display area, and four projection areas 305 to 308 are connected and displayed here.

  At this time, if the number of pixels corresponding to the projection area is smaller than the screen resolution of each projector 114 to 117 on the image captured by one imaging unit (single eye), the images captured by the plurality of imaging units are used as the basis. A process for stitching the projection areas together is performed. Note that the imaging units 201 to 209 all have the same angle of view and the number of sensor pixels and are directly facing the screen 304.

  Needless to say, the number of projectors is not limited to four, and may be any number such as six or nine.

  FIG. 4 is a functional configuration diagram of the image processing apparatus 100 according to the present embodiment. The configuration shown in FIG. 4 is realized as image processing application software. That is, it is realized by the CPU 101 executing various software (computer programs) stored in the HDD 105 or the like.

  The image processing apparatus 100 acquires image data of a calibration pattern image, imaging parameters, projector information, and image data of a display image from a storage device such as the input device 109 or the HDD 105. The imaging parameters include imaging conditions such as the exposure time and ISO sensitivity of the imaging units 201 to 209. The projector information includes information on the screen resolution of the projectors 114 to 117.

  The imaging device control unit 401 sends an imaging command to the imaging device 107 based on the imaging conditions included in the acquired imaging parameters. When the imaging device 107 performs imaging based on the received imaging command, the captured image data is sent to the correction coefficient calculation image generation unit 402 via the imaging I / F 106.

  The correction coefficient calculation image generation unit 402 generates correction coefficient calculation image data based on the screen resolution of the projector included in the acquired projector information and the captured image data received from the imaging device 107. The generated correction coefficient calculation image data is sent to the correction coefficient calculation unit 403.

  The correction coefficient calculation unit 403 calculates correction coefficients necessary for performing geometric correction on the display image based on the received correction coefficient calculation image data. The calculated correction coefficient is sent to the image correction unit 404.

  The image correction unit 404 performs geometric correction processing on the acquired display image data to generate corrected image data. The generated corrected image data is sent to the image output unit 405.

  The image output unit 405 outputs the calibration pattern image data and the corrected image data to the projectors 114 to 117 via the output I / Fs 110 to 113.

  FIG. 5 is a flowchart illustrating the flow of image processing in the image processing apparatus 100 according to the present embodiment. Actually, after a computer-executable program describing the following procedure is read from the ROM 103 or HDD 105 onto the RAM 102, the CPU 101 executes the program to execute the process.

  In step 501, the image processing apparatus 100 acquires image data of a geometric correction image as a calibration pattern image, imaging parameters, projector information, and image data of a display image. 6A and 6B are diagrams illustrating an example of a geometric correction image. FIG. 6A is a plain white image, and FIG. 6B is a lattice pattern lattice image. Hereinafter, a case where the white image of (a) is adopted as the geometric correction image will be described as an example.

  In step 502, the image output unit 405 outputs the acquired image data of the calibration pattern image to the projectors 114 to 117. In response to this, each of the projectors 114 to 117 projects a calibration pattern image (white image) onto the screen 304.

  In step 503, the imaging device control unit 401 gives an imaging command to the imaging device 107, and acquires data of a captured image (an image obtained by capturing a white image projected on the screen) from the imaging device 107. At this time, two types of captured image data of the entire projection state of the projectors 114 to 117 and the individual projection state for each projector are acquired. FIG. 7 is a diagram illustrating an example of captured image data acquired in this step. The image set 0 is a set of images obtained by capturing a state in which a white image is projected on the screen 304 by all projectors. Image set 1 to image set 4 are a set of images obtained by capturing a state in which a white image is projected onto the screen 304 by an individual projector. Image set 1 corresponds to projector 114, image set 2 corresponds to projector 115, image set 3 corresponds to projector 116, and image set 4 corresponds to projector 117. Each image set includes a total of nine images corresponding to the imaging units 201 to 209.

  In step 504, the correction coefficient calculation image generation unit 402 generates correction coefficient calculation image data based on the projector information acquired in step 501 and the captured image data acquired in step 503. Specifically, first, the screen resolution of each projector included in the projector information is compared with the number of pixels in the projection area of each projector in the captured image. If the screen resolution of the projector is smaller as a result of the comparison, an image captured by the image capturing unit 205 among the images included in the captured image is used as the correction coefficient calculation image. On the contrary, when the screen resolution of the projector is larger, a high-resolution process is performed using a plurality of images captured by the imaging units 201 to 209 to generate a correction coefficient calculation image. Details of this correction coefficient calculation image generation processing will be described later.

  In step 505, the correction coefficient calculation unit 403 calculates a correction coefficient to be used in the geometric correction process for the display image based on the generated correction coefficient calculation image data. Details of the correction coefficient calculation process will be described later.

  In step 506, the image correction unit 404 performs geometric correction processing on the display image using the generated correction coefficient, and generates corrected image data to be projected by the projectors 114 to 117. Details of this correction processing will be described later.

  In step 507, the image output unit 405 outputs the generated corrected image data to the projectors 114 to 117. In response to this, each of the projectors 114 to 117 projects a corrected image on the screen 304.

  Through the above processing, the display image data is projected as a single image by the plurality of projectors 114 to 117.

<Image generation processing for correction coefficient calculation>
FIG. 8 is a flowchart showing the flow of correction coefficient calculation image generation processing.

  In step 801, the correction coefficient calculation image generation unit 402 receives captured image data.

  In step 802, the correction coefficient calculation image generation unit 402 acquires screen resolutions of the projectors 114 to 117 from the projector information. The information of the screen resolution is given by the number of pixels of width: W and the number of pixels of height: H, for example, 1920 × 1080.

  In step 803, the correction coefficient calculation image generation unit 402 detects the projection areas of the projectors 114 to 117 from the received captured image data, and calculates the number of pixels in each projection area. In this pixel number calculation process, four images captured by predetermined image capturing units (here, the image capturing unit 205) included in each of the image sets 1 to 4 in the captured image data are used. These four images will be referred to as projection area detection image 1 to projection area detection image 4 in the order of projectors 114 to 117 for convenience. FIG. 9 is a diagram illustrating a process of calculating the number of pixels in each projection area. First, binarization is performed on the projection region detection images 1 to 4 by threshold processing. Thereby, the pixel values of the projection area detection images 1 to 4 are converted into luminance values. Then, the threshold value th set in advance is compared with the luminance value in each pixel of the projection region detection images 1 to 4. Binary images 1 to 4 are generated with pixels as other regions and different label values. Then, the number of pixels in the projection area included in each of the binary images 1 to 4 is counted. T1 to T4 represent the number of counted pixels.

  In step 804, the correction coefficient calculating image generation unit 402 compares the number of pixels obtained from the screen resolution of the projectors 114 to 117 acquired in step 802 with the number of pixels in each projection area acquired in step 803. Specifically, first, the product of the screen resolutions W and H is obtained, and the obtained value is set to S (W × H = S). Then, the obtained value S is compared with T1 to T4. As a result, if the pixel number S obtained from the screen resolution is larger than any one of the pixel numbers T1 to T4 of each projection area, the process proceeds to step 805. On the other hand, when the number S of pixels obtained from the screen resolution is smaller than any of T1 to T4, an image captured by a predetermined imaging unit included in each image set in the captured image data (for example, captured by the imaging unit 205) Image) is determined as a correction coefficient calculation image, and the process proceeds to step 806.

  In step 805, the correction coefficient calculation image generation unit 402 performs a resolution enhancement process (super-resolution process in this embodiment) based on the captured image to generate a correction coefficient calculation image. FIG. 10 is a diagram for explaining a state of high resolution processing in the present embodiment. As shown in FIG. 10, super-resolution processing is performed on each of the image sets 0 to 4 to generate high resolution images 0 to 4. As the super-resolution processing, an existing method such as a MAP (Maximum A Poster) method or an IBP (Iterative Back Projection) method can be applied. Hereinafter, a case where the high-resolution image 0 is generated from the image set 0 using the MAP method will be described. The MAP method is a method for estimating a high-resolution image that minimizes an evaluation function obtained by adding probability information of a high-resolution image to a square error. Specifically, the high resolution image is estimated as an optimal problem for maximizing the posterior probability by using certain look-ahead information for the high resolution image and a parameter representing the positional deviation between the images according to the following equation (1).

  Here, X is the high-resolution image 0, and Yk is the image data included in the image set 0. D is a matrix representing preset downsampling processing. B is a matrix for performing deterioration processing by PSF set in advance. M is a matrix representing the positional deviation between the image data, and is obtained by estimating the positional deviation amount of other images based on the image captured by the imaging unit 205. C is a matrix representing a linear filter to be applied to the estimated high resolution image 0. For example, a Laplacian filter can be applied. σ is a standard deviation of the noise amount of the image data included in the image set 0, and α is a parameter for adjusting the smoothing degree. The high-resolution image 0 is estimated from the image data included in the image set 0 by solving the minimization problem shown in Equation (1) using the steepest descent method. The high-resolution images 0 to 4 generated by performing such super-resolution processing on the image sets 0 to 4 are correction coefficient calculation images.

  In step 806, the correction coefficient calculation image generation unit 402 outputs the generated correction coefficient calculation image data to the correction coefficient calculation unit 403, and ends this processing.

  In step 804, the number of pixels obtained from the screen resolution of the projectors 114 to 117 is compared with the number of pixels in the projection area. However, for example, a method of detecting a rectangle circumscribing the projection area and comparing the width W ′ and height H ′ with the screen resolution W and H may be used.

  If the relationship between the screen resolution (number of pixels) of the projectors 114 to 117 and the number of pixels in the projection area is known in advance, the information is stored in the HDD 105 or the like, and steps 802 to 804 are omitted. It may be.

<Correction coefficient calculation process>
FIG. 11 is a flowchart showing the flow of correction coefficient calculation processing.

  In step 1101, the correction coefficient calculation unit 403 receives correction coefficient calculation image data. In the following description, it is assumed that the image data for high-resolution images 0 to 4 is included as the correction coefficient calculation image data.

  In step 1102, the correction coefficient calculation unit 403 determines a display area on which the display image is projected. Specifically, the user designates a rectangular area as a display area with a mouse or the like while referring to the high resolution image 0 in the received high resolution image data. Then, the designated area is set as a display area. In this case, the aspect ratio of the set display area is set to be the same as that of the display image. Here, as shown in FIG. 12, the pixels at the four corners of the display area are P00 to P03, respectively, and the pixels at the four corners of the display image corresponding to the P00 to P03 are Q00 to Q03, respectively.

In step 1103, the correction coefficient calculation unit 403 calculates correction coefficients (geometric transformation coefficients) between the display image and the high resolution image 0 and the four corners P 00 to P 03 of the display area set in step 1102 and the display image. Calculation is performed using the four corners Q00 to Q03. The calculated geometric transformation coefficients are indicated by a _{00 to} a ₀₇ in FIG. Hereinafter, a method for calculating the geometric transformation coefficient will be described.

The relationship between a _{00 to} a ₀₇ , x, y, x ″, y ″ when the pixel in the display image is (x, y) and the pixel in the high resolution image 0 is (x ″, y ″). Can be represented by formula (2).

  Expression (2) is developed, and the coordinates of the four corner pixels P00 to P03 of the high-resolution image 0 and the coordinates of the four corner pixels Q00 to Q03 of the display image corresponding to P00 to P03 are used. The simultaneous equations can be derived.

Here, each of the coordinates of _{P00~P03 (x 0, y 0)} ~ (x 3, y 3), respectively the coordinates of _{Q00~Q03 (x "0, y"} 0) ~ (x "3, y" ₃ ). The geometric transformation coefficients a _{00 to} a ₀₇ are calculated by solving the simultaneous equations represented by the equation (3).

In step 1104, the correction coefficient calculation unit 403 calculates a geometric transformation coefficient between the high resolution images 1 to 4 and the corrected image. Here, the corrected image data is composed of partial image data (partial images m, m = 1 to 4 and so on) that are respectively projected by the projectors 114 to 117. The calculated geometric transformation coefficients are shown by a _{m0 to} a _m7 (a _{10 to} a ₁₇ , a _{20 to} a ₂₇ , a _{30 to} a ₃₇ , a _{40 to} a ₄₇ ) in FIG. Hereinafter, a calculation method will be described.

First, corner detection processing is performed on each of the high resolution images 1 to 4 to detect the four corner pixels Qm0 to Qm3 of the high resolution images 1 to 4 corresponding to the four corner pixels Pm0 to Pm3 of the partial images 1 to 4. (See FIG. 13). Thereafter, four sets of geometric transformation coefficients a _{m0 to} a _m7 are calculated in the same manner as in step 1103 using the coordinates of the corresponding pixels. When the pixels in the high resolution images 1 to 4 are (x ″, y ″) and the pixels in the partial images 1 to 4 are (x ′, y ′), a _{m0 to} a ₀₇ , x ″, y ″, The relationship between x ′ and y ′ can be expressed by Equation (4).

  By developing the equation (4) and using the coordinates of the pixels Pm0 to Pm3 at the four corners of the partial images 1 to 4 and the coordinates of the pixels Qm0 to Qm3 of the high resolution images 1 to 4 corresponding to Pm0 to Pm3, The simultaneous equations of 5) can be derived.

Here, each of the coordinates of the _{Pm0~Pm3 (x "0, y"} 0) ~ (x "3, y" 3), Q 00 ~Q 03 of coordinates each _{(x '0, y' 0} ) ~ ( x ′ ₃ , y ′ ₃ ). The geometric transformation coefficients a _{m0 to} a _m7 are calculated by solving the simultaneous equations represented by the equation (5).

In step 1105, the correction coefficient calculation unit 403 calculates the geometric transformation coefficients a _{00 to} a ₀₇ and a _m0 to _m7 (a _{10 to} a ₁₇ , a _{20 to} a ₂₇ , a ₃₀ to ₃₀₎ calculated in steps 1103 and 1104, respectively. a _37, a ₄₀ ~a ₄₇₎ and outputs a correction coefficient.

  In this embodiment, the display area is set based on the user's manual designation in step 1102, but the display area setting method is not limited to this. For example, the projection area by the projectors 114 to 117 may be detected from the high-resolution image 0, and the display area may be automatically set so as to be inscribed in the detected projection area.

<Image correction processing>
FIG. 14 is a flowchart showing the flow of image correction processing.

  In step 1401, the image correction unit 404 receives display image data.

  In step 1402, the image correction unit 404 acquires a correction coefficient from the correction coefficient calculation unit 403.

  In step 1403, the image correction unit 404 selects an image to be processed. At the time immediately after the start of processing, the partial image 1 is selected as the processing target image by the initialization processing.

  In step 1404, the image correction unit 404 selects a processing target pixel {coordinate: (x ′, y ′)} in the selected processing target image. At the time immediately after the start of the process, (0, 0) is selected as the initial coordinate value by the initialization process.

In step 1405, the image correcting unit 404 calculates a pixel {coordinate: (x, y)} on the display image corresponding to the processing target pixel using the acquired correction coefficient (geometric transformation coefficient). At this time, as shown in FIG. 15, the geometric transformation coefficients a ₀₀ to ₀₇ and the geometric transformation coefficients (a _{10 to} a ₁₇ , a _{20 to} a ₂₇ , a _{30 to} a ₃₇ , a _{40 to} a ₄₇ ) to be processed. Calculation processing is performed using an image corresponding to the image. For example, if the partial image 1 is selected as the processing target image, first, the coordinates (x ″) on the processing target image corresponding to the processing target pixel selected using the geometric transformation coefficients a _{10 to} a _17. , Y ″) is calculated according to equation (4). Thereafter, using the geometric transformation coefficients a ₀₀ to ₀₇ , coordinates (x, y) on the display image corresponding to the calculated coordinates (x ″, y ″) are calculated according to Expression (2).

  In step 1406, the image correction unit 404 calculates a pixel value corresponding to the calculated coordinates (x, y) on the display image, and the selected processing target pixel {coordinates: (x ′, y ′)}. The pixel value.

  In step 1407, the image correction unit 404 determines whether pixel values have been calculated for all the pixels in the processing target image. If there is a pixel that has not been calculated, the process returns to step 1404 to select the next pixel. On the other hand, if pixel values have been calculated for all pixels, the process proceeds to step 1408.

  In step 1408, the image correction unit 404 determines whether the processing has been completed for all the partial images 1 to 4. If there is an unprocessed partial image, the process returns to step 1403 to select the next partial image as a processing target image. On the other hand, if the processing has been completed for all the partial images, the process proceeds to step 1409.

  In step 1409, the image correction unit 404 outputs the image data of the partial images 1 to 4 subjected to the geometric correction process to the image output unit 405.

  Then, the partial images 1 to 4 subjected to the geometric correction processing are projected onto the screen 304 by the projectors 114 to 117.

  As described above, according to the present embodiment, when images projected by a plurality of projectors are connected to display a single image, appropriate correction is performed on each image. As a result, it is possible to suppress the positional deviation between the projected images and improve the resolution.

(Example 2)
In the first embodiment, the aspect in which the calibration pattern image projected on the screen is captured using the imaging apparatus including a plurality of imaging units having the same angle of view has been described. Next, an embodiment in which an imaging apparatus including a plurality of imaging units having different angles of view is used will be described as a second embodiment. Note that description of parts common to the first embodiment is simplified or omitted, and here, differences will be mainly described.

  FIG. 16 is a diagram illustrating an example of a multi-lens imaging device 1610 including a plurality of imaging units according to the second embodiment. The imaging device 1610 according to the present embodiment is configured with two types of imaging units. One is an imaging unit 1601 located in the center, and the other is imaging units 1602 to 1609 arranged to surround the periphery. The imaging unit 1601 and the imaging units 1602 to 1609 have different focal lengths, and the latter focal length is longer. Therefore, as shown in FIG. 17, the imaging range by the imaging units 1602 to 1609 is smaller than the imaging range by the imaging unit 1601.

  FIG. 18 is a diagram illustrating an example of captured image data obtained by capturing the calibration pattern image projected on the screen by the imaging device 1610 according to the present embodiment. As in the first embodiment, the image set 0 is a set of images obtained by imaging the states projected by all projectors, and the image sets 1 to 4 are images obtained by imaging the states projected by separate projectors. It is a pair. The image sets 1 to 4 correspond to the projectors 114 to 117, respectively, and each image set includes a total of nine images corresponding to the imaging units 1601 to 1609.

  In the present embodiment in which the captured image data as described above is obtained, the content of the high resolution processing performed in step 805 of the flowchart in FIG. Hereinafter, the high resolution processing in this embodiment will be described.

  In step S705 in the present embodiment, image data including high resolution images 0 to 4 as shown in FIG. 19 is generated by performing stitch processing on the image sets 0 to 4 included in the captured image data 1801, respectively. As the stitch process, for example, the following method can be applied. First, a difference between an image captured by the imaging units 1602 to 1609 in each image set and a region in the image captured by the imaging unit 1601 corresponding to these images is calculated. And the correction process according to the calculated difference is performed with respect to the image imaged by the imaging parts 1602-1609. Finally, the corrected images are combined to generate a high resolution image. Note that, as another method of stitch processing, a method of joining images by matching feature points extracted from the captured image is also applicable.

  By applying the above processing to each of the image sets 0 to 4 shown in FIG. 19, high-resolution images 0 to 4 are generated, and these are output to the correction count calculation unit 403 as correction count calculation images. .

  Since the rest of the processing is the same as in the first embodiment, the following description is omitted.

Claims (9)

In a projection system that displays a single image by simultaneously projecting a plurality of images from a plurality of projection means, an image processing apparatus connected to the plurality of projection means,
Input means for inputting a plurality of captured images obtained by capturing a plurality of projection images projected from the plurality of projection means from different viewpoint positions using an imaging device having a plurality of imaging means;
In the plurality of captured images, a plurality of projection areas, each of which corresponds to one of the plurality of projection images, includes a projection area having a smaller number of pixels than the corresponding projection image And generating means for generating a high-resolution image in which the resolution of a projection region having a smaller number of pixels than at least the corresponding projection image is improved using the plurality of captured images;
An acquisition unit configured to acquire area information indicating an area corresponding to each image projected from the plurality of projection units in one image displayed by projection from the plurality of projection units;
In each area indicated by the area information, setting means for setting a reference point for alignment;
Detecting means for detecting a point corresponding to the reference point in the projection area;
A deformation parameter for deforming an image projected from the plurality of projection units based on the high-resolution image, and each image projected from the plurality of projection units corresponds to a region indicated by the region information respectively, as projected, it has a determining means for determining the deformation parameter,
The image processing apparatus , wherein the determining unit determines the deformation parameter so that a point detected by the detecting unit overlaps the reference point set by the setting unit. The plurality of captured images include a first captured image group obtained by capturing the first projected image projected from the first projecting unit from a plurality of different viewpoint positions, and a second group different from the first projecting unit. A second captured image group obtained by capturing the second projected image projected from the projecting means from a plurality of different viewpoint positions;
The generating means generates a first high-resolution image using the first captured image group, generates a second high-resolution image using the second captured image group,
The determining unit determines a first deformation parameter corresponding to the first projecting unit using the first high-resolution image, and determines the second projecting unit using the second high-resolution image. The image processing apparatus according to claim 1, wherein a second deformation parameter corresponding to the second deformation parameter is determined. The setting means sets the four corners of each region indicated by the region information as the reference point,
The image processing apparatus according to claim 1 , wherein the detection unit detects a point corresponding to the reference point by performing a corner detection process on the projection area. Said generating means by performing the super-resolution processing using the plurality of captured images, the image processing apparatus according to any one of claims 1 to 3, characterized in that to generate the high resolution image . The plurality of captured images include a plurality of telephoto images in which the number of pixels of the projection region is larger than the number of pixels of the corresponding projection image, and at least one wide-angle image having a wider angle of view than the plurality of telephoto images. ,
When the number of pixels of the projection area in the wide-angle image is larger than the number of pixels of the corresponding projection image, the determining means determines the deformation parameter using the wide-angle image,
When the number of pixels of the projection area in the wide-angle image is smaller than the number of pixels of the corresponding projection image,
The generation unit generates the high-resolution image by stitch-combining the plurality of telephoto images, and the determination unit determines the deformation parameter using the high-resolution image. The image processing device according to any one of claims 1 to 3 . A projection system having an image processing apparatus according, and said plurality of projection means connected to the image processing apparatus in any one of claims 1 to 5. The projection system according to claim 6 , wherein the plurality of projecting units display one image by projecting a deformed image using the deformation parameter determined by the determining unit. . In a projection system that displays a single image by simultaneously projecting a plurality of images from a plurality of projection means, the image processing method is executed by an image processing apparatus connected to the plurality of projection means,
An input step of inputting a plurality of captured images obtained by capturing a plurality of projected images projected from the plurality of projecting units from different viewpoint positions using an imaging device having a plurality of imaging units;
In the plurality of captured images, a plurality of projection areas, each of which corresponds to one of the plurality of projection images, includes a projection area having a smaller number of pixels than the corresponding projection image In addition, using the plurality of captured images, a generation step of generating a high-resolution image in which the resolution of a projection region having at least a smaller number of pixels than the corresponding projection image is improved,
An acquisition step of acquiring region information indicating a region corresponding to each image projected from the plurality of projection units in one image displayed by projection from the plurality of projection units;
In each region indicated by the region information, a setting step for setting a reference point for alignment;
A detection step of detecting a point corresponding to the reference point in the projection region;
A deformation parameter for deforming an image projected from the plurality of projection units based on the high-resolution image, and each image projected from the plurality of projection units corresponds to a region indicated by the region information as each of which is projected, the viewing including a determination step of determining the deformation parameter,
The image processing method according to claim 1, wherein in the determining step, the deformation parameter is determined so that the point detected in the detecting step and the reference point set in the setting step overlap .   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 5.

Images