JP2021114685A - Controller, projection control method, projection system, program, and storage medium - Google Patents

Abstract

To correct a positional deviation of a projection image by easily detecting a marker from the projection image even when the projection image is a moving image.SOLUTION: A controller includes: acquisition means for acquiring a photographed image obtained by photographing a projection image of an input image on which a marker including a plurality of basic patterns is superposed; detection means for generating an averaged image or added image by shifting and superposing the photographed image or copy image thereof on the photographed image itself so that a first basic pattern of the photographed image overlaps with a second basic pattern differing from the first basic pattern and removing a low frequency component from the averaged image or added image to detect the marker's position; and output means for outputting correction information for correcting a positional deviation of the projection image, on the basis of the detected marker.SELECTED DRAWING: Figure 2

Description

The present invention relates to a control device for correcting a projection position of a projection device, a projection control method, a projection system, a program, and a storage medium.

In recent years, the number of multi-projection systems that project with a plurality of projectors has increased. In multi-projection, even if the projection positions of a plurality of units are adjusted in advance, the projection positions may shift due to external factors (shaking, temperature, aging) during image projection. On the other hand, a technique for adjusting the deviation of the projection position during image projection has been proposed (for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2001-51346

The deviation of the projection position can be corrected by using, for example, a position adjustment marker superimposed on the projected image and displayed. By imaging the projection surface on which the position adjustment marker is displayed with the imaging device, the projection position with respect to the captured image can be specified. In order to adjust the projection position during image projection, it is necessary to lower the contrast of the marker so that it is not visible to the user, and it is necessary to devise to detect the marker from the captured image. Patent Document 1 discloses a technique of detecting a marker by taking a projected image in which positive and negative markers are superimposed on the projected content and taking a difference between the respective images. However, although this method works well when the projected image is a still image, it may be difficult to detect the superimposed marker when the projected image is a moving image.

Therefore, an object of the present invention is to easily detect a marker from a projected image and correct a misalignment of the projected image regardless of whether it is a still image or a moving image.

In order to achieve the above object, the first aspect of the present invention is
An acquisition means for acquiring a captured image obtained by capturing a projected image of an input image on which markers including a plurality of basic patterns are superimposed, and
The captured image or a duplicate image thereof is superposed on the captured image itself by shifting the position so that the first basic pattern of the captured image is overlapped with a second basic pattern different from the first basic pattern. A detection means for detecting the position of the marker by generating an averaged image or an added image and removing a low frequency component from the averaged image or the added image.
The control device is characterized by having an output means for outputting correction information for correcting the positional deviation of the projected image based on the detected position of the marker.

A second aspect of the present invention is
An acquisition step of acquiring a captured image obtained by capturing a projected image of an input image on which markers including a plurality of basic patterns are superimposed, and
The captured image or a duplicate image thereof is superposed on the captured image itself by shifting the position so that the first basic pattern of the captured image is overlapped with a second basic pattern different from the first basic pattern. A detection step of generating an averaged image or an added image and detecting the position of the marker by removing a low frequency component from the averaged image or the added image.
The projection control method is characterized by having an output step for outputting correction information for correcting the positional deviation of the projected image based on the detected marker.

A third aspect of the present invention is
A projection system including a projection device, a control device for controlling the projection device, and an imaging device for imaging a projection surface by the projection device.
The control device is
An acquisition means for acquiring a captured image obtained by capturing a projected image of an input image on which markers including a plurality of basic patterns are superimposed, and
The captured image or a duplicate image thereof is superposed on the captured image itself by shifting the position so that the first basic pattern of the captured image is overlapped with a second basic pattern different from the first basic pattern. A detection means for detecting the position of the marker by generating an averaged image or an added image and removing a low frequency component from the averaged image or the added image.
The projection system is characterized by having an output means for outputting correction information for correcting the positional deviation of the projected image based on the detected marker.

According to the present invention, it is possible to easily detect a marker from a projected image and correct a misalignment of the projected image regardless of whether it is a still image or a moving image.

It is a figure which illustrates the structure of the image projection system. It is a figure which illustrates the functional structure of the projection device and the control device. It is a figure explaining the geometrical deformation. It is a figure which illustrates the marker image and the image before and after superimposing the marker. It is a figure which shows the example of the basic pattern which constitutes a marker in a marker area. It is a figure explaining the instruction signal about marker superimposition. It is a flowchart which illustrates the position shift correction processing. It is a figure explaining the process of the deformation amount calculation part. It is a flowchart which illustrates the generation process of a marker extraction image. It is a figure which illustrates the captured image acquired by the image pickup apparatus. It is a figure which illustrates the average image which superposed on the captured image by shifting the duplicated image. It is a figure explaining the extraction process of a marker. It is a figure which illustrates the marker extraction image. It is a figure which illustrates the captured image of the test pattern projected at the initial projection position. It is a figure which illustrates the projection area in the captured image when the projection position shifts.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to the following embodiments. Further, the projected image is a moving image, but the projection image is not limited to this.

It should be noted that each functional block described in this embodiment does not have to be individual hardware. That is, the functions of the plurality of functional blocks may be executed by one hardware. Further, the functions of one functional block or a plurality of functional blocks may be executed by the linked operation of a plurality of hardware. Further, each functional block may be executed by a computer program expanded on the memory by the CPU.

In a multi-projection system that projects with multiple projectors, for example, blended projection in which multiple projected images are projected side by side on the projection surface to increase the resolution, and multiple projected images are projected on the projection surface at substantially the same position for brightness. Stack projection and the like are known to improve.

These techniques are implemented in projection mapping, public viewing, etc., which require high resolution and high brightness. Even if the projection positions of multiple units are adjusted before the event is held, the projection positions may shift due to external factors (shaking, temperature, aging) during image projection during the event. There is.

If the projection position is adjusted before the event is held, there is no concern that it will hinder the impression of the user who watches the event, so a position adjustment marker with a clear contrast completely separate from the image projection of the event. Can be projected to perform adjustment work. The position adjustment marker is a marker that can specify the projection position with respect to the imaging device by superimposing it on the projected image and imaging the projection surface with the imaging device.

However, when adjusting the projection position during image projection during an event, it is desirable to reduce the contrast of the markers so that they are not visible to the user. When the contrast of the marker is lowered, information other than the marker remains in the captured image in a moving image with a lot of movement, and it may be difficult to detect the marker. According to the present invention, even in the case of a moving image, a marker can be easily detected from the projected image and the misalignment of the projected image can be corrected.

<Embodiment>
(System configuration)
FIG. 1 is a diagram illustrating a configuration of an image projection system (projection system) according to an embodiment.

As shown in FIG. 1, the image projection system according to the embodiment includes a video reproduction device 100, a projection device A101, a projection device B102, a control device 103, an image pickup device 104, and a screen 105.

The video reproduction device 100 outputs the video signal ID 0 to the projection device A101 and outputs the video signal ID 1 to the projection device B102. The video signal ID 0 and the video signal ID 1 are panel resolution signals of the projection device A101 and the projection device B102, respectively. For example, if the panel resolution of the projection device is 4K resolution (3840 x 2160), the video reproduction device 100 outputs a video signal having a resolution of 3840 x 2160. In the following description, the color space of the input video signal will be described as a space represented by an RGB 8-bit signal (0 to 255 gradations).

The projection device A101 projects the video signal ID 0 output from the video reproduction device 100 onto the screen 105. The control device 103 outputs the deformation parameter PARAM_A (correction information) as a control signal, and controls the video signal ID0 projected by the projection device A101 on the screen 105.

The projection device B102 projects the video signal ID1 output from the video reproduction device 100 onto the screen 105. The control device 103 outputs the deformation parameter PARAM_B (correction information) as a control signal, and controls the video signal ID1 projected by the projection device A101 on the screen 105.

The projection devices A101 and B102 are communicably connected to a control device 103 that functions as a projection control device and an image processing device. The control device 103 may use other information processing devices such as smartphones and tablets. For communication between devices, for example, a local area network (LAN) using TCP / IP as a communication protocol can be used. In the following description, it is assumed that the number of images projected per unit time is 60.

The control device 103 controls the timing at which the projection device A101 and the projection device B102 superimpose the marker for adjusting (correcting) the positional deviation of the projected image on the input image. The timing of superimposing the markers is included in the deformation parameter PARAM_A output to the projection device A101 and the deformation parameter PARAM_B output to the projection device B. Further, the control device 103 controls the image pickup device 104 to take an image of the projected image displayed on the screen 105 and the peripheral portion of the projected image. The image pickup device 104 captures a projected image and a peripheral portion based on the image pickup instruction signal SIG generated by the control device 103. The image pickup device 104 outputs the captured image pickup image PIC to the control device 103.

The image pickup device 104 is, for example, a digital camera, a webcam, or a network camera. The image pickup device 104 may be a camera built in the control device 103, the projection device A101, or the projection device B102. It is assumed that the image pickup apparatus 104 is installed so as to include the entire projection region of the projection apparatus A101 and the projection apparatus B102 as an imaging range, and is fixed so that the imaging range does not change. When an external device is used as the image pickup device 104, the control device 103 is communicably connected to the image pickup device 104 directly or via a LAN. The control device 103 can control the operation of the image pickup device 104 by transmitting a predetermined command to the image pickup device 104. For example, the image pickup device 104 can take a picture in response to a request from the control device 103 and transmit the captured image data to the control device 103.

The screen 105 is a projection surface on which the projection device A101 and the projection device B102 project an image. FIG. 1 shows an example of stack projection in which the projection device A101 and the projection device B102 project an image at substantially the same coordinate position on the screen 105 to increase the brightness.

(Hardware configuration)
The device configuration of the control device 103 shown in FIG. 1 will be described. The control device 103 is a computer, workstation, or the like for controlling the projection of the projector 120 onto the projection object 130. The control device 103 has a processor such as a CPU and a DSP, a main storage unit such as a read-only memory (ROM) and a random access memory (RAM), and an auxiliary storage unit such as an EPROM, a hard disk drive (HDD), and a removable media. Further, the control device 103 has a communication unit for communicating with other devices such as the camera 110 and the projector 120.

The auxiliary storage unit stores an operating system (OS), various programs, various tables, and the like. The processor loads and executes the program stored in the auxiliary storage unit in the work area of the main storage unit, and can realize the function of at least a part of the functional blocks of the control device 103 through the execution of the program. However, some or all of the functions may be realized by hardware circuits such as ASICs and FPGAs. Further, the control device 103 is not limited to the case where it is realized by a single physical configuration, and may be configured by a plurality of computers that cooperate with each other. Further, the control device 103 may be integrally configured with any one of the plurality of projection devices.

(Functional configuration)
Next, the functional units of the projection device A101 and the control device 103 and the operation of each functional unit will be described in detail with reference to FIG. Since the projection device B102 has the same configuration as the projection device A101, the description thereof will be omitted.

The projection device A101 includes an image processing unit 201, a geometric deformation unit 202, a marker superimposing unit 203, a panel control unit 204, a panel unit 205, a light source control unit 206, and a light source unit 207.

The image processing unit 201 is a block that performs general high image quality processing such as noise reduction processing, edge enhancement processing, color correction processing, and gradation correction processing. The image processing unit 201 performs the above image processing on the input video signal ID 0 and outputs the image IDT to the geometric deformation unit 202.

The geometric transformation unit 202 geometrically transforms the original image so as to cancel the trapezoidal distortion that occurs in the projected image due to the deviation between the normal direction of the projection surface and the projection direction (generally, the optical axis of the projection optical system). (Geometric transformation) Make correction (Geometric correction). In the example of FIG. 2, the image IDT from the image processing unit 201 is used as the original image. Since the geometric transformation of the image can be realized by the projective transformation, the geometric transformation is realized by determining the parameters of the projective transformation, which is the correction amount of the geometric correction. For example, the deformation amount calculation unit 303 of the control device 103 can determine the parameters of the projective transformation based on the movement amount and the movement direction of each vertex of the original image on the rectangle and give it to the geometric deformation unit 202.

Here, the geometric deformation will be described with reference to FIG. As illustrated in FIG. 3, assuming that the coordinates of an arbitrary point on the original image 300 are (xs, ys), the coordinates (xd, yd) on the image 310 after being transformed by the projective transformation corresponding to the arbitrary point are It is expressed by the following equation 1.

In Equation 1, M is a 3 × 3 matrix, which is a projective transformation matrix from the original image 300 to the transformed image 310. The matrix M is generally obtained by solving simultaneous equations using the four-corner coordinates of the original image 300 and the four-corner coordinates of the deformed image 310. xso and yso are the coordinates of the upper left apex of the original image 300 shown by the solid line in FIG. xdo and ydo are the coordinate values of the vertices corresponding to the vertices (xso, yso) of the original image 300 before deformation in the deformed image 310 shown by the alternate long and short dash line in FIG.

変形量算出部３０３は、変形パラメータＰＡＲＡＭ（補正情報）として逆行列Ｍ^−１およびオフセット（ｘｓｏ，ｙｓｏ）、（ｘｄｏ，ｙｄｏ）を幾何変形部２０２に与える。幾何変形部２０２は、式２を用いて変形後の座標値（ｘｄ、ｙｄ）に対応する元画像３００の座標（ｘｓ、ｙｓ）を求める。 From the inverse matrix M ^{-1 of the} matrix M and the offsets (xso, yso), (xdo, ydo), Equation 1 can be transformed into Equation 2 below.

The deformation amount calculation unit 303 gives the inverse matrix M- ¹ and offsets (xso, yso) and (xdo, ydo) to the geometric deformation unit 202 as deformation parameters PARAM (correction information). The geometric deformation unit 202 obtains the coordinates (xs, ys) of the original image 300 corresponding to the coordinate values (xd, yd) after the deformation using Equation 2.

If the coordinates (xs, ys) of the original image 300 obtained by the equation 2 are all integers, the geometric transformation unit 202 uses the pixel values of the coordinates (xs, ys) of the original image 300 as they are in the image after geometric transformation. It can be a pixel value of coordinates (xd, yd).

On the other hand, when the coordinates of the original image 300 obtained by the equation 2 are not integers, the geometric deformation unit 202 uses the pixel values corresponding to the coordinates (xs, ys) of the original image 300 as the values of a plurality of peripheral pixels. It can be obtained by interpolation calculation. The geometric deformation unit 202 can use a known interpolation calculation such as bicubic as the interpolation calculation.

The region in the image 310 after geometric deformation such that the coordinates obtained by the equation 2 exist in the region of the original image 300 is called an effective pixel region. Regarding the coordinates other than the effective pixel area in the image 310 after the geometric deformation, the coordinates (xs, ys) obtained by the equation 2 are outside the area of the original image 300. That is, at coordinates other than the effective pixel region, at least one of xs <xso, ys <yso, xs> xeo, and ys> eo holds. In this case, since the reference pixel used for interpolation does not exist in the original image 300, the geometric transformation unit 202 sets the pixel value of the coordinates (xd, yd) of the image 310 after the geometric transformation to black (0 of 255 gradations) or The background color set by the user.

In this way, the geometrically deformed unit 202 can obtain the pixel values of each coordinate in the image 310 after geometrically deformed using Equation 2 and generate the image 310 after geometrically deformed. It is output to the marker superimposition unit 203 as an image KSDT generated by geometric deformation.

The marker superimposition unit 203 adds a projection position detection marker (hereinafter, also referred to as a marker) to the image KSDT based on the instruction signal TIM1 regarding the marker superimposition output by the marker control unit 301.

An image in which a marker is superimposed on the original image will be described with reference to FIG. FIG. 4A is an original image 400 before the markers are superimposed. FIG. 4B is a marker image 410, showing four marker regions 411a to 411d (also simply referred to as marker regions 411). FIG. 4C is a marker composite image in which a marker is superimposed on the original image 420. FIG. 4C shows four marker regions 421a to 421d (also simply referred to as marker regions 421) corresponding to the marker region 411.

In the example of FIG. 4B, the marker image 410 is an image in which a plurality of white (for example, gradation value 255) rectangular patterns are superimposed on a black background color (for example, gradation value 0). .. As shown in FIG. 4B, a cross-shaped unit in which white rectangular patterns are arranged vertically and horizontally around a black rectangular pattern is hereinafter referred to as a basic pattern. One basic pattern is arranged in the center of each marker area 411 included in the marker image 410. Further, in each marker area 411, four basic patterns are arranged at positions separated from the central basic pattern by a predetermined distance in four different directions. The positional relationship of each basic pattern in the marker region 411 will be described in detail in the description of FIG.

In this embodiment, the "marker" is a plurality of basic patterns included in the marker region 411. Further, the "marker position" is the position of the basic pattern at the center of the marker area 411, and in the following description, it is assumed to be the coordinates of the center of the basic pattern.

In the example of FIG. 4B, one marker image 410 includes four marker regions 411. The gradation value of the marker is not limited to 0 or 255 as long as the marker can be detected by the imaging device 104. Further, as illustrated in FIG. 4B, the marker region 411 is not limited to the four locations of the upper right, the upper left, the lower right, and the lower left. The marker region 411 may be located at a position different from that shown in FIG. 4 (B), or may be located at more than four locations. In the following description, it is assumed that the white rectangular pattern and the black rectangular pattern are squares having a 5-pixel angle, respectively.

FIG. 5 shows an arrangement example of a plurality of basic patterns constituting the marker in the marker area 411. FIG. 5 (A) shows an enlarged image of the marker region 411a on the upper right of FIG. 4 (B). FIG. 5 (B) shows an enlarged image of the marker region 411b on the upper left of FIG. 4 (B). As shown in FIG. 5, the marker superimposing unit 203 superimposes the marker on the image KSDT so that a plurality of basic patterns are included in the marker area 411. In the example of FIG. 5, each marker region 411 includes a centrally arranged basic pattern and four basic patterns arranged around the central basic pattern.

The marker area 411a shown in FIG. 5A has a basic pattern arranged at the center and four positions shifted from the center of the marker area 411a in the horizontal direction (horizontal direction) or the vertical direction (vertical direction). Includes basic patterns. Dxm− is the distance between the centrally located basic pattern and the leftwardly located basic pattern. In the example of FIG. 5, the distance between the basic patterns is the distance between the centers of the basic patterns. Dxm + is the distance between the basic pattern located in the center and the basic pattern located in the right direction. Dym− is the distance between the centrally located basic pattern and the downwardly located basic pattern. Dym + is the distance between the centrally located basic pattern and the upwardly located basic pattern.

The marker area 411b shown in FIG. 5B includes a basic marker arranged at the center and four basic patterns arranged at positions shifted obliquely upward or diagonally downward from the center of the marker area 411b. Dxym− + is the distance between the basic pattern located at the center and the basic pattern located diagonally to the upper left. Dxym + − is the distance between the basic pattern located at the center and the basic pattern located diagonally to the lower right. Dxym ++ is the distance between the basic pattern located at the center and the basic pattern located diagonally to the upper right. Dxym −− is the distance between the basic pattern located at the center and the basic pattern located diagonally to the lower left.

In the marker image 410 of FIG. 4B, the lower left marker region 411c is generated as a region in which the distances of the markers Dxm− and Dxm + and Dym− and Dym + of the marker region 411a shown in FIG. 5A are interchanged. can do. That is, the marker region 411c is a region in which the markers in the marker region 411a are inverted vertically and horizontally. Further, the marker region 411d at the lower right of the marker image 410 is generated as a region in which the distances of the markers Dxym−− and Dxym ++ and Dxym− + and Dxym + − of the markers of the marker region 411b shown in FIG. 5B are interchanged. be able to. That is, the marker region 411d is a region in which the markers in the marker region 411b are inverted vertically and horizontally.

Regarding the distance between the basic pattern located at the center of the marker region 411 and other basic patterns, for example, Dxm− can be 50 pixels and Dxm + can be 40 pixels, but the distance is not limited to this value. The absolute values of Dxm- and Dxm + may be values that are sufficiently smaller than the number of pixels in the horizontal direction (horizontal direction) of the image KSDT, and are set to, for example, 1/10 or less of the number of pixels in the horizontal direction of the image KSDT. Can be done.

Further, for example, Dym− can be 50 pixels and Dym + can be 40 pixels, but the value is not limited to this value. The absolute values of Dym− and Dym + may be values that are sufficiently smaller than the number of pixels in the vertical direction of the image KSDT, and can be set to, for example, 1/10 or less of the number of pixels in the vertical direction of the image KSDT.

In order to detect the marker (position) from the captured image of the projection surface, Dxm-, Dxm +, Dym-, and Dym + must all have a value equal to or larger than the size (length or width) of one basic pattern. Is preferable. The size of one basic pattern is, for example, when the white and black rectangular patterns are squares with 5 pixel squares, the vertical and horizontal lengths are 15 pixels each, and the distance between each basic pattern is 15 pixels or more. do it. Details of the marker detection method will be described later.

Further, for example, Dxym− + and Dxym ++ can be a diagonal distance of 50 pixels in the horizontal direction and 50 pixels in the vertical direction, and Dxym + − and Dxym−− can be a diagonal distance of 40 pixels in the horizontal direction and 40 pixels in the vertical direction. It is not limited to the value. These distances may be values sufficiently smaller than the number of pixels in the horizontal and vertical directions of the image KSDT, and are set to, for example, 1/10 or less than the average value of the number of pixels in the horizontal and vertical directions of the image KSDT. be able to.

In order to detect the marker from the captured image of the projection surface, it is preferable that Dxym− +, Dxym ++, Dxym + −, and Dxym−− are all values equal to or larger than the diagonal size of one basic pattern. The diagonal size of one basic pattern is, for example, when the white and black rectangular patterns are squares with 5 pixel angles, the diagonal distance is 15 pixels in the left-right direction and 15 pixels in the vertical direction, and the distance between each basic pattern. May be equal to or greater than the diagonal distance of 15 pixels in the left-right direction and 15 pixels in the up-down direction. Details of the marker detection method will be described later.

FIG. 5 shows an example in which the distances between the basic pattern located at the center of the marker region 411 and the four basic patterns are different, but the present invention is not limited to this. The distance between the basic pattern located at the center of the marker area 411 and the four basic patterns may be the same.

In FIG. 5, with respect to the basic pattern located at the center of each marker area 411, the direction in which the other basic patterns are located differs between the marker area 411a (up / down / left / right direction) and the marker area 411b (diagonal direction). Although shown, this is not the case. In each marker area 411, other basic patterns may be arranged in the same direction with respect to the centrally located basic pattern.

That is, in each marker area 411, the markers may be generated so that the distances between the centrally located basic pattern and the other basic patterns are different from each other, or the markers are generated so as to be the same distance. You may. Further, the markers may be generated so that the distance and direction of the centrally located basic pattern and the other basic patterns are the same between the marker regions 411.

The figure used to explain the marker is not drawn faithfully to the actual ratio of the number of pixels. In consideration of visibility, the basic pattern and the marker area are displayed in an enlarged state with respect to the entire screen.

Returning to the description of the marker superimposing portion 203 of FIG. The marker superimposing unit 203 can calculate the feature amount from the image KSDT from the geometric deformation unit 202. For example, the marker superimposing unit 203 acquires the APL (Average Picture Level) of the entire image KSDT. Further, the marker superimposing unit 203 can divide the image KSDT into blocks having a predetermined number of pixels on each side, and can acquire an APL for each block (hereinafter, referred to as a block APL). The marker superimposition unit 203 can acquire the difference value of the block APL of the previous frame and the current frame as a statistic. Further, the marker superimposing unit 203 may be able to acquire a histogram of the entire image KSDT, the number of pixels having a pixel value equal to or less than a predetermined threshold value, and the like.

The image MDT after the marker superimposition is calculated by the following formula 3. The marker superimposing unit 203 superimposes the marker image MARK (R, G, B) on the image KSDT (R, G, B) from the geometric deformation unit 202 by executing the process of the formula 3.
MDT (R) = KSDT (R) + MARK (R)
MDT (G) = KSDT (G) + MARK (G) ... (Equation 3)
MDT (B) = KSDT (B) + MARK (B)
(If MDT> 255, set it to 255)

It is preferable that the pixel value of the superimposed marker is a gradation value that is difficult for the user to see. Further, the pixel value of the superimposed marker may be controlled based on the feature amount of the original image on which the marker is superimposed. That is, the larger the pixel value in the original image and the brighter the region, the more difficult it is for the marker to be visually recognized by increasing the pixel value of the superimposed marker. For example, in a bright region where the pixel value in the original image is larger than a predetermined threshold value, the pixel value of the marker is set to 32 at 255 gradations, and in a dark region smaller than the predetermined threshold value, the pixel value of the marker is set to 8 at 255 gradations. Superimpose. By controlling in this way, the marker is less likely to be visually recognized by the user.

The feature amount of the original image for setting the pixel value of the marker is not limited to the pixel value, and the color or brightness of the area on which the marker is superimposed, such as the block APL and histogram of the image KSDT in the block on which the marker is superimposed, is determined. It may be a feature amount that can be produced.

The marker superimposition unit 203 calculates the difference value of the block APL between the previous frame and the current frame, determines that the region where the difference value is larger than the predetermined threshold value is the region where the movement is larger in the image KSDT, and the marker. May be superimposed. Since the marker is more easily detected in the region where the movement is large, the number of markers can be reduced by superimposing the marker on the region where the difference value of the block APL is larger than the threshold value. By reducing the number of markers, the markers are less visible to the user. In order to detect the movement, the feature amount is not limited to the difference value of the block APL between the previous frame and the current frame, and may be any feature amount that can represent the movement of each region of the moving image such as a histogram.

The marker superimposition unit 203 outputs the image MDT on which the markers are superposed to the panel control unit 204 and the light source control unit 206.

The panel control unit 204 generates a voltage signal PD for modulating the transmittance in the panel unit 205 based on the image MDT after the marker superimposition output by the marker superimposition unit 203.

The panel unit 205 outputs the projected image PDT to the screen 105 as image light by transmitting the light emitted by the light source unit 207 with a transmittance based on the voltage signal PD input from the panel control unit 204.

The light source control unit 206 transmits a control signal CON to the light source unit 207 based on the image information of the image MDT after the marker superimposition, the user designation via the operation unit (not shown), and the like, and turns the light source unit 207 on / off. Control the amount of light. For example, the light source control unit 206 controls the light source to be bright when the image MDT after superimposing the marker is a bright image, and to darken the light source when the image MDT is a dark image. The amount of light of the light source is not limited to the control amount, although the control amount is determined by using the image information of the image MDT after the marker is superposed by the marker superimposing unit 203. In order to suppress the luminance fluctuation due to geometric deformation or marker superimposition, the control amount of the light amount of the light source may be determined based on the image IDT from the image processing unit 201.

The light source unit 207 is, for example, a halogen lamp, a xenon lamp, a high-pressure mercury lamp, or the like, and irradiates the panel unit 205 with light based on the control signal CON.

Returning to FIG. 2, the operation of the control device 103 will be described. The control device 103 includes a marker control unit 301, a marker extraction unit 302, and a deformation amount calculation unit 303.

The marker control unit 301 transmits an instruction signal TIM1 instructing the marker superimposition unit 203 to superimpose the marker, and transmits an imaging instruction TIM2 instructing the image pickup apparatus 104 to image the projection surface. The instruction signal TIM1 includes a start instruction indicating the timing for starting the marker superimposition and an end instruction indicating the timing for ending the marker superimposition. The imaging instruction TIM2 includes an instruction indicating a start timing for starting the imaging of the projection surface and an end timing for ending the imaging of the projection surface. The image pickup instruction TIM2 may include an image pickup parameter for controlling the exposure time to the image sensor, the image pickup sensitivity, the aperture value, and the like included in the image pickup apparatus 104.

The instruction signal TIM1 that controls the temporal timing of superimposing the markers will be described with reference to FIG. FIG. 6 shows the timing relationship between Vsync, which represents the synchronization timing of the video signal, the image KSDT output from the geometric deformation unit 202, and the instruction signal TIM1 that gives an instruction regarding marker superimposition. The image KSDT from the geometric deformation unit 202 is processed in synchronization with Vsync.

The marker superimposing unit 203 operates so as not to superimpose the marker when the instruction signal TIM1 is Low, and to superimpose the marker when the instruction signal TIM1 is High. In the example of FIG. 6, since the image KSDT does not superimpose the marker between T1, T2, and T5, the image in which the marker is not displayed as shown in FIG. 4A is output as the image MDT after the marker superimposition processing. Will be done. Further, since the image KSDT superimposes the markers between T3 and T4, the image on which the markers are superposed is output as the image MDT after the marker superimposition processing as shown in FIG. 4C.

The marker extraction unit 302 acquires the captured image PIC from the imaging device 104, extracts the marker superimposed by the marker superimposing unit 203 from the captured image PIC, and detects the position of the marker. The marker extraction unit 302 transmits the marker extraction image MARKER, which is the extraction processing result, to the deformation amount calculation unit 303. The method of generating the marker extraction image MARKER generated by the marker extraction unit 302 will be described later.

The deformation amount calculation unit 303 calculates the deformation parameter PARAM used for superimposing the projection areas of the projection device A101 and the projection device B102 on the target projection area TAGT_POSI based on the marker extracted image MARKER. The deformation amount calculation unit 303 transmits the calculated deformation parameter PARAM to the geometric deformation unit 202. The target projection area TAGT_POSI indicates a reference position for superimposing the projection areas of the projection devices A101 and B102. Stack projection is realized by superimposing the projection areas of the projection device A101 and the projection device B102 on the target projection area TAGT_POSI. The target projection area TAGT_POSI may be a coordinate value on the captured image PIC of the four vertices of the projection region specified in advance on the captured image PIC by the user. Further, the target projection area TAGT_POSI may be designated as coordinate values on the captured image PIC of the four vertices of the projection area of any projection device selected in advance by the user.

The imaging device 104 can image the entire projection area of the projection device A101 and the projection device B102 based on the imaging instruction TIM2 input from the marker control unit 301, and transmit the acquired image PIC to the marker extraction unit 302. ..

(Position deviation correction processing)
The misalignment correction process will be described with reference to FIG. 7. In FIG. 7, the marker control unit 301 transmits an instruction signal TIM1 instructing the start of the marker superimposition display to the marker superimposition unit 203, and the deformation amount calculation unit 303 transmits the deformation parameter PARAM indicating the deformation amount to the geometric deformation unit 202. The processing up to this is shown. In the description of FIG. 7, the projection device A101 and the projection device B102 are also simply referred to as projection devices.

In S501, the marker control unit 301 determines whether or not the user has instructed the position deviation correction of the projection surface via an operation unit (not shown) such as a remote controller. When the user gives an instruction to correct the misalignment (S501: Yes), the process proceeds to S502. If there is no instruction for misalignment correction from the user (S501: No), the process returns to S501, and the projection device continues projection.

The process of S501 is a start condition of the position shift correction process by the control device 103, but the start condition is not limited to the instruction from the user. For example, the projection device is equipped with a sensor that detects a change in the physical position of the projection device itself, and the marker control unit 301 acquires the amount of change in the physical position of the projection device from the sensor, and the amount of change is equal to or greater than a predetermined threshold value. The misalignment correction process may be started when the result becomes. The sensor only needs to be able to detect the physical movement of the projection device, and examples thereof include an acceleration sensor, a speed sensor, and a gyro sensor.

Further, as a start condition, the marker control unit 301 may start the position shift correction process at the input execution interval based on the position shift correction execution interval previously input by the user by the operation unit.

Further, the marker control unit 301 may start the misalignment correction process as a start condition when it is determined that the scene is difficult for the user to visually recognize the marker. Whether or not the scene is difficult for the user to visually recognize the marker can be determined by the marker superimposing unit 203 calculating the feature amount of the image KSDT from the geometric deformation unit 202 and based on the calculated feature amount. The marker control unit 301 can receive the determination result from the marker superimposition unit 203 and determine whether or not to start the misalignment correction process.

Since human visual cognitive ability is relatively insensitive to minute changes in bright areas as compared to dark areas, it is desirable to superimpose markers in bright scenes. Specifically, the marker superimposing unit 203 acquires a block APL for each predetermined area of the image KSDT, and when there is no block APL below a predetermined threshold value, it is a bright scene in which it is difficult for the user to visually recognize the marker. You may judge.

In S502, the deformation amount calculation unit 303 determines whether or not the target projection region TAGT_POSI is specified. When the target projection area TAGT_POSI is specified (S502: Yes), the process proceeds to S504. If the target projection area TAGT_POSI is not specified (S502: No), the process proceeds to S503.

In S503, the deformation amount calculation unit 303 specifies the target projection region TAGT_POSI. The target projection area TAGT_POSI can be specified by the user via an operation unit (not shown). For example, in the target projection area TAGT_POSI, four vertices can be specified by the user on the captured image PIC acquired from the imaging device 104.

Further, the target projection area TAGT_POSI may be designated with four vertices on the projected image based on an instruction from an operation unit such as a remote controller. Specifically, the marker superimposing unit 203 generates an image MDT including a UI for designating the four vertices of the target projection area TAGT_POSI. The projection device projects a projected image based on an image MDT that includes a UI that specifies four vertices. The user adjusts the position of the UI while looking at the projected image. The UI for designating the four vertices of the target projection area TAGT_POSI may be projected by either the projection device A101 or the projection device B102. The deformation amount calculation unit 303 acquires the captured image PIC of the projection surface from the imaging device 104, detects the synthesized UI for specifying the target projection position, and obtains the coordinates on the captured image PIC.
It can be specified as the four vertices of the target projection area TAGT_POSI. After the designation of the target projection area TAGT_POSI is completed, the process proceeds to S504.

In S504, the marker control unit 301 transmits an instruction signal TIM1 instructing the marker superposition unit 203 to superimpose the markers. Upon receiving the marker superimposition instruction signal TIM1, the marker superimposition unit 203 generates an image MDT in which the marker is superposed on the image KSDT from the geometric deformation unit 202, and outputs the image MDT to the panel control unit 204. An image on which the marker is superimposed is projected on the screen 105. After the marker control unit 301 transmits the instruction signal TIM1 to the marker superimposition unit 203, the process proceeds to S505.

The marker superimposing unit 203 that has received the instruction signal TIM1 instructing the marker superimposition may generate an image MDT in which the marker is superposed on the image KSDT when it is determined that the movement of the projected image is larger. For example, the marker superimposing unit 203 can determine that the motion of the projected image is larger when the difference value of the feature amount calculated from the previous frame and the current frame of the image KSDT is equal to or more than a predetermined threshold value.

When the marker superimposing unit 203 determines that the movement of the projected image is larger and generates an image MDT on which the marker is superimposed, the marker superimposing unit 203 transmits a signal indicating that the marker is superimposed on the marker control unit 301 as a response of the instruction signal TIM1. do. The marker control unit 301 may proceed to S505 when it receives a signal indicating that the markers are superimposed, and proceeds to S511 when it does not receive a signal indicating that the markers are superimposed.

In S505, the marker control unit 301 transmits the image pickup instruction TIM2 to the image pickup apparatus 104. The image pickup instruction TIM2 is assumed to include a set value of the exposure time of the image pickup apparatus 104.

The marker control unit 301 gives an image pickup instruction to the image pickup apparatus 104 for a delay time from the time when the marker superimposition unit 203 receives the instruction signal TIM1 instructing the marker superimposition to the time when the image on which the marker is superposed is projected. The transmission of TIM2 may be delayed. After the marker control unit 301 transmits the image pickup instruction TIM2 to the image pickup apparatus 104, the process proceeds to S506.

In S506, the marker extraction unit 302 acquires the captured image PIC of the projection surface captured by the imaging device 104 that has received the imaging instruction TIM2. The set value of the exposure time of the imaging device 104 may be included in the imaging instruction TIM2, so that the projected image PDT of the frame on which the marker is superimposed can be reliably captured based on the frame rate of the projected projected image PDT. It may be determined in advance. After the marker control unit 301 acquires the captured image PIC from the imaging device 104, the process proceeds to S507.

In S507, the marker control unit 301 transmits an instruction signal TIM1 instructing the end of marker superimposition to the marker superimposition unit 203. The marker superimposing unit 203 can superimpose the marker on the projected image only during the imaging process by the imaging device 104. Therefore, the time for the marker to be superimposed on the projected image is reduced, and it becomes difficult for the user to recognize the marker.

Upon receiving the instruction signal TIM1 instructing the end of marker superimposition, the marker superimposition unit 203 outputs the image KSDT from the geometric deformation unit 202 as an image MDT to the panel control unit 204 as it is. In this case, an image on which the markers are not superimposed is projected on the screen 105. After the marker control unit 301 transmits the instruction signal TIM1 instructing the marker superimposition unit 203 to end the marker superimposition, the process proceeds to S508.

The instruction signal TIM1 transmitted to the marker superimposition unit 203 in S504 may include an instruction regarding the end of marker superimposition transmitted to the marker superimposition unit 203 in S507. Specifically, the instruction signal TIM1 includes the total time required for transmitting and receiving the exposure time set in the image pickup apparatus 104, the marker superimposition instruction TIM1, and the image pickup instruction TIM2 as the time for superimposing the markers. The marker control unit 301 can transmit an instruction regarding the end of marker superimposition to the marker superimposition unit 203 based on the time for superimposing the markers. When the instruction signal TIM1 transmitted in S504 includes an instruction regarding the end of marker superposition, the processing of S507 can be omitted.

In S508, the marker extraction unit 302 generates a marker extraction image MARKER, which is an image in which only the marker components are extracted. The marker extraction unit 302 first duplicates the captured image PIC, shifts the duplicated duplicated image CDTs in a predetermined direction by a predetermined distance, and superimposes the captured image PIC on the captured image PIC to generate an averaged image AVERAGE. .. Next, the marker extraction unit 302 removes the low frequency component from the averaged image AVERAGE and extracts the high frequency component to generate the marker extraction image MARKER. The marker extraction unit 302 transmits the generated marker extraction image MARKER to the deformation amount calculation unit 303. Details of the method for generating the marker-extracted image MARKER will be described later. After the marker extraction unit 302 transmits the marker extraction image MARKER to the deformation amount calculation unit 303, the process proceeds to S509.

In S509, the deformation amount calculation unit 303 calculates the amount (change amount) of the positional deviation between the projection area of the projection device and the target projection area TAGT_POSI using the marker extraction image MARKER. The deformation amount calculation unit 303 determines whether or not to correct the positional deviation based on the calculated change amount. The deformation amount calculation unit 303 can determine, for example, whether or not to correct the positional deviation depending on whether or not the amount of change is larger than a predetermined threshold value. When the deformation amount calculation unit 303 determines that the misalignment is corrected (S509: Yes), the process proceeds to S510, and when it is determined that the misalignment is not corrected (S509: No), the process proceeds to S511.

[Position deviation judgment processing]
Here, with reference to FIG. 8, the process of determining whether or not there is a positional deviation by the deformation amount calculation unit 303 will be described in detail. FIG. 8A shows a basic pattern located at the center of the four marker regions of the original marker image 600 superposed on the image KSDT by the marker superimposing portion 203. The black part of the background is shown in white. The coordinates of the four vertices of the marker image 600 are indicated by (xp_i, yp_i) (i = 1 to 4), and the coordinates of the center of gravity of each of the superimposed markers are indicated by (xm_i, ym_i) (i = 1 to 4). In the description of FIG. 8, it is assumed that the resolutions of the marker image 600 and the panel unit 205 are the same, and the coordinates of the four vertices of the marker image 600 and the panel unit 205 are the same at (xp_i, yp_i).

FIG. 8B shows a marker-extracted image MARKER, which shows a projection region 611 and a predetermined target projection region TAGT_POSI for the imaging range 610, respectively. The coordinates of the position of the center of gravity of each of the extracted markers are indicated by (xmc_i, ymc_i) (i = 1 to 4). The target projection area TAGT_POSI can be shown as shown in FIG. 8 (B) by sequentially connecting the coordinates (xtc_i, ytk_i) (i = 1 to 4) on the marker extraction image MARKER.

First, the deformation amount calculation unit 303 calculates the projection transformation matrix H representing the correspondence between the coordinates of the marker extraction image MARKER and the coordinates in the panel unit 205 by using the marker extraction image MARKER and the marker image 600 calculated in S508. The projective transformation matrix H is calculated from the center-of-gravity position coordinates (xm_i, ym_i) of the superimposed markers and the center-of-gravity position coordinates (xmc_i, ymc_i) of the markers extracted by the marker extraction unit 302. The projective transformation matrix H is a 3 × 3 matrix showing the relationship between the coordinates of the marker extracted image MARKER and the original marker image 600 at arbitrary points, and is represented by, for example, Equation 4. The method of obtaining the center of gravity position coordinates (xmc_i, ymc_i) of the marker from the captured image will be described later.

FIG. 8C is a diagram for explaining the positional relationship between the marker image 600 and the target projection area TAGT_POSI_p in which the target projection area TAGT_POSI is projected onto the marker image 600. The deformation amount calculation unit 303 substitutes the four-vertical coordinates (xtc_i, ytc_i) of the target projection area TAGT_POSI into the following equation 5 using the projective transformation matrix H, and projects the four-vertical coordinates (projected on the coordinate system of the marker image 600). xx_i, yt_i) (i = 1 to 4) are calculated.

The deformation amount calculation unit 303 compares the calculated coordinates (xt_i, yt_i) with the four vertex coordinates (xp_i, yp_i) of the marker image, and determines that there is no positional deviation when the coordinates match. Further, the deformation amount calculation unit 303 compares the calculated coordinates (xt_i, yt_i) with the coordinates of the four vertices (xp_i, yp_i) of the marker image, and determines that there is a positional deviation if there are coordinates that do not match. ..

In S509, the deformation amount calculation unit 303 determines that there is no positional deviation when the coordinates (xt_i, yt_i) of the target projection area TAGT_POSI_p and the coordinates (xp_i, yp_i) of the four vertices of the marker image match. An example of this is shown, but it is not limited to this. The determination standard in S509 may be any standard that can determine the necessity of misalignment correction.

For example, the amount of misalignment between the calculated coordinates (xt_i, yt_i) and the four vertex coordinates (xp_i, yp_i) of the marker image is defined as the Euclidean distance between the two corresponding points. The deformation amount calculation unit 303 may determine that the misalignment correction is not performed when the sum of the deviation amounts between the coordinates of the four vertices in the projection area and the coordinates of the four vertices in the target projection area is equal to or less than the threshold value.

Further, the deformation amount calculation unit 303 may determine that the misalignment correction is not performed when the deviation between the calculated coordinates of each vertex and the coordinates of the corresponding target projection area is equal to or less than a predetermined threshold value.

Further, the deformation amount calculation unit 303 may determine that the positional deviation correction is not performed when the deviation amount between the calculated coordinates of each vertex and the coordinates of each of the four vertices in the target projection area is equal to or less than the threshold value. In this case, if the deviation of the coordinates of even one vertex from each of the four vertices in the target projection area exceeds the threshold value, it may be determined that the displacement correction is not performed.

Subsequently, with reference to FIG. 8, a method in which the deformation amount calculation unit 303 calculates the barycentric coordinates (xmc_i, ymc_i) of the marker on the captured image PIC from the marker extracted image MARKER will be described. The coordinates of the center of gravity of the marker (xmc_i, ymc_i) will be described as an example of the coordinates of the center of the marker. Further, the origin of the coordinate system will be the upper left apex of the imaging range 610, and the right side in the horizontal direction (x-axis direction) and the lower side in the vertical direction (y-axis square) will be described as positive directions.

The deformation amount calculation unit 303 uses the marker region of a predetermined number of pixels LEN square centering on the basic pattern as the template T, and comprehensively searches for the region where the correlation coefficient between the marker extracted image MARKER and the template T is the highest. .. The deformation amount calculation unit 303 extracts four regions having a high correlation with the four marker regions (template T) including each basic pattern shown in FIG. 8 (A). Since the template T is an image of a predetermined number of pixels LEN square centered on the basic pattern, the deformation amount calculation unit 303 uses the coordinates of the center of the extracted region as the coordinates of the center of the extracted marker (xmc_i, ymc_i).

As shown in FIG. 8B, the deformation amount calculation unit 303 calculates the center coordinates (xmc_ave, ymc_ave) of the four extracted marker coordinates (xmc_i, ymc_i). xmc_ave is the value obtained by calculating the total value of xmc_i (i = 1 to 4) and dividing by the number of markers, and ymc_ave is the value obtained by calculating the total value of ymc_i (i = 1 to 4) and dividing by the number of markers. The value. The deformation amount calculation unit 303 sets the markers in the captured image and the markers in the marker image 600 based on the magnitude relationship between the extracted marker coordinates (xmc_i, ymc_i) and the center coordinates (xmc_ave, ymc_ave) of the four markers. Correspond.

When the x-coordinate xmc_i of the extracted marker is smaller than xmc_ave and the y-coordinate ymc_i is smaller than ymc_ave, the deformation amount calculation unit 303 determines that the barycentric coordinates (xmc_1, ymc_1) of this marker correspond to (xm_1, ym_1).

Further, when the x-coordinate xmc_i of the extracted marker is larger than xmc_ave and the y-coordinate ymc_i is smaller than ymc_ave, the deformation amount calculation unit 303 determines that the coordinates of the center of gravity (xmc_2, ymc_2) of this marker correspond to (xm_2, ym_2). to decide.

Further, the deformation amount calculation unit 303 determines that the barycentric coordinates (xmc_3, ymc_3) of the extracted marker correspond to (xm_3, ym_3) when the x-coordinate xmc_i of the extracted marker is larger than xmc_ave and the y-coordinate ymc_i is larger than ymc_ave. do.

Further, when the x-coordinate xmc_i of the extracted marker is smaller than xmc_ave and the y-coordinate ymc_i is larger than ymc_ave, the deformation amount calculation unit 303 determines that the barycentric coordinates (xmc_4, ymc_4) of this marker correspond to (xm_4, ym_4). ..

Note that FIG. 8 shows an example in which a basic pattern having the same shape is used, but the present invention is not limited to this. For example, in the marker image 410 of FIG. 4B, basic patterns having different shapes that are uniquely determined for each marker region 411 can be used. In this case, the coordinates of the center of gravity (xmc_i, ymc_i) of the extracted marker can uniquely specify the correspondence with the center coordinates (xm_i, ym_i) of the original marker.

Returning to FIG. 7, in S510, the deformation amount calculation unit 303 calculates the deformation parameter PARAM and transmits it to the geometric deformation unit 202. The process of S510 is executed when it is determined in S509 that the projection area deviates from the target projection area TAGT_POSI.

The transformation parameter PARAM includes a projective transformation matrix M_P and offsets (xt_1, yt_1), (xp_1, yp_1). The projective transformation matrix M_P is a projective transformation matrix from the original image represented by 3 × 3 to the deformed image in which the positional deviation is corrected, and satisfies the following equation 6.

The geometric deformation unit 202 deforms the image IDT from the image processing unit 201 based on the deformation parameter PARAM received from the deformation amount calculation unit 303 to generate an image KSDT. The geometric deformation unit 202 transmits the generated image KSDT to the marker superimposition unit 203.

Since the geometric deformation unit 202 geometrically deforms the image IDT using the received deformation parameter PARAM, the projection area of the projected image projected on the screen 105 coincides with the target projection position TAGT_POSI. After the deformation amount calculation unit 303 transmits the deformation parameter PARAM to the geometric deformation unit 202, the process proceeds to S511.

In S510, an example in which the deformation amount calculation unit 303 calculates the deformation parameter PARAM and the geometric deformation unit 202 corrects the projection position of the projected image has been described, but the present invention is not limited to this. When a projection device capable of optically moving the projection area by lens shift or zoom is used, the deformation amount calculation unit 303 calculates the control amount of the lens shift or zoom to correct the projection position. May be good. For example, when geometrically deforming each of the four vertices in the projection center direction, the minimum value of the deformation amount of each vertex may be corrected by lens shift or zoom, and the rest may be corrected by geometrical deformation.

In S511, the marker control unit 301 determines whether or not the end condition of the misalignment correction process shown in FIG. 7 is satisfied. The marker control unit 301 can determine that the end condition is satisfied when, for example, the user gives an end instruction by operating the remote controller. When it is determined that there is no end instruction from the user (S511: No), the process returns to S501, and when it is determined that there is an end instruction from the user (S511: Yes), the misalignment correction process shown in FIG. 7 is performed. finish.

[Marker extraction image generation process]
Subsequently, with reference to FIGS. 9 to 13, a method in which the marker extraction unit 302 generates the marker extraction image MARKER from the captured image in S508 will be described in more detail. FIG. 9 is a flowchart illustrating the details of the process of S508 of the position shift correction process shown in FIG. 7. The process of S508 is a process of generating a marker-extracted image MARKER from the captured image PIC.

FIG. 10 is a diagram illustrating an captured image PIC acquired by the imaging device 104. FIG. 11 is a diagram illustrating an average image in which a duplicate image of a captured image is superimposed on a captured image. FIG. 12 is a diagram illustrating a marker extraction process. FIG. 13 is a diagram illustrating a marker extracted image.

First, the captured image PIC will be described with reference to FIG. The black portion between the imaging range 610 and the projection area 611 indicates outside the projection range of the projection device A101 and the projection device B102.

The horizontal width Wc and the vertical width Hc of the projection area 611 correspond to the horizontal width W and the vertical width H of the original image 400 shown in FIG. 4 (A), respectively. The width Wc and the height Hc of the projection area 611 are the sizes on the captured image obtained by projecting the marker composite image 420 with the marker attached to the original image 400 onto the projection surface (screen 105). It is assumed that the pixels of the original image and the pixels on the captured image correspond to the same magnification, Wc is 3840 pixels like W, and Hc is 2160 pixels like H.

Dxmc− and Dxmc + indicate the distance in the left-right direction (horizontal direction) on the captured image between the basic pattern located at the center in the marker region and the basic pattern located on the left and right. Dymc and Dymc + indicate the vertical distance on the captured image between the centrally located basic pattern and the vertically located basic pattern in the marker region.

Dxmc−, Dxmc +, Dymc−, and Dymc + correspond to Dxm−, Dxm +, Dym−, and Dym + in FIG. 5 (A), respectively. Dxmc−, Dxmc +, Dymc−, and Dymc + are the sizes on the captured image obtained by projecting the marker composite image 420 with the marker attached to the original image 400 onto the projection surface (screen 105).

The pixels of the original image and the pixels on the captured image correspond to the same size. For example, Dymc− and Dxmc− may have 50 pixels, and Dymc + and Dxmc + may have 40 pixels.

In S5081 of FIG. 9, the marker extraction unit 302 generates a duplicate image CDT that duplicates the captured image PIC. The marker extraction unit 302 generates duplicate image CDTs in one marker area 421 for the number of basic patterns located around the central basic pattern. For example, in the marker region 421a of FIG. 4C, a marker is composed of a central basic pattern and four basic patterns whose positions are shifted by Dxm to the left and right and by Dym up and down from the central basic pattern. Therefore, the marker extraction unit 302 generates four duplicate image CDTs.

In S5082, the marker extraction unit 302 shifts the four duplicate image CDTs in a predetermined direction by a predetermined distance to generate an average image superimposed on the original captured image PIC. For example, the marker extraction unit 302 shifts the positions of the four duplicate image CDTs by Dxmc +, Dxmc−, Dymc +, and Dymc−, respectively, and superimposes them on the original captured image PIC to generate an average image.

Here, with reference to FIG. 11, the average image in which the marker extraction unit 302 superimposes the four duplicate image CDTs with respect to the original captured image PIC by shifting the positions by Dxmc +, Dxmc−, Dymc +, and Dymc− will be described. do. The non-projected area within the imaging range 610 is shown in white.

In the example of FIG. 11, the marker extraction unit 302 shifts each of the four duplicate image CDTs so that the four basic patterns located around the basic pattern at the center of the marker region 421a on the upper right overlap with the captured image. Synthesize with PIC. The central basic pattern is a basic pattern on the captured image PIC, and four surrounding basic patterns included on different duplicate image CDTs are superimposed on the central basic pattern on the captured image PIC. For example, when the central basic pattern and the basic pattern located on the left side are overlapped with each other, the marker extraction unit 302 generates an image in which the duplicate image CDT is shifted to the right by Dxmc− and synthesizes it with the captured image PIC. Similarly, when the central basic pattern and the basic pattern located on the right side are overlapped with each other, the marker extraction unit 302 generates an image in which the duplicate image CDT is shifted to the left by Dxmc + and synthesizes it with the captured image PIC. Similarly, when the central basic pattern and the basic pattern located on the upper side are overlapped with each other, the marker extraction unit 302 generates an image in which the duplicate image CDT is shifted downward by Dymc + and synthesizes it with the captured image PIC. Similarly, when the central basic pattern and the basic pattern located on the lower side are overlapped with each other, the marker extraction unit 302 generates an image in which the duplicate image CDT is shifted upward by Dymc − and synthesizes it with the captured image PIC.

Assuming that the duplicate image shifted to the right is CDT_R, the duplicate image shifted to the left is CDT_L, the duplicate image shifted upward is CDT_U, and the duplicate image shifted downward is CDT_D, the average image PIC_N after synthesis is as follows. It is calculated by the formula 7 of.
PIC_N (R) = {PIC (R) + CDT_R (R) + CDT_L (R) +
CDT_U (R) + CDT_D (R)} / 5
PIC_N (G) = {PIC (G) + CDT_R (G) + CDT_L (G) +
CDT_U (G) + CDT_D (G)} / 5
PIC_N (B) = {PIC (B) + CDT_R (B) + CDT_L (B) +
CDT_U (B) + CDT_D (B)} / 5
… (Equation 7)

In this way, by shifting the duplicated image CDT and superimposing it on the original captured image PIC one after another and averaging it, the marker extraction unit 302 can use the other basic pattern and the original image portion without blurring the central basic pattern. Can be blurred. In the example of FIG. 11, the marker extraction unit 302 generates the average image PIC_N after synthesis for the basic pattern located at the center of the marker region 421a on the upper right of the captured image. Similarly, the marker extraction unit 302 can generate the average image PIC_N after synthesis for the basic pattern located at the center of the other marker regions 421b to 421d in the captured image. In this case, the marker extraction unit 302 can generate the average image PIC_N after combining four images.

By the above processing, the marker extraction unit 302 can acquire the image PIC_N in which only the spatial frequency characteristic of the original image region is lowered without changing the spatial frequency characteristic of the extracted basic pattern.

Note that FIG. 11 has been described on the premise that the pixels of the original image and the pixels on the captured image have the same magnification. That is, the average image PIC_N illustrated in FIG. 11 is generated by shifting the duplicate image CDT by the distance between the basic patterns in the original image, but this is not the case. When the pixels of the original image and the pixels on the captured image are not the same size, the number of pixels on the captured image is calculated as described in FIGS. 14 and 15, and the average image PIC_N is the pixels on the obtained captured image. It may be generated using numbers. Further, the description has been made on the premise that the captured image has no distortion, but the present invention is not limited to this. When the captured image has distortion, the marker can be extracted from the captured image by extracting the marker with the distortion corrected and restoring the distortion.

In S5083, the marker extraction unit 302 crops (cuts out) a region outside the captured image PIC in the averaged image PIC_N to cut out an averaged image AVERAGE that cuts out a region corresponding to the original captured image PIC. get. Since four images PIC_N are generated, four averaged images AVERAGE are also generated.

In S5084, the marker extraction unit 302 extracts the marker extraction image MARKER_mini composed of the basic pattern located at the center of the marker region 421 by removing the low spatial frequency component from the averaged image AVERAGE. The marker extraction unit 302 extracts four different marker extraction images MARKER_mini corresponding to the four marker regions 421a to 421d. The marker extraction image corresponding to the upper right marker area 421a is MARKER_mini_1, and the marker extraction image corresponding to the upper left marker area 421b is MARKER_mini_1. Further, the marker extraction image corresponding to the lower right marker area 421d is MARKER_mini_3, and the marker extraction image corresponding to the lower left marker area 421c is MARKER_mini_4.

Here, a process in which the marker extraction unit 302 extracts the marker extraction image MARKER_mini from the averaged image AVERAGE will be described with reference to FIG. Specifically, the process of extracting the marker extraction image MARKER_mini_1 corresponding to the marker region 421a on the upper right in the captured image will be described. FIG. 12 shows the upper right part of the averaged image AVERAGE including the basic pattern to be extracted in the upper right marker region 421a in the captured image.

Image 801a shows the upper right quarter portion of the averaged image AVERAGE cut out in S5083. The basic pattern 801b indicates a basic pattern located at the center of the marker region 421a. Graph 801c is a graph showing changes in pixel values in the horizontal direction passing through the center of the basic pattern 801b.

The marker extraction unit 302 applies a Gaussian filter to the image 801a to acquire the image 802a from which the high frequency component of the image 801a has been removed. The basic pattern 802b shows a basic pattern after applying a Gaussian filter to the basic pattern 801b of the image 801a. The graph 802c is a graph showing the change of the pixel value in the horizontal direction passing through the center of the basic pattern 802b.

The filter size of the Gaussian filter can be determined based on the size of the basic pattern on the averaged image AVERAGE. For example, the filter size may be determined by acquiring the horizontal length LEN_MARKER of the basic pattern 801b on the averaged image AVERAGE and multiplying LEN_MARKER by a predetermined magnification. The predetermined magnification is 4 times the magnitude LEN_MARKER, but may be a numerical value other than 4 times. Specifically, the predetermined magnification is preferably 2 times or more of the size LEN_MARKER, and further preferably 4 times or more. The vertical filter size can be determined in the same manner as the horizontal filter size. The filter size is not limited to the value determined according to the size of the basic pattern, and a value obtained experimentally in advance may be used.

The horizontal length LEN_MARKER of the basic pattern 802a can be estimated from the following values.
-Horizontal resolution of the projection device PJ_X
-The horizontal length of the projected area on the captured image PIC PJ_CAM_X
-Horizontal length of the basic pattern on which the marker superimposing portion 203 is superposed MARKER_X

The number of display pixels on the captured image PIC of one pixel of the panel unit 205 of the projection device A101 can be calculated by PJ_CAM_X / PJ_X. Therefore, the horizontal length LEN_MARKER of the basic pattern 802a can be obtained by the following equation 8.
LEN_MARKER = MARKER_X × PJ_CAM_X / PJ_X ... (Equation 8)

The horizontal length PJ_CAM_X of the projected region on the captured image PIC can be obtained as follows. First, the marker extraction unit 302 includes an image captured when the projection device A101 projects an image having a maximum pixel value (all white image) on all pixels, and an image having a minimum pixel value on all pixels (all black image). ) Is projected, and the captured image is acquired. The marker extraction unit 302 calculates the difference images of these captured images, and determines that the region where the pixel value is equal to or greater than a predetermined threshold value is the projection region. The marker extraction unit 302 can set the number of pixels in the horizontal direction of the region determined to be the projection region to PJ_CAM_X.

The image 803a is a marker extraction image MARKER_mini_1 representing a difference image between the image 801a and the image 802a to which the Gaussian filter is applied to the image 801a. The basic pattern 803b is a basic pattern extracted by a process of generating a difference image between the image 801a and the image 802a. The graph 803c is a graph showing the change in the pixel value in the horizontal direction in which the basic pattern 803b is located.

In the image 801a which is a part of the averaged image AVERAGE, the basic pattern 801b is hardly blurred, but the other basic patterns and the original image portion are in a blurred state. Therefore, the basic pattern other than the basic pattern 801b and the original image portion have little change from the image 801a even in the image 802a to which the Gaussian filter is applied.

On the other hand, since the vicinity of the basic pattern 801b of the image 801a has a high frequency component, by applying the Gaussian filter, it becomes a blurred state as shown in 802b. Therefore, the marker extraction unit 302 can extract only the extracted image 803b corresponding to the basic pattern 801b by generating the difference image between the image 801a and the image 802a. As described above, the marker extraction unit 302 can generate the marker extraction image MARKER_mini_1 in which only the marker (basic pattern 801b) is extracted by removing the low frequency component of the spatial frequency from the captured image PIC. The marker extraction unit 302 can generate the marker extraction images MARKER_mini_2-4 in the same manner as the marker extraction image MARKER_mini_1.

In S5085, the marker extraction unit 302 integrates the generated marker extraction images MARKER_mini_i (i = 1 to 4) by comparative bright synthesis to generate one marker extraction image MARKER.

FIG. 13 shows a marker extraction image MARKER extracted by the marker extraction unit 302. The projection area 611 on the captured image PIC does not actually appear clearly in the marker-extracted image MARKER, but is shown for reference. The method for synthesizing the marker-extracted image MARKER_mini_i (i = 1 to 4) in S5085 is not limited to comparative bright synthesis. The marker extraction unit 302 may be combined by adding or averaging the marker extraction images MARKER_mini_i (i = 1 to 4), as long as the four marker information can be expressed in one image.

This completes the detailed description of the process of S508 shown in FIG. In the above embodiment, in the marker region 421 in the captured image, the respective distances between the basic pattern located at the center and the other basic patterns located in the periphery are equal to the corresponding distances in the marker image 410. I explained on the premise that it is double. However, the distance in the captured image and the corresponding distance in the marker image 410 are more often not the same in actual use than in the actual use. In the following, a method of acquiring the distance between the central pattern located in the center of the marker region in the captured image and other basic patterns located in the periphery from the captured image in advance will be described.

[How to get the distance between basic patterns]
The marker extraction unit 302 adjusts the distance between each basic pattern in the captured image by, for example, the user by operating a remote controller or the like to adjust the projection position of each projection device in order to configure the stack projection, and each projection device is initially set. Acquired when it is the projection position. The marker extraction unit 302 projects the dedicated test pattern described with reference to FIG. 14 and acquires the distance between each basic pattern in the captured image by using the captured image. The test pattern is an image in which a marker is drawn.

It is preferable that the marker extraction unit 302 executes a process of acquiring the distance between each basic pattern in the captured image at a timing that does not hinder the user's impression. By projecting the test pattern at such a timing, it is possible to project the test pattern having a clearer contrast. The test pattern may be an image in which a marker is superimposed on a background image having a low spatial frequency. In this case, the marker superimposed on the background image of the test pattern may be a marker having a higher contrast than the marker superimposed on the image KSDT.

FIG. 14 shows a captured image PIC0 captured by projecting a test pattern for acquiring a distance between each basic pattern in the captured image. 610 is the imaging range and 611 is the projection area. The test pattern also includes marker regions 612a to 612d (also simply referred to as marker regions 612). The test pattern used in FIG. 14 is the same as the marker image shown in FIG. 4 (B), but unlike the method described in FIG. 4 (B), it is not superimposed on the input image and is displayed on the screen 105 as it is. Be projected. The test pattern is, for example, an image in which the black background portion is set to the minimum gradation value (= 0 gradation value) and the white dot portion is set to the maximum gradation value (= 255 gradation value). In this case, the position detection marker (basic pattern) is recorded in the captured image PIC0 with higher contrast. The test pattern is not limited to the one in which the black background is the lowest gradation value and the white dot part is the maximum gradation value. The test pattern may be an image in which a basic pattern having a higher contrast is superimposed on a background image having a lower spatial frequency.

The marker extraction unit 302 uses the captured image PIC0 to obtain the distance between the basic pattern A0 located in the center of the marker region 612a and the basic pattern B0 located on the left side. For example, the marker extraction unit 302 detects the number of basic patterns (20 in the example of FIG. 14) included in the marker image by, for example, pattern matching.

The marker extraction unit 302 determines the marker region 612 from the positional relationship of each basic pattern in the captured image. Identify a centrally located marker within each marker area 612. As an example, the marker extraction unit 302 may identify the basic pattern A0 located in the center of the upper right marker area 612a and obtain the distance from the basic pattern B0 in the same marker area 612a. The distance between the basic patterns is the distance between the centers of gravity of each basic pattern. The marker extraction unit 302 may obtain the distance between the centers of gravity as the distance between the basic patterns. The marker extraction unit 302 can also obtain the distance between the basic pattern A0 and the basic pattern other than the basic pattern B0 in the marker area 612a in the same manner.

The marker extraction unit 302 records the obtained distance between the basic patterns in a RAM (not shown) mounted on the projection device, and uses it in the process of generating the marker extraction image MARKER from the captured image in S508 of FIG. Can be done. As described above, the marker extraction unit 302 can acquire the distance between each basic pattern from the captured image.

Next, with reference to FIG. 15, it will be described that the distance between each basic pattern in the captured image obtained at the initial projection position is useful even after the projection position shift occurs at the time of projecting the video signal. FIG. 15 is a diagram illustrating the state of the projected region in the captured image when the projected position is deviated. Specifically, the state of the projection region in a state where the right end portion of the full width of the projected image is shifted downward by 5 pixels with respect to the left end of the full width of the projected image due to the deviation of the projection position during projection of the video signal is shown.

The horizontal width W of the marker image is assumed to be, for example, 3840 pixels. Further, the projection position deviation dH during the projection of the video signal will be described as having 5 pixels. The distance P between the basic pattern A and the basic pattern B in the marker image is 50 pixels. The displacement amount dP in which the projected displacement amount dH affects the basic pattern A and the basic pattern B can be obtained by Equation 9.
dP = dH x P / W ... (Equation 9)
The dP is about 0.065 pixels when calculated using Equation 9.

When the projection position of the projection device shifts, even if the distance between the basic patterns obtained at the initial installation position is used as it is in S508, the shift of the overlap between the basic patterns in the captured image can be suppressed to 0.065 pixels. .. Therefore, even if one of the duplicate image CDTs is superimposed on the captured image PIC by shifting the distance between the basic patterns obtained at the initial installation position of the projection device to generate the averaged image AVERAGE, the basics to be extracted are the basics. The pattern is almost unblurred. On the other hand, since the basic pattern other than the extraction target and the original image portion can be blurred, the basic pattern of the extraction target can be detected from the captured image without any problem.

Further, since the error of the center of gravity position of the basic pattern to be extracted that is overlapped and averaged is suppressed to 0.065 pixels, the projection position deviation due to the marker can be detected with an accuracy of 0.065 pixels.

Further, if the distance between the plurality of basic patterns on the marker image is set to be larger than 50 pixels, the overlap deviation becomes large. In order to detect the deviation of the projection position with an accuracy of 0.5 pixel or less, the distance P between the plurality of basic patterns on the marker image may be suppressed to 1/10 or less of the horizontal width W of the marker image. .. That is, when the horizontal width W of the marker image is 3840 pixels, the distance P between the basic patterns may be 384 pixels or less.

[Method of correcting distortion of captured image]
In this embodiment, an example is shown in which the projected image in the captured image PIC has a similar figure to the original image 420. In this case, the basic pattern A on the captured image PIC and the adjacent basic pattern B are shapes that are congruent with each other and can be superimposed. However, when the image pickup device 104 is arranged at an angle with respect to the screen 105, the projected image of the captured image PIC is not similar to the original image 420 to which the marker is attached, but is deformed into a trapezoidal shape. The basic pattern A adjacent to the basic pattern A on the captured image PIC is not a congruent figure to the extent that it cannot be ignored. In this state, even if the duplicated image CDT is shifted by the distance P as a whole and overlapped with the original captured image PIC, the two basic patterns are not congruent and therefore do not overlap. As a result, the basic pattern at the position of the basic pattern A in the averaged image AVERAGE becomes blurred, and the position of the basic pattern becomes difficult to detect.

In order to solve the problem due to the deformation of the captured image PIC, the marker extraction unit 302 uses a projective transformation matrix described later so that the basic pattern A and the basic pattern B have substantially the same shape before the processing of S5081. The distortion of the captured image PIC may be corrected. The corrected image PIC is a converted image PIC'(corrected image). The marker extraction unit 302 performs the processing of S5081 to S5085 shown in FIG. 9 on the converted image PIC'and extracts the converted marker image MARKER' (correction marker). The marker extraction unit 302 can obtain the marker extraction image MARKER whose distortion is restored by converting the conversion marker image MARKER'with the inverse matrix of the projection transformation matrix obtained by converting the captured image PIC. After that, the processing after S509 may be executed.

The marker extraction unit 302 can use the projection transformation matrix H obtained at the time of initial projection position adjustment as the projection transformation matrix used for converting the deformed captured image PIC. The projection transformation matrix H used for adjusting the initial projection position is a projection transformation matrix representing the correspondence between the panel coordinates in the panel unit 205 and the coordinates of the marker extracted image MARKER.

The projection plane is distorted because the projection device does not face the screen in the initial installation position. However, the difference in shape between the basic pattern A and the basic pattern B due to this distortion can be corrected by projecting the captured image with the projective transformation matrix H and then extracting the marker. Therefore, even if the projection position shift increases, the marker can be extracted from the captured image.

From the above, it can be seen that it is useful to use the distance between each basic pattern in the captured image obtained at the initial projection position even after the projection position shift occurs at the time of projecting the video signal.

[Superposition of basic patterns]
In the present embodiment, a duplicated image CDT that duplicates the captured image PIC is generated, and the duplicated image CDT is shifted and averaged by superimposing the duplicated image PIC on the original captured image PIC one after another, but the present invention is not limited to this. Even if the duplicate image CDT is not necessarily explicitly generated, as the processing of the image data, the pixel values of the pixels separated from each pixel by a certain number of pixels are accumulated in the pixel values of each pixel of the captured image PIC. Image processing that obtains the same result can be performed by averaging. Moreover, averaging does not necessarily have to be performed. As long as the pixel values accumulated one after another do not cause an error such as overflow due to the bit width limitation of the processing system, they may be processed as an added image with the accumulated values as they are without averaging. In this way, a process of shifting the position of the captured image or a duplicated image thereof so that the surrounding basic pattern overlaps the central basic pattern and superimposing the captured image itself on the captured image itself to generate an averaged image or an additive image. Alternatively, image processing may be performed to obtain the same result.

[Filter for extracting marker components]
In the present embodiment, an example in which a Gaussian filter is used as a frequency filter for extracting a marker component is shown, but the present invention is not limited to this. The filter for extracting the marker component may be any other frequency filter, algorithm, or parameter as long as it can reduce the low frequency component of the image on which the marker is superimposed. Further, the spatial frequency component of the marker may be extracted by acquiring the distribution of the spatial frequency component of the captured image PIC by Fourier transform, removing the low frequency component, and then reconstructing the image by inverse Fourier transform. In addition, a marker-extracted image MARKER containing only markers may be generated by directly applying a high-pass filter to the captured image PIC.

[Target projection area]
In the present embodiment, in S503, an example in which the four vertices designated on the captured image PIC are set as the four vertices of the target projection area TAGT_POSI has been described, but the target projection area TAGT_POSI can also be used as the projection area of the projection device designated by the user. good. For example, the user designates any one of the projection devices constituting the multi-projection through an operation unit (not shown) mounted on the control device 103. The target projection area TAGT_POSI may be the coordinates of four vertices on the captured image PIC of the projection area of the projection device designated by the user. The coordinates of the four vertices on the captured image PIC of the projection area of the designated projection device can be generated by executing S504 to S509 in advance.

[others]
In the present embodiment, a stack projection in which the brightness is increased by projecting the two projection devices at the same coordinate position on the screen 105 has been described as an example, but the description is not limited to the two. Even when the number of projection devices is three or more, the same effect can be obtained by performing the same operation of each part described in the present embodiment. Further, even when there is only one projection device, it is possible to adjust the position of the projection onto a predetermined target projection area without interfering with the projection of the image.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

103: Control device 301: Marker control unit 302: Marker extraction unit 303: Deformation amount calculation unit

Claims (18)

An acquisition means for acquiring a captured image obtained by capturing a projected image of an input image on which markers including a plurality of basic patterns are superimposed, and
The captured image or a duplicate image thereof is superposed on the captured image itself by shifting the position so that the first basic pattern of the captured image is overlapped with a second basic pattern different from the first basic pattern. A detection means for detecting the position of the marker by generating an averaged image or an added image and removing a low frequency component from the averaged image or the added image.
A control device including an output means for outputting correction information for correcting a positional deviation of the projected image based on the detected position of the marker. The control device according to claim 1, wherein the marker includes the first basic pattern and a plurality of the second basic patterns arranged in different directions from the first basic pattern. The detection means sets and sets the distance between the first basic pattern and the second basic pattern based on at least one of the horizontal and vertical lengths of the projected image. The control device according to claim 1 or 2, wherein the captured image or a duplicated image thereof is superposed on the captured image itself by shifting the position based on the distance. The detection means obtains the captured image or a duplicate image thereof based on the distance between the first basic pattern and the second basic pattern in the image captured by projecting the test pattern on which the marker is drawn. The control device according to claim 1 or 2, wherein the image is superposed on the captured image itself by shifting the position. The test pattern is an image in which the marker is superimposed on a background image having a low spatial frequency, and the marker having a higher contrast than the marker superimposed on the input image is superimposed on the background image. The control device according to claim 4, wherein the control device is characterized by the above. The detection means generates a corrected image that corrects the distortion of the projected image caused by the deviation between the normal direction of the projection surface and the projection direction, detects the position of the correction marker whose distortion is corrected from the corrected image, and detects the position of the correction marker. The control device according to any one of claims 1 to 5, wherein the position of the marker is detected by restoring the distortion of the correction marker. The detection means is characterized in that a projection transformation matrix for correcting distortion of the projected image is generated from the coordinates of the marker superimposed on the input image and the coordinates of the marker detected in the captured image. The control device according to claim 6. The control device according to any one of claims 1 to 7, wherein the acquisition means instructs a timing to start and a timing to end the superposition of the marker on the input image. The control device according to any one of claims 1 to 8, wherein the acquisition means instructs a timing to start and a timing to end the acquisition of the captured image. The control device according to any one of claims 1 to 9, further comprising an input means for causing the user to specify a target projection area for projecting the input image. Any one of claims 1 to 10, wherein the detection means uses at least one of a Gaussian filter, a high-pass filter, and a Fourier transform to remove low-frequency components from the averaged image or the additive image. The control device according to item 1. When a plurality of the markers are superimposed on the input image, the plurality of basic patterns included in the first marker among the plurality of markers may be the first of the plurality of basic patterns included in the second marker. The control device according to any one of claims 1 to 11, wherein the basic pattern of 1 is inverted vertically and horizontally. An acquisition step of acquiring a captured image obtained by capturing a projected image of an input image on which markers including a plurality of basic patterns are superimposed, and
The captured image or a duplicate image thereof is superposed on the captured image itself by shifting the position so that the first basic pattern of the captured image is overlapped with a second basic pattern different from the first basic pattern. A detection step of generating an averaged image or an added image and detecting the position of the marker by removing a low frequency component from the averaged image or the added image.
A projection control method comprising: an output step for outputting correction information for correcting a positional deviation of the projected image based on the detected marker. A projection system including a projection device, a control device for controlling the projection device, and an imaging device for imaging a projection surface by the projection device.
The control device is
An acquisition means for acquiring a captured image obtained by capturing a projected image of an input image on which markers including a plurality of basic patterns are superimposed, and
The captured image or a duplicate image thereof is superposed on the captured image itself by shifting the position so that the first basic pattern of the captured image is overlapped with a second basic pattern different from the first basic pattern. A detection means for detecting the position of the marker by generating an averaged image or an added image and removing a low frequency component from the averaged image or the added image.
A projection system comprising: an output means for outputting correction information for correcting a positional deviation of the projected image based on the detected marker. The projection device has a superimposing means for superimposing the marker on the input image.
When the acquisition means instructs the superimposing means to start superimposing the marker on the input image and receives a response from the superimposing means indicating that the marker is superposed on the input image. The projection system according to claim 14, further comprising instructing the image pickup apparatus of a timing at which the image pickup image is to be started. The projection device has a correction means for optically moving the projection area of the input image.
The projection system according to claim 14 or 15, wherein the output means calculates a control amount for moving the projection region with respect to the correction means and transmits the control amount to the correction means. A program for causing a computer to function as each means of the control device according to any one of claims 1 to 12. A computer-readable storage medium containing a program for causing the computer to function as each means of the control device according to any one of claims 1 to 12.

Images