JP2021061513A - Device, method, program, and storage medium - Google Patents

Abstract

To provide a projection control device capable of detecting pixel shifts of a plurality of projectors based on machine learning.SOLUTION: A device includes an acquisition unit that acquires a captured image obtained by capturing a region including a superimposed region in which a first projection image projected by a first projector and a second projection image projected by a second projector overlap, and an estimation unit that estimates the magnitude and the direction of pixel deviation between the first projection image and the second projection image in the superimposed region from the captured image using a learning model generated using teacher data classified on the basis of the pixel deviation of a plurality of projection images in a region of a model image indicating a region in which the plurality of projection images projected by the plurality of projectors overlap each other.SELECTED DRAWING: Figure 2

Description

The present invention relates to a device that detects deviations in a plurality of projected images projected by a plurality of projectors.

When an image is projected from a projector onto a screen and displayed, pixels may be projected in an overlapping manner using a plurality of projectors in order to improve the projection brightness or the projection resolution.

In Patent Document 1, in order to analyze the pixel overlap of a projection using a plurality of projectors, a test pattern is used to detect the pixel overlap deviation (hereinafter referred to as pixel deviation) and the plurality of projectors. A technique for aligning the projection position of is disclosed.

Further, Patent Document 2 discloses a technique for constructing a learning model for performing machine learning and determining the quality of laser light by using a set of laser light observation data and evaluation label as teacher data.

Japanese Unexamined Patent Publication No. 2011-182076 JP-A-2019-38027

When pixels are actually projected using a plurality of projectors in an overlapping manner, pixel shift in units of several pixels may occur due to changes over time.

However, in the prior art described in Patent Document 1 above, since a test pattern is projected to detect pixel deviation, it is necessary to stop the projection of the content being viewed in order to detect the pixel deviation.

Further, in the conventional technique described in Patent Document 2 above, it is not possible to generate a learning model of machine learning that detects pixel deviation when projected by a plurality of projectors.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a device capable of detecting pixel deviations of a plurality of projectors based on machine learning.

The apparatus according to the present invention includes a plurality of acquisition means for acquiring an image captured by capturing a region including a superposed region in which the first projected image projected by the first projector and the second projected image projected by the second projector overlap. Using a learning model generated using teacher data in which model images showing a region in which a plurality of projected images projected by the projector of the above overlap each other are classified based on the pixel deviation of the plurality of projected images in the region, the above-mentioned It is characterized by including an estimation means for estimating the magnitude and direction of pixel deviation between the first projected image and the second projected image in the superimposed region from the captured image.

According to the present invention, it is possible to detect pixel shifts of a plurality of projectors based on machine learning.

The figure which shows the structure of the projection system which concerns on 1st Embodiment of this invention. The block diagram which shows the relationship between the projector and the learning apparatus of 1st Embodiment. Conceptual diagram of neural network. A flowchart illustrating the processing of the learning device. The conceptual diagram for explaining the teacher data of 1st Embodiment. It is a figure which shows the teacher data which estimates the pixel shift direction. It is a figure which shows the teacher data for estimating the pixel shift amount. The flowchart for demonstrating the processing of the pixel shift estimation part. The conceptual diagram for explaining the estimation result of a pixel shift. The block diagram which shows the relationship between the projector and the learning apparatus of 2nd Embodiment. The conceptual diagram for explaining the teacher data of the 2nd Embodiment. The conceptual diagram for explaining the teacher data of the 3rd Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

(First Embodiment)
FIG. 1 is a diagram showing a configuration of a projection system including a plurality of projectors according to the first embodiment of the present invention. The projection system 160 includes a projector 100a, a projector 100b, a screen 150, a signal source 152, and a learning device 154.

The projector 100a and the projector 100b are projection devices that project a projected image onto a screen based on image data input from the signal source 152. The projectors 100a and 100b detect the pixel shift of each projected image based on the learning result of the learning device 154. Details of the functions of the projector 100a and the projector 100b will be described later. When the projector 100a and the projector 100b are described together, it is referred to as the projector 100. When each is shown individually, it is referred to as a projector 100a and a projector 100b. The projection system 160 is a stack projection system in which the projection image A projected by the projector 100a and the projection image B projected by the projector 100b are superimposed and projected on the screen.

The screen 150 is a projection surface on which an image is projected by the projector 100a and the projector 100b. The screen 150 may be a wall surface or may not be a flat surface.

The signal source 152 is an output device that outputs image data to be input to the projector 100a and the projector 100b. The signal source 152 is, for example, a PC or a video distributor.

In this embodiment, two projectors 100a and 100b are shown as a plurality of projectors, but it goes without saying that three or more projectors may be used.

FIG. 2 is a block diagram showing the relationship between the projector 100 and the learning device 154.

The learning device 154 has a teacher data storage unit 10, a pixel shift learning unit 11, and a learned model 12 in which a large number of teacher data 10a are stored. The pixel shift learning unit 11 performs machine learning based on the teacher data 10a and generates a learned model (learning model) 12.

The projector 100 is a projector that projects an image onto the screen 150 based on the input image data (image signal) or the image data stored inside. As described above, the projection system 160 is configured to use a plurality of projectors 100 to superimpose pixels to improve the projection brightness for projection. The projector 100 can detect pixel deviation from an image captured on the projection surface of the screen 150 by using the trained model 12 generated by the learning device 154.

In the learning device 154, the teacher data 10a is composed of images classified by pixel shift amount and direction, and data to which a teacher label (hereinafter referred to as a label) indicating a pixel shift state (shift type, direction, shift amount) is added. Will be done. Details will be described later.

The pixel shift learning unit 11 generates a trained model 12 by performing supervised learning of machine learning using the teacher data 10a. The pixel shift learning unit 11 is composed of a device in which a GPU (Graphical Processing Unit) is built in a personal computer (not shown). Further, the pixel shift learning unit 11 may be composed of an LSI (Large-scale Integrated Circuit) (not shown) or an FPGA (Field Programmable Gate Array).

The neural network 50 will be described with reference to FIG. The neural network 50 is composed of a plurality of neurons 51 that model nerve cells and a plurality of signals 52 (having a coupling weighting coefficient) that connect the neurons. The neurons in the left column of the neural network 50 are called the input layer, the neurons in the rightmost column are called the output layer, and the neurons in the middle column are called the intermediate layer. The pixel shift learning unit 11 inputs data to the input layer, performs calculations by the intermediate layer and the output layer, and acquires the output of the neural network 50.

The pixel shift learning unit 11 includes an error detection unit (not shown) and an update unit. The error detection unit obtains an error between the output data output from the output layer and the teacher data 10a according to the teacher data 10a input to the input layer of the neural network 50. Based on the error obtained by the error detection unit, the update unit updates the coupling weighting coefficient and the like between the neurons of the neural network 50 (signal 52) so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error back propagation method is a method of adjusting the coupling weighting coefficient and the like of the signal 52 of the neural network 50 so that the above error becomes small.

In this way, the pixel shift learning unit 11 learns the features of the teacher data 10a and generates the trained model 12. Here, the trained model 12 is composed of a coupling weighting coefficient of the neural network 50 and a program that performs arithmetic processing using the neural network 50.

The pixel shift learning unit 11 is not limited to the neural network, and may use an SVM (Support Vector Machine) or a decision tree algorithm.

Subsequently, the configuration of the projector 100 of FIG. 2 will be described.

The projector 100 includes an image pickup unit 20, a pixel shift estimation unit 21, an output unit 22, an image input unit 23, an image processing unit 24, a memory 25, a light source 26, a panel 27, a projection optical system 28, a CPU 30, and a bus 31.

The imaging unit 20 can capture the periphery of the screen 150, and captures an image projected on the screen 150 by a plurality of projectors 100 (projectors 100a and 100b in this embodiment) to generate an captured image. The imaging unit 20 acquires an captured image (superimposed image) by imaging an area of the screen 150 including a region (superimposed region) in which a plurality of images projected on the screen 150 by a plurality of projectors 100 overlap each other. In the present embodiment, the entire image projected by the projector 100a and the entire image projected by the projector 100b are projected so as to overlap each other. Therefore, the imaging unit 20 captures the images projected by the projectors 100a and 100b, respectively. The imaging unit 20 outputs the captured image to the pixel shift estimation unit 21.

The image pickup unit 20 is obtained by a lens that acquires an optical image of a subject, an actuator that drives the lens, a microprocessor that controls the actuator, an image pickup element that converts an optical image acquired via the lens into an image signal, and an image pickup element. It is provided with an AD conversion unit or the like that converts the generated image signal into a digital signal. The image pickup unit 20 may be provided as an external device (imaging device) of the projector 100. In that case, the projector 100 includes a connection interface that connects to the image pickup unit 20, and acquires an captured image from the image pickup unit 20 via the connection interface.

The pixel deviation estimation unit 21 estimates the pixel deviation of each of the images projected by the plurality of projectors 100 by using the captured image generated by the imaging unit 20. The pixel shift estimation unit 21 estimates the pixel shift using the trained model 12.

The output unit 22 notifies the outside of the pixel deviation estimation result of the pixel deviation estimation unit 21. For example, the pixel shift detection result can be transmitted to an external server via a network (not shown). The memory 25 temporarily stores a control program and data as a work memory.

The image input unit 23 has a function of receiving a video signal from an external device (signal source 152 in FIG. 1), and is, for example, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal or an SDI (Serial Digital Interface) terminal. , DisplayPort® (DP) terminal and the like. Here, the external device is a personal computer, a camera, a mobile phone, a smartphone, a hard disk recorder, a game machine, or the like that outputs a video signal. The image input unit 23 transmits the received video signal to the image processing unit 24.

The image processing unit 24 has a function of changing the number of frames, the number of pixels, the image shape, etc. of the video signal received from the video input unit 23 and transmitting it to the panel 27, for example, from a microprocessor for image processing. Become. The image processing unit 24 does not need to be a dedicated microprocessor. For example, the CPU 30 may execute the same processing as the image processing unit 24 by a program stored in the memory 25. The image processing unit 24 can execute functions such as frame thinning processing, frame interpolation processing, resolution conversion processing, OSD superimposition processing such as menus, distortion correction processing (keystone correction processing), and edge blending processing. In addition to the video signal received from the video input unit 23, the image processing unit 24 can also perform the above-mentioned change processing on the image or video reproduced by the CPU 30.

The light source 26 is a light source that outputs (irradiates) light to the panel 27. The light source 26 may be any member as long as it can emit light, and includes, for example, a halogen lamp, a xenon lamp, a high-pressure mercury lamp, an LED (Light Emitting Diet), and a laser.

The panel 27 is a modulation panel that modulates the light emitted from the light source 26 and outputs an image (light representing an image). The panel 27 transmits the light emitted from the light source 26 with a transmittance (transmittance distribution) based on the image data input from the image processing circuit 24. Alternatively, the panel 27 reflects the light emitted from the light source 26 with a reflectance (reflectance distribution) based on the image data input from the image processing circuit 24. In this embodiment, it is assumed that the panel 27 is a transmissive liquid crystal panel. The panel 27 is not limited to the liquid crystal panel.

The projection optical system 28 is an optical system for projecting the image output from the panel 27 onto the screen 150. The projection optical system 28 is composed of a plurality of lenses, an actuator for driving the lens, and the like. By driving the lens by the actuator, the projection image is enlarged / reduced, the projection position is shifted, and the focus of the projection optical system 28 is adjusted. Make adjustments. The projected image is an image displayed on the screen by projecting the image output from the panel 27, and the projection position is a position on the screen 150 on which the image output from the panel 27 is projected.

The CPU 30 is a processor for controlling the operation of the projector 100. The CPU 30 executes a program read from the memory 25 to execute the control (function) described later. The CPU 30 may be composed of a plurality of processors. Further, a part or all of the control executed by the CPU 30 may be executed by an electronic circuit without using a program.

FIG. 4 is a flowchart showing the processing of the learning device 154. This flowchart is a process performed after the learning device 154 starts learning.

In S1, the teacher data used by the user to learn the pixel shift is prepared. The teacher data classifies images with pixel misalignment for which the amount of pixel misalignment is known in advance, and assigns a label indicating the state of pixel misalignment to the classified images. This work is manually performed by the user, and the created teacher data 10a is stored in the teacher data storage unit 10.

In addition, a projection image superimposed on the projection surface in the pre-input pixel shift size and direction is created using a projection simulator, and the pixel shift size and direction corresponding to the created image are labeled. May be given as.

Here, the teacher data 10a used for learning will be described with reference to FIG. FIG. 5A is a diagram for explaining pixel deviation. FIG. 5B is a diagram for explaining an image in which the pixels are horizontally, vertically, and obliquely shifted, and a label thereof. Further, FIG. 5C is a diagram for explaining an image in which the pixels are shifted by 2, 4, or 8 pixels in the horizontal direction and a label thereof.

Here, the pixel shift image of the teacher data 10a is projected onto the same projection surface by two image projection devices in advance, the shift direction and the shift amount are adjusted to generate pixel shift, and then the projection surface is photographed. Is acquired. Further, the pixel shift image of the teacher data 10a can be generated by duplicating one image and synthesizing it into one image based on the shift direction and the shift amount. The label of the teacher data 10a is generated based on the deviation direction and the deviation amount at this time.

FIG. 5A will be described with reference to the pixel shift when the two projectors 100 project the pixels so as to overlap each other on the screen.

It is assumed that the pixel 100 of the projected image projected by the projector 100a is projected on the screen 150 so as to be superimposed on the corresponding pixel 101 of the projected image projected by the projector 100b.

However, the projection position of the projected image may shift due to changes in the projection position of the projector 100 over time. In this case, the projection positions of the pixel 100 and the pixel 101 do not match.

5 (a) to 5 (d) are schematic views showing the relationship between the projection positions of the pixel 100 and the pixel 101, respectively. Needless to say, pixels other than the pixels 100 and 101 are also projected on the screen 150.

FIG. 5A shows the pixels 100 of the projector 100a (circles filled with white) and the pixels 101 of the projector 100b (diagonal lines) when the two projectors 100a and 100b project the pixels on the screen 150 so as to overlap each other. Indicates a state in which (circles) overlap (a state in which there is no pixel shift).

FIG. 5B shows a state in which the pixels 100 and the pixels 101 are vertically displaced. FIG. 5C shows a state in which the pixels 100 and the pixels 101 are displaced in the horizontal direction. FIG. 5D shows a state in which the pixels 100 and the pixels 101 are obliquely displaced.

It is also possible to classify pixel deviations based on the amount of pixel deviations. For example, a state in which the pixel 100 and the pixel 101 are displaced by 3 pixels is classified as "3 pixel deviation". It is also possible to classify the pixel shift using the pixel shift direction and the pixel shift amount. For example, a state in which two pixels are displaced in the horizontal direction is classified as "horizontal two pixel deviation".

The teacher data 10a (model image) for estimating the pixel shift direction (no shift, horizontal, vertical, diagonal) will be described with reference to FIG.

The image 200 of FIG. 6A is a plurality of image data obtained by collecting images that are not pixel-shifted. “No pixel shift” 201 is the label of the image 200. Similarly, the image 202 of FIG. 6B is a plurality of image data obtained by collecting images displaced in the horizontal direction for each deviation direction. “Horizontal shift” 203 is a label indicating the pixel shift direction of the image 202. Image 204 of FIG. 6C is a plurality of image data obtained by collecting images displaced in the vertical direction. “Vertical shift” 205 is a label indicating the pixel shift direction of the image 204. Image 206 of FIG. 6D is a plurality of image data obtained by collecting images displaced in the oblique direction. “Diagonal deviation” 207 is a label indicating the pixel deviation direction of the image 206.

Next, the teacher data 10a for estimating the amount of pixel shift will be described with reference to FIG. 7.

The image 220 of FIG. 7A is a plurality of image data obtained by collecting images that are not pixel-shifted. “No pixel shift” 221 is the label of the image 220. Similarly, the image data is classified according to the amount of pixel shift. The image 222 of FIG. 7B is a plurality of image data obtained by collecting images shifted by two pixels. Here, the "two pixel shift" is a state in which the pixel is moved to a place where the position is shifted by two pixels from the state where there is no pixel shift. The same applies to the case where the deviation is two or more pixels. “2 pixel shift” 223 is a label indicating the amount of pixel shift of the image 222. The image 224 of FIG. 7C is a plurality of image data obtained by collecting images shifted by 4 pixels. “4 pixel shift” 225 is a label indicating the amount of pixel shift of the image 224. The image 226 of FIG. 7D is a plurality of image data obtained by collecting images shifted by 6 pixels. “6 pixel shift” 227 is a label indicating the amount of pixel shift of the image 226.

Returning to the flowchart of FIG. 4, in S2, the pixel shift learning unit 11 performs learning using the teacher data 10a. The pixel shift learning unit 11 generates the learned data 12 when the learning process is completed.

In S3, the learned data 12 generated by the pixel shift learning unit 11 is stored in the storage unit, and the learned data 12 is set in the pixel shift estimation unit 21 of the projector 100.

The pixel shift correction process will be described with reference to the flowchart of FIG. The deviation correction process is executed in a state where the projector 100 is projecting an image by stack projection. It is assumed that the deviation correction process is repeatedly executed when the user has previously set to enable the deviation correction function. The deviation correction process may be executed by both the projectors 100a and 100b, or may be executed by either one. In the present embodiment, it is assumed that the deviation correction function of the projector 100a is set to be valid and the deviation correction function of the projector 100b is set to be invalid. That is, the projector 100a detects the pixel shift characteristic and corrects the projection position of the projected image of the projector 100a according to the detected pixel shift characteristic. Before starting the processing of this flowchart, it is assumed that the trained model 12 is set in the pixel shift estimation unit 21 by the user.

In S100, the imaging unit 20 captures an image projected on the screen to generate a captured image.

In S101, the pixel shift estimation unit 21 estimates whether or not the pixel shift of the stack projection has occurred using the captured image generated in S100.

Here, the estimation result of the pixel shift estimation unit 21 will be described with reference to FIG.

The estimation result 400 is an estimation result of the pixel shift estimation unit 21. The estimation result 400 has a label and an item of classification probability classified into the label. The label indicates the state of pixel shift given to the teacher data 10a. The CPU 30 selects the label having the highest classification probability from the estimation result 400 and detects the pixel shift. In the case of the estimation result 400, the CPU 30 sets the "horizontal deviation" having the highest classification probability of 85% as the pixel deviation detection result.

Returning to the flowchart of FIG. 8, in S102, it is determined whether or not the CPU 30 has detected the pixel shift based on the estimation result 400. If the label with the highest probability of the estimation result 400 indicates a pixel shift, the CPU 30 determines that there is a pixel shift and proceeds to S103. If the label with the highest probability of the estimation result 400 indicates that there is no pixel shift, the CPU 30 determines that there is no pixel shift, and proceeds to step S105.

In S103, the CPU 30 estimates the pixel shift direction and the pixel shift amount based on the estimation result 400.

In S104, pixel shift correction processing is performed. The CPU 30 adjusts the shift amount of the projection position of the projection optical system 28 based on the pixel shift direction and the pixel shift amount obtained in S103. It is also possible to correct pixel misalignment by using the distortion correction processing function of the image processing unit 24.

In S105, the CPU 30 determines whether or not the user has performed an end operation (an operation to end the flowchart of FIG. 8). When the end operation is performed, the flowchart of FIG. 8 ends, and when the end operation is not performed, the process returns to S100 and the process is continued.

In the first embodiment, when the pixel shift is detected, the pixel shift is corrected. However, when the pixel shift is detected, the output unit 22 sends the pixel shift to the external server via the network without correction. Information may be output.

Further, in the above description, it has been described that a functional portion for detecting pixel deviation, which is composed of an imaging unit 20, a pixel deviation estimation unit 21, and an output unit 22, is provided inside the projector 100. However, all or part of the functional parts for detecting these pixel shifts may be mounted on an external control device of the projector 100. As the external control device, a control device created for projection control, a personal computer, or the like can be used.

In the above description, the projectors are projected in a stack, but it is also possible to detect and correct the pixel deviation of the edge blending portion when the projectors are multi-projected and projected at high resolution.

As described above, according to the first embodiment, the pixel shift on the projection surface can be detected by using the trained model based on the captured images of the projection images of the plurality of projectors 100. Further, the correction amount can be calculated from the detected pixel deviation to correct the pixel deviation.

(Second embodiment)
Hereinafter, a second embodiment of the present invention will be described. In the first embodiment, a configuration for detecting and correcting pixel deviation on a projection surface using a trained model based on images obtained by capturing projected images of a plurality of projectors 100 has been described. However, this method has problems that a large amount of teacher data must be prepared for learning and that the scale of the neural network used becomes large. Therefore, in the second embodiment, a configuration in which the teacher data 10a for estimating the pixel deviation of the first embodiment is preprocessed and then learned will be described. Since the configuration of the image projection system 160 of the second embodiment is the same as the configuration of the first embodiment shown in FIG. 1, the description of overlapping contents will be omitted. However, the projector 100 is changed to the projector 300, and the learning device 154 is changed to the learning device 301.

FIG. 10 is a block diagram showing the relationship between the projector 300 and the learning device 301 in the second embodiment. The difference from the first embodiment is that preprocessing is applied before the teacher data 10a is trained, and preprocessing is applied to the captured image of the projection surface before estimating the pixel deviation.

The learning device 301 has a teacher data storage unit 10, a Fourier transform unit 310, a pixel shift learning unit 11, and a trained model 311.

The Fourier transform unit 310 applies a two-dimensional Fourier transform to the teacher data 10a, generates a frequency spectrum image, and stores the teacher data in a storage unit (not shown). Here, in general, a frequency spectrum image generated by a two-dimensional Fourier transform contains high-frequency components of the image near the center and low-frequency components near the four corners of the frequency spectrum image. In the following, when the Fourier transform unit 310 divides the coordinate plane having the origin at the center of the frequency spectrum image into four, the first quadrant (upper right), the third quadrant (lower left), and the second quadrant (upper left) are It is assumed that a frequency spectrum image in which the fourth quadrant (lower right) is exchanged is generated. The frequency spectrum image generated by this replacement contains a low frequency component near the center and a high frequency component as the distance from the center increases.

Here, the teacher data converted by the Fourier transform unit 310 used for learning will be described with reference to FIG. The teacher data in FIG. 11 is data for estimating the deviation direction (horizontal, vertical, diagonal).

The image 500 of FIG. 11A is an image without pixel shift, and is given the label of “no pixel shift” 501. Similarly, image 502 in FIG. 11B is a horizontally displaced image and is given the label “horizontal deviation” 503. Similarly, the image 504 of FIG. 11 (c) is given the label "vertical shift" 505. Further, the image 506 of FIG. 11D is given the label of "diagonal deviation" 507.

The Fourier transform unit 310 Fourier transforms the image included in the teacher data to generate a frequency spectrum image. The image 500 of FIG. 11 (a) is Fourier transformed to generate the frequency spectrum image 510 of FIG. 11 (e). Similarly, the image 502 of FIG. 11 (b) is shown in the frequency spectrum image 512 of FIG. 11 (f), the image 504 of FIG. 11 (c) is shown in the frequency spectrum image 514 of FIG. 11 (g), and FIG. 11 (d). Image 506 is converted into the frequency spectrum image 516 of FIG. 11 (h) and generated.

Then, the same label as that given to the image before conversion to the frequency spectrum image is given. The frequency spectrum image 510 is given the label "no pixel shift" 511. The frequency spectrum image 512 is labeled "horizontal shift" 512. The frequency spectrum image 514 is given the label "vertical deviation" 515. Further, the frequency spectrum image 516 is given the label "diagonal deviation" 517.

Returning to FIG. 10, the trained model 311 is a trained model generated by the pixel shift learning unit 11 learning using the teacher data converted into the frequency spectrum image.

The projector 302 includes an imaging unit 20, a Fourier transform unit 320, a pixel shift estimation unit 21, an output unit 22, an image input unit 23, an image processing unit 24, a memory 25, a light source 26, a panel 27, a projection optical system 28, a CPU 30, and a bus. Has 31.

The Fourier transform unit 320 acquires an image obtained by capturing a projected image of the screen from the imaging unit 20, then generates a frequency spectrum image by a two-dimensional Fourier transform process and outputs the frequency spectrum image to the pixel shift estimation unit 21. Here, the frequency spectrum image has the first quadrant (upper right), the third quadrant (lower left), and the second quadrant when the coordinate plane having the origin at the center of the frequency spectrum image is divided into four like the Fourier transform unit 310. Use the one in which (upper left) and the fourth quadrant (lower right) are exchanged.

As described above, according to the second embodiment, pixel misalignment is detected by using a trained model for images obtained by capturing projection images of a plurality of projectors 100 with a limited amount of teacher data and a neural network scale. And can be corrected.

(Third Embodiment)
Hereinafter, a third embodiment of the present invention will be described. In the first and second embodiments, a configuration has been described in which pixel deviations are detected and corrected by using a learned model for estimating pixel deviations in images obtained by capturing projected images of a plurality of projectors 100 (300). However, there is also a problem that there is actually an image in which pixel deviation is difficult to detect. For example, it is difficult to detect pixel shift in an image having many low frequency components. Therefore, in the third embodiment, a configuration will be described in which images that do not detect pixel deviation are collected and a label that does not detect pixel deviation is given. Since the configuration of the image projection system 160 of the third embodiment is the same as the configuration of the first embodiment shown in FIG. 1, the description of overlapping contents will be omitted. The third embodiment differs from the first embodiment only in the teacher data.

The teacher data of the third embodiment will be described with reference to FIG.

The images 200, 202, 204, and 206 of FIGS. 12A to 12D are the same as those of the first embodiment, and thus the description thereof will be omitted. Further, the labels of those images, "no pixel shift" 201, "horizontal shift" 203, "vertical shift" 205, and "diagonal shift" 207, are the same as those in the first embodiment.

The image 600 of FIG. 12E is a collection of image data having a low detection rate of pixel shift, such as a test pattern. "Not detected" 601 is a label given to image 600.

The pixel shift learning unit 11 performs learning using the teacher data of FIG. 12 of the third embodiment, and generates a trained model. Then, a process of estimating the pixel deviation is performed according to the flowchart of the projector 100 of FIG. 8, so that the pixel deviation is not discriminated in the image in which the pixel deviation is difficult to occur. Thereby, the detection rate of pixel shift can be improved.

As described above, in the third embodiment, it is possible to improve the detection accuracy when detecting the pixel shift using the trained model for the images obtained by capturing the projected images of the plurality of projectors 100.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

10: Teacher data, 11: Pixel shift learning unit, 12: Trained model, 20: Imaging unit, 21: Pixel deviation estimation unit, 22: Output unit, 23: Image input unit, 24: Image processing unit, 25: Memory , 26: Light source, 27: Panel, 28: Projection optics, 100: Projector, 150: Screen, 152: Signal source, 154: Learning device

Claims (19)

An acquisition means for acquiring an image captured by capturing a region including a superposed region in which the first projected image projected by the first projector and the second projected image projected by the second projector overlap.
Using a learning model generated using teacher data, which classifies model images showing a region where a plurality of projected images projected by a plurality of projectors overlap each other based on pixel deviations of the plurality of projected images in the region, An estimation means for estimating the magnitude and direction of pixel deviation between the first projected image and the second projected image in the superimposed region from the captured image.
A device characterized by comprising. The apparatus according to claim 1, wherein the teacher data is generated by attaching a label indicating the magnitude and direction of pixel deviation to the model image. The claim data is characterized in that the teacher data is generated by attaching a label indicating the magnitude and direction of the deviation of the corresponding pixel to the frequency spectrum image obtained by applying the two-dimensional Fourier transform to the model image. The device according to 1. The third aspect of the invention is characterized in that the estimation means estimates the magnitude and direction of the pixel shift by inputting a frequency spectrum image obtained by applying a two-dimensional Fourier transform to the captured image into the learning model. Equipment. The position where the first projector projects the first projected image and the position where the second projector projects the second projected image based on the amount of pixel shift estimated by the guessing means and the direction of the shift. The apparatus according to any one of claims 1 to 4, further comprising a correction means for correcting at least one of them. The device according to any one of claims 1 to 5, further comprising a connecting means for connecting to an image pickup device that images a region including a first projected image and a superposed region of the second projected image. The apparatus according to any one of claims 1 to 6, further comprising an output means for outputting the estimation result of the guessing means to the outside. The device according to any one of claims 1 to 7.
Projection means for projecting images and
A projection device characterized by comprising. A learning means for generating a learning model by using teacher data obtained by classifying a superposed image obtained by superimposing a plurality of projected images using the displacement of the positions of the plurality of projected images in the superposed image is provided.
In the learning model, an image captured by capturing an superimposed region in which the first projected image projected on the projection surface by the first projector and the second projected image projected on the projection surface by the second projector overlap on the projection surface is input. In this case, the apparatus is characterized by estimating the positional deviation between the first projected image and the second projected image in the superimposed region. An acquisition step of acquiring an image captured by capturing a region including a superposed region in which the first projected image projected by the first projector and the second projected image projected by the second projector overlap.
Using a learning model generated using teacher data, which classifies model images showing a region where a plurality of projected images projected by a plurality of projectors overlap each other based on pixel deviations of the plurality of projected images in the region, a learning model is used. An estimation step of estimating the magnitude and direction of pixel deviation between the first projected image and the second projected image in the superimposed region from the captured image.
A method characterized by having. The method according to claim 10, wherein the teacher data is generated by attaching a label indicating the magnitude and direction of pixel deviation to the model image. The claim data is characterized in that the teacher data is generated by attaching a label indicating the magnitude and direction of the deviation of the corresponding pixel to the frequency spectrum image obtained by applying the two-dimensional Fourier transform to the model image. 10. The method according to 10. 12. The estimation step according to claim 12, wherein a frequency spectrum image obtained by applying a two-dimensional Fourier transform to the captured image is input to the learning model to estimate the magnitude and direction of the pixel shift. the method of. A position where the first projector projects the first projected image and a position where the second projector projects the second projected image based on the amount of the pixel shift estimated in the estimation step and the direction of the shift. The method according to any one of claims 10 to 13, further comprising a correction step for correcting at least one of them. The method according to any one of claims 10 to 14, further comprising a connection step of connecting to an image pickup apparatus that images a region including a superimposed region of the first projected image and the second projected image. The method according to any one of claims 10 to 15, further comprising an output step of outputting the result of the guess in the guess step to the outside. It has a learning step of generating a learning model by using teacher data in which a superposed image obtained by superimposing a plurality of projected images is classified by using the displacement of the positions of the plurality of projected images in the superposed image.
In the learning model, an image captured by capturing an superimposed region in which the first projected image projected on the projection surface by the first projector and the second projected image projected on the projection surface by the second projector overlap on the projection surface is input. In this case, a method characterized by estimating the positional deviation between the first projected image and the second projected image in the superimposed region. A program for operating a computer as each means of the device according to any one of claims 1 to 7. A computer-readable storage medium that stores a program for causing the computer to function as each means of the apparatus according to any one of claims 1 to 7.

Images