JP2018160891A - Image processing apparatus, projector, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of recording, as suitable images, a display image and other subjects as well.SOLUTION: The image processing apparatus 10 comprises a captured image acquisition part, a content acquisition part, a display area identification part, a processing information creation part, and an image processing part. The captured image acquisition part acquires a captured image of an area that includes a display area in which a content is displayed by a display device and a peripheral area of the display area. The content acquisition part acquires the content. The display area identification part identifies a display area in the captured image. The processing information creation part creates process information based on the captured image. The image processing part performs image processing that includes combining an image based on the content with the display area of the captured image by using the process information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, a projector, an image processing method, and an image processing program.

  Speakers sometimes give lectures while displaying materials on large display devices or projectors. Such a lecture may be recorded so that it can be viewed by many people as well as the audience at the lecture hall, or to be stored as a record.

  For example, Patent Document 1 discloses a presentation apparatus that includes a liquid crystal display that displays video and the like related to a presentation, and a video camera that records from the start to the end of the presentation.

JP 2014-102598 A

  When recording a lecture, it is preferable that both the material displayed on the display device and the state of the lecturer be recorded beautifully. However, if the focus of the camera is adjusted so that, for example, the speaker's face is in focus, the display image such as the document may not be in focus. In addition, when a document is displayed by a projector, geometric correction may be performed to eliminate distortion of the projected image. Such geometric correction generally reduces the quality of the displayed image.

  An object of the present invention is to provide an image processing apparatus, a projector, an image processing method, and an image processing program capable of recording a display image and other subjects as appropriate images.

  According to an aspect of the present invention, the image processing apparatus includes a captured image acquisition unit that acquires a captured image of an area including a display area in which content is displayed by a display device and a peripheral area of the display area, and the content Using the content acquisition unit, the display region specifying unit for specifying the display region in the captured image, the processing information generating unit for generating processing information based on the captured image, and the processing information. An image processing unit that performs image processing including synthesizing an image based on the content in the display area of the image.

  According to an aspect of the present invention, the projector generates the captured image, the projection unit as the display device that outputs the image projected onto the display area based on the content, and the captured image. And a photographing unit.

  According to an aspect of the present invention, an image processing method acquires a captured image of an area including a display area where content is displayed by a display device and a peripheral area of the display area, and acquires the content. Identifying the display area in the captured image, creating processing information based on the captured image, and using the processing information based on the content in the display area of the captured image Performing image processing including combining images.

  According to an aspect of the present invention, the image processing program acquires a captured image of an area including a display area in which content is displayed by a display device and a peripheral area of the display area, and acquires the content. Identifying the display area in the captured image, creating processing information based on the captured image, and using the processing information based on the content in the display area of the captured image Causing a computer to perform image processing including combining images.

  According to the present invention, it is possible to provide an image processing apparatus, a projector, an image processing method, and an image processing program capable of recording a display image and other subjects as appropriate images.

FIG. 1 is a diagram illustrating an outline of a configuration example of a lecture video creation system according to an embodiment. FIG. 2 is a flowchart showing an outline of an example of the operation of the lecture video creation system. FIG. 3 is a flowchart showing an outline of an example of pre-processing performed in the lecture moving image creating system. FIG. 4 is a diagram illustrating an outline of an example of a positional relationship between a shooting area and a display area in a shot image. FIG. 5 is a diagram schematically illustrating an example of the luminance of each coordinate in the imaging region. FIG. 6 is a diagram schematically showing an example of alpha blend mask image data. FIG. 7 is a flowchart showing an outline of an example of a lecture recording process performed in the lecture video creation system. FIG. 8 is a flowchart showing an outline of an example of the lecture video creation process performed in the lecture video creation system. FIG. 9 is a flowchart illustrating an outline of an example of a lecture moving image creation process according to the second embodiment. FIG. 10 is a diagram for explaining a situation according to the third embodiment. FIG. 11 is a flowchart illustrating an outline of an example of a lecture moving image creation process according to the third embodiment. FIG. 12 is a flowchart illustrating an outline of an example of a lecture moving image creation process according to the third embodiment. FIG. 13 is a diagram for explaining a situation according to the fourth embodiment. FIG. 14 is a flowchart illustrating an outline of an example of a lecture moving image creation process according to the fourth embodiment.

(First embodiment)
[Outline of lecture video creation system]
The lecture video creation system according to the present embodiment is used in a situation where, for example, lecture materials are projected onto a screen using a projector, and a lecturer is giving a lecture beside the screen. Lectures include various classes, classes, seminars, presentations, etc. This lecture video creation system has a digital camera. The digital camera is configured to shoot a moving image with respect to an angle of view including, for example, displayed lecture materials and a lecturer. In addition, the lecture video creation system records image data projected on a screen using a projector. The lecture movie creation system creates a lecture movie in which an image to be projected is inserted in an area where a document is projected in a movie shot by a digital camera.

[Configuration of lecture video creation system]
FIG. 1 shows an outline of a configuration example of the lecture video creation system 1 according to the present embodiment. The lecture video creation system 1 includes an image processing apparatus 10, a personal computer (PC) 50, a digital camera 60, a microphone 70, and a projector 80.

  In the PC 50, for example, application software for presentation is operating. The user operates this software to operate an image to be projected on the screen 340. The PC 50 outputs content data that is an image to be projected on the screen 340 in accordance with a user operation. The content data output from the PC 50 is input to the image processing apparatus 10.

  The projector 80 has a projection unit, acquires content data output from the PC 50 via the image processing apparatus 10, and projects an image of content based on the data on the screen 340. An area on which the image is projected is referred to as a display area 320.

  The digital camera 60 has a photographing unit, photographs a photographing region including a screen 340 on which an image is projected, and a lecturer 330 in the vicinity thereof, and photographing that is image data including the screen 340 and the lecturer 330. Create image data. A range including the screen 340 and the lecturer 330 and shot by the digital camera 60 is referred to as a shooting area 310. The digital camera 60 outputs captured image data. The captured image data output from the digital camera 60 is input to the image processing apparatus 10. In the present embodiment, the captured image data input from the digital camera 60 to the image processing device 10 is time-sequential moving image data.

  The microphone 70 generates an audio signal based on the audio of the lecture hall including mainly the audio uttered by the speaker. This audio signal is input to the image processing apparatus 10.

  The image processing apparatus 10 includes a processor 11, a memory 12, a mass storage 13, an input device 14, a content input interface (I / F) 15, and an image input interface (connected to each other via a bus line 19. I / F) 16, an audio input interface (I / F) 17, and an image output interface (I / F) 18.

  The content data output from the PC 50 is input to the image processing apparatus 10 via the content input I / F 15. The captured image data output from the digital camera 60 is input to the image processing apparatus 10 via the image input I / F 16. The audio signal output from the microphone 70 is input to the image processing apparatus 10 via the audio input I / F 17. The content data is output from the image processing apparatus 10 to the projector 80 via the image output I / F 18.

  The processor 11 controls the operation of each unit of the image processing apparatus 10. The processor 11 may include an integrated circuit such as a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a graphics processing unit (GPU). The processor 11 may be a single integrated circuit or a combination of a plurality of integrated circuits.

  The memory 12 is, for example, a random access memory (RAM) realized by a semiconductor integrated circuit, and functions as a main memory, a cache memory, and the like. The mass storage 13 is a hard disk or a semiconductor memory.

  The content data input from the PC 50 is recorded and recorded in the mass storage 13. In other words, on the large-capacity storage 13, an image displayed on the projector 80 in time series is recorded as a moving image based on the content data input from the PC 50. The image recorded in this way is referred to as a content moving image. The captured image data input from the digital camera 60 is recorded and recorded in the large-capacity storage 13. In other words, on the large-capacity storage 13, images taken in time series based on the taken image data input from the digital camera 60 are recorded as moving images. The image recorded in this way is referred to as a captured moving image. The mass storage 13 can also store programs used in the processor 11 and various parameters.

  The input device 14 includes, for example, a button switch and a touch panel. The user inputs an instruction related to the operation of the image processing apparatus 10 using the input device 14.

  For example, the processor 11 that operates using an image processing program recorded in the large-capacity storage 13 serves as a captured image acquisition unit, a content acquisition unit, a display area specifying unit, a processing information generation unit, and an image processing unit, as will be described later. Realize the function. Here, the captured image acquisition unit acquires the captured image data generated by being captured by the digital camera 60 via the image input I / F 16. The photographed image data is data of a photographed image generated by photographing a photographing region including a display region in which content is displayed by the projector 80 as a display device and a peripheral region of the display region. The content acquisition unit acquires content data for display by the projector 80 as a display device via the content input I / F 15. The display area specifying unit specifies a display area in the captured image based on the captured image data. The processing information creation unit creates processing information based on the captured image data. The image processing unit creates a synthesized moving image as a lecture moving image by synthesizing the moving image of the embedded image created by performing image processing based on the content moving image in the display area of the moving image of the captured image based on the captured image data. At this time, the image processing unit may use the processing information.

[Operation of the lecture video creation system]
<Outline of actions related to lecture video creation>
The operation of the lecture video creation system 1 is roughly divided into three stages. FIG. 2 is a flowchart showing an outline of an example of the operation of the lecture moving image creating system 1.

  In step S101, the lecture video creation system 1 performs pre-processing. The pre-processing is processing performed prior to the lecture in which the lecture video is created. In the pre-processing, the display area by the projector 80 that occupies the photographing area by the digital camera 60 is specified.

  In step S102, the lecture video creation system 1 performs a lecture recording process. The lecture recording process is a process of recording the lecture in which a lecture video is created. The digital camera 60 shoots the state of the shooting area including the speaker 330 and the screen 340 who are giving a lecture, and the shooting result by the digital camera 60 is recorded as a moving image. Further, an image projected by the projector 80 during the lecture, that is, the content is recorded as a moving image.

  In step S103, the lecture video creation system 1 performs a lecture video creation process. The lecture video creation process is a process performed after the lecture ends. In the lecture video creation process, the captured image and the embedded image are synthesized to create a synthesized video as a lecture video. The photographed image used here is a photographed image recorded and recorded by the digital camera 60. The inset image is recorded content and is an image created so as to correspond to the projection image projected by the projector 80.

<Pre-processing>
The processing of the image processing apparatus 10 performed in the pre-processing in step S101 will be described with reference to the flowchart shown in FIG. The pre-processing is performed by, for example, instructing the start of the pre-processing before the lecture. The pre-processing start instruction is given, for example, by pressing a pre-processing start button included in the input device 14.

  In step S201, the processor 11 causes the projector 80 to project a display area recognition pattern. That is, the processor 11 creates image data for the display area recognition pattern, and outputs the created image data to the projector 80 via the image output I / F 18. As a result, the projector 80 projects the display area recognition pattern on the screen 340. Here, the display area recognition pattern is preferably an image from which a display area in an image taken by the digital camera 60 is easily extracted. The display area recognition pattern may be, for example, a monochromatic image that is bright on the entire surface.

  It is preferable that the projected image is appropriately subjected to geometric correction such as so-called trapezoidal correction so that the display area 320 is rectangular. Moreover, it is preferable that the projected image is accurately focused on the screen 340.

  In step S202, the processor 11 acquires photographed image data. That is, the digital camera 60 shoots the shooting area 310 to create photographic image data and outputs it. The processor 11 acquires photographed image data related to the photographing region 310 onto which the display region recognition pattern is projected via the image input I / F 16.

  In step S <b> 203, the processor 11 analyzes the captured image data acquired in step S <b> 202 and specifies a display area in the captured area in the captured image. For example, when a display area recognition pattern that is a bright monochromatic image is projected on the display area 320 of the screen 340, the processor 11 analyzes the luminance value of each coordinate in the shooting area in the shot image, and By detecting the contrast (difference in luminance), the display area in the captured image can be specified.

  FIG. 4 shows an outline of an example of the positional relationship between the shooting area 410 and the display area 420 in the shot image 400. As shown in this figure, the lower left coordinates of the imaging area 410 are coordinates (0, 0), and the upper right coordinates of the imaging area 410 are coordinates (Xm, Ym). A display area 420 exists in the imaging area 410. In FIG. 4, the display area 420 is hatched. The lower left coordinates of the display area 420 are coordinates (Xc1, Yc1), and the upper right coordinates of the imaging area 410 are coordinates (Xc2, Yc2).

  In the case of the example of FIG. 4, an outline of an example of the luminance of each coordinate in the imaging region 410 analyzed by the processor 11 is schematically shown in FIG. As shown in this figure, a luminance value is acquired for each coordinate in the imaging region 410. In FIG. 5, the coordinates corresponding to the display area 420 are hatched.

  In step S204, the processor 11 creates a mask image αm for alpha blending. That is, the processor 11 sets the α value of the dark point determined to be outside the display area 420 in step S203 to 1 and sets the α value of the bright point determined to be the display area 420 to 0.

  FIG. 6 schematically shows an example of alpha blend mask image data. In the coordinates corresponding to the display area 420 with hatching in FIG. 6, the α value is set to 0. In other coordinates, the α value is set to 1. In the alpha blend mask image data in which the α value is set, the hatched display area 420 in FIG. 4 is completely transparent, and the other areas are completely opaque image data.

  As described above, the pre-processing is completed when the alpha blend mask image data is created.

  Here, one simple method for specifying the display area 420 is shown, but the method for specifying the display area 420 is not limited to the above. Various methods can be used to identify the display area 420.

<Lecture recording process>
The process of the image processing apparatus 10 performed in the lecture recording process in step S102 will be described with reference to the flowchart shown in FIG. The lecture recording process is performed, for example, by the user instructing the start of the lecture recording process immediately before the start of the lecture. The instruction to start the lecture recording process is performed, for example, by pressing a recording start button included in the input device 14.

  In step S <b> 301, the processor 11 starts recording the captured image data acquired from the digital camera 60 and the content data acquired from the PC 50. The captured image and the content are each associated with time information and recorded as a moving image in the large-capacity storage 13.

  In step S302, the processor 11 determines whether to end the recording. For example, when an instruction to end recording is input by the user, it is determined to end recording. The instruction to end the recording is given by, for example, pressing a recording end button included in the input device 14 after the end of the lecture. When it is determined in step S302 that recording is not to be ended, the process repeats step S302. At this time, the recording of the captured image and the content continues, and the captured image and the content associated with the time information are continuously recorded as a moving image in the large-capacity storage 13. If it is determined in step S302 that recording is to be ended, the process proceeds to step S303.

  In step S <b> 303, the processor 11 converts the captured moving image Mb, which is a moving image related to the captured image, and the content moving image Mc, which is a moving image related to the content, into files and stores them in the large-capacity storage 13.

  As described above, the lecture video recording process ends when the captured video Mb and the content video Mc are stored.

<Lecture video creation process>
The processing of the image processing apparatus 10 performed in the lecture video creation processing in step S103 will be described with reference to the flowchart shown in FIG. The lecture video creation process is performed, for example, by the user instructing the start of the lecture video creation process at an arbitrary time after the lecture ends. The instruction to start the lecture video creation process is performed, for example, when a lecture video creation button included in the input device 14 is pressed.

  In step S401, the processor 11 adjusts the size and position of the content moving image Mc and creates a moving image as an embedded image. In the present embodiment, a moving image created by adjusting the size and position of the content moving image Mc in this way is referred to as an adjusted moving image Mn. The overall image size of the adjusted moving image Mn is adjusted to be the same as the image size of the captured moving image Mb. In other words, the overall image size of the adjusted moving image Mn is adjusted to be a size corresponding to (0, 0) − (Xm, Ym) when expressed in coordinates in the example illustrated in FIG. 4, for example. The processor 11 reduces or enlarges the content moving image Mc in accordance with the size of the portion corresponding to the display area 420. Further, the processor 11 adjusts the position of the reduced or enlarged content moving image Mc to a position corresponding to the display area 420. In this way, the adjusted moving image Mn in which the position and size of the content moving image Mc are adjusted to a portion corresponding to the display area 420 is created. That is, for example, when expressed in coordinates in the example shown in FIG. 4, the size and position of the content moving image Mc are adjusted in accordance with (Xc1, Yc1)-(Xc2, Yc2). The time information included in the content moving image Mc is also carried over to the adjusted moving image Mn. The created adjusted moving image Mn is stored in the large-capacity storage 13, for example.

  In step S402, the processor 11 combines the photographed moving image Mb and the adjusted moving image Mn to create a moving image. In the present embodiment, a moving image in which the captured moving image Mb and the adjustment moving image Mn are combined in this way is referred to as a combined moving image Ma. For example, the processor 11 creates the synthesized moving image Ma by alpha blending using the captured moving image Mb as the foreground side and the mask image αm for alpha blending and the adjusted moving image Mn. At this time, using the time information included in the captured moving image Mb and the adjusted moving image Mn, these moving images are adjusted to be synchronized. The created composite video Ma is stored in the large-capacity storage 13, for example. This synthesized video Ma becomes the target lecture video. The above is the operation of the lecture video creation system 1.

  According to the lecture video creation system 1 according to the present embodiment, a composite video Ma is created in which an image at a position corresponding to the display area 320 is replaced with a content video Mc among the shot video Mb shot by the digital camera 60. The This synthesized video Ma becomes an image with excellent visibility. That is, in the shooting by the digital camera 60, the speaker 330 is generally focused. As a result, the projected image projected on the screen 340 can be out of focus. Further, in the projected image, geometric correction such as so-called trapezoidal correction can be performed so that the image is projected onto the rectangular display area 320. By such correction, in the projected image, the luminance is lowered or the image quality is deteriorated. Further, noise may occur in the projected image, such as spoke light that may be generated when a DLP (registered trademark) projector is used as the projector 80. Spoke light is difficult to recognize when the projected image is observed with the naked eye, but may be recognized in a captured image. In the synthetic video Ma created in the present embodiment, the image in the display area 420 is created based on the content data, so that the above-described defocus, luminance reduction, image quality degradation, noise superposition, etc. are displayed. It does not occur in the image of the area 420.

(Second Embodiment)
A second embodiment will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. In the present embodiment, a composite moving image Ma that is processed according to the captured moving image Mb obtained using the digital camera 60 is created.

  The color tone of the image in the display area 420 included in the photographed moving image Mb and the color tone of the content movie Mc may differ depending on the photographing condition and the like. In the present embodiment, when the adjustment moving image Mn is created, for example, a process of matching the color tone of the content moving image Mc with the color tone of the image in the display area 420 included in the captured moving image Mb is performed. A composite video Ma is created using the adjusted video Mn created in this way. As a result, a synthetic moving image Ma having a natural color tone on the entire moving image screen can be created. There are various methods for matching the color tone of the content moving image Mc with the color tone of the image in the display area 420 included in the captured moving image Mb. An example is shown below.

[First method]
The first method of this embodiment will be described. The lecture video creation process according to this method will be described with reference to the flowchart shown in FIG.

  In step S501, the processor 11 analyzes the captured moving image Mb. More specifically, the processor 11 specifies color tone information in the display area 420 included in the images constituting the captured moving image Mb. Here, the color tone can change due to the following factors, for example. That is, the color tone can change depending on shooting conditions such as setting of white balance when shooting with the digital camera 60. Further, the color tone can change depending on projection conditions such as characteristics of the light source of the projector 80. In addition, the color tone may change depending on environmental conditions such as lighting conditions at the lecture venue.

  In step S502, the processor 11 adjusts the size of the content moving image Mc and creates the adjusted moving image Mn, as in the case of the first embodiment. At this time, the processor 11 adjusts the color tone of the content moving image Mc according to the analysis result of step S501. That is, the processor 11 performs image processing so that the color tone of the content moving image Mc matches the color tone of the image in the display area 420 included in the captured moving image Mb. In this way, an adjusted moving image Mn is created in which the color tone is adjusted and the position and size of the content moving image Mc are adjusted to a portion corresponding to the display area 420.

  Note that the captured moving image Mb may be analyzed for the entire period of the moving image, and the color tone of the adjusted moving image Mn may be adjusted under different conditions for each time. The analysis of the captured moving image Mb may be performed only for the initial state of the lecture, for example, and the color tone of the adjusted moving image Mn may be adjusted under the same conditions over the entire period of the moving image. Further, the analysis of the captured moving image Mb may be repeatedly performed, for example, every predetermined period, and the color tone of the adjusted moving image Mn may be adjusted under a condition that is reset every fixed period.

  The time information included in the content moving image Mc is also carried over to the adjusted moving image Mn. The created adjusted moving image Mn is stored in the large-capacity storage 13, for example.

  In step S503, the processor 11 creates a composite video Ma in which the captured video Mb and the adjustment video Mn are combined, as in the case of the first embodiment. The created composite video Ma is stored in the large-capacity storage 13, for example. This synthesized video Ma becomes the target lecture video.

  As described above, the synthetic moving image Ma having natural characteristics in terms of color tone can be created in the entire moving image screen.

  Here, an example of image processing in which the color tone of the adjusted moving image Mn created based on the content moving image Mc is adjusted is shown, but the present invention is not limited to this. Instead of the adjustment movie Mn created based on the content movie Mc, the color tone of the photographed movie Mb may be adjusted by image processing. In this case, the processor 11 performs image processing for adjusting the color tone of the captured moving image Mb so that the color tone of the region other than the display region 420 of the captured moving image Mb matches the color tone of the content moving image Mc that has not been adjusted. . The processor 11 combines the photographed moving image Mb with the adjusted color tone and the adjusted moving image Mn to create a combined moving image Ma. In addition, the processor 11 may perform a process of adjusting the color tone of both the adjusted moving image Mn and the captured moving image Mb so that the color tones of both moving images are matched.

[Second method]
The second method of this embodiment will be described. In the first method, the color tone information in the display area 420 is specified by analyzing the captured moving image Mb. On the other hand, in this method, the color tone information in the display area 420 is specified based on various conditions at the time of shooting recorded in association with the data of the shooting moving image Mb.

  For example, in step S303 of the lecture recording process, one or more of various settings such as the white balance setting of the digital camera 60 and the projector 80 at the time of shooting are recorded in association with the data of the shooting moving image Mb.

  In the lecture movie creation process, the processor 11 reads the above-described information associated with the photographed movie Mb together with the photographed movie Mb. Based on this information, the processor 11 adjusts the color tone of the content moving image Mc and creates the adjusted moving image Mn, similarly to the process of step S502. Others are the same as in the case of the first method.

  Even in the above-described manner, the synthesized moving image Ma having a natural color tone characteristic on the entire moving image screen can be created.

(Third embodiment)
A third embodiment will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. Also in the present embodiment, a composite video Ma that is processed according to the captured video Mb obtained using the digital camera 60 is created.

  In the first embodiment, the example in which the display area in the captured image is specified in advance as the fixed area by the pre-processing has been described. On the other hand, for example, when a speaker, a pointer, or the like enters the display area, the boundary between the display area and other peripheral areas may change with time. For example, as shown in FIG. 10, when a part 352 of the speaker 330 overlaps with the original display area 320, when the process as in the first embodiment is performed, the original display area 320 is displayed in the synthesized video Ma. The part 352 of the speaker 330 that overlaps with will disappear. Further, when the processing as in the first embodiment is performed, for example, the bright spot 354 indicated by the laser pointer 332 included in the speaker 330 is not reproduced in the synthesized moving image Ma. In order to cope with such a situation, the image processing apparatus 10 may be configured such that the display area changes with time.

  The boundary between the display area and the peripheral area can be easily specified by comparing, for example, a content image and a captured image. That is, when the speaker 330 or the like does not enter the display area 320, the content image and the captured image display area 320 match. On the other hand, when a speaker or the like has entered the display area 320, the content image and the image in the display area of the photographed image do not match at the intruding portion. Therefore, the boundary between the display area and the peripheral area can be specified based on the area that does not match.

  Further, when a bright spot 354 formed by the laser pointer 332 exists in the captured image, the bright spot 354 has a very high luminance, and thus the bright spot 354 can be specified as a point having a high brightness.

  In any case, the alpha value of the alpha blend at the position where the photographed image should be displayed in the display area 420, for example, when a lecturer is shown, may be changed to 1 even in the display area 420. In this way, a natural lecture video is created even if there is something other than the display image in the display area 420. Hereinafter, some examples of the processing method according to the present embodiment will be further described with reference to the drawings.

[First method]
The lecture video creation processing according to this method will be described with reference to the flowchart shown in FIG.

  In step S601, the processor 11 analyzes the initial display area of the captured moving image Mb and the content moving image Mc, and detects a pixel difference at each time point during the lecture. Here, the initial display area is, for example, the display area 420 of the captured moving image Mb in an initial state where the speaker 330 or the like has not entered the area of the screen 340. Further, the display area specified in the pre-processing described with reference to FIG. 3 may be set as the initial display area. The difference between the initial display area of the photographed moving picture Mb and the content moving picture Mc is an area in the synthesized moving picture Ma that should indicate a pixel that does not correspond to the pixel of the content moving picture Mc in the initial display area. For example, an image of the speaker 330 who has entered the initial display area is detected as a difference. Further, an image of the bright spot 354 irradiated to the initial display area by the laser pointer 332 is detected as a difference.

  In step S602, the processor 11 updates the α value of the alpha blend mask image αm for the difference portion detected in step S601. More specifically, the α value is set to 1 for the portion where the difference is detected. In this way, the alpha blend mask image αm in which the α value may be different in each pixel of the initial display region at each time point in time series is created.

  That is, the initial display area other than the difference portion becomes a new display area. An image based on the content moving image Mc is synthesized in this new display area.

  In step S603, the processor 11 adjusts the size of the content video Mc, and creates and stores the adjusted video Mn, as in the first embodiment.

  In step S604, the processor 11 uses the alpha blend mask image αm generated in step S602 to combine the captured moving image Mb and the adjustment moving image Mn to generate a combined moving image Ma. The created composite video Ma is stored in the large-capacity storage 13, for example. This synthesized video Ma becomes the target lecture video. In the synthesized moving image Ma, an image based on the captured moving image Mb is displayed instead of the content moving image Mc in a portion corresponding to the bright spot 354 of the speaker 330 or the laser pointer 332 that has entered the initial display area.

  As described above, even if the captured moving image Mb includes the image of the lecturer 330 or the bright spot 354 between the digital camera 60 and the display area 320, the combined moving image Ma that is a natural moving image is obtained. Can be created.

  Note that, as in the second embodiment, this method may include a process of adjusting the color tone of the content moving image Mc based on the captured moving image Mb.

  Further, the adjustment of the color tone is not limited to being performed uniformly on all the pixels of the content moving image Mc or the captured moving image Mb as in the second embodiment. Another example of the color tone adjustment will be described below as the second method.

[Second method]
The lecture video creation process according to this method will be described with reference to the flowchart shown in FIG.

  In step S701, the processor 11 analyzes the color tone of the captured moving image Mb, as in step S501 of the second embodiment. That is, the processor 11 specifies the color tone information in the display area 420 included in the image constituting the captured moving image Mb.

  In step S702, as in step S601 of the first method, the processor 11 analyzes the initial display area of the captured moving image Mb and the content moving image Mc, and detects a difference between corresponding pixels at each time point during the lecture.

  In step S703, the processor 11 updates the α value of the alpha blend mask image αm for the difference portion detected in step S702, as in step S602 of the first method.

  In step S704, the processor 11 adjusts the size of the content moving image Mc and creates the adjusted moving image Mn, as in step S502 of the second embodiment. Further, the processor 11 adjusts the color tone of the content moving image Mc by image processing based on the analysis result of step S701, and creates an adjusted moving image Mn. The processor 11 performs image processing for adjusting the color tone so that the images are smoothly connected particularly at the boundary between the captured moving image Mb and the content moving image Mc. At this time, the processor 11 may adjust the color tone uniformly for the entire content moving image Mc. In addition, the processor 11 specifies a position that becomes a boundary between the image based on the captured moving image Mb and the image based on the content moving image Mc when combined based on the difference detected in step S702, and uses this to determine the color tone. May be adjusted. This boundary corresponds to the edge of a new display area obtained by removing the difference portion from the initial display area. The processor 11 may adjust the color tone of the content moving image Mc stepwise toward the boundary at the boundary portion so that the specified boundary looks natural. Such color tone adjustment may be performed on the entire length of the edge of the display area, or may be performed on the edge of the display area corresponding to the periphery of the difference portion, for example.

  In step S705, the processor 11 adjusts the color tone of the captured moving image Mb by image processing based on the analysis result in step S701. In the present embodiment, a moving image created by adjusting the color tone of the captured moving image Mb is referred to as an adjusted captured moving image Md. The processor 11 performs image processing for adjusting the color tone so that the images are smoothly connected particularly at the boundary between the captured moving image Mb and the content moving image Mc. At this time, the processor 11 may adjust the color tone uniformly for the entire captured moving image Mb. Further, the processor 11 may specify the edge of the display area specified based on the difference detected in step S702, and adjust the color tone using this. The processor 11 may adjust the color tone of the captured moving image Mb in a stepwise manner toward the boundary at the boundary portion so that the specified edge looks natural. Such color tone adjustment may be performed on the entire length of the edge of the display area, or may be performed on the edge of the display area corresponding to the periphery of the difference portion, for example.

  In step S706, the processor 11 combines the adjusted shooting moving image Md and the adjusted moving image Mn using the alpha blend mask image αm generated in step S703 to generate a combined moving image Ma. The created composite video Ma is stored in the large-capacity storage 13, for example. This synthesized video Ma becomes the target lecture video.

  In the synthesized moving image Ma, an image based on the captured moving image Mb, not the content moving image Mc, is represented in the portion of the speaker 330 and the bright spot 354 of the laser pointer 332 that enter the display area 320. In addition, according to the present method, the boundary between the adjusted captured moving image Md and the adjusted moving image Mn to be combined becomes smooth.

  In the above example, an example in which the color tone of both the captured moving image Mb and the content moving image Mc is adjusted has been described, but the present invention is not limited thereto. The color tone of either one of the captured moving image Mb and the content moving image Mc may be adjusted. Further, the boundary between the captured moving image Mb and the content moving image Mc is smoothed by using the alpha blend mask image αm regardless of the color tone of the captured moving image Mb or the content moving image Mc or in addition to the adjustment of the color tone. Good. That is, in the boundary portion, the synthesis ratio of the adjusted shooting moving image Md and the adjusted moving image Mn may be changed stepwise according to the distance from the boundary.

(Fourth embodiment)
A fourth embodiment will be described. Here, differences from the first embodiment will be described, and the same portions will be denoted by the same reference numerals and description thereof will be omitted. Also in the present embodiment, a composite video Ma that is processed according to the captured video Mb obtained using the digital camera 60 is created. In the present embodiment, based on the captured moving image Mb, an image that did not exist in the actual lecture is combined with the combined moving image Ma.

  For example, as shown in FIG. 13, when the speaker 330 performs a predetermined operation using the laser pointer 332, a display corresponding to the operation is included in the synthesized moving image Ma. For example, when the position of the bright spot is changed so as to reciprocate a predetermined number of times, a line 464 corresponding to the locus of the bright spot is superimposed on the synthesized video Ma as a superimposed image and included in the synthesized video Ma. For example, when the lecturer 330 wants to add an underline to a portion to be emphasized in the content, the lecturer 330 may reciprocate the position of the bright spot a predetermined number of times at the position. Further, when the position of the bright spot is changed so as to surround a certain position, a surrounding line 466 corresponding to the locus of the bright spot is included in the synthesized moving image Ma. For example, when the lecturer 330 wants to surround a portion to be emphasized in the content with a rectangle or a circle, the speaker 330 may operate the laser pointer 332 so as to surround the portion with a bright spot. Further, in order to emphasize the position pointed by the laser pointer 332, for example, an arrow graphic 462 may be displayed at the bright spot portion in the synthesized moving image Ma.

  The lecture video creation processing according to the present embodiment will be described with reference to the flowchart shown in FIG.

  In step S801, the processor 11 adjusts the size of the content moving image Mc to create an adjusted moving image Mn.

  In step S802, the processor 11 analyzes the initial display area of the captured moving image Mb and the content moving image Mc, and detects pixel differences at each time point during the lecture. In this embodiment, the processor 11 specifies the bright spot by the laser pointer 332 based on this difference. The processor 11 identifies the bright spot as an indicated position that is the position indicated by the speaker 330 within the initial display area.

  In step S803, the processor 11 analyzes the movement of the bright spot that is the designated position identified in step S802. For example, the processor 11 analyzes whether the luminescent spot moves so as to reciprocate the same position a predetermined number of times along the time series, or whether the luminescent spot moves so as to surround an arbitrary position.

  In step S804, the processor 11 analyzes whether or not the content display page has been changed in the content moving image Mc. The processor 11 can specify whether or not the page of the content has been turned, for example, by analyzing the content moving image Mc. Further, in the lecture recording process, the timing of the page turning operation of the content by the speaker 330 may be recorded, and the processor 11 may specify whether or not the page of the content has been turned based on the recording.

  In step S805, the processor 11 creates a moving image to be displayed superimposed on the captured moving image Mb and the adjusted moving image Mn in the combined moving image Ma based on the analysis results in steps S803 and S804, and stores the moving image in, for example, the large-capacity storage 13. . A moving image including an image not included in the captured moving image Mb and the content moving image Mc as created here is referred to as a superimposed moving image Me. For example, the processor 11 creates the superimposed moving image Me including the line 464 at the position so that an underline is displayed at the position when the bright spot reciprocates the same position a predetermined number of times. In addition, when the bright spot surrounds an arbitrary position, the processor 11 creates the superimposed moving image Me including the surrounding line 466 at the position so that the surrounding line is displayed along the position of the bright spot. Further, the processor 11 creates the superimposed moving image Me in which the arrow graphic 462 is included at the position of the bright spot so that the arrow graphic 462 is displayed in accordance with the position of the bright spot.

  The line 464 and the like in the superimposed video Me may be configured to be deleted when the content page is changed. In other words, even if the content page changes, the underline etc. corresponding to the content of the previous page will remain and it will be difficult to understand, so the underline etc. will be deleted at the same time the content page is turned In addition, the superimposed moving image Me is created.

  In step S806, the processor 11 combines the captured moving image Mb and the adjusted moving image Mn using the alpha blend mask image αm. Furthermore, the processor 11 synthesizes the superimposed moving image Me to create a combined moving image Ma. The created composite video Ma is stored in the large-capacity storage 13, for example. This synthesized video Ma becomes the target lecture video.

  In this synthesized moving image Ma, an image corresponding to the movement of the bright spot by the laser pointer 332 is included. As a result, the lecture video is an easy-to-understand video with more information added to the actual lecture.

  As shown here, a line is displayed when the luminescent spot has reciprocated a predetermined number of times, or a surrounding line is displayed when the luminescent spot surrounds a predetermined position. It is an example of the relationship with the image to be performed.

  In the above-described example, an example in which the bright spot is detected based on the difference between the photographed moving picture Mb and the content moving picture Mc is shown, but the method of detecting the bright spot is not limited to this. For example, a point included in the photographed moving image Mb and having a luminance higher than a predetermined level may be specified as a bright spot by the laser pointer. Further, as the laser pointer 332, a laser pointer that shows a predetermined blinking pattern that is not recognized by the naked eye but can be specified in the captured moving image Mb may be used. In this case, the position of the bright spot can be specified based on the blinking pattern.

  In the above-described embodiment, the bright spot of the laser pointer 332 is used as the designated position, and the superimposed moving image Me is created based on the bright spot. However, the present invention is not limited to this, and an image to be superimposed may be created based on the gesture of the speaker 330. For example, the position indicated by the speaker 330 with the hand may be set as the designated position, and when a predetermined operation by the speaker 330 is detected, an image corresponding to the operation may be created as a superimposed image.

  Further, when it is possible to create an image to be superimposed in real time, the image can be projected by the projector 80. In this case, since an underline or the like is displayed on the screen 340, an image easy to understand for a person who is actually listening to the lecture is displayed.

  The present embodiment may be configured to be implemented in combination with the second embodiment or the third embodiment.

[Modification]
The case where the image of the document was presented by the display device and the lecturer gave a lecture beside the image was explained. Here, the case where a projector is used as the display device has been described as an example. A display device for presenting materials is not limited to a projector. For example, various devices such as a liquid crystal display may be used instead of the projector. In this case, in the description of any of the above-described embodiments, the portion described as “projection” can be replaced with “display”. The same applies to the modifications shown below. In the third embodiment, the case where a speaker 330 enters the display area 320 is considered. In such a case, when the display device is a projector, a speaker 330 enters between the screen and the projector, and an image is projected onto the speaker 330. On the other hand, when the display device is a liquid crystal display or the like, the speaker 330 enters between the display and the digital camera.

  The case where the projection state is adjusted so that the display area 320 is rectangular has been described as an example, but the display area 320 may not be rectangular. The shape of the adjustment moving image Mn to be synthesized can be appropriately corrected by image processing in accordance with the display area 420 specified based on the captured image. By such image processing, an appropriate lecture video can be created even if the shape of the display area 320 does not match the shape of the content image.

  When changing the projection image, the user operates the PC 50 or the projector 80. For example, when the projected image is paged, the user operates the PC 50, and when performing the keystone correction of the projected image, the user operates the projector 80. The processor 11 may acquire information on an operation performed on the PC 50 or the projector 80, and create the adjustment moving image Mn using the information.

  Further, for example, even when there is no change in the content data output from the PC 50, a part of the projected image is enlarged using the function of the projector 80, or the enlarged part is moved up, down, left, and right. Sometimes. In such a case, the processor 11 may acquire change information related to an operation input to the projector 80, and create the adjustment moving image Mn using the acquired change information. By using such a method, it is easier to create the adjusted moving image Mn than to determine the adjusted moving image Mn to be generated by image processing using only the captured moving image Mb.

  The image as the content output from the PC 50 and projected by the projector 80 may be a still image or a moving image. Further, the content may include audio. The audio included in the content can be output by the projector 80 or other means. The audio related to the content included in the lecture video is recorded using the microphone 70, and may be processed in the same manner as the audio generated by the speaker, for example, or from the PC 50 to the image processing apparatus 10. It may be created on the basis of voice data input to.

  In the above-described embodiment, the image synthesis by the method using the alpha blend has been described, but the synthesis method is not limited to this. Various methods can be used for image synthesis.

  Further, in each of the above-described embodiments, the image processing apparatus 10 has been described as performing any of the pre-processing, the lecture recording process, and the lecture moving image creation process, but the present invention is not limited thereto. For example, the lecture moving image creation process may be performed by an apparatus different from the apparatus that has performed the pre-processing and the lecture recording process. That is, if the captured moving image Mb, the content moving image Mc, and the mask image αm are provided, a lecture moving image creation process may be performed by a PC or a server based on these.

  In each of the above-described embodiments, the image processing apparatus 10 has been described as creating a lecture video by image synthesis, but the present invention is not limited thereto. For example, when the image quality of a part of the display area 420 of the photographed moving image Mb is deteriorated, only that part may be corrected based on the content moving image Mc. That is, it is conceivable to perform correction using information on a part corresponding to the part of the content moving image Mc according to the degree of deterioration in image quality of the part. The display area 420 for the captured moving image Mb only needs to be image processing that is sufficiently high in quality for a person viewing the lecture moving image, and is not limited to the synthesis of the fitted image.

  Further, the composition of the captured moving image Mb and the content moving image Mc may be performed in real time if the processing capacity of the processor 11 is sufficiently high. If a lecture video can be created in real time, the created lecture video can be transmitted to a remote location so that a remote audience can view the lecture with high-quality images at the same time as the lecture.

  The example in the case where the image processing apparatus 10 is provided independently from the PC 50 and the projector 80 has been described, but the function as the image processing apparatus 10 may be provided in the projector 80, for example. It may be provided in the PC 50. For example, the above-described function may be provided as a projector in which the projector 80, the digital camera 60, the microphone 70, and the image processing apparatus 10 illustrated in FIG. The position where the projector is arranged is a position suitable for projecting the display image, and this is a position suitable for photographing the projected image and the speaker at the same time. Further, content data is generally input to the projector to perform projection, and this is content data for creating a composite image at the same time. Therefore, it is reasonable to place the digital camera and the image processing apparatus in the housing of the projector.

  Note that the PC 50 in the above-described embodiment is an example, and the PC 50 can be replaced with various devices that operate application software for presentation, such as a tablet information terminal and a smartphone. Even if these devices are not provided, for example, the projector 80 may perform the function. In this case, a recording medium on which data related to the presentation is recorded is connected to the projector 80, and the projector 80 performs projection based on this data. At this time, the projector 80 transmits the data of the projected image to the image processing apparatus 10 as content data. Further, the image processing apparatus 10 may perform the function of the PC 50. In this case, a recording medium on which data related to the presentation is recorded is connected to the image processing apparatus 10, or the data is recorded on the large-capacity storage 13. The image processing apparatus 10 transmits content data to the projector 80 based on these data.

  Any of these modifications can be combined in various ways, and any of these modifications and combinations thereof can be combined with any of the first to fourth embodiments described above.

  In addition, this invention is not limited to the said embodiment, In the implementation stage, it can change variously in the range which does not deviate from the summary. Further, the embodiments may be implemented in combination as appropriate, and in that case, the combined effect can be obtained. Furthermore, the present invention includes various inventions, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if several constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the problem can be solved and an effect can be obtained, the configuration from which the constituent requirements are deleted can be extracted as an invention.

Hereinafter, the invention described in the scope of claims of the present application will be appended.
[1]
A captured image acquisition unit that acquires a captured image of an area including a display area in which content is displayed by the display device and a peripheral area of the display area;
A content acquisition unit for acquiring the content;
A display area specifying unit for specifying the display area in the captured image;
A processing information creation unit for creating processing information based on the captured image;
An image processing unit that performs image processing including synthesizing an image based on the content in the display area of the captured image using the processing information;
An image processing apparatus comprising:
[2]
The processing information creation unit identifies the display area that changes over time based on the captured image,
The image processing unit performs image processing for synthesizing an image based on the content in the display area of the captured image that changes with time.
The image processing device according to [1].
[3]
The modification information creation unit identifies the display area that changes with time by comparing an image inside the initial display area, which is the display area in the initial state of the captured image, with the content. 2].
[4]
The processing information creation unit creates the processing information including information on at least one of the color tone of the captured image and the color tone of the image based on the content,
The image processing unit is configured to match the captured image and the content with each other so that the color tone of the captured image matches the color tone of the image based on the content in at least a part of the edge of the display area in the synthesized image. Performing image processing for adjusting the color tone for at least a part of at least one of them,
The image processing device according to [3].
[5]
The processing information creation unit creates the processing information including information on at least one of the color tone of the captured image and the color tone of the image based on the content,
The image processing unit adjusts the color tone of at least one of the captured image and the content so that the color tone of the captured image in the synthesized image matches the color tone of the image based on the content. Perform image processing,
The image processing device according to any one of [1] to [3].
[6]
The processing information creation unit identifies a designated position inside the display area based on the captured image, creates a superimposed image based on the designated position,
The image processing unit further synthesizes the superimposed image with an image created by combining the captured image and an image based on the content;
The image processing device according to any one of [1] to [5].
[7]
The image processing apparatus according to [6], wherein the processing information creation unit creates the superimposed image including at least one of a figure indicating the designated position and a figure showing a locus of the designated position.
[8]
The image processing device according to [7], wherein the processing information creation unit creates the superimposed image from which a graphic indicating the locus of the designated position is deleted when the content page is changed.
[9]
The processing information creation unit identifies the designated position, which is a bright spot by a laser pointer, or the designated position specified by a speaker's action, according to any one of [6] to [8]. Image processing device.
[10]
The image processing unit obtains change information related to the image processing from the display device that performs image processing on the content and displays the content, and performs image processing based on the change information. ] To [9].
[11]
The image processing apparatus according to any one of [1] to [10];
A projection unit as the display device that outputs an image projected on the display area based on the content;
A projector comprising: a photographing unit that generates the photographed image.
[12]
Acquiring a captured image of an area including a display area in which content is displayed by the display device and a peripheral area of the display area;
Obtaining the content;
Identifying the display area in the captured image;
Creating processing information based on the captured image;
Performing image processing including synthesizing an image based on the content in the display area of the captured image using the processing information;
An image processing method including:
[13]
Acquiring a captured image of an area including a display area in which content is displayed by the display device and a peripheral area of the display area;
Obtaining the content;
Identifying the display area in the captured image;
Creating processing information based on the captured image;
Performing image processing including synthesizing an image based on the content in the display area of the captured image using the processing information;
An image processing program for causing a computer to execute.

  DESCRIPTION OF SYMBOLS 1 ... Lecture animation production system, 10 ... Image processing apparatus, 11 ... Processor, 12 ... Memory, 13 ... Mass storage, 14 ... Input device, 15 ... Content input interface (I / F), 16 ... Image input interface (I / F), 17 ... Audio input interface (I / F), 18 ... Image output interface (I / F), 19 ... Bus line, 50 ... Personal computer (PC), 60 ... Digital camera, 70 ... Microphone, 80 ... Projector, 310 ... shooting area, 320 ... display area, 330 ... speaker, 340 ... screen.

Claims (13)

A captured image acquisition unit that acquires a captured image of an area including a display area in which content is displayed by the display device and a peripheral area of the display area;
A content acquisition unit for acquiring the content;
A display area specifying unit for specifying the display area in the captured image;
A processing information creation unit for creating processing information based on the captured image;
An image processing unit that performs image processing including synthesizing an image based on the content in the display area of the captured image using the processing information;
An image processing apparatus comprising: The processing information creation unit identifies the display area that changes over time based on the captured image,
The image processing unit performs image processing for synthesizing an image based on the content in the display area of the captured image that changes with time.
The image processing apparatus according to claim 1.   The processing information creation unit identifies the display area that changes with time by comparing an image inside the initial display area, which is the display area in the initial state of the captured image, with the content. Item 3. The image processing apparatus according to Item 2. The processing information creation unit creates the processing information including information on at least one of the color tone of the captured image and the color tone of the image based on the content,
The image processing unit is configured to match the captured image and the content with each other so that the color tone of the captured image matches the color tone of the image based on the content in at least a part of the edge of the display area in the synthesized image. Performing image processing for adjusting the color tone for at least a part of at least one of them,
The image processing apparatus according to claim 3. The processing information creation unit creates the processing information including information on at least one of the color tone of the captured image and the color tone of the image based on the content,
The image processing unit adjusts the color tone of at least one of the captured image and the content so that the color tone of the captured image in the synthesized image matches the color tone of the image based on the content. Perform image processing,
The image processing apparatus according to any one of claims 1 to 3. The processing information creation unit identifies a designated position inside the display area based on the captured image, creates a superimposed image based on the designated position,
The image processing unit further synthesizes the superimposed image with an image created by combining the captured image and an image based on the content;
The image processing apparatus according to any one of claims 1 to 5.   The image processing apparatus according to claim 6, wherein the processing information creating unit creates the superimposed image including at least one of a figure indicating the designated position and a figure showing a locus of the designated position.   The image processing device according to claim 7, wherein when the content page is changed, the processing information creation unit creates the superimposed image from which a graphic indicating the locus of the designated position is deleted.   The image according to any one of claims 6 to 8, wherein the processing information creation unit identifies the designated position, which is a bright spot by a laser pointer, or the designated position specified by an action of a speaker. Processing equipment.   The image processing unit obtains change information related to the image processing from the display device that performs image processing on the content and displays the content, and performs image processing based on the change information. The image processing apparatus according to any one of 1 to 9. An image processing apparatus according to any one of claims 1 to 10,
A projection unit as the display device that outputs an image projected on the display area based on the content;
A projector comprising: a photographing unit that generates the photographed image. Acquiring a captured image of an area including a display area in which content is displayed by the display device and a peripheral area of the display area;
Obtaining the content;
Identifying the display area in the captured image;
Creating processing information based on the captured image;
Performing image processing including synthesizing an image based on the content in the display area of the captured image using the processing information;
An image processing method including: Acquiring a captured image of an area including a display area in which content is displayed by the display device and a peripheral area of the display area;
Obtaining the content;
Identifying the display area in the captured image;
Creating processing information based on the captured image;
Performing image processing including synthesizing an image based on the content in the display area of the captured image using the processing information;
An image processing program for causing a computer to execute.