JP6102181B2 - Image processing apparatus, program, and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, a program, and an image processing method.

  In order to enhance the impression when presenting advertisements and other contents to customers and viewers, there is a technology for projecting content images, etc., in a shape that matches the external shape of the screen onto a projection screen that is shaped like a human figure It has been proposed (see, for example, Patent Document 1). This technology is used for the production of novel and effective exhibitions at exhibitions, etc., or for explanation of products for human beings such as “virtual mannequins”.

  As the above-mentioned human type screen, a film for rear projection, for example, a special film provided with a black light absorption layer in the gap between the spherical beads and the layer on which the microbead lens is arranged on the back side to be projected of the transparent vinyl chloride film, etc. It is often used by sticking it to a base material such as a transparent acrylic board molded into a human shape, and it is relatively bright and bright from a wide angle and has a presence as if a human is standing there. A projected image is obtained.

  In the above technology, if the human figure (projected person image) protrudes outside the outer shape of the humanoid screen, the face or shoulder of the human figure in that part may be missing, thinned, or unnaturally shaped. become. Further, when a person image is taken and used, the outline of the person image moves or changes. Therefore, projection is performed so that a blank area is formed between the humanoid screen and the human image.

JP 2011-150221 A

  However, conventionally, since the blank area has not been used, the maximum rendering effect has not been obtained.

  The present invention has been made in view of the above-described problems, and an object thereof is to improve a rendering effect by using a blank area between a screen and an image of a projection object. .

In order to solve the above-mentioned problem, the invention described in claim 1
An image processing apparatus connected to a projection apparatus that projects video content including a person onto a screen molded into the shape of the person ,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer shape extraction means for extracting the outer shape of the screen from the acquired photographed image;
Discrimination means for discriminating the gender of the person based on video data of the video content;
The person area is extracted from the video data of the video content, and the video data in the video data is projected onto the screen based on the extracted person area and the outer shape of the screen extracted from the captured image. A synthesizing unit that synthesizes an image of a design according to the gender determined by the determining unit in a region corresponding to the outline of the person and the outer shape of the screen when
On the basis of the synthesized image data, and control means for projecting onto the screen by the projection apparatus a video content which the image is synthesized during the contour and contour of the screen of the person,
Is provided.

  According to the present invention, it is possible to improve the effect by using the blank area between the screen and the image of the projection target.

It is a figure which shows the example of whole structure of the content reproduction system in this embodiment. (A) is a front view of the content reproduction system before video projection, and (b) is a front view of the content reproduction system at the time of video projection. It is a block diagram which shows the functional structure of the image processing apparatus of FIG. It is a flowchart which shows the reproduction | regeneration control processing performed in the control part of FIG. It is a flowchart which shows the position alignment process performed in step S1 of FIG. It is a flowchart which shows the screen external shape extraction process performed in step S2 of FIG. (A) is a diagram schematically showing the captured image acquired in step S202 of FIG. 6, (b) is a diagram schematically showing a frame image to which a dividing line is added, and (c) is (a). FIG. 6 is a diagram schematically showing a blank area extracted based on (b). 5 is a flowchart showing a composition image generation process (composition image generation process A) executed in step S3 of FIG. 4 in the first embodiment. It is a flowchart which shows the blank area extraction process performed in step S302 of FIG. It is a flowchart which shows the projection control process performed in step S4 of FIG. It is a figure which shows the example by which the image was synthesize | combined to the blank area | region corresponded between the external shape of the screen of a frame image, and the outline of a person area. It is a figure which shows the example of a whole structure of the content reproduction system in 2nd Embodiment. It is a flowchart which shows the image generation process for a synthesis | combination (image generation process for a synthesis | combination B) performed in 2nd Embodiment. It is a flowchart which shows the image generation process for a synthesis | combination (image generation process C for a synthesis | combination) performed in 3rd Embodiment. It is a flowchart which shows the image generation process for a synthesis (image generation process for a synthesis D) performed in 4th Embodiment. It is a flowchart which shows the image generation process for a synthesis (image generation process for a synthesis E) performed in 5th Embodiment. It is a flowchart which shows the reproduction | regeneration control process B performed in 6th Embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The present invention is not limited to the illustrated example.

[First Embodiment]
<Configuration of Content Playback System 100>
1 and 2 are diagrams showing a schematic configuration of a content reproduction system 100 according to the present embodiment. FIG. 1 is a side view of the content reproduction system 100, and FIG. 2 is a front view of the content reproduction system 100. FIG. 2A is a front view before image projection, and FIG. 2B is a front view during image projection.

  The content reproduction system 100 is a system that reproduces video content for projection and projects a projection target included in the content onto a screen that is molded into the shape of the projection target. In the present embodiment, the case where the projection target is a person and the screen is a human type will be described as an example. However, the present invention is not limited to this. The projection object may be, for example, a character or an animal.

  As shown in FIG. 1, the content reproduction system 100 includes an image processing device 1, a projector 2, a camera 3, a speaker 4, a screen 5, a microphone 7, and the like. The projector 2, camera 3, speaker 4, and microphone 7 are connected to the image processing apparatus 1.

The image processing apparatus 1 synthesizes an image in an area (margin area) corresponding to the outline of the person area and the outline of the screen 5 when the video data is projected onto the screen 5 in the video data of the video content for projection. Then, the composite image is transmitted to the projector 2 and projected onto the screen 5. As the image processing apparatus 1, for example, a PC (Personal Computer), a terminal device, or the like is applicable.
Here, the outline of the screen 5 indicates the outline of the screen 5, and the outline area of the screen 5 indicates an area surrounded by the outline of the screen 5. An area between the outline of the screen 5 and the outline of the person area (person image) of the projected image is referred to as a margin area. In the video data (frame image) of the projection video content, A region corresponding to the outline of the person image is called a blank region.

  FIG. 3 shows a functional configuration example of the image processing apparatus 1. As shown in FIG. 3, the image processing apparatus 1 includes a control unit 10, an operation unit 11, a display unit 12, a storage unit 13, an I / F 14, an I / F 15, an audio output unit 16, an I / F 17, and the like. Has been.

Although not shown, the control unit 10 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The CPU of the control unit 10 reads the program stored in the program storage unit 131 of the storage unit 13, expands it in the work area of the RAM, and executes various processes including a reproduction control process described later according to the expanded program. To do. The RAM of the control unit 10 is a volatile memory and has a work area for storing various programs executed by the CPU and data related to these various programs.
The control unit 10 functions as a screen image acquisition unit, a screen outer shape extraction unit, a synthesis unit, and a control unit by executing a reproduction control process.

  The operation unit 11 has various function keys, cursor keys, and the like, receives key input by the user, and outputs the operation information to the control unit 10.

  The display unit 12 is configured by an LCD (Liquid Crystal Display) or the like, and performs various displays according to a display control signal from the control unit 10.

  The storage unit 13 includes an HDD (Hard Disk Drive), a semiconductor nonvolatile memory, or the like. As shown in FIG. 3, the storage unit 13 includes a program storage unit 131, a video data storage unit 132, and a setting information storage unit 133.

  The program storage unit 131 stores a system program executed by the control unit 10 and various processing programs, data necessary for executing these programs, and the like. The program is stored in the program storage unit 131 in the form of computer readable program code. The control unit 10 sequentially executes operations according to the program code.

  The video data storage unit 132 stores video data of projection video content. The video data is composed of a plurality of frame image data constituting the moving image data and audio data corresponding to each frame image.

  The setting information storage unit 133 stores setting information indicating the magnification of the person area and the amount of movement of the position when the video data of the projection video content is projected on the screen 5. For example, the magnification setting information is obtained by extracting a person area from a frame image of the projection video content (for example, a frame image of the first frame) by known image processing such as image recognition, and the size of the extracted person area, Based on the enlargement ratio of the projector 2, the size of the screen 5, etc., it is calculated and registered in advance. As the setting information for the amount of movement of the position, for example, a person area is extracted from the frame image of the video content for projection by image processing, the centroid of the person area is obtained, and the movement is performed so that the obtained centroid position matches the center of the image. The amount of movement and the direction of movement are registered in advance.

The I / F 14 is an interface for connecting to the projector 2 and performing data transmission / reception.
The I / F 15 is an interface for connecting to the camera 3 and performing data transmission / reception.

The audio output unit 16 is connected to the speaker 4, converts audio data instructed by the control unit 10 into an audio signal, and outputs the audio signal from the speaker 4.
The I / F 17 is connected to the microphone 7 and inputs an electrical electrical signal from the microphone 7.

  The projector 2 is a rear projection type projection device that projects an image from the back surface of the screen 5 based on the frame image transmitted from the image processing device 1. For example, the projector 2 individually turns on / off each tilt angle of a plurality of micromirrors arranged in an array (in the case of XGA, horizontal 1024 pixels × vertical 768 pixels) based on a frame image at a high speed. Thus, a DLP (Digital Light Processing) (registered trademark) projector using a DMD (digital micromirror device) which is a display element that forms a light image by reflected light by performing a display operation can be applied. When the projection target is a person area in the frame image, the person area can be projected by turning on the minute mirror at a position corresponding to the person area and turning off the other areas.

  The camera 3 is installed at a position where the outer shape of the screen 5 falls within the field of view, images the screen 5 in accordance with an instruction from the image processing apparatus 1, and transmits the obtained captured image to the image processing apparatus 1 via the I / F 15. To do.

  The screen 5 is a human-type screen supported by the support base 6 and arranged in the output light irradiation direction of the projector 2. As shown in FIG. 1, the screen 5 has a screen film 52 for rear projection attached to the upper front side of a base 51 such as a transparent acrylic plate molded into a human shape to be projected, and a person to be projected on the lower side. A film 53 on which the lower half of the body is printed is attached. By using a commercially available screen film having high brightness and high contrast, the content projected on the screen 5 can be visually recognized even in the daytime or in a bright room. can do.

  In the present embodiment, the projection video content in which the upper body of a person is photographed is projected onto the screen 5 as an example. However, a screen film 52 for rear projection is attached to the entire surface of the base 51 of the screen 5. The projection video content in which the whole body of the person is photographed may be projected onto the screen 5.

  The microphone 7 detects sound and converts it into an electric signal.

<Operation of Image Processing Apparatus 1>
Next, the operation in this embodiment will be described.
In the configuration of the content reproduction system 100 shown in FIG. 1, the frame image of the projection video content and the captured image of the projection image projected on the screen 5 are reversed from the left and right of the front image. In the figure to be used, left-right reversal is omitted, and all are shown as front images.

  FIG. 4 shows a flowchart of the reproduction control process executed by the image processing apparatus 1. The reproduction control process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.

First, the control unit 10 executes an alignment process (step S1).
FIG. 5 shows a flowchart of the alignment process. The alignment process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.

In the alignment process, first, the control unit 10 reads and acquires the frame image (for example, the first frame) of the video content for projection from the video data storage unit 132 of the storage unit 13 (step S101).
Next, the control unit 10 acquires setting information from the setting information storage unit 133 of the storage unit 13 and creates a frame image in which the magnification and position of the person area are changed based on the acquired setting information (step S102).

  Next, the control unit 10 extracts a person area from the created frame image, transmits the position information of the extracted person area and the created frame image to the projector 2 via the I / F 14, and the projector 2 transmits the frame image. The person area inside is projected on the screen 5 (step S103). The person region can be extracted by a known image processing technique such as image recognition, for example.

Next, the control unit 10 causes the camera 3 to capture a projected image including the screen 5 to acquire a captured image (step S104), and extracts an outer region of the screen 5 from the acquired captured image (step S105).
The outline area of the screen 5 in step S105 is extracted by, for example, performing blurring processing and binarization processing on the captured image, but is not limited thereto, and is extracted using other known image processing techniques. May be.

  Next, the control unit 10 extracts a person area of the projected image from the captured image (step S106). For example, a person region is extracted by performing binarization processing on a captured image using a threshold value of brightness (brighter) higher than that in step S105. However, the present invention is not limited to this, and other known image processing techniques may be used. May be used for extraction.

  Next, the control unit 10 compares the extracted outline area of the screen 5 with the person area, and aligns the two (step S107). For example, the center of gravity of the outline area of the screen 5 and the center of gravity of the person area are obtained, and the person area is moved so that they match, and then the person area has a predetermined size with respect to the outline area of the screen 5 (for example, , The magnification of the person area is changed to be about 0.9 times.

  Next, based on the magnification and movement amount (and movement direction) of the person area acquired in the alignment in step S107, the control unit 10 calculates the magnification and position movement amount (and movement direction) of the person area in the frame image at the time of projection. ) Is updated, the setting information stored in the setting information storage unit 133 is updated based on the calculation result (step S108), and the process proceeds to step S2 in FIG.

  In step S2 of FIG. 4, a screen outer shape extraction process is executed (step S2).

  FIG. 6 shows a flowchart of the screen outer shape extraction process. The screen outer shape extraction process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.

  First, the control unit 10 divides an image having the same size as the frame image of the projection video content into areas of n × n (n is a positive integer), and the image including the dividing line is projected via the I / F 14 to the projector. 2 to be projected (step S201).

  Next, the control unit 10 causes the camera 3 to capture a projected image including the screen 5 being projected, acquires a captured image (step S202), and extracts the outer shape of the screen 5 from the acquired captured image (step S203). Then, the process proceeds to step S3 in FIG. The extraction of the outer shape of the screen 5 can be performed by extracting the outer region of the screen 5 by the same process as described in step S105 of FIG. 5 and extracting the contour. FIG. 7A shows a schematic diagram of the captured image acquired in step S202.

In step S3 of FIG. 4, a composite image generation process is executed (step S3). In the present embodiment, a composite image generation process A shown in FIG. 8 is executed as the composite image generation process.
FIG. 8 shows a flowchart of the composite image generation process. The composite image generation process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.

  First, the control unit 10 reads and acquires the frame image of the projection video content from the video data storage unit 132 of the storage unit 13 (step S301).

Next, the control unit 10 performs a blank area extraction process (step S302).
FIG. 9 shows a flowchart of blank area extraction processing. The blank area extraction process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.

  First, the control unit 10 acquires the setting information of the magnification and position movement amount of the person area when projecting the frame image of the projection video content from the setting information storage unit 133 of the storage unit 13, and the acquired setting Based on the information, a frame image in which the magnification and position of the person area are changed is created (step S2021).

Next, the control unit 10 divides the created frame image into n × n, and adds a dividing line to the frame image (see step S2022, FIG. 7B).
Next, the control unit 10 extracts a person region from the frame image (step S2023). The extraction of the person area can be performed by a known image processing technique such as image processing such as image recognition.

  Then, the control unit 10 compares the outline of the screen extracted by the screen outline extraction process in step S2 of FIG. 4 with the extracted outline of the person area, and extracts a blank area from the frame image (step S2024). For example, by taking a correspondence between a captured image and a frame image by using a dividing line of a captured image and a dividing line of a frame image captured by the screen outer shape extraction process, changing the magnification of the captured image and changing the divided line of the captured image to a frame image Align with the dividing line. Thereby, a position corresponding to the outer shape of the screen 5 in the frame image can be specified. Next, the outline of the screen 5 in the frame image is compared with the outline of the person area in the frame image, and the area between the outline of the person area and the outline of the screen is extracted as a blank area. FIG. 7C shows a blank area extracted based on the screen outer shape shown in FIG. 7A and the person area shown in FIG. 7B. In FIG. 7C, a hatched portion is a blank area. After extracting the blank area, the control unit 10 proceeds to the process of step S303 in FIG.

  In step S303 of FIG. 8, a composition image in which a predetermined color is added to the extracted blank area is generated (step S303). The predetermined color can be preset by the user from the operation unit 11, for example. In addition to the color, a pattern or the like may be attached to the blank area.

  Then, the control unit 10 stores the generated composition image in the storage unit 13 in association with the frame image (the frame image acquired in step S301) (step S304). For example, an identification number or the like for identifying the frame image is written and stored in the header area of the composition image.

  Next, the control unit 10 determines whether or not the next frame image exists in the video data of the projection video content (step S305). When it is determined that the next frame image exists (step S305; YES), the control unit 10 returns to the process of step S301 and repeatedly executes the processes of step S301 to step S304 for the next frame image. When it is determined that the next frame image does not exist (step S305; NO), the control unit 10 proceeds to the process of step S4 in FIG.

  As the extraction of the blank area, the person area extracted from the frame image is projected on the screen 5, the screen 5 including the projected image is photographed by the camera 3, and the outer shape of the screen 5 and the person area are obtained from the obtained photographed image. And a blank area may be extracted based on the extracted outline of the screen 5 and the outline of the person area. In this case, the screen outline extraction process in step S2 of FIG. 2 is not necessary.

In step S4 of FIG. 4, the control unit 10 executes a projection control process (step S4).
FIG. 10 shows a flowchart of the projection control process. The projection control process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.

  First, the control unit 10 reads and acquires the frame image of the projection video content from the video data storage unit 132 of the storage unit 13 (step S401).

  Next, the control unit 10 acquires the setting information of the magnification and position movement amount of the person area when projecting the frame image of the projection video content from the setting information storage unit 133 of the storage unit 13, and the acquired setting Based on the information, a frame image in which the magnification and position of the person area are changed is created (step S402).

  Next, the control unit 10 extracts a person region from the created frame image (step S403). The extraction of the person area can be performed by a known image processing technique such as image processing such as image recognition.

  Next, the control unit 10 searches the storage unit 13 and determines whether or not there is a composition image corresponding to the acquired frame image (step S404). When it is determined that there is a composition image corresponding to the acquired frame image (step S404; YES), the control unit 10 combines the person region extracted in step S403 and the composition image associated therewith and projects it. The image is set (step S405), and the process proceeds to step S407.

  FIG. 11 shows an example of the projected image. As shown in FIG. 11, in the projected image, a predetermined color, pattern, or the like is synthesized in a blank area between the screen outer shape and the outline of the person area at the time of projection. Therefore, for example, by synthesizing, for example, a color or pattern of the same color as the background of the screen 5 in this blank area, a three-dimensional projection or the like in which a human image emerges from the surroundings is made possible, and the effect of production is improved. It becomes possible to make it.

  On the other hand, if it is determined that there is no composition image associated with the acquired frame image (step S404; NO), the person region extracted in step S403 is used as a projection image (step S406), and the process proceeds to step S407. To do.

  In step S <b> 407, the control unit 10 transmits the combined projection image and position information of the projection target area in the projection image (person area + margin area position information) to the projector 2 via the I / F 14. The projection target area in the projection image is projected on the screen 5. Also, the audio data corresponding to the original frame image is converted into an audio signal by the audio output unit 16 and output from the speaker 4 (step S407).

  Next, the control unit 10 determines whether or not the next frame image exists in the video data of the projection video content (step S408). When it is determined that the next frame image exists (step S408; YES), the control unit 10 returns to the process of step S401 and repeatedly executes the processes of step S401 to step S407 for the next frame image. When it is determined that the next frame image does not exist (step S408; NO), the control unit 10 ends the reproduction control process.

[Second Embodiment]
Next, a second embodiment will be described.
As shown in FIG. 12, the content reproduction system 200 in the second embodiment is provided with a camera 8 on the front surface of the screen 5 of the content reproduction system 100 shown in FIG. The camera 8 captures the screen 5 and its background, and is connected to the image processing apparatus 1. The configuration of the content reproduction system 200 and other devices constituting the second embodiment according to the second embodiment are the same as those described in the first embodiment, and the description is used. The operation will be described.

In the second embodiment, the composition image generation process executed in step S3 of FIG. 4 is different from that described in the first embodiment. Since the other processes are the same as those described in the first embodiment, the description is used. Hereinafter, the image generation process for synthesis (image generation for synthesis is executed in step S3 in FIG. 4 in the second embodiment). The process B) will be described.
The control unit 10 functions as a screen image acquisition unit, a screen outer shape extraction unit, a synthesis unit, a control unit, and a background image acquisition unit by executing a reproduction control process including the synthesis image generation process B.

FIG. 13 shows a flowchart of the composition image generation process B. The composition image generation process B is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.
First, the control unit 10 causes the camera 8 to capture the background of the screen 5 and acquires the captured image (step S311).
Next, the control unit 10 reads and acquires the frame image of the projection video content from the video data storage unit 132 of the storage unit 13 (step S312).

  Next, the control unit 10 performs a blank area extraction process (step S313). Since the blank area extraction process is the same as that described with reference to FIG. 9 in the first embodiment, the description is used.

  Next, the control unit 10 generates a synthesis image in which the background captured image acquired in step S311 is arranged in the extracted blank area (step S314). In the case of the configuration shown in FIG. 12, for example, the left and right sides of the captured image of the background are reversed, the outline of the screen 5 is extracted from the captured image of the background, and this is extracted from the extracted blank area corresponding to the screen outline of the frame image. The magnification of the photographed image is changed so as to match the outer shape, and an image for synthesis is generated by cutting out a part of the image corresponding to the blank area of the photographed image whose magnification has been changed.

  Then, the control unit 10 stores the generated composition image in the storage unit 13 in association with the frame image (the frame image acquired in Step S312) (Step S315). For example, an identification number or the like for identifying the frame image is written and stored in the header area of the composition image.

  Next, the control unit 10 determines whether or not the next frame image exists in the video data of the projection video content (step S316). If it is determined that the next frame image exists (step S316; YES), the control unit 10 returns to the process of step S311 and repeatedly executes the processes of step S311 to step S315 for the next frame image. When determining that the next frame image does not exist (step S316; NO), the control unit 10 proceeds to the process of step S4 in FIG. 4 and executes the projection control process in FIG. By executing the projection control process, the captured image of the background of the screen 5 is projected onto the blank area between the screen 5 and the person image.

  According to the second embodiment, since the background shot image is projected on the blank area between the screen 5 and the person image, natural projection is performed as if a person exists in front of the background. It becomes possible to improve the production effect.

[Third Embodiment]
Next, a third embodiment will be described.
Since the configuration of the content playback system and each of the devices constituting the content playback system according to the third embodiment is the same as that described in the first embodiment, the description will be used and the operation of the third embodiment will be described below. To do.

The operation of the third embodiment differs from that of the first embodiment in the composition image generation process executed in step S3 of FIG. Since the other processes are the same as those described in the first embodiment, the description is used. Hereinafter, the image generation process for synthesis (image generation for synthesis is executed in step S3 in FIG. 4 in the third embodiment). The process C) will be described.
The control unit 10 functions as a screen image acquisition unit, a screen outer shape extraction unit, a synthesis unit, a control unit, and a text acquisition unit by executing a reproduction control process including the synthesis image generation process C.

FIG. 14 shows a flowchart of the composition image generation process C. The composition image generation process C is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.
First, the control unit 10 acquires the sound of the projection video content (step S321). Specifically, the control unit 10 reads the audio data of the projection video content from the video data storage unit 132, converts the audio data into an audio signal by the audio output unit 16, and causes the speaker 4 to output the audio signal. Then, the voice waveform of the voice signal input by the microphone 7 is acquired.

  Next, the control unit 10 analyzes the acquired speech waveform, performs speech recognition, and creates a text corresponding to the speech (step S322).

Next, the control unit 10 detects a break of the created text, determines a display timing of the text for each detected break, and stores each frame image of the projection video content and the text in association with each other (step S323). ). For example, the control unit 10 determines a text break using a known morphological analysis technique or the like, and displays a display timing of a series of frame images displayed (projected) at a timing when a sound of the divided text is reproduced. It is determined as the display timing of the group of texts. For each text break, the text is stored in association with a frame image that is displayed at the same display timing as the text display timing. For example, the text is stored in association with an identification number for identifying the frame image.
For example, if the frame image displayed at the timing when the voice “I welcome” is played is an image with frame numbers 1 to 5, the display timing of the text “I welcome” is set to the frame image with frame numbers 1 to 5. Determine the display timing. Then, frame numbers 1 to 5 are stored in association with the text “I welcome you”.

  Next, the control unit 10 reads and acquires the frame image of the projection video content from the video data storage unit 132 of the storage unit 13 (step S324), and executes a blank area extraction process (step S325). Since the blank area extraction process is the same as that described with reference to FIG. 9 in the first embodiment, the description is used.

  Next, the control unit 10 arranges the text associated with the frame image in the extracted blank area and generates a composition image (step S326).

  Then, the control unit 10 stores the generated composition image in the storage unit 13 in association with the frame image (step S327). For example, an identification number or the like for identifying the frame image is written and stored in the header area of the composition image.

  Next, the control unit 10 determines whether or not the next frame image exists in the video data of the projection video content (step S328). If it is determined that the next frame image exists (step S328; YES), the control unit 10 returns to the process of step S324 and repeatedly executes the processes of step S324 to step S327 for the next frame image. When it is determined that the next frame image does not exist (step S328; NO), the control unit 10 proceeds to the process of step S4 in FIG. 4 and executes the projection control process in FIG. By executing the projection control process, the output voice text is displayed in a blank area between the person image projected when the voice is output and the screen 5.

  According to the third embodiment, since the output voice text is displayed in the blank area between the person image and the screen 5, the audience is projected even when the voice is difficult to hear. It is possible to easily recognize what the person image says.

[Fourth Embodiment]
Next, a fourth embodiment will be described.
Since the configuration of the content reproduction system and each device configuring the content reproduction system according to the fourth embodiment is the same as that described in the first embodiment, the description will be used and the operation of the fourth embodiment will be described below. To do.

The operation of the fourth embodiment is different from that of the first embodiment in the composition image generation process executed in step S3 of FIG. Since the other processes are the same as those described in the first embodiment, the description is used, and hereinafter, the image generation process for synthesis (image generation for synthesis is executed in step S3 in FIG. 4 in the fourth embodiment). Process D) will be described.
The control unit 10 functions as a screen image acquisition unit, a screen outer shape extraction unit, a synthesis unit, and a control unit by executing a reproduction control process including the synthesis image generation process D. The control unit 10 functions as a detection unit in cooperation with the microphone 7.

FIG. 15 shows a flowchart of the composition image generation process D. The composition image generation process D is executed by the cooperation of the CPU of the control unit 10 and the program stored in the program storage unit 131.
First, the control unit 10 acquires the sound of the projection video content (step S331). Specifically, the control unit 10 reads the audio data of the projection video content from the video data storage unit 132, converts the audio data into an audio signal by the audio output unit 16, and causes the speaker 4 to output the audio signal. Then, the voice waveform of the voice signal input by the microphone 7 is acquired.

  Next, the control unit 10 analyzes the acquired speech waveform and detects the strength of the speech (step S332).

Next, the control unit 10 determines whether the detected sound intensity is strong, medium, or weak based on whether or not the detected sound intensity value exceeds a predetermined threshold ( Step S333). When the sound intensity is high (step S333; strong), the control unit 10 determines a bright color as the color of the blank area at the time of sound output (step S334). When the sound intensity is medium (step S333; medium), the control unit 10 determines an intermediate color as the color of the blank area at the time of sound output (step S335). When the sound intensity is weak (step S333; weak), the control unit 10 determines a dark color as the color of the blank area at the time of sound output (step S336). Here, the color corresponding to each of the bright color, the intermediate color, and the dark color is determined in advance.
The control unit 10 repeatedly executes the processing of Steps S332 to S336 until the analysis and the determination of the color of the blank area for all the audio waveforms acquired in Step S332 are completed, and when the determination of the color of the blank area is completed ( Step S337; YES), the process proceeds to Step S338.

  In step S338, the control unit 10 determines the display timing of each color according to the audio reproduction timing, and stores the color in association with the frame image displayed at the same display timing as the determined display timing (step S338). S338), the process proceeds to step S339. For example, the identification number and color of the frame image are stored in association with each other.

  In step S339, the control unit 10 reads and acquires the frame image of the projection video content from the video data storage unit 132 of the storage unit 13 (step S339).

  Next, the control unit 10 performs a blank area extraction process (step S340). Since the blank area extraction process is the same as that described with reference to FIG. 9 in the first embodiment, the description is used.

  Next, the control unit 10 generates a composition image in which the extracted blank area is arranged with the color associated in step S338 (step S341). Then, the control unit 10 stores the generated composition image in the storage unit 13 in association with the frame image (step S342). For example, an identification number or the like for identifying the frame image is written and stored in the header area of the composition image.

  Next, the control unit 10 determines whether or not the next frame image exists in the video data of the projection video content (step S343). When it is determined that the next frame image exists (step S343; YES), the control unit 10 returns to the process of step S339 and repeatedly executes the processes of step S339 to step S342 for the next frame image. When it is determined that the next frame image does not exist (step S343; NO), the control unit 10 proceeds to the process of step S4 of FIG. 4 and executes the projection control process of FIG. By executing the projection control process, a color corresponding to the sound intensity is combined with the blank area of the frame image.

  According to the fourth embodiment, the blank area between the person image and the screen 5 is displayed in a color corresponding to the sound intensity. Accordingly, when the sound is emphasized in the projection video content, the surroundings of the person image become bright colors, so that the part to be emphasized can be impressed by the audience.

  In the fourth embodiment, the sound intensity of the projection video content is analyzed and the margin area is projected with a color corresponding to the sound intensity. However, the sound volume is analyzed and the sound volume is analyzed according to the sound volume. The margin area may be projected by color.

[Fifth Embodiment]
Next, a fifth embodiment will be described.
The configuration of the content reproduction system and each device constituting the same in the fifth embodiment is the same as that described in the first embodiment, so the description will be used and the operation of the fifth embodiment will be described below. To do.

The operation of the fifth embodiment is different from that of the first embodiment in the composition image generation process executed in step S3 of FIG. Since the other processes are the same as those described in the first embodiment, the description is used. Hereinafter, the image generation process for synthesis (image generation for synthesis is executed in step S3 in FIG. 4 in the fifth embodiment). Processing E) will be described.
The controller 10 functions as a screen image acquisition unit, a screen outer shape extraction unit, a synthesis unit, a control unit, and a determination unit by executing a reproduction control process including the synthesis image generation process E.

  FIG. 16 is a flowchart of the composition image generation process E. The composition image generation process E is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.

  First, the control unit 10 reads out and obtains a frame image (for example, a frame image of the first frame) of the projection video content from the video data storage unit 132 (step S351).

  Next, the control unit 10 determines whether the person included in the frame image is male or female (step S352). Specifically, a known face recognition technique is used to determine whether the person included in the frame image is male or female. For example, an average of a plurality of predetermined characteristics (for example, the length and width of the nose) is extracted from a plurality of male and female face images in advance. Then, the feature of the face of the person included in the frame image is analyzed, and it is determined whether the character is male or female depending on whether each feature is close to male or female.

If it is determined in step S352 that the person is male (step S352; male), the control unit 10 determines the design of the blank area as a male-like design (step S353), and the process proceeds to step S355.
On the other hand, if it is determined in step S352 that the woman is a woman (step S352; woman), the control unit 10 determines the design of the margin area as a feminine design (step S354), and proceeds to the process of step S355.
Images of a masculine design and a feminine design are stored in the storage unit 13 in advance.

  In step S355, the control unit 10 reads and acquires the frame image of the projection video content from the video data storage unit 132 of the storage unit 13 (step S355).

  Next, the control unit 10 performs a blank area extraction process (step S356). Since the blank area extraction process is the same as that described with reference to FIG. 9 in the first embodiment, the description is used.

  Next, the control unit 10 generates a synthesis image in which the determined design is arranged in the extracted blank area (step S357). Then, the control unit 10 stores the generated composition image in the storage unit 13 in association with the frame image (step S358). For example, an identification number or the like for identifying the frame image is written and stored in the header area of the composition image.

  Next, the control unit 10 determines whether or not the next frame image exists in the video data of the projection video content (step S359). When it is determined that the next frame image exists (step S359; YES), the control unit 10 returns to the process of step S355, and repeatedly executes the processes of step S355 to step S358 for the next frame image. When it is determined that the next frame image does not exist (step S359; NO), the control unit 10 proceeds to the process of step S4 in FIG. 4 and executes the projection control process in FIG. By executing the projection control process, a design corresponding to the person to be projected is projected between the person image and the screen 5.

  According to the fifth embodiment, it is determined whether the person to be projected is male or female, and a design corresponding to the determination result is displayed in a blank area between the person image and the screen 5. Accordingly, since a design suitable for the person is displayed around the person to be projected, a natural projection can be achieved.

[Sixth Embodiment]
Next, a sixth embodiment will be described.
The configuration of the content reproduction system and each device that configures the content reproduction system in the sixth embodiment is the same as that described in the first embodiment, so the description will be used and the operation of the sixth embodiment will be described below. To do.

FIG. 17 shows a flowchart of the reproduction control process (reproduction control process B) executed in the sixth embodiment. The reproduction control process B is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131.
By executing the reproduction control process B, the control unit 10 functions as a screen image acquisition unit, a screen outer shape extraction unit, a synthesis unit, a control unit, a number of people acquisition unit, and an audience image acquisition unit.

  First, the control unit 10 executes alignment processing (step S11). Since the alignment processing is the same as that described with reference to FIG.

  Next, the control unit 10 executes screen outer shape extraction processing (step S12). The screen outer shape extraction processing is the same as that described with reference to FIG.

  Next, the control unit 10 reads out and acquires the frame image (for example, the frame image of the first frame) of the video content for projection from the video data storage unit 132 (step S13).

  Next, the control unit 10 causes the camera 3 to photograph the audience to obtain a photographed image (step S14), and analyzes the obtained photographed image to obtain the number of audiences (step S15). For example, the number of people whose faces can be recognized in the captured image is acquired. In addition, it is possible to acquire the number of people who have stopped, that is, the number of people whose positions have not moved more than a predetermined threshold among the people whose faces have been recognized in the previous photographed image. Note that the audience may be photographed by a camera provided separately from the camera 3.

When the number of persons is 10 or more (step S16; 10 or more), the control unit 10 determines a color to be arranged in a blank area between the projected person image and the screen 5 as a bright color (step S17). The process proceeds to step S20. When the number of persons is 5 to 9 or more (step S16; 5 to 9 persons), the control unit 10 determines the color to be arranged in the blank area between the person image and the screen 5 as an intermediate color (step S18). ), The process proceeds to step S20. When the number of persons is 0 to 4 (step S16; 0 to 4 persons), the control unit 10 determines the color to be arranged in the blank area between the person image and the screen 5 as a dark color (step S19). The process proceeds to step S20.
Here, the color corresponding to each of the bright color, the intermediate color, and the dark color is determined in advance.

  In step S20, the control unit 10 performs a blank area extraction process (step S20). Since the blank area extraction process is the same as that described with reference to FIG. 9 in the first embodiment, the description is used.

  Next, the control unit 10 combines the image of the determined color with the extracted blank area and generates a projection image (step S21). Then, the control unit 10 transmits the combined projection image and position information of the projection target area in the projection image (person area + margin area position information) to the projector 2 via the I / F 14. The projection target area inside is projected onto the screen 5. Also, the audio data corresponding to the original frame image is converted into an audio signal by the audio output unit 16 and output from the speaker 4 (step S22).

  Next, the control unit 10 determines whether or not the next frame image exists in the video data of the projection video content (step S23). When it is determined that the next frame image exists (step S23; YES), the control unit 10 returns to the process of step S13 and repeatedly executes the processes of step S13 to step S22 for the next frame image. When it is determined that the next frame image does not exist (step S23; NO), the control unit 10 ends the reproduction control process B.

  According to the sixth embodiment, the color to be arranged in the blank area between the person image and the screen 5 is changed according to the number of the audience watching the projection video content. Specifically, bright colors are arranged when the number of people is large. Therefore, when a large number of audiences are gathered, the surroundings of the person to be projected are highlighted in a brighter color, thereby making it possible to increase the degree of attention and make the production more effective. Also, the projection video content provider can easily grasp the approximate number of audiences by simply looking at the projected images.

As described above, the control unit 10 of the image processing apparatus 1 extracts the human area that is the projection target from the video data of the video content, and the screen 5 extracted from the outline of the extracted human area and the captured image. In the video data, an image is synthesized with a blank area corresponding to the outline of the person area and the outline of the screen 5 when the video data is projected onto the screen 5. Then, based on the synthesized video data, the projector 2 projects the video content on which the image is synthesized between the outline of the person image and the outline of the screen 5 on the screen 5.
Therefore, since the synthesized image is projected on the blank area between the screen and the person image, it is possible to improve the effect by using the blank area between the screen and the person image.

  For example, by synthesizing an image of a color or pattern of the same color as the background in an area corresponding to the outline of the person area and the outline of the screen 5 in the video data, a three-dimensional image in which a person image is raised from the surroundings. Projection and the like can be performed, and the production effect can be improved.

  Further, by synthesizing the captured image of the background of the screen 5 in an area corresponding to the outline of the person area and the outline of the screen 5 in the video data, it is natural as if a person exists in front of the background. Projection can be performed, and the production effect can be improved.

  Further, for example, by synthesizing the audio text information of the video content in an area corresponding to the outline between the outline of the person area and the outline of the screen 5 in the video data, the audio text output at the time of projection becomes a human image. Since it is displayed in a margin area with the screen 5, the audience can easily recognize what the projected person image is saying even when it is difficult to hear the sound.

  Further, for example, by synthesizing an image of a color corresponding to the audio intensity of the video content (a brighter color as the intensity is higher) in an area corresponding to the outline of the person area in the video data and the outer shape of the screen 5, At the time of projection, when the sound is emphasized, the person image can be brightly colored, and the part to be emphasized can be impressed by the audience.

  In addition, for example, by synthesizing an image having a design according to the gender of the person image in an area corresponding to the outline of the person area in the video data and the outer shape of the screen 5, the person image is displayed around the person image at the time of projection. It is possible to display an image with a suitable design.

  Further, for example, a color image corresponding to the number of audiences is synthesized in an area corresponding to the outline of the person area in the video data and the outline of the screen 5 (specifically, a bright color is synthesized when the number of people is large). By doing so, it is possible to make the surroundings of the person to be projected stand out in a brighter color when many audiences are gathered, and it is possible to make the production effective. Also, the projection video content provider can easily grasp the approximate number of audiences by glanceping at the projected image.

  In addition, the description content in the said embodiment is a suitable example of the content reproduction | regeneration system which concerns on this invention, and is not limited to this.

  For example, in the above-described embodiment, the image processing apparatus 1 and the projector 2 have been described as separate bodies, but may be integrally configured. Further, the projector 2 and the camera 3 may be integrally configured.

  In the above-described embodiment, the configuration in which the output light of the projector 2 is directly applied to the back surface of the screen 5 has been described as an example. However, a mirror that reflects the output light of the projector 2 is provided, and the reflected light of the mirror is supplied to the projector 2. It is good also as projecting on the back surface.

  In the above embodiment, the video data of the projection video content has been described as being stored in the storage unit 13. However, as a configuration in which the video data is distributed from a server connected to the image processing apparatus 1 via a communication network. Also good. Further, the video data may be captured from a USB memory or various recording media.

  In the first to fifth embodiments described above, first, a synthesis image such as a color, text, and design to be arranged in the blank area is generated, and then the synthesis image is synthesized with each frame image. However, the determined color, text, design, etc. may be directly combined with each frame image.

  In the above-described embodiment, the control unit 10 controls the camera 3 and the camera 8 to perform shooting and acquires various shot images. However, the shot image shot in advance is stored in the storage unit 13. It is good also as memorize | storing and acquiring by reading from the memory | storage part 13 when the control part 10 is needed.

  Also, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory, a portable recording medium such as a CD-ROM, in addition to a ROM, a hard disk, etc. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

  In addition, the detailed configuration and detailed operation of each device constituting the content reproduction system can be changed as appropriate without departing from the spirit of the invention.

(Appendix)
Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
<Claim 1>
An image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer shape extraction means for extracting the outer shape of the screen from the acquired photographed image;
An area of the projection target is extracted from the video data of the video content, and the video data in the video data is extracted based on the extracted area of the projection target and the outer shape of the screen extracted from the captured image. A synthesizing unit that synthesizes an image in a region corresponding to the outline of the projection object and the outline of the screen when projected onto the screen;
Control means for projecting on the screen by the projection device video content in which an image is synthesized between the outline of the projection object and the outline of the screen based on the synthesized video data;
An image processing apparatus comprising:
<Claim 2>
The image processing apparatus according to claim 1, wherein the synthesizing unit synthesizes an image of a predetermined color or pattern in an area corresponding to an outline between the projection object and the outline of the screen in the video data.
<Claim 3>
A background image acquisition means for acquiring a captured image of the background of the screen;
The image processing apparatus according to claim 1, wherein the synthesizing unit synthesizes a captured image of the background of the screen in an area corresponding to the outline of the projection object and the outline of the screen in the video data.
<Claim 4>
Audio output means for outputting audio of the video content;
A text acquisition means for analyzing the audio of the video content and acquiring audio text information;
The image processing apparatus according to claim 1, wherein the synthesizing unit synthesizes audio text information of the video content in an area corresponding to an outline between the projection object and the screen in the video data.
<Claim 5>
Audio output means for outputting audio of the video content;
Detecting means for analyzing the sound of the video content and detecting the sound intensity;
The said synthetic | combination means synthesize | combines the image of the color corresponding to the audio | voice intensity | strength of the said video content to the area | region corresponded between the outline of the said projection target object in the said video data, and the external shape of the said screen. Image processing device.
<Claim 6>
Determining means for determining the gender of the projection object based on video data of the video content;
2. The composition unit according to claim 1, wherein the composition unit synthesizes an image having a design corresponding to the gender determined by the determination unit, in a region corresponding to the outline of the projection object and the outline of the screen in the video data. Image processing apparatus.
<Claim 7>
Comprising number-of-people obtaining means for obtaining the number of audiences of the video content projected on the screen;
2. The composition means for compositing an image of a color corresponding to the number of persons acquired by the number of persons acquisition means in an area corresponding to the outline of the projection object and the outline of the screen in the video data. The image processing apparatus described.
<Claim 8>
Audience image acquisition means for acquiring a captured image of the audience of the video content,
The image processing apparatus according to claim 7, wherein the number-of-persons acquisition unit analyzes a photographed image of an audience of the video content and acquires the number of audiences.
<Claim 9>
An image processing method in an image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Obtaining a captured image including the screen;
Extracting the outer shape of the screen from the acquired captured image;
An area of the projection target is extracted from the video data of the video content, and the video data in the video data is extracted based on the extracted area of the projection target and the outer shape of the screen extracted from the captured image. Synthesizing an image in a region corresponding to the outline of the projection object and the outline of the screen when projected onto the screen;
A step of projecting on the screen by the projection device video content in which an image is synthesized between the outline of the projection object and the outline of the screen based on the synthesized video data;
An image processing method including:
<Claim 10>
A computer used in an image processing apparatus connected to a projection apparatus that projects video content including the projection object onto a screen molded into the shape of the projection object;
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer shape extracting means for extracting the outer shape of the screen from the acquired photographed image;
An area of the projection target is extracted from the video data of the video content, and the video data in the video data is extracted based on the extracted area of the projection target and the outer shape of the screen extracted from the captured image. A synthesizing unit that synthesizes an image in a region corresponding to an outline between the outline of the projection object and the outline of the screen when projected onto the screen;
Control means for projecting video content, in which an image is synthesized between the outline of the projection object and the outline of the screen, on the screen by the projection device based on the synthesized video data;
Program to function as.

DESCRIPTION OF SYMBOLS 100 Content reproduction system 200 Content reproduction system 1 Image processing apparatus 10 Control part 11 Operation part 12 Display part 13 Storage part 131 Program storage part 132 Video data storage part 133 Setting information storage part 14 I / F
15 I / F
16 Audio output unit 17 I / F
2 Projector 3 Camera 4 Speaker 5 Screen 51 Base Material 52 Screen Film 53 Film 6 Support Stand 7 Microphone

Claims (7)

An image processing apparatus connected to a projection apparatus that projects video content including a person onto a screen molded into the shape of the person ,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer shape extraction means for extracting the outer shape of the screen from the acquired photographed image;
Discrimination means for discriminating the gender of the person based on video data of the video content;
The person area is extracted from the video data of the video content, and the video data in the video data is projected onto the screen based on the extracted person area and the outer shape of the screen extracted from the captured image. A synthesizing unit that synthesizes an image of a design according to the gender determined by the determining unit in a region corresponding to the outline of the person and the outer shape of the screen when
On the basis of the synthesized image data, and control means for projecting onto the screen by the projection apparatus a video content which the image is synthesized during the contour and contour of the screen of the person,
An image processing apparatus comprising: An image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer shape extraction means for extracting the outer shape of the screen from the acquired photographed image;
Number-of-people acquisition means for acquiring the number of audiences of the video content projected on the screen;
An area of the projection target is extracted from the video data of the video content, and the video data in the video data is extracted based on the extracted area of the projection target and the outer shape of the screen extracted from the captured image. A synthesizing unit that synthesizes an image of a color corresponding to the number of people acquired by the number of people acquiring unit in a region corresponding to an outline between the outline of the projection object and the outer shape of the screen when projected onto the screen;
Control means for projecting video content, in which the image is synthesized between the outline of the projection object and the outline of the screen, on the screen by the projection device based on the synthesized video data;
An image processing apparatus comprising: Audience image acquisition means for acquiring a captured image of the audience of the video content,
The image processing apparatus according to claim 2 , wherein the number-of-persons acquiring unit acquires a number of the audiences by analyzing a photographed image of the audience of the video content. An image processing method in an image processing apparatus connected to a projection apparatus that projects video content including a person onto a screen molded into the shape of the person ,
Obtaining a captured image including the screen;
Extracting the outer shape of the screen from the acquired captured image;
Determining the gender of the person based on video data of the video content;
The person area is extracted from the video data of the video content, and the video data in the video data is projected onto the screen based on the extracted person area and the outer shape of the screen extracted from the captured image. A step of synthesizing an image of the design according to the determined gender in a region corresponding to the outline of the person and the outline of the screen when
On the basis of the synthesized image data, a step of projecting a video content which the image is synthesized during the outer contour and the screen of the person on the screen by the projection device,
An image processing method including: An image processing method in an image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Obtaining a captured image including the screen;
Extracting the outer shape of the screen from the acquired captured image;
Obtaining the number of audiences of the video content projected on the screen;
An area of the projection target is extracted from the video data of the video content, and the video data in the video data is extracted based on the extracted area of the projection target and the outer shape of the screen extracted from the captured image. Synthesizing an image of a color corresponding to the acquired number of people in a region corresponding to the outline of the projection object and the outline of the screen when projected onto the screen;
Projecting on the screen by the projection device video content in which the image is synthesized between the outline of the projection object and the outline of the screen based on the synthesized video data;
An image processing method including: A computer used in an image processing apparatus connected to a projection apparatus that projects video content including a person onto a screen molded into the shape of the person ,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer shape extracting means for extracting the outer shape of the screen from the acquired photographed image;
Discrimination means for discriminating the gender of the person based on video data of the video content;
The person area is extracted from the video data of the video content, and the video data in the video data is projected onto the screen based on the extracted person area and the outer shape of the screen extracted from the captured image. Combining means for combining an image of a design according to the gender determined by the determining means in a region corresponding to the outline of the person and the outer shape of the screen when
On the basis of the synthesized image data, control means for projecting onto the screen the video content which the image is synthesized by the projection apparatus during the outer contour and the screen of the person,
Program to function as. A computer used in an image processing apparatus connected to a projection apparatus that projects video content including the projection object onto a screen molded into the shape of the projection object;
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer shape extracting means for extracting the outer shape of the screen from the acquired photographed image;
Number acquisition means for acquiring the number of audiences of the video content projected on the screen;
An area of the projection target is extracted from the video data of the video content, and the video data in the video data is extracted based on the extracted area of the projection target and the outer shape of the screen extracted from the captured image. A synthesizing unit that synthesizes an image of a color corresponding to the number of people acquired by the number of people acquiring unit in a region corresponding to an outline between the outline of the projection object and the outer shape of the screen when projected onto the screen;
Control means for projecting video content, in which the image is synthesized between the outline of the projection object and the outline of the screen, on the screen by the projection device based on the synthesized video data;
Program to function as.