JP6079135B2 - Image processing apparatus, video data correction method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a video data correction method, and a program.

  In order to enhance the impression when presenting advertisements and other contents to customers and viewers, there is a technology for projecting content images, etc., in a shape that matches the external shape of the screen onto a projection screen that is shaped like a human figure It has been proposed (see, for example, Patent Document 1). This technology is used for the production of novel and effective exhibitions at exhibitions, etc., or for explanation of products for human beings such as “virtual mannequins”.

  As the above-mentioned screen, a film for rear projection, for example, a special film having a black light-absorbing layer in the gap between spherical beads and a layer on which microbead lenses are arranged on the back side of a transparent vinyl chloride film projected, Affixed to a base material such as a transparent acrylic plate that is molded into a mold, it is often used as a projection image with a presence as if a human is standing on it from a wide angle, relatively bright and bright. Is obtained.

JP 2011-150221 A

However, in the above technique, effective projection cannot be performed unless the outline of the projection image and the outline of the screen are closely matched. For this reason, there is a problem that it takes time and labor to adjust the positional relationship between the projector or its projected image and the screen.
For example, if the projected image protrudes outside the outer shape of the screen, the face, shoulders, etc. of that portion may be missing, thinned, or become unnatural. In this way, if the projected image and the screen outline do not match, the power, reality, and three-dimensional presence of the “virtual mannequin” will be impaired, and the effects of exhibition and production will be halved. .

  In addition, there is a method of providing a dedicated stand that can fix the positional relationship, angle, distance, etc. between the projector and the mirror or screen in advance, but installation and adjustment are easy, but the required area and installation location are limited. It is difficult to set up ad hoc freely. The dedicated stand is slightly larger in shape and costs more. In addition, when a person's video is shot and used as projection video content, it is necessary to shoot so that the outline of the person image does not move or change. In other words, the person who becomes the subject model will be forced to perform the demonstration while maintaining almost the same posture so that it will not move, and it is necessary to prepare and photograph the model, photographer, studio etc. It was.

  Further, the screen is cut out of a thick acrylic plate or the like according to the outline of the projection object. However, if a screen having a different outer shape is recreated for each content to be projected, there is a problem that it is expensive. If you want to use the screen for other content, the common size and outline are preconditions, so you can use the same posture for demonstrations and videos that hold down gestures and body movements. There was also a problem that was limited and lacked originality and freshness.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to appropriately project even if the projection object moves when projecting onto a screen having a shape corresponding to the projection object. Is to be able to do it.

In order to solve the above-mentioned problem, the invention described in claim 1
An image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer area extraction means for extracting the outer area of the screen from the acquired photographed image;
Target area extracting means for extracting the area of the projection target from the video data of the video content;
Movement information acquisition means for acquiring information indicating movement of the projection object in the video content based on the area of the projection object extracted by the target area extraction means;
The outline area of the screen extracted by the screen outline area extraction unit, the area of the projection object extracted by the target area extraction unit, and the movement of the projection object acquired by the motion information acquisition unit Correction means for correcting the position or magnification of the video data of the video content based on the information;
Control means for projecting the projection object onto the screen by the projection device based on the corrected video data;
Is provided.

  ADVANTAGE OF THE INVENTION According to this invention, when projecting on the screen of the shape according to a projection target object, it becomes possible to project appropriately even if a projection target moves.

It is a figure which shows the example of whole structure of the content reproduction system in this embodiment. (A) is a front view of the content reproduction system before video projection, and (b) is a front view of the content reproduction system at the time of video projection. It is a block diagram which shows the functional structure of the image processing apparatus of FIG. It is a flowchart which shows the reproduction | regeneration control processing performed in the control part of FIG. It is a flowchart which shows the initial position alignment process of FIG. It is a flowchart which shows the reproduction | regeneration and correction process of FIG. FIG. 5 is a diagram schematically showing a concept of a person area correction operation in the reproduction control process of FIG. 4. It is a flowchart which shows the reproduction | regeneration control process B performed by the control part of FIG. 3 in 2nd Embodiment. It is a figure for demonstrating the comparison method of the screen external region of a picked-up image, and the largest area | region of a frame image.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The present invention is not limited to the illustrated example.

[First Embodiment]
<Configuration of Content Playback System 100>
1 and 2 are diagrams showing a schematic configuration of a content reproduction system 100 according to the present embodiment. FIG. 1 is a schematic side view of the content reproduction system 100, and FIG. 2 is a front view of the content reproduction system 100. FIG. 2A is a front view before image projection, and FIG. 2B is a front view during image projection.

  The content reproduction system 100 is a system that reproduces the projection video content and projects the projection target onto a screen molded into the shape of the projection target. In the present embodiment, the case where the projection target is a person and the screen is a human type will be described as an example. However, the present invention is not limited to this. The projection object may be, for example, a character or an animal.

  As shown in FIG. 1, a content reproduction system 100 includes an image processing device 1, a projector 2, a camera 3, a speaker 4, a screen 5, and the like. The projector 2, the camera 3, and the speaker 4 are connected to the image processing apparatus 1.

  The image processing apparatus 1 uses the person in the video data of the projection video content so that the person area projected on the projector 2 fits in the outline area of the screen 5 according to the movement of the person who is the projection target of the projection video content. This is a device that corrects the magnification and position of an area, transmits the corrected video data to the projector 2, and projects it onto the screen 5. The outline area of the screen 5 refers to an area surrounded by the outline (outline) of the screen 5. As the image processing apparatus 1, for example, a PC (Personal Computer), a terminal device, or the like is applicable.

  FIG. 3 shows a functional configuration example of the image processing apparatus 1. As shown in FIG. 3, the image processing apparatus 1 includes a control unit 10, an operation unit 11, a display unit 12, a storage unit 13, an I / F 14, an I / F 15, an audio output unit 16, and the like.

Although not shown, the control unit 10 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The CPU of the control unit 10 reads the program stored in the program storage unit 131 of the storage unit 13, expands it in the work area of the RAM, and executes various processes including a reproduction control process described later according to the expanded program. To do.
The control unit 10 functions as screen image acquisition means, screen outline area extraction means, target area extraction means, second target area extraction means, correction means, motion information acquisition means, and control means by executing reproduction control processing. To do.

  The operation unit 11 has various function keys, cursor keys, and the like, receives key input by the user, and outputs the operation information to the control unit 10.

  The display unit 12 is configured by an LCD (Liquid Crystal Display) or the like, and performs various displays according to a display control signal from the control unit 10.

  The storage unit 13 includes an HDD (Hard Disk Drive), a semiconductor nonvolatile memory, or the like. As shown in FIG. 3, the storage unit 13 includes a program storage unit 131, a video data storage unit 132, and a setting information storage unit 133.

  The program storage unit 131 stores a system program executed by the control unit 10 and various processing programs, data necessary for executing these programs, and the like. The program is stored in the program storage unit 131 in the form of computer readable program code. The control unit 10 sequentially executes operations according to the program code.

  The video data storage unit 132 stores video data of projection video content. The video data is composed of a plurality of frame image data constituting the moving image data and audio data corresponding to each frame image.

  The setting information storage unit 133 stores setting information indicating the magnification of the person area and the amount of movement of the position when the video data (frame image) of the video content for projection is projected on the screen 5. For example, the setting information of the magnification is obtained by extracting a person area from an arbitrary frame image (for example, the first frame) of the projection video content by known image processing such as image recognition, and the size of the extracted person area, projector Based on the enlargement ratio of 2, the size of the screen 5, etc., it is calculated and registered in advance. As the setting information of the amount of movement of the position, for example, a person area is extracted from the frame image of the video content for projection by image processing, the center of gravity position of the person area is obtained, and the obtained center of gravity position matches the center of the image. The movement amount and the movement direction when moved are registered in advance.

The I / F 14 is an interface for connecting to the projector 2 and performing data transmission / reception.
The I / F 15 is an interface for connecting to the camera 3 and performing data transmission / reception.

  The audio output unit 16 is connected to the speaker 4, converts audio data instructed by the control unit 10 into an audio signal, and outputs the audio signal from the speaker 4.

  The projector 2 is a rear projection type projection device that projects an image from the back surface of the screen 5 based on the frame image transmitted from the image processing device 1. For example, the projector 2 individually turns on / off each tilt angle of a plurality of micromirrors arranged in an array (in the case of XGA, horizontal 1024 pixels × vertical 768 pixels) based on a frame image at a high speed. Thus, a DLP (Digital Light Processing) (registered trademark) projector using a DMD (digital micromirror device) which is a display element that forms a light image by reflected light by performing a display operation can be applied. When the projection target is a person area in the frame image, it is possible to project only the person area by turning on the minute mirror at the position corresponding to the person area and turning off the other areas.

  The camera 3 is installed at a position where the outer area of the screen 5 falls within the field of view, and the screen 5 is photographed in accordance with an instruction from the image processing apparatus 1, and the obtained photographed image is transferred to the image processing apparatus 1 via the I / F 15. Send.

  The screen 5 is a human-type screen supported by the support base 6 and arranged in the output light irradiation direction of the projector 2. As shown in FIG. 1, the screen 5 has a screen film 52 for rear projection attached to the upper front side of a base 51 such as a transparent acrylic plate molded into a human shape to be projected, and a person to be projected on the lower side. A film 53 on which the lower half of the body is printed is attached. By using a commercially available screen film having high brightness and high contrast, the content projected on the screen 5 can be visually recognized even in the daytime or in a bright room. can do.

  In the present embodiment, the projection video content in which the upper body of a person is photographed is projected onto the screen 5 as an example. However, a screen film 52 for rear projection is attached to the entire surface of the base 51 of the screen 5. The projection video content in which the whole body of the person is photographed may be projected onto the screen 5.

<Operation of Image Processing Apparatus 1>
Next, the operation in this embodiment will be described.
In the configuration of the content reproduction system 100 shown in FIG. 1, the frame image of the projection video content and the captured image of the projection image projected on the screen 5 are reversed from the left and right of the front image. In the figure to be used, left-right reversal is omitted, and all are shown as front images.

  FIG. 4 shows a flowchart of the reproduction control process executed by the image processing apparatus 1. The reproduction control process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131 of the storage unit 13.

  First, the control unit 10 sets a correction threshold according to an input from the operation unit 11 (step S1). The correction threshold is a value used when determining whether or not the movement of the person area in the projection video content is equal to or greater than the threshold in the reproduction and correction processing described later. For example, the position of the person area of the frame image used for projection is shifted by% or more (how many pixels or more) to the left, right, top, or bottom with respect to the standard position (set in the initial alignment process described later). If so, the value indicates whether correction is performed. The correction threshold value input from the operation unit 11 is stored in the program storage unit 131 by the control unit 10.

Next, the control unit 10 executes an initial alignment process (step S2).
FIG. 5 shows a flowchart of the initial alignment process executed by the image processing apparatus 1. The initial alignment process is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131 of the storage unit 13.

In the initial alignment process, first, the control unit 10 reads and acquires the frame image (for example, the first frame) of the projection video content from the video data storage unit 132 of the storage unit 13 (step S201).
Next, the control unit 10 acquires setting information from the setting information storage unit 133 of the storage unit 13 and creates a frame image in which the magnification and position of the person area are changed based on the acquired setting information (step S202). .

  Next, the control unit 10 extracts a person area from the created frame image, transmits the position information of the extracted person area and the created frame image to the projector 2 via the I / F 14, and the projector 2 transmits the frame image. The person area inside is projected on the screen 5 (step S203). The person region can be extracted by a known image processing technique such as image recognition, for example.

Next, the control unit 10 causes the camera 3 to capture a projected image including the screen 5 being projected, acquires a captured image (step S204), and extracts an outer region of the screen 5 from the acquired captured image (step S205). .
In step S205, the outline region of the screen 5 is extracted by performing, for example, blurring processing and binarization processing on the captured image, but is not limited thereto, and is extracted using other known image processing techniques. May be.

  Next, the control unit 10 extracts a person area of the projected image from the captured image (step S206). In step S206, for example, the person area of the projected image is extracted by performing binarization processing on the captured image using a threshold value of brightness (brighter) higher than that in step S205. The image may be extracted using a known image processing technique.

  Next, the control unit 10 compares the outline area of the screen 5 extracted from the captured image with the person area, and aligns the two (step S207). For example, the center of gravity of the outline area of the screen 5 and the center of gravity of the person area are obtained, and the person area is moved so that they match, and then the person area has a predetermined size with respect to the outline area of the screen 5 (for example, , The magnification is changed to about 0.9 times.

  Next, the control unit 10 calculates the magnification and the moving amount (and moving direction) of the frame image at the time of projection based on the magnification and moving amount (and moving direction) of the person area in the alignment in step S207, The setting information in the setting information storage unit 133 is updated with the calculation result. Further, based on the updated setting information, a frame image (frame image after alignment) in which the magnification and position of the frame image acquired in step S201 are changed is created (step S208).

Next, the control unit 10 extracts a person region from the frame image after alignment (step S209). The extraction of the person area can be performed by a known image processing technique such as image processing such as image recognition.
And the control part 10 calculates | requires a gravity center from the extracted person area | region, memorize | stores the coordinate of the calculated | required gravity center position in RAM as a standard position of a person area (step S210), and a process transfers to step S3 of FIG.

In step S3 of FIG. 4, reproduction and correction processes are executed.
FIG. 6 shows a flowchart of the reproduction and correction process executed by the image processing apparatus 1. The reproduction and correction processing is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131 of the storage unit 13.

In the reproduction and correction processing, the control unit 10 reads out and acquires the projection target frame image of the video content for projection from the video data storage unit 132 of the storage unit 13 (step S301).
Next, the control unit 10 acquires the setting information of the magnification and the moving amount of the position from the setting information storage unit 133 of the storage unit 13, and changes the magnification and the position of the person area of the frame image based on the acquired setting information. A frame image is created (step S302).

  Next, the control unit 10 extracts a person region from the created frame image (step S303). The extraction of the person area can be performed by a known image processing technique such as image processing such as image recognition.

  Next, the control unit 10 obtains the centroid position of the extracted person area, compares the centroid position with the standard position stored in the RAM, and moves the centroid position of the person area from the standard position (and the movement direction). Is acquired as information indicating the movement of the person area (step S304). Then, based on whether or not the movement amount of the center of gravity position of the person area from the standard position is equal to or greater than a preset correction threshold value, it is determined whether or not the movement of the person area is equal to or greater than the threshold value (step S305). .

  If it is determined in step S305 that the movement of the person area is not equal to or greater than the threshold (step S305; NO), the control unit 10 proceeds to the process of step S307. If it is determined that the movement of the person area is equal to or greater than the threshold (step S305; YES), the control unit 10 corrects the frame image so that the center of gravity of the extracted person area becomes the standard position (step S306). The process proceeds to S307.

  In step S307, the control unit 10 extracts a person area from the frame image (corrected frame image; if not corrected, the frame image created in step S302), and extracts the position information of the extracted person area and the frame. The image is transmitted to the projector 2 via the I / F 14, and the projector 2 causes the person area in the frame image to be projected on the screen 5. Corresponding audio data is converted into an audio signal by the audio output unit 16 and output from the speaker 4 (step S307).

  Next, the control unit 10 determines whether or not there is a next frame image in the projection video content (step S308). If there is a next frame image (step S308; YES), the control unit 10 returns to the process of step S301, and executes the processes of step S301 to step S307 for the next frame image. When it is determined that there is no next frame image (step S308; NO), the control unit 10 ends the reproduction control process.

  FIG. 7 schematically shows the concept of the human region correction operation in the reproduction control process. FIG. 7A is a diagram illustrating an example of a captured image obtained by capturing the screen 5 and the projected image when the projection video content is projected onto the screen 5. When the photographed image of the projected image is as shown in FIG. 7A, the contour area of the screen 5 of the photographed image and the person area that is the projected image are extracted, the center of gravity position of the person area is moved, and the contour of the screen 5 is extracted. Along with the position of the center of gravity of the area, the magnification of the person area is changed so that the person area has a predetermined magnification that fits on the screen 5. Based on the moving amount (and moving direction) at this time and the changed magnification, a frame image in which the magnification and moving amount of the original frame image are changed is created as shown in FIG. When a video based on this frame image is projected onto the screen 5, a person region is appropriately projected so as to be within the screen 5 as shown in FIG. In addition, the center of gravity position of the person area in the frame image at this time is stored as a standard position, and when the person area of the projection video content to be projected has a motion, the position of the center of gravity of the person area of each frame image is the standard position. By correcting the frame image in such a manner, even if there is a movement in the person area, it is possible to project the person image onto an appropriate position in the screen 5 without protruding or missing a part of the person image.

  In the first embodiment, the person area is extracted from the frame image after the alignment in step S209 of the initial alignment process, and the center of gravity position of the extracted person area is stored as the standard position. A face area may be extracted from the frame image using a known face recognition technique, and the center of gravity position of the face area may be stored as a standard position. In step S303 of the reproduction and correction process, a face area is extracted from the created frame image. In step S304, the center of gravity position of the extracted face area is compared with the standard position. In step S305, the standard position is extracted. Whether the movement of the person area is equal to or greater than the threshold may be determined based on whether the amount of movement of the center of gravity of the face area from the threshold is equal to or greater than a preset correction threshold. In this case, the correction threshold value set in step S1 of the reproduction control process is such that the position of the face area is shifted by% or more (how many pixels or more) to the left, right, top, or bottom with respect to the standard position. Is a value indicating whether correction is to be performed.

  In addition, although it has been described that the position of the center of gravity of the person area is used as a standard for calculating the standard position and the movement amount, the present invention is not limited to this. For example, the center of the person area (the diagonal line of the rectangular area circumscribing the person area) It is also possible to use a crossing position).

[Second Embodiment]
Next, a second embodiment will be described.
In the first embodiment, an example has been described in which, when each frame image of the projection video content is projected, the movement of the person area in the projected frame image is corrected and projected. In the second embodiment, an example will be described in which the projection video content is projected after the motion of all the frame images is corrected.

  In the second embodiment, the program storage unit 131 of the storage unit 13 stores a program for executing a reproduction control process B described later. Since the configurations of the content reproduction system 100 and other devices constituting the content reproduction system 100 according to the second embodiment are the same as those described in the first embodiment, the description thereof is incorporated, and the second embodiment is hereinafter described. The operation will be described.

  FIG. 8 shows a flowchart of the reproduction control process B executed by the image processing apparatus 1 in the second embodiment. The reproduction control process B is executed in cooperation with the CPU of the control unit 10 and the program stored in the program storage unit 131 of the storage unit 13.

  First, the control unit 10 performs an initial alignment process (step S10). Specifically, the control unit 10 performs the processes of steps S201 to S207 of the initial alignment process of FIG. 5, and performs projection based on the magnification and movement amount (and movement direction) of the person area in the alignment of step S207. The magnification and position movement amount (and movement direction) of the person area at the time are calculated, and the setting information in the setting information storage unit 133 is updated with the calculation result.

  Next, the control unit 10 divides an image having the same size as the frame image of the projection video content into areas of n × n (n is a positive integer), and the image including the dividing line is projected via the I / F 14 to the projector. 2 to project the entire image (step S11).

  Next, the control unit 10 causes the camera 3 to capture a projected image including the screen 5 being projected, acquires a captured image (step S12), and extracts an outline area of the screen 5 from the acquired captured image (step S13). . The extraction of the outer region of the screen 5 can be performed by the same process as described in step S205 of FIG. FIG. 9A shows a schematic diagram of the captured image acquired in step S12.

  Next, the control unit 10 acquires setting information on the magnification and position movement amount of the person area when projecting the frame image of the projection video content from the setting information storage unit 133 of the storage unit 13 (step S14). Based on the acquired setting information, a frame image in which the magnification and position of the person area are changed is created (step S15).

  Next, the control unit 10 extracts a person region from the created frame image (step S16). The extraction of the person area can be performed by a known image processing technique such as image processing such as image recognition.

  Next, the control unit 10 determines whether or not the maximum area is set in the RAM (step S17). Here, the maximum area is the maximum person area when a plurality of frame images constituting the projection video content are superimposed. In the RAM, an area obtained by ORing the person areas of a plurality of frame images from the first frame image to the immediately preceding frame image (referred to as an OR area) is sequentially set as the maximum area. In the case of the first frame image, since the OR region has not yet been calculated, the maximum region is not set.

  If it is determined that the maximum area is not set (step S17; NO), the control unit 10 sets the extracted person area as the maximum area in the RAM (step S18), and proceeds to the process of step S21.

  If it is determined that the maximum area is set (step S17; YES), the control unit 10 compares the extracted person area with the maximum area, and determines whether or not the person area has a portion protruding from the maximum area. Judgment is made (step S19). Here, in the present embodiment, the extracted person area is compared with the maximum area, and the movement of the person area is detected depending on whether or not there is a protruding part. If it is determined that there is a portion protruding from the maximum area (step S19; YES), the control unit 10 adds the protruding portion to the maximum area and updates the maximum area of the RAM (step S20). That is, the OR area between the maximum area so far and the extracted person area is set as the maximum area.

Next, the control unit 10 determines whether there is a next frame image in the projection video content (step S21). When there is a next frame image (step S21; YES), the control unit 10 returns to the process of step S14 and repeatedly executes the processes of step S14 to step S20. When determining that there is no next frame image (step S21; NO), the control unit 10 proceeds to step S22.
The maximum area obtained by superimposing the person areas of all the frame images is information indicating the range of movement of the person area in the projection video content.

In step S22, the control unit 10 compares the outline area of the screen 5 extracted in step S13 with the maximum area, and determines the magnification of the person area (step S22). For example, the control unit 10 divides the image in which the maximum area is formed into n × n, and adds a dividing line at that time to the image. FIG. 9B schematically shows an image of the maximum area to which the dividing line is added. The maximum area is an area indicated by a dotted line. Then, the screen 5 and the maximum area are compared by taking the correspondence between the captured image and the frame image by using the dividing line of the captured image captured in step S12 and the dividing line of the image of the maximum area (see FIG. 9C). The maximum magnification at which the maximum area does not protrude from the screen 5 is determined (see FIG. 9D), and the magnification of the setting information is updated with the determined magnification.
In this way, by updating the magnification of the setting information so that the maximum area does not protrude from the outer area of the screen 5, a frame image with the magnification corrected in the next step S23 can be created.

  Next, the control unit 10 causes the projector 2 to sequentially project the person area of each frame image of the projection video content based on the updated magnification and position setting information, and the corresponding audio data is output to the audio output unit 16. The signal is converted into a signal, and the sound is output from the speaker 4 (step S23). Specifically, a frame image in which the magnification and position of each frame image is changed (corrected) based on the setting information is created, a person area is extracted from the created frame image, and the position information and the created person area are created. The frame images are sequentially transmitted to the projector 2 via the I / F 14, and the projector 2 projects the person area in the frame image onto the screen 5. Corresponding audio data is converted into an audio signal by the audio output unit 16 and output from the speaker 4.

  As described above, in the second embodiment, since the frame image is corrected so that the maximum area when the frame images are superimposed is within the outline area of the screen 5, even if there is a movement in the person area, A part of the image can be projected to an appropriate position in the screen 5 without protruding or missing.

As described above, according to the content reproduction system 100, the control unit 10 extracts the outer area of the screen 5 from the captured image including the human-shaped screen 5, and the person to be projected from the frame image of the projection video content. The area is extracted, and the frame image of the projection video content is corrected based on the extracted outline area and person area of the screen 5. A person is projected on the screen 5 by the projector 2 based on the corrected frame image.
Therefore, by correcting the frame image so that the person area fits on the screen 5, even if a person in the projection video content moves, the person in the screen 5 is not adjusted without adjusting the positional relationship between the projector 2 and the screen 5. It is possible to perform projection appropriately so that the image fits.

  For example, based on the person area extracted from the plurality of frame images of the video content for projection, information indicating the movement of the person area in the video content is acquired, and extracted from the acquired information indicating the movement of the person and the captured image By correcting each frame image of the video content based on the outline area of the screen, it is possible to appropriately perform projection so that the person image fits within the screen 5 even if a person in the projection video content moves. .

  Specifically, the maximum area when a plurality of person areas extracted from a plurality of frame images are superimposed is acquired as information indicating the range of movement of the person area in the projection video content, and extracted from the maximum area and the photographed image. By correcting the frame image of the video content so that the maximum area can be accommodated on the screen 5 based on the outer shape area of the screen 5, the person image can be accommodated in the screen 5 even if a person in the projection video content moves. Thus, it becomes possible to perform projection appropriately.

  Further, for example, based on the outline area of the screen 5 extracted from the captured image and the person area extracted from each frame image of the projection video content, information indicating the movement of the person area in the projection video content is acquired. By correcting each frame image based on the acquired information indicating the movement of the person area, it is possible to appropriately perform projection so that the person image fits within the screen 5 even if the person in the projection video content moves. It becomes possible.

  Specifically, a person area is extracted together with an outline area of the screen 5 from a photographed image including the screen 5 on which the person image is projected, and a plurality of pieces of projection video content are based on the extracted outline area and person area of the screen. The standard position of the person area in the frame image is determined. Then, the movement amount from the standard position of the person area in each of the plurality of frame images is acquired as information indicating the movement of the person area, and each frame image is corrected based on the acquired information indicating the movement of the person area. As a result, even if a person in the projection video content moves, it is possible to appropriately perform projection so that the person image fits within the screen 5.

  In addition, the description content in the said embodiment is a suitable example of the content reproduction | regeneration system which concerns on this invention, and is not limited to this.

  For example, in the above-described embodiment, the image processing apparatus 1 and the projector 2 have been described as separate bodies, but may be integrally configured. Further, the projector 2 and the camera 3 may be integrally configured.

  In the above-described embodiment, the configuration in which the output light of the projector 2 is directly applied to the back surface of the screen 5 has been described as an example. However, a mirror that reflects the output light of the projector 2 is provided, and the reflected light of the mirror is supplied to the projector 2. It is good also as projecting on the back surface.

  In the above embodiment, the video data of the projection video content has been described as being stored in the storage unit 13. However, as a configuration in which the video data is distributed from a server connected to the image processing apparatus 1 via a communication network. Also good. Further, the video data may be captured from a USB memory or various recording media.

  Also, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory, a portable recording medium such as a CD-ROM, in addition to a ROM, a hard disk, etc. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

  In addition, the detailed configuration and detailed operation of each device constituting the content reproduction system can be changed as appropriate without departing from the spirit of the invention.

(Appendix)
Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
<Claim 1>
An image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer area extraction means for extracting the outer area of the screen from the acquired photographed image;
Target area extracting means for extracting the area of the projection target from the video data of the video content;
Movement information acquisition means for acquiring information indicating movement of the projection object in the video content based on the area of the projection object extracted by the target area extraction means;
The outline area of the screen extracted by the screen outline area extraction unit, the area of the projection object extracted by the target area extraction unit, and the movement of the projection object acquired by the motion information acquisition unit Correction means for correcting the video data of the video content based on the information;
Control means for projecting the projection object onto the screen by the projection device based on the corrected video data;
An image processing apparatus comprising:
<Claim 2>
The target area extracting means extracts the area of the projection target from each of a plurality of frame images constituting the video data,
The motion information acquisition means is a range of motion of the projection object in the video content that is the maximum area when the areas of the plurality of projection objects extracted from the plurality of frame images by the target area extraction means are overlapped. Is obtained as information indicating
The correction means corrects video data of the video content based on the maximum area acquired by the motion information acquisition means and the outline area of the screen extracted by the screen outline area extraction means. Image processing apparatus.
<Claim 3>
The image processing according to claim 1, wherein the motion information acquisition unit further acquires information indicating the movement of the projection target in the video content based on the screen outer region extracted by the screen outer region extraction unit. apparatus.
<Claim 4>
The screen image acquisition means acquires a captured image including the screen on which the projection object is projected by the projection device,
The target area extracting means extracts the area of the projection target from each of a plurality of frame images constituting the video data,
A second target area extracting unit that extracts a region of the projection target from a captured image including the screen onto which the projection target is projected;
The motion information acquisition unit is configured to select the screen data in the video data based on the outline region of the screen extracted by the screen outline region extraction unit and the region of the projection target extracted by the second target region extraction unit. A standard position of the projection object is determined, and a movement amount of the area of the projection object from the standard position in each of the plurality of frame images extracted by the target area extraction means indicates the movement of the projection object. The image processing apparatus according to claim 3, which is acquired as information.
<Claim 5>
A video data correction method in an image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Obtaining a captured image including the screen;
Extracting an outer area of the screen from the acquired captured image;
Extracting a region of the projection object from video data of the video content;
Obtaining information indicating the movement of the projection object in the video content based on the extracted region of the projection object;
Correcting the video data of the video content based on the extracted outline area of the screen, the extracted area of the projection object, and the acquired information indicating the movement of the object;
Projecting the projection object onto the screen by the projection device based on the corrected video data;
Video data correction method including
<Claim 6>
A computer used in an image processing apparatus connected to a projection apparatus that projects video content including the projection object onto a screen molded into the shape of the projection object;
Screen image acquisition means for acquiring a captured image including the screen;
A screen outer region extraction means for extracting an outer region of the screen from the acquired photographed image;
Target area extracting means for extracting the area of the projection target from the video data of the video content;
Movement information acquisition means for acquiring information indicating movement of the projection object in the video content based on the area of the projection object extracted by the target area extraction means;
The outline area of the screen extracted by the screen outline area extraction unit, the area of the projection object extracted by the target area extraction unit, and the movement of the projection object acquired by the motion information acquisition unit Correction means for correcting the video data of the video content based on the information;
Control means for projecting the projection object onto the screen by the projection device based on the corrected video data;
Program to function as.

100 content reproduction system 1 image processing apparatus 10 control unit 11 operation unit 12 display unit 13 storage unit 131 program storage unit 132 video data storage unit 133 setting information storage unit 14 I / F
15 I / F
16 Audio output unit 2 Projector 3 Camera 4 Speaker 5 Screen 51 Base material 52 Screen film 53 Film 6 Support stand

Claims (5)

An image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Screen image acquisition means for acquiring a captured image including the screen;
Screen outer area extraction means for extracting the outer area of the screen from the acquired photographed image;
Target area extracting means for extracting the area of the projection target from the video data of the video content;
Movement information acquisition means for acquiring information indicating movement of the projection object in the video content based on the area of the projection object extracted by the target area extraction means;
The outline area of the screen extracted by the screen outline area extraction unit, the area of the projection object extracted by the target area extraction unit, and the movement of the projection object acquired by the motion information acquisition unit Correction means for correcting the position or magnification of the video data of the video content based on the information;
Control means for projecting the projection object onto the screen by the projection device based on the corrected video data;
An image processing apparatus comprising: The target area extracting means extracts the area of the projection target from each of a plurality of frame images constituting the video data,
The motion information acquisition means is a range of motion of the projection object in the video content that is the maximum area when the areas of the plurality of projection objects extracted from the plurality of frame images by the target area extraction means are overlapped. Is obtained as information indicating
The correction means corrects video data of the video content based on the maximum area acquired by the motion information acquisition means and the outline area of the screen extracted by the screen outline area extraction means. Image processing apparatus. The screen image acquisition means acquires a captured image including the screen on which the projection object is projected by the projection device,
The target area extracting means extracts the area of the projection target from each of a plurality of frame images constituting the video data,
A second target area extracting unit that extracts a region of the projection target from a captured image including the screen onto which the projection target is projected;
The motion information acquisition unit is configured to select the screen data in the video data based on the outline region of the screen extracted by the screen outline region extraction unit and the region of the projection target extracted by the second target region extraction unit. A standard position of the projection object is determined, and a movement amount of the area of the projection object from the standard position in each of the plurality of frame images extracted by the target area extraction means indicates the movement of the projection object. The image processing apparatus according to claim 1, which is acquired as information. A video data correction method in an image processing apparatus connected to a projection apparatus that projects video content including a projection object onto a screen molded into the shape of the projection object,
Obtaining a captured image including the screen;
Extracting an outer area of the screen from the acquired captured image;
Extracting a region of the projection object from video data of the video content;
Obtaining information indicating the movement of the projection object in the video content based on the extracted region of the projection object;
The position or magnification of the video data of the video content is corrected based on the extracted outline area of the screen, the extracted area of the projection object, and information indicating the movement of the acquired object. And a process of
Projecting the projection object onto the screen by the projection device based on the corrected video data;
Video data correction method including A computer used in an image processing apparatus connected to a projection apparatus that projects video content including the projection object onto a screen molded into the shape of the projection object;
Screen image acquisition means for acquiring a captured image including the screen;
A screen outer region extraction means for extracting an outer region of the screen from the acquired photographed image;
Target area extracting means for extracting the area of the projection target from the video data of the video content;
Movement information acquisition means for acquiring information indicating movement of the projection object in the video content based on the area of the projection object extracted by the target area extraction means;
The outline area of the screen extracted by the screen outline area extraction unit, the area of the projection object extracted by the target area extraction unit, and the movement of the projection object acquired by the motion information acquisition unit Correction means for correcting the position or magnification of the video data of the video content based on the information;
Control means for projecting the projection object onto the screen by the projection device based on the corrected video data;
Program to function as.