JP6556680B2 - VIDEO GENERATION DEVICE, VIDEO GENERATION METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to a technique for generating an image to be superimposed on an object.

  A technique is known in which an image that gives an apparent movement is superimposed on an object and perceived as if the object is moving (Patent Document 1).

International Publication No. WO / 2015/163317

  It is difficult to guess how an object on which an image giving an apparent motion is superimposed is perceived. Therefore, it is not easy to generate an image that gives a desired visual effect when superimposed on an object.

  An object of the present invention is to easily generate an image that gives a desired visual effect when superimposed on an object.

  A designation unit that is a user interface for setting information for generating a video presents an image area included in the image of the target object so that the user can visually recognize the video, and generates the video. The video generation unit generates a video corresponding to the specification information received by the designation unit, and the video presentation unit uses the target region, which is the target object region, and the designation unit. The video corresponding to the designation information received by the designation unit is presented so that the user can visually recognize how the video generated by the video generation unit corresponding to the received designation information is superimposed. As a result, in the state in which the user can visually see how the target region that is the target region and the video generated by the video generation unit corresponding to the specification information received by the specification unit are superimposed, It is possible to present an image area included in the image of the object and to accept input of designation information for generating a video.

  This makes it possible to easily generate an image that gives a desired visual effect when superimposed on an object.

FIG. 1 is a block diagram illustrating a video system according to an embodiment. FIG. 2 is a conceptual diagram illustrating a user interface screen according to the embodiment. FIG. 3 is a conceptual diagram illustrating a parameter setting unit (before video superimposition) of the user interface. FIG. 4 is a conceptual diagram illustrating a parameter setting unit (at the time of video superimposition) included in the user interface. FIG. 5 is a conceptual diagram illustrating a user interface screen according to the embodiment. FIG. 6 is a conceptual diagram illustrating a user interface screen according to the embodiment. FIG. 7 is a block diagram illustrating a video system according to the embodiment.

Embodiments of the present invention will be described below.
[Overview]
First, an outline will be described.
In this embodiment, when generating a “video” to be superimposed in order to give an apparent motion (motion impression) to the “object”, (1-1) information for generating the “video” is “utilized”. “Designator”, which is a user interface for setting by “user”, presents “image area” included in the image of “object” so that “user” can see it, and generates “video” (1-2) The “video generation unit” generates “video” corresponding to the “designation information” received by the “designation unit”, and (1-3) The “video presentation unit” superimposes the “target region”, which is the region of the “object”, and the “video” generated by the “video generation unit” corresponding to the “designated information” received by the “designation unit”. In order for the “user” to be able to see the appearance of the To present the "Video". The “video presentation unit” may be a projection device or a transmissive display. In the former case, the “video presentation unit” that is a projection device projects the “video” onto the “target area”, thereby superimposing the “target area” and the “video”. In the latter case, the “video presentation unit” which is a transmissive display displays a transparent “video” and is placed behind the “video” displayed in the “video presentation unit” and the “video presentation unit”. The “target region” of the “target” that has been added is superimposed. As a result, the “target region” that is the region of the “target object” and the “video” generated by the “video generation unit” corresponding to the “designation information” received by the “designation unit” are superimposed. In the state where the “user” is visible, the “designation unit” presents the “image area” included in the “object” image and accepts the input of “designation information” for generating the “video”. It becomes possible. (1-4) The “target region” that is the region of the “object” and the “video” generated by the “video generation unit” corresponding to the “specified information” received by the “designating unit” are superimposed. When the state is visible to the “user”, the presentation of the image area included in the image of the object in the “designating part” and the reception of the designation information for generating the “video” are received. Then, the “video generation unit” generates the “video” in which the received “designated information” is immediately reflected as “updated video”, and (1-5) the “video presentation unit” is in the “object” area. The “updated video” is presented so that the user can visually recognize a certain “target region” and the “updated video” generated by the “video generation unit”. When the “video presentation unit” presents the “updated video”, the “video generation unit” corresponds to the “target region” that is the region of the “target object” and the “designation information” received by the “designation unit”. The “updated video” generated by “” is in a state where the “user” can visually recognize the superimposed state. As a result, the “user” can input “designation information” from the user interface and confirm in real time how the “video” corresponding to the “designation information” is superimposed on the “target area”. Since the “user” can set “designated information” while watching this state, it is possible to easily generate “video” that gives a desired visual effect when superimposed on the “target region”.

  For example, (2-1) a “designating unit” which is a user interface for the “user” to set the “object” area so that the “user” can visually recognize the image of the “object”. In addition to presenting, the input of “first designation information” for specifying the “image area” included in the image of the “object” is accepted, and (2-2) “video generation unit” is accepted by the “designation unit” A video corresponding to the “image area” specified by the “first designation information” is generated, and (2-3) “video presentation unit” corresponds to the “image area” specified by the “first designation information”. “Video” generated by the “video generation unit” corresponding to the “target region” that is the region of the “target object” and the “image region” specified by the “first designation information” received by the “designation unit” The “first finger” received by the “designating part” so that the “user” can see the superimposed state Corresponding to the "image region" specified by the information "may present a" picture "of" image generator "is generated. As a result, the “video generation unit” corresponding to the “target region” that is the region of the “target object” and the “image region” specified by the “first designation information” received by the “designation unit” is generated. In a state where the user can visually recognize the state in which the “video” is superimposed, the “designation unit” presents the image area included in the “object” image and inputs the “first designation information”. Acceptance is possible. “User” inputs “first designation information” from the user interface and confirms that “video” is superimposed on “target area” corresponding to “image area” specified by “first designation information”. it can. Since the “user” can set the “target region” while watching this state, it is possible to easily generate a “video” that gives a desired motion to the desired region when superimposed on the “target region”.

  For example, (3-1) Movement given to the image of “image area” and “target area” by the “designating unit” which is a user interface for the “user” to set the movement given to the image of “target area” The parameter corresponding to is presented so that the user can visually recognize it, and the input of “second designation information” that identifies the parameter corresponding to the movement is accepted. (3-2) The “video generation unit” A video that gives a motion corresponding to the parameter specified by the “second specification information” received by the “designation unit” to the “image region” specified by the “first specification information” received by the “part”, (3-3) The “video presentation unit” includes “target region” that is the region of “object” and “image region” that is specified by “first designation information” received by the “designation unit”. The parameter specified by the “second designation information” received by the “designation section” In the “image area” specified by the “first designation information” received by the “designation unit” so that the user can visually recognize the superposition of the “video” that gives the movement corresponding to the meter, The “video” generated by the “video generation unit” that gives a motion corresponding to the parameter specified by the “second specification information” received by the “designation unit” may be presented. As a result, the “target region” that is the region of the “object” and the “second region” received by the “designation unit” are identified by the “image region” identified by the “first designation information” received by the “designation unit”. In the state where the user can visually recognize the superimposed state of the “video” generated by the “video generation unit” that gives a motion corresponding to the “parameter” specified by the “specification information”, It is possible to present parameters corresponding to movement and to accept input of second designation information. The “user” inputs “second designation information” from the user interface, and can confirm that the “video” giving the corresponding motion is superimposed on the “target area”. Since the “user” can set the “second designation information” while watching this state, it is possible to easily generate a “video” that gives a desired motion when superimposed on the “target region”. The “movement” may be periodic or non-periodic.

  For example, (4-1) a “designating unit” which is a user interface for a “user” to set a time range of motion given to an image of “target region” is on the time axis of motion given by “video” The parameter corresponding to the position and the time range of the motion given by the “video” is presented so that the “user” can visually recognize, and the “third designation information” for specifying the parameter corresponding to the time range of the motion is input. (4-2) The “video generation unit” receives the “third designation information” received by the “designation unit” in the “image area” specified by the “first designation information” received by the “designation unit”. ”Is generated in the time range corresponding to the parameter specified by“ (4-3) ”, and the“ video presentation unit ”receives the“ target object ”region and the“ designation unit ”. Parameters specified by “third specification information” The “designation unit” accepts the “video” giving motion in the corresponding time range, and the position on the time axis of the motion given by “video” so that the user can visually recognize it. Even if the “image area” specified by the “first designation information” is presented with “video” that gives motion in the time range corresponding to the parameter specified by the “third designation information” received by the “designation unit” Good. As a result, the “target area” that is the area of the “object” and the “image area” that is specified by the “first designation information” that is received by the “designation section” are the “third area” that is received by the “designation section”. The movement of the “video” generated by the “video generation unit” that gives motion in the time range corresponding to the parameter specified by the “specification information” and the position on the time axis of the movement that “video” gives , The parameter corresponding to the time range of movement in the “designating part” and the acceptance of the input of “third designated information” can be received. The “user” inputs “third designation information” from the user interface, and can confirm that the “video” that gives the motion with the corresponding time change is superimposed on the “target region”. Since the “user” can set the “third designation information” while watching this state, it is possible to easily generate a “video” that gives a motion in a desired time range when superimposed on the “target region”. The motion time range may be specified by the start time and end time of the motion, or may be specified by the start time of the motion, the duration of the motion, and the number of frames.

  The “object” may be an object having a three-dimensional shape (for example, a vase, a ball, a model), or a plane (for example, paper, a board, a wall, a screen). When the “object” is a plane, it is desirable that the pattern includes the plane. Examples of patterns included in the plane include a photograph printed on paper, an image, a photograph projected on a predetermined plane, and an image. When the “object” is a screen such as a display, examples of the pattern include an image displayed on the screen such as a display. The “target area” may be the entire area of the “target object” or may be a partial area of the “target object”.

The “video” is a moving image that gives an apparent movement to the image of the “target area”. In other words, the “video” is a transparent moving image representing the movement given to the image of the “target area”. For example, the “video” is a moving image including a “luminance motion component” representing a motion given to the image of the “target region”, and preferably a moving image including only the “luminance motion component” (see, for example, Patent Document 1). . By superimposing such an “image” on the “object”, an apparent movement can be given to the “object”. An example of such a “video” generation method is as follows. A still image of the “object” is expressed as M _static . The still image M _static is a color image including a chromatic color, and is composed of pixel values representing intensity changes in the R channel, the G channel, and the B channel.
However, {R _static (x, y)}, {G _static (x, y)}, and {B _static (x, y)} are pixel values R _static (x, y) and G _static (x, y), respectively. y) is a two-dimensional matrix having a two-dimensional space with B _static (x, y) as elements. Pixel values R _static (x, y), G _static (x, y), and B _static (x, y) represent the intensities of the R channel, the G channel, and the B channel at the horizontal position x and the vertical position y, respectively. x and y are integers indicating the horizontal position and the vertical position when a still image is expressed in a two-dimensional coordinate system, respectively. Let the lower limit and upper limit of the horizontal position be x _min and x _max (x _min <x _max ), and the lower limit and upper limit of the vertical position y _min and y _max (y _min <y _max ). x and y satisfy x _min ≦ x ≦ x _max and y _min ≦ y ≦ y _max .

Next, the pixel values R _static (x, y), G _static (x, y), and B _static (x, y) are weighted and added according to the degree that each color contributes to the luminance, and the luminance component image of the image M _static Y _static (x, y) (where x _min ≤ x ≤ x _max , y _min ≤ y ≤ y _max ) is obtained by the following equation.
Y _static (x, y) = α _R R _static (x, y) + α _G G _static (x, y) + α _B B _static (x, y)
However, α _R , α _G and α _B are weighting coefficients (constants) (for example, α _R = 0.299, α _G = 0.587, α _B = 0.114).

A moving image in which a motion is given to the luminance component image Y _static of the still image of the “object” is expressed as Y ₁ . The movement given to the luminance component image Y _static is not limited, and any movement may be given. Video Y ₁ can be expressed as follows.
However, {Y (x, y, t)} is a three-dimensional matrix having spatial two-dimensional and temporal one-dimensional information whose pixel value Y (x, y, t) is an element. The pixel value Y (x, y, t) represents the intensity of the luminance component at the horizontal position x, the vertical position y, and the frame number t. x, y, and t are integers indicating the horizontal position, the vertical position, and the frame number when a moving image is expressed in a three-dimensional coordinate system. The lower and upper limits of the horizontal position are x _min and x _max (x _min <x _max ), the lower and upper limits of the vertical position are y _min and y _max (y _min <y _max ), and the lower and upper limits of the frame number are t _min And t _max (t _min <t _max ). x, y, and t satisfy x _min ≤ x ≤ x _max , y _min ≤ y ≤ y _max , and t _min ≤ t ≤ t _max .

Further, the luminance motion component Y _motion (x, y, t) is obtained by subtracting the luminance component image Y _static (x, y) from the luminance component image Y ₁ (x, y, t) of each frame t.
Y _motion (x, y, t) = Y ₁ (x, y, t) -Y _static (x, y)

The luminance motion component Y _motion (x, y, t) thus obtained may be used as the above-mentioned “video”, or may be filtered into the luminance motion component Y _motion (x, y, t) (for example, high spatial frequency filtering). Alternatively, a video obtained by low spatial frequency filtering may be used as an “image”. Alternatively, as described later, the luminance motion component Y _motion (x, y, t) is given a weight, corrected for the reflectance of the “object” and the albedo, or added with a constant. It may be “video”. An example of such an “image” is a “phantom light” (registered trademark).

  In addition, the “non-transparent” moving image representing the movement given to the “target area” image may be “video”.

  The “object image” is, for example, an image obtained by capturing image information of the “object”. For example, the “object image” may be an image obtained by photographing the “object” or may be an image obtained by scanning the “object”. The “object image” may be an image obtained by performing image processing on an image obtained by capturing image information of the “object”, or an image obtained by drawing the “object”. It may be a computer graphic image of an “object”. Alternatively, the “object image” may be an image representing only some features of the “object”. The “object image” may be a color image, a monotone image, or a simple monochrome image (binary image). For example, the “object image” may be an image representing only the edges and contours of the “object”. The “image region” may be the entire region of the “object image” or may be a partial region of the “object image”.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Configuration of video system>
As illustrated in FIG. 1, the video system 1 of the present embodiment includes a video generation device 11 that generates a video to be superimposed on the target 14, a shooting device 12 that captures an image of the target 14 by capturing the target 14, and A projection device 13 (image presentation unit) that projects an image on the object 14 is provided. The video generation device 11 may be directly connected to the imaging device 12 and the projection device 13, may be connected through a local network such as a LAN, or may be connected through a public network such as the Internet. Good.

  The video generation device 11 includes a pixel correspondence calculation unit 11a, a superimposed video generation unit 11b, a projection target image acquisition unit 11c, an albedo calculation unit 11d, a position / orientation conversion unit 11e, an image search unit 11f, a video generation unit 11g, and a motion editing unit 11h. An interface unit 11i, an input unit 11j, a display unit 11k, a database (DB) unit 11m, and a memory 11r. The interface unit 11i, the input unit 11j, and the display unit 11k are user interfaces (designating units) that accept input of information from the user and present (output) information to the user. The video generation apparatus 11 includes, for example, a processor (hardware processor) such as a central processing unit (CPU), a memory such as random-access memory (RAM) / read-only memory (ROM), a mouse, a touch panel, a liquid crystal display, and the like A general-purpose or dedicated computer having a user interface or the like executes a predetermined program. The computer may include a single processor and memory, or may include a plurality of processors and memory. This program may be installed in a computer, or may be recorded in a ROM or the like in advance. In addition, some or all of the processing units are configured using an electronic circuit that realizes a processing function without using a program, instead of an electronic circuit (circuitry) that realizes a functional configuration by reading a program like a CPU. May be. In addition, an electronic circuit constituting one device may include a plurality of CPUs. The video generation device 11 is configured, for example, by mounting a program on a computer such as a personal computer, a smartphone terminal device, a tablet terminal device, or a server device.

  The photographing device 12 may be a camera capable of photographing a visible light image, or may be a camera capable of photographing ultraviolet rays or infrared rays. The projection device 13 is a projector that has a light source, for example, and projects a transparent image.

The DB unit 11m includes, for each of a plurality of “objects”, information representing the “object”, a “reference image” corresponding to the “object”, and “motion information” of the “reference image”. Are stored in association with each other. A set of information corresponding to each object (“information representing the object”, “reference image”, “motion information”) is referred to as a “scene”. Examples of the “information representing the object” include an image pattern and a feature amount (for example, SIFT feature amount) of the object. The “motion information” includes information (coordinates, etc.) of one or more distortion windows (DW: Deformation Window) which are image areas to which motion is applied, a spatial range of distortion (motion) corresponding to each distortion window DW, distortion ( Motion) time range (distortion start time, duration, fade-in time, fade-out time, etc.), distortion basis function type, amplitude, spatial frequency, speed, angle, phase, and file name information to play Including. The “motion information” in the initial state includes, for example, information on a predetermined distortion window DW, a distortion spatial range, a distortion time range, and a distortion basis function type corresponding to the distortion window DW. Information on amplitude, spatial frequency, speed, angle, phase, and sound effect ID. The distortion basis function is a function representing a basic distortion (motion), and the movement is given by a linear combination of the distortion basis functions. The motion specified by the “motion information” is, for example, periodic. For example, the movement in the distortion window DW is a reciprocating movement in the distortion window DW. As described above, by storing not the video (projection pattern) itself but the information specifying the motion as the motion information, the storage capacity of the DB unit 11m can be saved. Furthermore, according to the image of the object, a video is generated from the information specifying the movement, so that the configuration is robust against displacement. For example, as shown below, a list of scenes that associates a “scene number” that identifies a “scene”, “information representing an object”, and “reference image” and “motion information” corresponding to the “object” Is stored in the DB unit 11m (Table 1). When it is desired to always give a specific movement regardless of the type of the object, the DB may include a “scene” that does not have “information representing the object”. In this case, a white image of an arbitrary size is associated with the scene as a “reference image”.

When the “object” is a surface pattern of a three-dimensional object (three-dimensional object), a plurality of image patterns (three-dimensional data) obtained by photographing one “three-dimensional object” from a plurality of directions, A plurality of feature amounts corresponding to such a plurality of image patterns may be used as “information representing an object”. For example, as shown below, a “three-dimensional object”, “information representing the object” of a plurality of objects corresponding to the three-dimensional object, a “reference image” corresponding to the object, and the “reference image” A DB that stores a list of “scenes” associated with “motion information” may be stored in the DB unit 191 (Table 2).

<Operation>
Next, the operation of the video system 1 of this embodiment will be described.
≪Startup≫
When activating the image generating apparatus 11, a projection target image acquisition unit 11c is, indicates the projection of uniform light brightness B ₁ to the projection unit 13, and instructs the acquisition of the image to the imaging device 12. The image 101 of the object 14 acquired by the imaging device 12 is sent to the interface unit 11i. The motion editing unit 11h extracts the coordinates and motion information of the outer frame of the reference image included in the initially set scene (for example, the scene with the scene number i = 1) from the DB unit 11m, and is generated using these. The interface unit 11i is instructed to display the user interface screen 100-1. The interface unit 11i displays the user interface screen 100-1 (FIG. 2) on the display unit 11k. As illustrated in FIG. 2, the user interface screen 100-1 includes an image 101, a projection area 110 obtained from the coordinates of the outer frame of the reference image of the initially set scene, and each DW of the motion information of the scene. Image regions 120 and 121 obtained from the coordinates, a time axis information display unit 130 that is a region representing time axis information, and a parameter setting unit 150 are included. The image area 121 is a distortion window (DW) included in motion information of a predetermined scene (for example, a scene with scene number i = 1) extracted by the motion editing unit 11h. The positions of the projection area 110 and the image areas 120 and 121 are determined by the scene and do not necessarily correspond to the actual area of the image 101. The user can manually adjust the position and orientation of the imaging device 12 and the position and orientation of the object 14 while viewing the user interface screen 100-1.

As illustrated in FIG. 3, the parameter setting unit 150 of the user interface screen 100-1 displays setting units 151a, 152a to 152h, 153a to 153e, 154a to 154d, 155a to 155c, and 156a to 156d. An outline of the setting units 151a, 152a to 152h, 153a to 153e, 154a to 154d, 155a to 155c, and 156a to 156d is as follows.
Setting section 151a: Switching to the effect edit mode. Switching to the effect edit mode is performed by an input operation to the setting unit 151a. The “effect editing mode” is a mode in which the movement is set without superimposing the “video” for giving the apparent movement to the object 14.
Setting unit 152a: Starts projection of an “image” (for example, an illusion lamp) that gives an apparent movement to the object 14. The “video” is projected onto the object 14 by an input operation to the setting unit 152a.
Setting unit 152b: Starts gray code pattern projection. By the input operation to the setting unit 152b, projection of a gray code pattern for calibration is started.
Setting unit 152c: Projecting test image (ON / OFF switching). Whether or not to project the test image can be switched by an input operation to the setting unit 152c. By browsing the projected test image, the appropriateness of the positional relationship between the object 14 and the projection device 13 can be visually confirmed.
Setting unit 152d: Frame setting of the projection area 110. The projection area 110 can be changed by an input operation to the setting unit 152d.
Setting unit 152e: Spotlight-like mask setting. An area of light projected from the projection device 13 can be limited (masked) by an input operation to the setting unit 152e.
Setting unit 152f: ON / OFF switching of the display of the captured image. By an input operation to the setting unit 152f, it is possible to switch whether an image photographed by the photographing device 12 is displayed from the display unit 11k in real time (ON) or not (OFF).
Setting unit 152g: Projects brighter uniform light (ON / OFF) among the brightnesses B ₁ and B ₂ projected at the time of capturing a still target image. Of the two types of brightness (B ₁ , B ₂ ) projected at the time of image capturing, the brighter uniform light is projected (ON) or the darker uniform light by the input operation to the setting unit 152g. Can be switched.
Setting unit 152h: saves a photographed image. The image 101 can be saved by an input operation to the setting unit 152h.
Setting unit 153a: Left-right reversal of the captured image. The left and right sides of the image 101 can be reversed by an input operation to the setting unit 153a.
Setting unit 153b: Gain setting of the photographing device 12. A gain parameter can be set by an input operation to the setting unit 153b.
Setting unit 153c: Setting of the shutter speed of the photographing apparatus 12. A shutter speed parameter can be set by an input operation to the setting unit 153c.
Setting unit 153d: White balance setting of the photographing apparatus 12. A white balance parameter can be set by an input operation to the setting unit 153d.
Setting unit 153e: White balance setting of the photographing apparatus 12. White balance parameters can be set by an input operation to the setting unit 153e.
Setting unit 154a: Gamma value setting of the projection device 13. A gamma value parameter can be set by an input operation to the setting unit 154a.
Setting unit 154b: setting of the maximum value of the output of the projection device 13. The parameter of the maximum output value can be set by an input operation to the setting unit 154b.
Setting unit 154c: Minimum value setting of the output of the projection device 13. The parameter of the minimum value of the output can be set by an input operation to the setting unit 154c.
Setting unit 154d: Average luminance setting of “video” that gives an apparent movement to the object 14. The average brightness of the “video” can be set by an input operation to the setting unit 154d.
Setting unit 155a: The gray code pattern stripe is thickened.
Setting unit 155b: The stripe of the gray code pattern is thinned.
Setting unit 155c: switches the stripe direction of the gray code pattern.
Setting unit 156a: Advances the display scene. By an input operation to the setting unit 156a, the scene number i of the scene corresponding to the image area 121 to be displayed is incremented by one. That is, every time the setting unit 156a is clicked, the image area 121 displayed on the user interface screen 100-1 is switched.
Setting unit 156b: Moves the display scene back. By the input operation to the setting unit 156a, the scene number i of the scene corresponding to the image area 121 to be displayed is decreased by one. That is, every time the setting unit 156a is clicked, the image area 121 displayed on the user interface screen 100-1 is switched.
Setting unit 156c: saves the current state. By the input operation to the setting unit 156c, the scene number i and the like at that time are stored.
Setting unit 156d: application termination The user performs an input operation (click, tap, numerical input, etc.) to each setting unit using the input unit 11j, and sets each parameter or switches the function. In the setting section where a parameter can be set, the current value of the parameter is displayed. When a parameter is updated by an input operation to the setting unit, the updated parameter is displayed on the setting unit.

≪Calibration≫
The calibration for calculating the correspondence map between the imaging device 12 and the projection device 13 will be described. The calibration is performed by projecting a set of black and white striped patterns called gray code patterns from the projection device 13 onto the object 14 and photographing this with the photographing device 12.

  First, the projection device 13 is adjusted while changing the thickness and direction of the stripe so that the Gray code pattern is correctly captured. That is, by the input operation to the setting units 155a to 155c, the thickness and direction of the gray code pattern stripes sent from the pixel correspondence calculation unit 11a to the projection device 13 and projected onto the object 14 can be changed. The gain, shutter speed, white balance, etc. of the photographing device 12 are adjusted by an input operation to 153b to 153e.

  When an input operation to the setting unit 152b is performed, calibration is started. When calibration is started, the pixel correspondence calculation unit 11 a sends a gray code pattern to the projection device 13, and the projection device 13 projects the gray code pattern onto the object 14. The imaging device 12 captures the projected gray code pattern and sends the captured image to the pixel correspondence calculation unit 11a. The pixel correspondence calculation unit 11a examines the order in which black and white images are captured in each pixel, so that the coordinates (Cx, Cy) in the coordinate system of the imaging device 12 are any coordinates ( C → P map indicating whether or not it corresponds to Px, Py). On the contrary, the pixel correspondence calculation unit 11a determines which coordinate (Px, Py) in the coordinate system of the projection apparatus 13 corresponds to which coordinate (Cx, Cy) in the coordinate system of the imaging apparatus 12 from the C → P map. , P → C map is calculated. The missing part of the P → C map is compensated by interpolation (interpolation). The P → C map is sent to the superimposed video generation unit 11b. When an input operation to the setting unit 156c is performed, the superimposed video generation unit 11b stores the P → C map in the memory 11r. Thereafter, the superimposed video generation unit 11b reads and uses the P → C map from the memory 11r as necessary. By the input operation to the setting unit 152c, it is possible to confirm whether the inspection image is projected from the projection device 13 and the calibration is correctly completed. If the positional relationship between the imaging device 12 and the projection device 13 and the object 14 are the same, the calibration can be omitted.

≪Optical calibration≫
When the projection pattern by the projection device 13 is inappropriate, or when the object 14 photographed by the photographing device 12 is not correctly recognized, or the effect by the “video” (the effect of giving an apparent movement to the object 14) is maximized. In such a case, optical calibration may be performed. Optical calibration is performed by an input operation of the setting units 152g, 153b to 153e, and 154a to 154d. For example, in order to make the input / output between the imaging device 12 and the projection device 13 as linear as possible, the gamma value of the projection device 13 may be input to the setting unit 154a. In addition, an input operation to the setting unit 152g projects the projection device 13 with the brightness at the time of image recognition, and adjusts the gain of the imaging device 12 with the setting unit 153b while confirming the image captured by the imaging device 12. May be. The parameters input to the setting units 153b to 153e and 154a to 154d are displayed on the setting units 153b to 153e and 154a to 154d, respectively.

≪Selection of display scene≫
The user can select a scene corresponding to the projection area 110 and the image areas 120 and 121 displayed on the user interface screen 100-1 by an input operation to the setting units 156a and 156b. Each time the user performs an input operation on the setting unit 156a, the scene number i is incremented, and every time an input operation is performed on the setting unit 156b, the scene number i is incremented. When an input operation to the setting units 156a and 156b is performed, the scene number i of the selected scene is sent to the motion editing unit 11h. The motion editing unit 11h reads the coordinates of the outer frame of the reference image corresponding to the sent scene number i and the coordinates of the distortion window DW of the motion information corresponding to the scene number i from the DB unit 11m, and sends them to the interface unit 11i. send. The interface unit 11i sends the outer frame of the sent reference image as the projection area 110, and sends the coordinates of the sent DW as the image areas 120 and 121 to the display unit 11k. The display unit 11k displays the projection area 110 and the image areas 120 and 121 corresponding to the scene number i, and the time axis information display unit 130 on the user interface screen 100-1. The information displayed here includes a projection area 110 indicating an area where a reference image corresponding to the scene number i exists, image areas 120 and 121 indicating specific image areas of the reference image, and a time axis. This is a time axis information display unit 130 that is an area representing the information (FIG. 2). As described above, the projection area 110 is defined by the coordinates of the outer frame of the reference image of the scene that is initially set at the time of activation, and does not necessarily correspond to the actual area of the object 14. The user can also set the projection area 110 manually. In this case, after the input operation to the setting unit 152d, the user selects an area by clicking or tapping. The selected area is set as the projection area 110.

≪Start of processing for giving an apparent movement to the object 14≫
Hereinafter, a process for giving an apparent motion to the object 14 (for example, an “alternative light effect” which is an apparent motion given by superimposing an illusion lamp on the object 14) will be described. When the user performs an input operation to the setting unit 152a, the projection target image acquisition unit 11c _first instructs the projection device 13 to project uniform light with brightness B1. The projection device 13 projects uniform light with brightness B ₁ onto the object 14, and the imaging device 12 captures an image of the object 14 onto which uniform light with brightness B ₁ is projected. An image I _{B1 of the} object 14 photographed under uniform light with brightness B ₁ is stored in the memory 11r. Next, the projection target image acquisition unit 11c instructs the projection device 13 to project uniform light with a brightness B ₂ that is brighter than the brightness (light intensity) B ₁ (for example, B ₂ = 2B ₁ ). The projection device 13 projects uniform light with brightness B ₂ onto the object 14, and the imaging device 12 captures an image of the object 14 onto which uniform light with brightness B ₂ is projected. An image I _{B2 of the} object 14 photographed under uniform light of brightness B ₂ is stored in the memory 11r. Rather than obtain an image I _B2 after obtaining the image I _B1, may be obtained images I _B1 after obtaining the image I _B2. B ₁ and B ₂ are positive values representing the light intensity, and the values increase as the light intensity increases. The acquired images I _B1 and I _B2 are sent to the albedo calculation processing unit. At least one of the images I _B1 and I _B2 is sent to the image search unit. Furthermore, one of the images I _B1 and I _B2 is sent to the video generation unit 11g.

≪Albedo calculation process≫
The albedo calculation unit 11d calculates an albedo (in this case, indicates the ratio of the increment of the captured pixel value to the increase of the projection light intensity) from the images I _B1 and I _B2 received from the projection target image acquisition unit 11c, and calculates the memory 11r. To store. The albedo K is calculated as K = (I _B2 −I _B1 ) / (B ₂ −B ₁ ). As described above, B ₁ and B ₂ represent the light intensity projected from the projection device 13 when the images I _B1 and I _B2 are captured.

≪Image Search≫
The image search unit 11f receives the images I _B1 and I _B2 output from the projection target image acquisition unit 11c, and extracts information representing the image from one or both of the images I _B1 and I _B2 . The information representing the image may be, for example, the SIFT feature quantity of the image I _B1 or I _B2 , or the SIFT feature quantity of the image I _{B1 and} the SIFT feature quantity of the image I _B2 . Image retrieval section 11f, for example, using SIFT feature quantity of the image _{I B1} or _{I B2} to retrieve a list of scenes stored in the DB unit 191, adapted to image _{I B1} or _{I B2} (match) to "reference image" Extract scenes that contain For example, the image search unit 11f extracts a scene including a “reference image” corresponding to “information representing an object” that matches or approximates information representing the image I _B1 or I _B2 . For example, when the “information representing the object” stored in the DB is a SIFT feature amount and the “information representing the image” is the SIFT feature amount of the image I _B1 or I _B2 , the image search unit 11f displays the image I The SIFT feature value of _B1 or I _B2 is calculated, a SIFT feature value including a predetermined number or more of feature points that match or approximate the SIFT feature value of the image I _B1 or I _B2 is searched from the DB, and the searched SIFT feature value A scene including the corresponding “reference image” is extracted. For example, when “information representing an object” stored in the DB is a SIFT feature quantity, and “information representing an image” is a SIFT feature quantity of the image I _{B1 and} a SIFT feature quantity of the image I _B2 , the image search unit 11f calculates a SIFT feature quantity of the image I _{B1 and} a SIFT feature quantity of the image I _B2 , and searches the DB for a SIFT feature quantity that includes a predetermined number or more of feature points that match or approximate at least one of these SIFT feature quantities. Then, a scene including a “reference image” corresponding to the searched SIFT feature value is extracted. Hereinafter, the “reference image” included in the extracted scene is referred to as reference image I _data . A set of the feature point coordinates included in the image I _B1 or I _B2 and the feature point coordinates in the reference image I _data matched therewith is sent to the position / orientation conversion unit 11e. In addition, the image search unit 11f extracts “motion information” associated with the reference image I _data in the DB. Hereinafter, “motion information” associated with (associated with) the reference image I _data is referred to as motion information A. The motion information A is stored in the memory 11r. The image search unit 11f may detect a plurality of “reference images” that match the image I _B1 or I _B2 . In this case, pairs of feature point coordinates in all detected reference images I _data are sent to the position / orientation conversion unit 11e, and all detected motion information A is stored in the memory 11r. In addition, for a scene that does not include “information representing the object”, the image search unit 11f does not perform the matching process, and always extracts the scene regardless of the information representing the image I _B1 or I _B2. Scene motion information A is stored in the memory 11r.

≪Generation of position and orientation transformation matrix≫
The position / orientation conversion unit 11e receives the set of feature point coordinates output from the image search unit 11f, and uses the set of feature point coordinates to convert the image I _B1 or I _B2 into an image having the same posture and scale as the reference image I _data. A homography matrix (position and orientation conversion matrix) T to be converted is calculated. However, when the homography matrix T for converting the image I _B1 or I _B2 into an impossible posture (for example, a posture in which the image is reversed) is obtained, the matching in the image search unit 11f is misidentified. Subsequent processing may be canceled. The homography matrix T can be estimated by a well-known method using, for example, an OpenCV findHomography function. The generated homography matrix T is stored in the memory 11r. When the image search unit 11f detects a plurality of “reference images” that match the image I _B1 or I _B2 , the homography matrix T is calculated for each of a plurality of scenes including these and stored in the memory 11r. In addition, when the image search unit 11f extracts a scene that does not include “information representing the object”, the homography matrix T is obtained from the coordinates of the four corners of the projection region 110 set by the input operation to the setting unit 152d. Calculated as a transformation matrix into the coordinates of the four corners of the “reference image”.

≪Switching user interface screen≫
When the calculation of the homography matrix T is completed for all scenes matched by the image search unit, the information is sent to the video generation unit 11g. At the same time, the interface unit 11i gives an instruction to the display unit 11k, and the display unit 11k displays the user interface screen 100-3 (FIG. 5). As illustrated in FIG. 5, the user interface screen 100-3 includes an image 101, a projection area 110, an image area 121, a time axis information display unit 130, and a parameter setting unit 150. However, the projection area 110 included in the user interface screen 100-3 is an area in the captured image 101 corresponding to the reference image I _data extracted by the image search unit 11f, and the image area 121 is extracted by the image search unit 11f. Corresponds to motion information A. However, when the image search unit 11f extracts a scene that does not include “information representing the target”, the projection region 110 is an initially set region or a region set by an input operation to the setting unit 152d. In the parameter setting unit 150 of the user interface screen 100-3 illustrated in FIG. 4, setting units 161a to 161h, 162a to 162k, 162m, 162n, 162p to 162r, and 163a to 163f are displayed. The outline of the setting units 161a to 161h, 162a to 162k, 162m, 162n, 162p to 162r, 163a to 163f is as follows (FIG. 4).
Setting unit 161 a: Stops projection of “video” by the projection device 13. The projection of the “video” by the projection device 13 can be stopped by an input operation to the setting unit 161a.
Setting unit 161b: Restart of projection of “video” by the projection device 13. By the input operation of the setting unit 161b, the process is resumed from << start of the process of giving an apparent movement to the object 14 >>.
Setting unit 161c: ON / OFF switching of the display of the captured image. By an input operation to the setting unit 161c, it is possible to switch whether an image captured by the imaging device 12 is displayed from the display unit 11k in real time (ON) or not (OFF).
Setting unit 161d: temporarily stops projection of “video” by the projection device 13. The projection of the “video” by the projection device 13 can be temporarily stopped by an input operation to the setting unit 161a.
Setting unit 161e: Color compensation switching (ON / OFF). By the input operation of the setting unit 161e, it is possible to switch whether “video” includes a color component (ON) or not (OFF).
Setting unit 161f: Brightness compensation switching (ON / OFF). By an input operation to the setting unit 161f, it is possible to switch whether to project "video" considering the albedo of the object 14 (ON) or projecting "video" not considering the albedo of the object 14 (OFF). .
Setting unit 161g: Projects an albedo (ON / OFF). By an input operation to the setting unit 161f, it is possible to switch whether an image is projected (ON) or not (OFF) using an albedo of each pixel as a pixel value.
Setting unit 161h: Contrast (reflected on the whole) setting. An input operation to the setting unit 161h can amplify or attenuate the contrast of the entire “video”.
Setting unit 162a: Playback speed setting. The speed of time advance can be set by an input operation to the setting unit 162a.
Setting unit 162b: Sets the number of frames per loop of the entire motion information. The number of frames per loop of the entire motion information can be set by an input operation to the setting unit 162b.
Setting unit 162c: Creates a copy of the currently selected DW (image area). A copy of the selected DW can be added as a new DW by an input operation to the setting unit 162c.
Setting unit 162d: Selection of DW. A DW can be selected by inputting the ID of the DW by an input operation to the setting unit 162d.
Setting unit 162e: Setting of a frame at which movement in DW is started. By the input operation to the setting unit 162e, it is possible to set a frame in which a motion in the selected DW is started.
Setting unit 162f: Sets the number of continuous frames of motion in DW. By the input operation to the setting unit 162f, the number of continuous frames of motion in the selected DW can be set.
Setting unit 162g: Sets the number of frames in the fade-in section of the motion. By the input operation to the setting unit 162g, the number of frames in the fade-in section of the motion in the selected DW can be set.
Setting unit 162h: Setting of the number of frames in the motion fade-out section. By the input operation to the setting unit 162h, the number of frames in the fade-out section of the motion in the selected DW can be set.
Setting unit 162i: DW area change. The area of the currently selected DW can be changed by an input operation to the setting unit 162i.
Setting unit 162j: Setting of motion pattern ID (0-4). A motion pattern of the currently selected DW can be selected by an input operation to the setting unit 162j.
Setting unit 162k: Movement amplitude setting. By the input operation to the setting unit 162k, the amplitude of the movement of the selected DW can be set.
Setting unit 162m: Contrast setting of “video” read from the outside. Exception processing when an “image” that gives an apparent movement to the object 14 is read from the outside. The contrast of the “video” read from the outside can be set by an input operation to the setting unit 162m.
Setting unit 162n: Spatial frequency setting of motion video. The spatial frequency of the selected DW motion can be set by an input operation to the fixing unit 162n.
Setting part 162p: Speed setting of movement. The speed of movement of the selected DW can be set by an input operation to the setting unit 162p.
Setting unit 162q: setting of the angle of movement. The direction of movement of the selected DW can be set by an input operation to the setting unit 162q.
Setting unit 162r: Movement phase setting. The phase of the selected DW movement can be set by an input operation to the setting unit 162r.
Setting unit 163a: setting of sound effect ID (where ID = -1 is no sound effect). A sound effect can be set to the selected DW by an input operation to the setting unit 163a.
Setting unit 163b: Deletes the selected DW. The selected DW can be deleted by an input operation to the setting unit 163b.
Setting unit 163c: Advances the display scene. The scene number i of the motion editing target scene is incremented by 1 by an input operation to the setting unit 163c.
Setting unit 163d: Returns the display scene to the front. By the input operation to the setting unit 163d, the scene number i of the motion editing target scene is decreased by one.
Setting unit 163e: Saves the current state. The set information is saved by an input operation to the setting unit 163e.
Setting unit 163f: End of application.

  The user performs an input operation to each setting unit using the input unit 11j to set each parameter or switch functions. In the setting section where a parameter can be set, the current value of the parameter is displayed. When a parameter is updated by an input operation to the setting unit, the updated parameter is displayed on the setting unit. The image 101 acquired by the imaging device 12 is displayed on the user interface screen 100-3. However, after starting the process of giving an apparent motion to the object 14, a real-time live image is used to reduce the burden. Acquisition may be stopped, and the image 101 acquired last by the imaging device 12 may be displayed instead. However, this is not indispensable, and the imaging device 12 may acquire a live image and display it on the user interface screen 100-3 even after the “process for giving an apparent movement to the object 14” is started. As described above, the acquisition of the live video can be started or stopped by an input operation to the setting unit 161c.

≪Video generation≫
The video generation unit 11g generates a video P that is superimposed on the object 14. When the image search unit 11f has extracted a plurality of scenes (hereinafter referred to as “scene extracted by the image search unit 11f”), the following procedures (a) to (c) are executed for all of them. (A) First, the video generation unit 11g reads the homography matrix T corresponding to the scene from the memory 11r, and receives the image I _B1 or I _B2 (hereinafter referred to as the homography matrix T received from the projection target image acquisition unit 11c). , and _{I B} for simplicity apply to), and stores the obtained image I _'B obtained by converting the image _{I B} in the same attitude scale image and the reference image _{I data} in the memory 11r. (B) then the image generation unit 11g applies from the memory 11r, the motion information A tied to the reference image _{I data,} and 'read the _B, and the motion information A image I' image I to _B, A video I ′ _D that is a moving image is obtained. For example, the image generation unit 11g for all distortion window DW of the motion information A, the motion vector of each pixel of the image I _'B is calculated based on the motion information A, then, based on their motion vector image I to obtain a _D 'moves each pixel of the _B image I by'. The video I ′ _D is a moving image in which a motion corresponding to the motion information A is given to the region represented by the distortion window DW. The video I ′ _D is stored in the memory 11r. (C) Furthermore, the video generation unit 11g reads the video I ′ _D and the homography matrix T from the memory 11r, applies the inverse transformation T ⁻¹ of the homography matrix T to the video I ′ _D , and converts the video I ′ _D to _Returning to the coordinate system of the imaging device 12, the video _IDs are obtained. Video _{I Ds} is stored in the memory 11r. By performing the above steps (a) to (c) for all the scenes extracted by the image search unit, one video _ID in which videos of all the scenes are drawn in the coordinate system of the imaging device 12 is obtained. Finally, the video generation unit 11g reads the video I _D , the original image I _B and the albedo K from the memory 11r, obtains the following video P, and outputs the video P to the memory 11r.
_{P = w (I D -I B} ) / K + B (1)
Here, B represents the brightness of the uniform light that has been output from the projection device 13 when taking the I _B (pixel value). The reason why B is added is to make the image P a positive value. If B is not added, a negative image P may be emitted. The reason why w (I _D −I _B ) is divided by K is to obtain an image P having an optimum or near contrast in consideration of the reflectance of the object 14 or the like. If the photographing device 12 and the projection device 13 are optically linear, an image P having an optimum contrast can be obtained. However, K = 1 may be set (same as not dividing by K). The user performs an input operation to the setting unit 1621f (FIG. 4), whether K is set to a value obtained by the albedo calculation unit 11d (ON: reflectivity is considered) or K = 1 (OFF: reflection). Can be switched). The setting contents in the setting unit 161f are sent from the interface unit 11i to the video generation unit 11g and used for the calculation of the above formula (1) in real time. Also, information on whether K is a value obtained by the albedo calculation unit 11d (ON) or K = 1 (OFF) is displayed on the setting unit 161f (FIG. 4). W is a positive parameter for adjusting the contrast. Although the initial value of w is 1.0, the user can input the value of w by an input operation to the setting unit 161h (setting the contrast of the image P and reflecting it on the entire image P) (FIG. 4). The input value of w is sent from the interface unit 11i to the video generation unit 11g and used for the calculation of the above equation (1) in real time. The value of w is displayed on the setting unit 161h. The video P is stored in the memory 11r.

«Projection of the superimposed image generation and image P _C of the object 14»
Superimposed image generation unit 11b converts, from the memory 11r, reads the image P is P → C maps and videos, to apply the P → C mapped to video P, the image P on the image P _C of the coordinate system of the projection device 13 and sends the image _{P C} to the projector 13. Superimposed image generation unit 11b repeats the reproduction of the video P _C until there is some indication, it continues continuously sends image P _C to the projector 13. Projection device 13 is projected onto the object 14 image _{P C,} thereby superimposed on the object 14 image _{P C.} User viewed object 14 to the image P _C is superimposed is perceived as motion apparent in the region of the object 14 (object region) corresponding to the distortion window DW is given.

≪Selection of display scene≫
If the image retrieval section 11f has extracted a plurality of scenes, for all of which the process of the «projected to the object 14 in the video generation »« superimposed image generation and image P _C» described above is performed. The user selects scenes corresponding to the projection area 110 and the image areas 120 and 121 displayed on the user interface screen 100-3 from the scenes extracted by the image search unit 11f by an input operation to the setting units 163c and 163d. it can. When an input operation to the setting units 163c and 163d is performed, the scene number i of the selected scene is sent to the motion editing unit 11h. The motion editing unit 11h reads the coordinates of the outer frame of the reference image I _data of the scene corresponding to the sent scene number i and the coordinates of each DW of the motion information A from the memory 11r and sends them to the interface unit 11i. Using this information, the interface unit 11i sends the projection region 110 and the image region 121 corresponding to the scene number i to the display unit 11k, and the display unit 11k displays them on the user interface screen 100-3 (FIG. 5). ).

This process will be described in more detail. Immediately after << switching the user interface screen >>, the scene having the smallest scene number i is selected from the scenes extracted by the image search unit 11f. Each time the user performs an input operation on the setting unit 163c, the scene number i is incremented, and every time an input operation is performed on the setting unit 163d, the scene number i is incremented. The motion editing unit 11h stores the coordinates of the outer frame of the reference image I _data corresponding to the scene number i selected in this way and the coordinates of each distortion window DW of the motion information A corresponding to the scene number i from the memory 11r. Read and send to interface unit 11i. The interface unit 11i receives from the motion editing unit 11h using the inverse matrix T- ¹ of the homography matrix corresponding to the selected scene number i among the homography matrices T obtained by the position / orientation conversion unit 11e. The coordinates are converted into the coordinate system of the imaging device 12 and sent to the display unit 11k. The display unit 11k displays the outer frame of the reference image I _data after coordinate conversion as the projection area 110 on the user interface screen 100-3, and displays the area of the distortion window DW after the coordinate conversion as the image area 121 ( FIG. 5). In the time axis information display unit 130, a horizontal straight line 131 representing the time range in the scene of the image area 121 is displayed, and a bar 140 representing the current time position of the scene is displayed.

≪Motion Edit≫
Not performed editing video P _C, when the motion information A is the initial state, the motion editing unit 11h from the memory 11r, setting unit 163c, corresponds to a scene selected by the input operation to 163d The coordinates of the outer frame of the reference image I _data and the motion information A are read. The motion editing unit 11h sends the coordinates of the outer frame of the reference image I _data corresponding to the currently selected scene and the coordinates of the distortion window DW of the motion information A to the interface unit 11i as described above. The interface unit 11i converts the coordinates received from the motion editing unit 11h into the coordinate system of the imaging device 12 using the inverse matrix T- ¹ of the homography matrix obtained by the position / orientation conversion unit 11e as described above, Send to display 11k. As described above, the display unit 11k displays the outer frame of the reference image I _data after coordinate conversion as the projection area 110 on the user interface screen 100-3, and displays the area of the distortion window DW after the coordinate conversion as the image area 121. (FIG. 5). The image area 121 is an area included in the projection area 110. The image area 121 is represented by, for example, a polygon having colored sides. In the example of FIG. 5, a rectangular image area 121 is displayed in a partial area of the projection area 110. In the time axis information display unit 130, a horizontal straight line 131 of the same color as the image area 121 is displayed, and a bar 140 representing the current time position of the selected scene is displayed. The straight line 131 represents the position on the time axis of the movement given to the image area 121 (in other words, the position on the time axis of the apparent movement given to the target area of the object 14 corresponding to the image area 121). The bar 140 is arranged orthogonal to the straight line 131, and its position changes according to the current time position of the selected scene. In the example of FIG. 5, the bar 140 moves in the τ direction along the straight line 131 as the scene time advances. Thus, the user can know which time position in the entire scene corresponds to the movement currently given to the object, and which image area gives the movement.

User interface screen 100-3 is visible by the user, and a user interface for setting the information for generating an image P _C to be superimposed on the target object 14 is the user. User interface screen 100-3 with user image area 121 presents (displayed) as visible included in the image of the object 14 receives input of designation information for generating an image P _C. The functions of the user interface screen 100-3 are realized by the interface unit 11i, the input unit 11j, and the display unit 11k. The input designation information is sent to the motion editing unit 11h. The motion editing unit 11h updates the motion information A associated with the aforementioned reference image I _data stored in the DB unit 11m according to the designation information, and stores the updated motion information A in the memory 11r. To do. When the motion information A is updated, based on the new motion information A, the process line of «selection of projection »« display scene to the object 14 in the video generation »« superimposed image generation and image P _C» described above Is called. Thus, updates to the motion information A is reflected in the image P _C to be projected to the object 14 in real time. Therefore, the user, with reference to the user interface screen 100-3, enter the designation information using the input unit 11j, to edit the image P _C to be superimposed on the object 14. Hereinafter, this process will be specifically described.

≪Editing parameters related to the operation of the entire scene≫
The user performs an input operation from the input unit 11j to the setting units 162a and 162b in accordance with the motion editing content, and designates the scene playback speed and the number of frames per scene loop. The designated contents are displayed on the setting sections 162a and 162b. The designation information input by these input operations is sent from the interface unit 11i to the video generation unit 11g. The video generation unit 11g changes the playback speed of the scene and the number of frames until the scene loops in real time according to the designation information. The video P generated based on the updated parameters are sent to the processing «projection onto superimposed image generation and image P _C of the object 14» above. Thus, the user interface screen 100-4 to the displayed editing result is reflected on the image P _C to be superimposed on the target object 14 in real time.

≪Editing image area / distortion window DW≫
The user can add, move, change, and delete an image area that gives an apparent movement included in the projection area. In synchronization with this, a target area (corresponding to the distortion window DW) of the target object 14 to which an apparent movement is given is added, changed, moved, or deleted.

When adding an image area, the user uses the input unit 11j to perform an input operation (for example, click or tap on the image area 121) to the image area 121 displayed on the user interface screen 100-3. Select. The setting unit 162d displays the ID of the distortion window DW corresponding to the selected image area 121. The user can also select the image area 121 by directly inputting the ID of the distortion window DW to the setting unit 162d. Next, the user performs an input operation to the setting unit 162c. The designation information input by the input operation is sent from the interface unit 11i to the motion editing unit 11h. In response to this designation information, the motion editing unit 11h duplicates the image area 122 having the same shape and posture as the image area 121, and further generates a straight line 132 having the same color as the image area 122. Furthermore, the motion editing unit 11h updates the distortion window DW of the motion information A associated with the above-described reference image I _data stored in the DB unit 11m according to the added image region 122. In other words, the motion editing unit 11h duplicates a new distortion window DW having the same shape and posture as the distortion window DW corresponding to the image area 121 that is the copy source. The motion corresponding to the added distortion window DW is the same as the motion corresponding to the original distortion window DW until there is further motion editing. The image area 122 and the straight line 132 are sent from the interface unit 11i to the display unit 11k. The display unit 11k displays a user interface screen 100-4 to which a new image area 122 is added. A straight line 132 corresponding to the image region 122 is added to the time axis information display unit 130 of the user interface screen 100-4 (FIG. 6).

When moving the position of the image area 122, the user performs an input operation (for example, dragging or swiping the image area 122) to the image area 122 displayed on the user interface screen 100-4 using the input unit 11j. The designation information input by this input operation is sent to the interface unit 11i, and the coordinates of the image area 122 are moved according to this designation information. The interface unit 11i uses the homography matrix T obtained by the position / orientation conversion unit 11e to change the coordinates of the moved image region 122 from the coordinate system of the photographing apparatus 12 to the coordinate system of the reference image of the scene in the DB. And sent to the motion editing unit 11h. The motion editing unit 11h updates the distortion window DW of the motion information A associated with the reference image I _data stored in the DB unit 11m in accordance with the movement of the coordinates of the image region 122. That is, the motion editing unit 11h moves the distortion window DW corresponding to the image area 122 in accordance with the movement of the image area 122. The movement corresponding to the distortion window DW after movement is the same as the movement corresponding to the distortion window DW before movement until further movement editing is performed. The coordinates of the image area 122 after movement are sent from the interface unit 11i to the display unit 11k, and the display unit 11k displays the image area 122 of the new coordinates on the user interface screen 100-4 (FIG. 6).

When transforming the image area 123 obtained by copying / moving the image area 122, first, the user uses the input unit 11j to perform an input operation on the image area 123 displayed on the user interface screen 100-3 (for example, The image area 123 is selected by clicking or tapping the image area 121. The setting unit 162d displays the ID of the distortion window DW corresponding to the selected image area 123. The user may select the image region 123 by directly inputting the ID of the distortion window DW into the setting unit 162d. Next, the user performs an input operation to the setting unit 162i using the input unit 11j and selects the change of the image region, and then specifies the image region 123 after the change on the user interface screen 100-4. Perform input operations. An example of this input operation is an operation of designating the apexes of the updated image region 123 by left clicking or the like and confirming the designation by right clicking. The designation information input by these input operations is sent to the interface unit 11i, and the image area 123 is deformed according to the designation information. The interface unit 11i uses the homography matrix T obtained by the position / orientation conversion unit 11e to change the coordinates of the deformed image region 123 from the coordinate system of the photographing apparatus 12 to the coordinate system of the reference image of the scene in the DB. Converted and sent to the motion editing unit 11h. The motion editing unit 11h updates the distortion window DW of the motion information A associated with the above-described reference image I _data stored in the DB unit 11m according to the transformation of the coordinates of the image region 123. That is, in accordance with the deformation of the image area 123, the motion editing unit 11h also deforms the distortion window DW corresponding to the image area 123. The motion corresponding to the post-deformation distortion window DW is the same as the motion corresponding to the pre-deformation distortion window DW until further motion editing is performed. The transformed image area 123 is sent from the interface unit 11i to the display unit 11k, and the display unit 11k displays the transformed image area 123 on the user interface screen 100-4. In the time axis information display unit 130 of the user interface screen 100-4, a straight line 133 corresponding to the image area 123 is displayed when the image area 123 is copied (FIG. 6).

When deleting the image area 121, first, the user selects an image area 121 by performing an input operation on the image area 121 displayed on the user interface screen 100-3 using the input unit 11j. The setting unit 162d displays the ID of the distortion window DW corresponding to the selected image area 121. The user may select the image area 121 by directly inputting the ID of the distortion window DW to the setting unit 162d. Next, the user performs an input operation to the setting unit 163b using the input unit 11j. The designation information input by this input operation is sent from the interface unit 11i to the motion editing unit 11h. The motion editing unit 11h deletes the image area 121 according to the designation information. Furthermore, the motion editing unit 11h updates the distortion window DW of the motion information A associated with the above-described reference image I _data stored in the DB unit 11m in response to the deletion of the image area 121. That is, the motion editing unit 11h deletes the distortion window DW corresponding to the deleted image area 121. These pieces of information are sent from the interface unit 11i to the display unit 11k, and the display unit 11k displays the user interface screen 100-4 from which the image area 121 is deleted.

Adding more image areas, moving, changing, based on the motion information A that has been updated with the deletion, the process of the «projected to the object 14 in the video generation »« superimposed image generation and image P _C» described above Done. Thus, updates to the motion information A is reflected on the image P _C to the object 14 in real time, the modified image P _{C (updated} image) is projected onto the object 14. That is, as a result edited displayed on the user interface screen 100-4 is reflected in the image P _C to be superimposed on the target object 14 in real time. In the example of FIG. 6, the bar 140 moves in the τ direction along the straight lines 131 to 133 as the scene time advances. Thus, the position on the time axis of the video P _C superimposed on the object 14 can be visually represented.

  As described above, the user interface screen 100-4 functions as a user interface for the user to set the area of the object 14, and presents the image of the object 14 so that the user can visually recognize the image. An input of first designation information for specifying an image region included in the image of the object 14 is received. The function of the user interface screen 100-4 is also realized by the interface unit 11i, the input unit 11j, and the display unit 11k.

≪Editing how to move image area / distortion window DW≫
The user can edit how the image area displayed on the user interface screen 100-3 or 4 moves. In synchronization with this, the apparent movement given to the target region (corresponding to the distortion window DW) of the target object 14 can also be edited. When editing the movement method given to the image area 121 and the distortion window DW corresponding to the image area 121, first, the user uses the input unit 11j to perform an input operation to the image area 121 displayed on the user interface screen 100-3 or 4. Go to select the image area 121. The setting unit 162d displays the ID of the distortion window DW corresponding to the selected image area 121. The user may select the image area 121 by directly inputting the ID of the distortion window DW to the setting unit 162d. Next, the user uses the input unit 11j to perform input operations to the setting units 162j to 162r in accordance with the motion editing content, so that the motion pattern, motion amplitude, contrast, spatial frequency, speed, angle, phase, etc. Specify parameters. The designated contents are displayed on the setting sections 162j to 162r. The designation information input by these input operations is sent from the interface unit 11i to the motion editing unit 11h. The motion editing unit 11h changes the motion of the image area 121 according to the designation information. Furthermore, the motion editing unit 11h updates the motion information A associated with the above-described reference image I _data stored in the DB unit 11m in accordance with a change in the motion of the image region 121. That is, the motion editing unit 11 h changes the motion of the distortion window DW corresponding to the image region 121 in accordance with the change of the motion of the image region 121. Also based on the updated motion information A, the processing of the «projected to the object 14 in the video generation »« superimposed image generation and image P _C» described above is performed. Thus, the user interface screen 100-4 to the displayed editing result is reflected on the image P _C to be superimposed on the target object 14 in real time. Also by the display of the bar 140, the position on the time axis of the video P _C superimposed on the object 14 can be represented visually (Figure 6).

  As described above, the user interface screens 100-3 and 4 function as a user interface for the user to set the movement to be given to the image of the target area that is the area of the target object 14, and are included in the image of the target object 14. In addition to presenting the parameters corresponding to the movement given to the image area and the image of the target area so that the user can visually recognize, the input of the second designation information specifying the parameter corresponding to the movement is accepted.

≪Editing the time range of movement given to the image area / distortion window DW≫
The user can edit the movement time range set in the image area displayed on the user interface screen 100-3 or 4. In synchronization with this, the apparent motion time range given to the target area (corresponding to the distortion window DW) of the target object 14 can also be edited. In this embodiment, it is possible to edit the frame that starts the motion in the distortion window DW, the number of continuous frames of motion in the DW, the number of frames in the motion fade-in section, and the number of frames in the motion fade-out section. When editing the time range of movement of the image area 121, first, the user uses the input unit 11j to perform an input operation on the image area 121 displayed on the user interface screen 100-3 or 4 to display the image area 121. select. The setting unit 162d displays the ID of the distortion window DW corresponding to the selected image area 121. The user may select the image area 121 by directly inputting the ID of the distortion window DW to the setting unit 162d. Next, the user uses the input unit 11j to perform an input operation to the setting units 162e to 162h according to the editing content of the time range, and to start the frame in which the movement in the distortion window DW corresponding to the image region 121 is started, Specify parameters such as the number of frames, the number of frames in the motion fade-in section, and the number of frames in the motion fade-out section. The designated content is displayed on each setting unit 162e to 162h. The designation information input by these input operations is sent from the interface unit 11i to the motion editing unit 11h. The motion editing unit 11h sets a time range of motion of the image area 121 (disclosure point, end point, fade-in period, fade-out period) and the like according to the designation information. Furthermore, the motion editing unit 11h updates the motion information A associated with the above-described reference image I _data stored in the DB unit 11m, according to the motion time range of the image region 121. That is, the motion editing unit 11h changes the time range of motion of the distortion window DW corresponding to the image region 121 according to the time range of motion of the image region 121. Also based on the updated motion information A, the processing of the «projected to the object 14 in the video generation »« superimposed image generation and image P _C» described above is performed. Thus, the user interface screen 100-4 to the displayed editing result is reflected on the image P _C to be superimposed on the target object 14 in real time. Also by the display of the bar 140, the position on the time axis of the video P _C superimposed on the object 14 can be represented visually (Figure 6).

As described above, the user interface screen 100-3,4 functions as a user interface for setting the user a time range of motion that gives the image of the target area of the object 14, the motion picture P _C gives position on the time axis, and with user parameters corresponding to a time range of motion is presented to visible video P _C gives the input of the third specifying information for specifying a parameter corresponding to a time range of motion Accept.

≪Sound effect setting≫
The user can also set sound effects for the movement of the image area. When sound effects are set for the movement of the image area 121 displayed on the user interface screen 100-3 or 4, first, the user uses the input unit 11j to display the image area displayed on the user interface screen 100-3 or 4 The image area 121 is selected by performing an input operation on the screen 121. The setting unit 162d displays the ID of the distortion window DW corresponding to the selected image area 121. The user may select the image area 121 by directly inputting the ID of the distortion window DW to the setting unit 162d. Next, the user uses the input unit 11j to perform an input operation to the setting unit 163a to specify the sound effect ID (non-negative integer). When no sound effect is set, ID = -1. The designated ID is displayed on the setting unit 163a. The designation information input by these input operations is sent from the interface unit 11i to the motion editing unit 11h. The motion editing unit 11h sets a sound effect according to the time range of motion of the image area 121 according to the designation information. That is, a setting is made to output a sound effect only in a time range in which the movement of the image area 121 is given. It is also possible to change the volume of the sound effect according to the fade-in / fade-out of the movement.

≪Save settings≫
The user can save the edited result, that is, the edited motion information A, in the DB unit 11m by performing an input operation to the setting unit 163e. At the next startup, the motion information A edited from the DB unit 11m is read, and the above-described processes are executed.

[Second Embodiment]
In the first embodiment, it projected on the object 14 an image P _C by a projection device 13, and superimposes the image P _C to the object 14. However, it may be superimposed the image P _C displayed on the transmissive display to the image of the object 14.

As illustrated in FIG. 7, the video system 2 of the present embodiment includes the video generation device 11, the imaging device 12, and the transmissive display 23 (video presentation unit) described in the first embodiment. The object 14 is fixed to the wall 23 and is disposed between the wall 23 and the transmissive display 23. Image P _C described in the first embodiment is displayed on transmissive display 23. The user views the object 14 through the transmissive display 23 in the U direction. The user through display 23 image P _C displayed on the perceived image superimposed on the object 14, thereby perceived as the movement of apparent object 14 is given.

[Other variations]
The present invention is not limited to the embodiment described above. For example, the functions of the video generation device 11 may be distributed and arranged on the network.

  In the above-described embodiment, the motion information is stored in the DB of the DB unit 11m as an initial state. However, in the initial state, the motion information is not stored in the DB, and the motion information corresponding to the distortion window DW and the motion content may be stored in the DB after << distortion editing >> is specified. Alternatively, the motion information stored in the DB of the DB unit 11m as an initial state includes information on a plurality of predetermined distortion windows DW and a distortion spatial range, a distortion time range corresponding to each distortion window DW, It may include information on the type of distortion basis function, amplitude, spatial frequency, speed, angle, phase, and sound effect ID.

In the first embodiment, the relative position (relative coordinates) between the imaging device 12 and the projection device 13 may be determined in advance. Thereby, calibration can be omitted. Also, it may be a video image P _C is composed of only the luminance motion component (e.g., see Patent Document 1). In this case, the video P may be generated by the following equation instead of the equation (1).
_{P = w (Y D -Y B} ) / K + B (2)
However, Y _D is a moving image consisting of only a luminance component of the image I _D, Y _B is an image consisting of only a luminance component of the image I _B.

Other video used for general projection mapping may be a P _C.

In the first embodiment, in the «image search», if the image I conforming reference image I _data to _B is more detected, the homography matrix T for all the reference images I _data that has been detected is estimated, respectively Video I _Ds was generated for the scene corresponding to the reference image I _data . However, if a matching reference image I _data to the image I _B is more detected, the user of the scene to be processed later than it may be selectable.

  In the above-described embodiment, the interface unit 11i, the input unit 11j, and the display unit 11k are user interfaces (designation units) that receive input of information from the user and present information to the user. . However, at least a part of the information displayed from the display unit 11k described above may be projected from the projection device 13. In that case, the projection device 13 may be a part of the user interface (designation unit). Further, when all information displayed by the display unit 11k is projected from the projection device 13, the display unit 11k may be omitted. In this case, the interface unit 11i, the input unit 11j, and the projection device 13 serve as a user interface (designation unit).

  The various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. Needless to say, other modifications are possible without departing from the spirit of the present invention.

  When the above configuration is realized by a computer, the processing contents of the functions that each device should have are described by a program. By executing this program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. An example of a computer-readable recording medium is a non-transitory recording medium. Examples of such a recording medium are a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, and the like.

  This program is distributed, for example, by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own storage device, and executes a process according to the read program. As another execution form of the program, the computer may read the program directly from the portable recording medium and execute processing according to the program, and each time the program is transferred from the server computer to the computer. The processing according to the received program may be executed sequentially. The above-described processing may be executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by an execution instruction and result acquisition without transferring a program from the server computer to the computer. Good.

  In the above embodiment, the processing functions of the apparatus are realized by executing a predetermined program on a computer. However, at least a part of these processing functions may be realized by hardware.

1, 2 Video generation system 11 Video generation device

Claims (11)

A video generation device that generates a video to be superimposed to give an apparent movement to an object,
It is a user interface for a user to set information for generating the video, and presents the image area included in the image of the target object so that the user can visually recognize, and generates the video. A designation unit that accepts input of designation information for
A video generation unit that generates the video corresponding to the designation information received by the designation unit;
In order for the user to visually recognize a state in which the target area that is the target object area and the video generated by the video generation unit corresponding to the designation information received by the designation unit are superimposed, A video presentation unit that presents the video corresponding to the designation information received by the designation unit;
In a state where the user can visually recognize a state in which the target region that is the target object region and the video generated by the video generation unit corresponding to the designation information received by the designation unit are superimposed, It is possible to present the image area included in the image of the object in the designation unit and accept an update input of the designation information for generating the video,
The video generation device further includes a database unit and an image search unit,
In the database unit, information including a set of information that can specify a registration target and motion information that specifies a motion of an image area included in the image of the registration target is stored,
The image search unit includes, from the database unit, information that can identify the registration target that matches the image of the target, and an image area included in the image of the registration target that matches the image of the target Extracting the movement information for identifying movement;
The video generation unit is configured to store the image area included in the image of the object based on parameters obtained from the image of the object and information that can identify the registration object extracted by the image search unit. On the other hand, the video is generated by giving a motion based on the motion information extracted by the image search unit, and the designation information specifies a parameter corresponding to the motion,
The video generation unit updates the video by giving a motion corresponding to the parameter specified by the designation information to the image area included in the image of the target object when the designation information is updated. , Video generation device. A video generation device that generates a video to be superimposed to give an apparent movement to an object,
It is a user interface for a user to set an area of the object, presents the image of the object so that the user can visually recognize, and specifies an image area included in the image of the object A designation unit that receives input of the first designation information;
A video generation unit that generates the video corresponding to the image area specified by the first specification information received by the specification unit;
The target area corresponding to the image area specified by the first designation information and the video generation corresponding to the image area identified by the first designation information received by the designation unit The video generation unit corresponding to the image area specified by the first designation information received by the designation unit so that the user can visually recognize a state in which the video generated by the unit is superimposed on the video image. A video presentation unit for presenting the generated video,
A state in which a target region that is a region of the target object and the video generated by the video generation unit corresponding to the image region specified by the first specification information received by the specification unit are superimposed In a state where the user can visually recognize, it is possible to present the image area included in the image of the object in the designation unit and to accept input of the first designation information,
The video generation device further includes a database unit and an image search unit,
In the database unit, information including a set of information that can specify a registration target and motion information that specifies a motion of an image area included in the image of the registration target is stored,
The image search unit includes, from the database unit, information that can identify the registration target that matches the image of the target, and an image area included in the image of the registration target that matches the image of the target Extracting the movement information for identifying movement;
The image generation unit is configured to identify the image area specified by the first designation information based on a parameter obtained from an image of the object and information that can identify the registration target extracted by the image search unit. In contrast, generating the video by giving a motion based on the motion information extracted by the image search unit,
The designating unit is a user interface for the user to set a movement to be given to the image of the target area, and the user visually recognizes a parameter corresponding to the movement to be given to the image area and the image of the target area. Presenting as possible and accepting an update input of the second designation information specifying the parameter corresponding to the movement,
In response to the update input of the second designation information, the video generation unit uses the second designation information received by the designation unit in the image area specified by the first designation information received by the designation unit. Updating the video with a movement corresponding to the identified parameter;
The video presenting unit is specified by the second designation information received by the designation unit in the target area that is the area of the target object and the image area specified by the first designation information received by the designation unit. In the image area identified by the first designation information received by the designation unit, the image area giving the movement corresponding to the parameter to be viewed can be visually recognized by the user, Presenting the video updated by the video generation unit, giving a motion corresponding to the parameter specified by the second designation information received by the designation unit;
The target area that is the area of the target object and the image area that is specified by the first designation information received by the designation section correspond to the parameters that are identified by the second designation information received by the designation section. Presentation of parameters corresponding to the movement in the designation unit and the second designation information in a state where the user can visually recognize a state in which the video and the video updated by the video generation unit are superimposed. An image generation device capable of accepting an input of. The video generation device according to claim 2,
The designation unit is a user interface for the user to set a time range of movement given to the image of the target area, and a position on the time axis of the movement given by the video and a movement given by the video Presenting the parameter corresponding to the time range so as to be visible to the user, accepting the input of the third designation information specifying the parameter corresponding to the time range of the movement,
The video generation unit moves to the image area specified by the first specification information received by the specification unit in a time range corresponding to the parameter specified by the third specification information received by the specification unit. Generating the video,
The video presenting unit is identified by the third designation information received by the designation unit in the target area that is the area of the target object and the image area identified by the first designation information received by the designation unit. The designation unit is configured so that the user can visually recognize the superimposed state of the video that gives motion in the time range corresponding to the parameter and the position on the time axis of the motion given by the video. Presenting the video giving motion in a time range corresponding to the parameter specified by the third specification information received by the specification unit in the image region specified by the received first specification information;
Corresponds to the target area that is the area of the object and the parameter specified by the third designation information received by the designation section in the image area that is identified by the first designation information received by the designation section. The designation is made in a state where the user can visually recognize the superimposed state of the video generated by the video generation unit and the position on the time axis of the motion given by the video. A video generation device capable of presenting a parameter corresponding to the time range of movement in the unit and accepting input of the third designation information. The video generation device according to claim 2 or 3,
The video generation apparatus, wherein the video is a luminance motion component representing a motion given to an image of the target area. The video generation device according to any one of claims 1 to 4,
The parameter corresponding to the motion represents one or more of a motion pattern, motion amplitude, motion video contrast, motion video spatial frequency, motion speed, motion angle, and motion phase. apparatus. The video generation device according to any one of claims 1 to 5,
The information that can specify the registration object includes a feature amount of the image of the registration object and a reference image of the registration object,
The registered object that matches the image of the object is the registered object that includes at least a predetermined number of feature points having feature quantities that match or approximate the feature quantities of the image of the object,
The video generation unit
Converting the image of the object into a converted image having the same posture and scale as the reference image of the registered object extracted by the image search unit;
Giving the converted image a motion based on the motion information extracted by the image search unit to generate a converted video;
A video generation device that reversely converts the converted video into the video having the same posture and scale as the image of the object. The video generation device according to any one of claims 1 to 6,
The image of the object is obtained by photographing the object on which projection light is projected,
An albedo calculation unit for calculating an albedo representing a ratio of an increment of a pixel value of an image of the object to an increase in intensity of the projection light;
The video generation device is further configured to generate the video based on the albedo. The video generation device according to any one of claims 1 to 7,
The designation unit is a user interface for allowing the user to set an average luminance of the video, presents the average luminance so that the user can visually recognize, and sets a parameter corresponding to the average luminance. Accepts the input of the fourth specified information to identify,
The video generation unit is configured to generate the video having an average luminance corresponding to the parameter specified by the fourth designation information received by the designation unit. An image generation method for generating an image to be superimposed to give an apparent movement to an object,
(A) A designation unit, which is a user interface for a user to set information for generating the video, presents an image area included in the image of the object so that the user can visually recognize the image area. Receiving an input of designation information for generating the video;
(B) a video generation unit generating the video corresponding to the designation information received by the designation unit;
(C) The video presentation unit uses a state in which a target region that is a region of the target object and the video generated by the video generation unit corresponding to the designation information received by the designation unit are superimposed on each other Presenting the video corresponding to the designation information received by the designation unit so that a person can visually recognize,
(D) The user can visually recognize a state in which the target area that is the target object area and the video generated by the video generation unit corresponding to the designation information received by the designation unit are superimposed. In the state, when the presentation of the image area included in the image of the object in the designation unit and the input of the designation information for generating the video are accepted, the received designation information is immediately reflected The generated image as the updated image by the image generation unit;
(E) The video presentation unit is configured so that the user can visually recognize a state in which the target region that is the region of the target object and the updated video generated by the video generation unit are superimposed. Presenting the updated video, and
The updated video generated by the video generation unit corresponding to the target area that is the target object area and the designation information received by the designation unit when the video presentation unit presents the updated video. And the user can see the superimposed state,
The database unit stores information including a set of information that can specify the registration target and motion information that specifies the motion of the image area included in the image of the registration target,
The step (b)
(B-1) The image search unit includes, from the database unit, information that can identify the registration target that matches the target image and the registration target image that matches the target image. Extracting the motion information identifying the motion of the image area to be
(B-2) Based on parameters obtained from the image of the object and information that can identify the registered object extracted in the step (b-1), the video generation unit Generating the video by giving a motion based on the motion information extracted in the step (b-1) to the image region included in the image,
The designation information includes information for specifying a parameter corresponding to movement,
The video generation method includes:
(F) The specification unit, which is a user interface for the user to set the movement to be given to the image of the target area, allows the user to set parameters corresponding to the movement to be given to the image area and the image of the target area. Including the step of presenting the information so as to be visually recognized and receiving an update input of the specified information for specifying a parameter corresponding to the movement,
The step (d) corresponds to a parameter specified by the designation information received in the step (f) with respect to the image area included in the image of the object, triggered by an update input of the designation information. An image generation method including a step of giving a motion and updating the image generated in the step (b) to the updated image. An image generation method for generating an image to be superimposed to give an apparent movement to an object,
(A) A designation unit that is a user interface for a user to set a region of the object presents the image of the object so that the user can visually recognize the image, and is included in the image of the object Receiving input of first designation information for specifying an image area to be displayed;
(B) a step of generating a video corresponding to the image region identified by the first designation information received by the designation unit;
(C) The video presenting unit is specified by a target region that is a region of the target object corresponding to the image region specified by the first specifying information, and the first specifying information received by the specifying unit. The image area specified by the first designation information received by the designation unit so that the user can visually recognize a state in which the video generated by the video generation unit corresponding to the image area is superimposed. Presenting the video generated by the video generation unit corresponding to
(D) The target area which is the area of the target object and the video generated by the video generation unit corresponding to the image area specified by the first specification information received by the specification unit are superimposed. In a state where the user can visually recognize the state, it is possible to present the image area included in the image of the object in the designation unit and to accept input of the first designation information,
The database unit stores information including a set of information that can specify the registration target and motion information that specifies the motion of the image area included in the image of the registration target,
The step (b)
(B-1) images the search unit, from the database unit, said a registration object identifiable information that match the image of the object, the image of the registration object that matches the image of the object Extracting the motion information identifying the motion of the included image region;
(B-2) The video generation unit is specified by the first designation information based on a parameter obtained from an image of the target and information capable of specifying the registration target extracted by the image search unit. Generating the video by giving a motion based on the motion information extracted in the step (b-1) to the image region, wherein the video generation method further includes (e) the target The designating unit, which is a user interface for the user to set a motion to be given to the image of the region, allows the user to visually recognize parameters corresponding to the motion to be given to the image of the image region and the target region. Presenting and receiving an update input of second designation information specifying a parameter corresponding to the movement;
(F) The video generation unit accepts in the step (e) the image area specified by the first designation information accepted in the step (a) when the second designation information is updated. Updating the video by giving a motion corresponding to the parameter specified by the second designation information;
(G) The video presentation unit has received the target area that is the target area and the image area specified by the first designation information received in the step (a) in the step (e). Identified by the first designation information received in the step (a) so that the user can visually recognize the superimposed state of the video that gives a motion corresponding to the parameter specified by the second designation information. Presenting the image updated in step (f) to give a motion corresponding to the parameter specified by the second designation information received in step (e) to the image region to be Have
The target area, which is the area of the object, and the image area specified by the first specification information received in step (a) are specified by the second specification information received in step (e). Presenting the parameter corresponding to the motion in the designating unit in a state where the user can visually see the superimposed state of the video updated in step (f), which gives the motion corresponding to the parameter. A video generation method capable of accepting input of the second designation information.   A program for causing a computer to function as the video generation device according to claim 1.