JP2020068444A - Lighting control/photographing device, composite image generation device, and program - Google Patents

Abstract

To provide a lighting control/photographing device, a composite image generation device, and a program that increase the degree of freedom of lighting effects and improve S/N for a captured image.SOLUTION: A lighting control/photographing device 1 according to the present invention includes a lighting device 10 that illuminates a subject, a lighting control unit 11 that controls the lighting by the lighting device 10 on the basis of a time code (time information) and externally outputs the light, and a camera 12 that generates a captured image of the subject and outputs the image externally. A composite image generation device 3 according to the present invention includes an image processing unit 4 branches a plurality of captured image frames included in a captured image in time series on the basis of time information obtained from the lighting control/photographing device 1 according to the present invention, classifies the branched image frames as a predetermined number of base lighting frames, multiplies the gains individually, combines two or more of the predetermined number of base lighting frames after the multiplication, and generates time-series composite image, and an operation unit 5 that specifies the specified illumination frame to be a target of the composite image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an illumination control / imaging device for performing illumination control when a subject imaged by a lighting device in a studio or the like is imaged by a camera, and a synthetic image generation for synthesizing image signals photographed through the illumination control. A device and a program.

  2. Description of the Related Art Conventionally, a large number of lighting devices have been used in order to generate various lighting effects in taking photographs and videos in general studios. Then, the subject is photographed after appropriately adjusting the posture of each illuminating device, the projected area, the brightness, the color, and the like (hereinafter, illumination state). Normally, the illumination state of a subject is changed in part or all of the illumination device during shooting.

  By the way, conventionally, there is known a composite video generation system that composites a foreground video of a live-action image and a background video to be composited in a so-called virtual studio (for example, refer to Patent Document 1).

  In Patent Document 1, a shooting camera shoots a foreground object in a virtual studio to generate a foreground image, a background image to be combined is generated, and the background image and the foreground image are combined. To generate a composite video. At this time, a subject-side illumination pattern representing the illumination position and the illumination amount on the subject side is calculated based on the difference in illumination amount between the background image and the foreground image, and the illumination device projects with the illumination pattern for the illumination device. The subject of the foreground is photographed by the photographing camera by controlling the light ray direction and the illumination amount.

Patent No. 52227883

  First, in the conventional photography of a photo or video in a general studio, the illumination state needs to be appropriately set at the time of photography (before or during photography). In other words, at the time of shooting, it is necessary to balance all the lighting devices with respect to the irradiation angle, the irradiation range, the irradiation light amount, and the irradiation color of the lighting device, so that the lighting effect of each lighting device is limited, and Only an image (including a still image or a moving image) in the light condition in the lighting environment at the time of shooting can be obtained.

  Further, also in the generation of a captured image in the virtual studio disclosed in Patent Document 1, it is necessary to control the light ray direction and the illumination amount projected by the illumination device based on the difference in the illumination amount between the background image and the foreground image at the time of shooting. There is. Therefore, in this case as well, with respect to the irradiation angle, irradiation range, irradiation light amount, and irradiation color of the lighting device, only the subject-side lighting pattern in which the balance of all the lighting devices is adjusted at the time of shooting can be calculated. The lighting effect is limited, and only an image (including a still image or a moving image) in the state of ambient light at the time of shooting can be obtained.

  Therefore, in view of the above-mentioned problems, an object of the present invention is to increase the degree of freedom of the illumination effect and improve the S / N so that a camera projects an image of a subject projected using the illumination device. An object of the present invention is to provide a lighting control / photographing device that performs lighting control, a synthetic video generation device that synthesizes a video image (including a still image or a moving image) captured through the lighting control, and a program.

  The illumination control / imaging device of the present invention uses one or a plurality of illumination devices for irradiating a subject with illumination light, and the illumination according to a predetermined illumination control pattern based on time information with which a captured video frame can be specified. An illumination control unit that controls the device and outputs the time information to the outside, and a camera that generates a captured image by capturing the subject illuminated by the illumination device and outputs the captured image to the outside. And

  Further, in the illumination control / imaging device of the present invention, the illumination control means may use one or more of the illumination angle, the illumination range, the illumination light amount, and the illumination color according to the predetermined illumination control pattern. It is characterized by controlling the lighting device of.

  Further, in the illumination control / imaging device of the present invention, the illumination control means, as the predetermined illumination control pattern, controls lighting of the plurality of illumination devices or dimming and extinction of irradiation light amount and a frame frequency of the captured image. The feature is that it is controlled to switch by.

  Further, in the illumination control / imaging device of the present invention, the illumination control means controls, as the predetermined illumination control pattern, dimming and extinguishing the amount of irradiation light with respect to one of the illumination devices. The feature is that it is controlled to switch by.

  Further, the synthetic image generation device of the present invention inputs the time information and the captured image directly or indirectly output from the illumination control / shooting device of the present invention, and includes the captured image based on the time information. A plurality of captured video frames that are time-sequentially branched and classified as a predetermined number of base illumination frames, a multiplying unit that individually multiplies the predetermined number of base illumination frames by a gain, and a multiplying unit. An image synthesizing unit that synthesizes two or more of the predetermined number of base illumination frames to generate a time-series synthesized image.

  Further, in the combined image generation device of the present invention, the operation unit is configured to operate the gain used in the multiplication means, and the gain that allows the predetermined number of base illumination frames to be individually multiplied is multiplied by zero. It is characterized in that it is configured to operate any one of, 1 ×, attenuation, and increase.

  Further, in the composite image generation device of the present invention, the multiplication unit is a pixel after performing conversion by a predetermined function indicating a linear or curved monotonically increasing function with respect to input pixel values in the predetermined number of base illumination frames. It is characterized in that the value is multiplied by the gain.

  Further, in the composite video generation device of the present invention, the image composition unit may generate the composite video at each frame point of the captured video by a moving composite process, or may generate a plurality of frames of the captured video by a segmental composite process. It is characterized in that the synthetic image is generated in units of minutes.

  Further, in the synthetic image generating apparatus of the present invention, the image synthesizing unit has means for performing conversion by a predetermined function indicating a monotonically increasing function of a straight line or a curve after generating the synthetic image, and outputting the result. .

  Further, the program of the present invention is configured as a program for causing a computer to function as the synthetic image generation device of the present invention.

  According to the illumination control / imaging device of the present invention, while changing the illumination of one or a plurality of illumination devices momentarily, the time taken to associate the captured image captured under the illumination of the illumination control pattern with the illumination control pattern. Since the information can be output so as to be transmitted or recorded together with the information, it is possible to further increase the degree of freedom of the lighting effect and improve the S / N for the captured image in the studio or the like.

  Further, according to the combined image generation device of the present invention, a captured image frame (base illumination frame) corresponding to the illumination control pattern of the illumination device is obtained from the captured image acquired by the illumination control / imaging device of the present invention, and the gain is obtained. Is operated to generate a composite image in virtual illumination distribution, the degree of freedom of the illumination effect can be further increased and the S / N can be improved.

  Therefore, according to the present invention, it is possible to further increase the degree of freedom of the lighting effect and improve the S / N for a video image shot in a studio or the like. For example, it is possible to obtain a new image with the lighting effect variably adjusted for the captured image after shooting, change the lighting condition after shooting, change the lighting condition at a remote place, acquire images under multiple different lighting conditions. Is possible.

FIG. 1 is a block diagram showing an example of a connection form of a lighting control / photographing device and a composite image generation device according to an embodiment of the present invention. FIG. 1 is a block diagram showing an example of a schematic configuration of an illumination control / imaging device of an embodiment according to the present invention. FIG. 6A is a block diagram showing an example of a schematic configuration of a lighting control unit and a lighting device in a lighting control / imaging device according to an embodiment of the present invention, and FIG. 6B shows an example of a lighting control pattern in this case. It is a figure. (A) is a figure which shows another example of the illumination control pattern by the illumination control part in the illumination control and imaging device of one Embodiment by this invention, (b) is a figure which shows another example of an illumination control pattern. . (A) is a block diagram showing another example of the schematic configuration of the illumination control unit and the illumination device in the illumination control / imaging device of one embodiment according to the present invention, and (b) is an example of the illumination control pattern in this case. FIG. FIG. 9 is a block diagram showing another example of the schematic configuration of the illumination control / imaging device of one embodiment according to the present invention. FIG. 6A is a block diagram showing an example of a lighting control unit that controls one lighting device in the lighting control / imaging device of one embodiment according to the present invention, and FIG. 6B is an example of a lighting control pattern in this case. FIG. FIG. 1 is a block diagram showing an example of a schematic configuration of a composite video generation device of one embodiment according to the present invention. (A) is a figure which shows an example of the moving type synthetic | combination process performed when producing | generating a synthetic | combination video in the synthetic | combination image production | generation apparatus of one Embodiment by this invention, (b) shows one by this invention. It is a figure which shows an example of the division type synthetic | combination process performed when producing | generating a synthetic | combination video in the image synthetic | combination part in the synthetic | combination video generation apparatus of embodiment. (A) is a diagram showing an illumination effect on a subject of a video signal obtained by photographing a picture or a video in a general studio as a conventional technique, and (b) is a lighting control / photographing device according to an embodiment of the present invention; It is a figure which shows the illumination effect with respect to the to-be-photographed object of the video signal obtained by the synthetic | combination video production | generation apparatus.

  Hereinafter, an illumination control / photographing apparatus 1 and a composite image generation apparatus 3 according to an embodiment of the present invention will be described with reference to the drawings.

(overall structure)
FIG. 1 is a block diagram showing an example of a connection form of an illumination control / imaging device 1 and a composite image generation device 3 according to an embodiment of the present invention. As shown in FIG. 1, the lighting control / photographing apparatus 1 and the composite video image generating apparatus 3 may be directly connected, or the photographed video image generated by the lighting control / photographing apparatus 1 is transmitted by wire or wirelessly and temporarily stored. It may be connected via a transmission device, a storage device, or both (in FIG. 1, shown as "transmission device / storage device 2").

  The illumination control / imaging device 1 will be described in detail later with reference to FIGS. 2 to 7, but one or a plurality of illumination devices for illuminating the subject Sbj (one unit as described later, N or M lighting devices 10), and a lighting control unit 11 that controls lighting by the lighting devices 10 according to a predetermined lighting control pattern based on a time code (time information) that enables specification of a captured video frame. , A camera 12 that generates a captured image of the subject Sbj illuminated by the one or more illumination devices 10.

  Then, the illumination control / imaging device 1 is configured to output the time code and the captured image to the outside.

  The detailed configuration of the composite video generation device 3 will be described later with reference to FIGS. 8 and 9, but the time code and the captured video output from the illumination control / shooting device 1 are input, and the time code is set to the time code. An image processing unit 4 that generates a new time-series composite image based on the captured video based on the captured image, and an operation unit 5 that operates generation of the composite video in the image processing unit 4 are provided.

  When passing through the transmission device, a delay may occur. When the storage device is used, the lighting control / shooting device 1 writes the captured video and time code (time information) to the storage device, and the composite video generation device 3 writes the captured video and time code from the storage device. Is configured to read. This writing and reading may be executed at different times (however, of course, when reading a frame from the storage device, the frame needs to be written to the storage device before the reading. ).

  Further, as shown in FIG. 1, another composite video generation device 3B having the same configuration can be connected in parallel with the composite video generation device 3. Although not shown, when the captured video and the time code are transmitted via the transmission device, the transmission from the lighting control / photographing device 1 to the synthetic video generation device 3 and the transmission from the lighting control / photographing device 1 to the synthetic video are performed. The transmission to the generation device 3B may be a different route. In addition, when the captured video and the time code are transmitted through the storage device, the time point when the composite video generation device 3 reads from the storage device and the time point when the composite video generation device 3B reads from the storage device. Can be different from.

(Lighting control / imaging device)
FIG. 2 is a block diagram showing an example of a schematic configuration of the illumination control / imaging device 1 according to the embodiment of the present invention. The illumination control / imaging device 1 shown in FIG. 2 includes N illumination devices 10 (N is an integer of 2 or more) for illuminating the subject Sbj, an illumination control unit 11, and a camera 12.

  The lighting control unit 11 is a device that individually controls the lighting by the N lighting devices 10 according to a lighting control pattern based on a time code that allows a captured video frame in a video captured by the camera 12 to be specified. It is configured to output to the outside.

  The camera 12 is configured to generate a captured image of the subjects Sbj illuminated by the N illumination devices 10 and output the captured image to the outside.

  Further, FIG. 3A is a block diagram showing an example of a schematic configuration of the lighting control unit 11 and the N lighting devices 10 in the lighting control / imaging device 1 according to the embodiment of the present invention, and FIG. 8A is a diagram showing an example of a lighting control pattern in this case. FIG.

Here, as shown in FIG. 2, the N lighting devices 10 are identified as lighting numbers L ₀ , L ₁ , L ₂ , L ₃ , ..., L _N−1 , respectively, and are shown in FIG. As described above, each of the N lighting devices 10 includes a lighting lamp 101, a lighting power supply 102 that supplies power to the lighting lamp 101, and a switch 103 that controls the power supply.

  In the example shown in FIG. 3A, the illumination lamp 101 in each of the N illumination devices 10 is, for example, white, or a red, green, or blue monochromatic lamp (for example, a fluorescent lamp or an LED lamp). By intermittently turning the switch 103 on and off, it is possible to control lighting (or dimming of the irradiation light amount) / lighting off at a frequency higher than the frame frequency of the image captured by the camera 12.

  Moreover, as shown in FIG. 3A, the illumination control unit 11 includes a time code generation unit 111 and a time code decoder unit 112. Although the illumination control unit 11 can be configured as one device including the time code generation unit 111 and the time code decoder unit 112, the time code generation unit 111 and the time code decoder unit 112 are configured as separate devices. You may.

  The time code generation unit 111 is a functional unit that generates a time code that enables specification of a captured video frame in a video captured by the camera 12, and outputs the time code to the time code decoder unit 112 and the outside.

  The time code decoder unit 112, based on the time code input from the time code generation unit 111, a predetermined lighting for controlling the lighting (switch 103 on / off) of each of the N lighting devices 10 in time series. It is a functional unit that generates a control signal toward each of the N lighting devices 10 in a control pattern.

  As an example of this lighting control pattern, as shown in FIG. 3B, in the case of controlling lighting on / off of each of the N lighting devices 10, the time code input from the time code generation unit 111 is used. Based on this, it is possible to determine which of the lights in each of the N lighting devices 10 should be turned on.

When the time code is associated with the captured video frame number F starting from the frame at time 00:00:00 00:00 (0th frame), the captured video by the camera 12 is captured video frame number F = It is expressed as F ₀ , F ₁ , F ₂ , F ₃ , F ₄ , F ₅ , F ₆ , ...

Then, the time code decoder unit 112 obtains a remainder when the value of the captured video frame number F is divided by the total number N of the illumination devices 10 to be controlled, and the illumination device having the illumination number designated by the numerical value of the remainder. Lights up and turns off other lighting devices. As a result, the time code decoder unit 112, for the captured image by the camera 12 represented by the captured image frame number F = F ₀ , F ₁ , F ₂ , F ₃ , F ₄ , F ₅ , F ₆ ,. The illumination by the illumination device 10 is switched every N consecutive frames.

  Since the time code controls the lighting of the lighting device 10, it is not necessary to synchronize with the start (or end) of the captured video frame of the camera 12 in time, and the lighting control is switched by the frame frequency of the captured video. Any pattern can be used as long as it can form a pattern. In other words, even if there is a time lag between the lighting start of the lighting device 10 and the start (or end) of the captured video frame, if the lighting device 10 is switched on in successive captured video frames. As long as it is good, it is possible to specify the captured video frame.

For example, when controlling N lighting devices (for example, N = 5) as in the example illustrated in FIG. 3B, the time code decoder unit 112 calculates i by the following [Equation 1]. closing only the switch 103 of the illumination device having an illumination number L _i such that only illumination device having an illumination number L _i is turned to open the switch 103 of another lighting apparatus.

  Note that in [Equation 1], the operator% is defined as the remainder when the numerical value on the front side (F in this example) is divided by the numerical value on the back side (N in this example).

  By the way, in the example shown in FIG. 3B, only one of the N lighting devices 10 is turned on and the other lighting devices are turned off. However, the control is limited to turning on only one device. do not have to. For example, FIG. 4A is a diagram showing another example of the illumination control pattern by the illumination control unit 11 in the illumination control / imaging device 1 according to the embodiment of the present invention, and FIG. It is a figure which shows another example.

As in the example shown in FIG. 4A, the time code decoder unit 112 closes only the switch 103 of the lighting device having the lighting numbers L ₀ and L ₁ , opens the switch 103 of the other lighting device, and then It is possible to follow a lighting control pattern in which only the switch 103 of the lighting device having the lighting numbers L ₁ and L ₂ is closed and the switch 103 of the other lighting device is repeatedly opened.

Alternatively, as in the example shown in FIG. 4B, the time code decoder unit 112 closes only the switch 103 of the lighting device having the lighting numbers L ₀ and L ₃ , and opens the switch 103 of the other lighting device, Next, it is possible to follow a lighting control pattern in which only the switch 103 of the lighting device having the lighting numbers L ₁ and L ₄ is closed and the switch 103 of the other lighting device is repeatedly opened.

  In addition, in the example illustrated in FIGS. 3B and 4A and 4B, the time code decoder unit 112 has been described as an example of switching ON / OFF of one of the N lighting devices 10. The lighting control pattern for controlling various dimming / extinction of the irradiation light amount can be followed. In controlling various dimming of the irradiation light amount, it is realized by intermittently turning on and off the switch 103 included in each of the lighting devices 10 shown in FIG. 3A. It may be configured to control the power supply of the illumination power supply 102 that supplies power to the lamp 101.

  For example, FIG. 5A is a block diagram showing another example of the schematic configuration of the illumination control unit 11 and the illumination device 10 in the illumination control / imaging device 1 according to the embodiment of the present invention, and FIG. It is a figure which shows an example of the illumination control pattern in this case. In FIG. 5, the same components as those shown in FIG. 3 are designated by the same reference numerals.

The N lighting devices 10 shown in FIG. 5A are identified as lighting numbers L ₀ , L ₁ , L ₂ , L ₃ , ..., L _N−1 , respectively, and each of the N lighting devices 10 is Each includes an illumination lamp 101 and a dimming illumination power source 104 that supplies power to the illumination lamp 101.

  Also in the example shown in FIG. 5A, the illumination lamp 101 in each of the N illumination devices 10 is, for example, a white, or red, green, or blue single-color lamp (for example, a fluorescent lamp or an LED lamp). By adjusting the power supply by the dimming illumination power source 104, it is possible to control dimming / extinction of the irradiation light amount at a frequency higher than the frame frequency of the video image captured by the camera 12.

For example, as in the example illustrated in FIG. 5B, the time code decoder unit 112 sets A ₀ = light control 100% lighting and A ₁ = light control 50, where the lighting state of the lighting device to be lighted is A _j. % Lighting and A ₂ = light control 30% lighting are sequentially repeated, i is calculated by [Equation 1], and when i = 0, only the lighting device having the lighting number L ₀ is in the lighting state A. The lighting device is controlled to be turned on at ₀ (light control 100%), and the other lighting devices are turned off. When i = 1, only the lighting device having the lighting number L ₁ is controlled to be turned on in the lighting state A ₁ (light control 50%), and the other lighting devices are turned off. Then, when i = 2, only the lighting device having the lighting number L ₂ is controlled to be turned on in the lighting state A ₂ (light control 30%), and the other lighting devices are turned off. Subsequently, when i = 3, only the lighting device having the lighting number L ₃ is controlled to be turned on in the lighting state A ₀ (dimming control 100%), and the other lighting devices are turned off. Can be followed.

  Further, in the example shown in FIG. 3B, FIG. 4A, and FIG. 5B, and FIG. 5A and FIG. 5B, the time code decoder unit 112 is one of the N lighting devices 10. The example of switching between lighting and extinguishing has been described, but the invention is not limited to this.

  For example, FIG. 6 is a block diagram showing another example of the schematic configuration of the illumination control / imaging device 1 according to the embodiment of the present invention. In the example shown in FIG. 6, as the illuminating device, M (RGB) color switching type illuminators having red (R) color, green (G) color, and blue (B) color light sources and capable of switching each light source The apparatus 10 (M is an integer of 2 or more) is used. In this case, the time code decoder unit 112 discriminates each light emitting source in the M RGB color switching type illumination devices 10 by N illumination numbers based on [Equation 1], as shown in FIG. b), FIG. 4B, or FIG. 5B, the lighting control pattern can be followed.

  In addition, in each of the examples shown in FIGS. 2 and 6 described above, the lighting control unit 11 turns on (or adjusts the amount of irradiation light) of any one of the N lighting devices 10 as a predetermined lighting control pattern. Although the example in which the turn-off is switched by the frame frequency of the captured video has been described, the present invention is not limited to this, and the dimming / extinguishing of the irradiation light amount of one lighting device 10 may be switched by the frame frequency of the captured video. You can also

  For example, FIG. 7A is a block diagram showing an example of a lighting control unit 11 that controls one lighting device 10 in the lighting control / imaging device 1 of one embodiment according to the present invention, and FIG. FIG. 6 is a diagram showing an example of a lighting control pattern in this case.

  One illumination device 10 shown in FIG. 7A includes an illumination lamp 101 and a dimming illumination power source 104 that supplies power to the illumination lamp 101, as in the case of FIG. 5A. I shall. Therefore, the illumination lamp 101 in this case can be, for example, a white color, or a single color lamp of red, green, and blue (for example, a fluorescent lamp or an LED lamp), and the power supply for the dimmable illumination power source 104 adjusts the power supply. The dimming / extinguishing of the irradiation light amount can be controlled at a frequency higher than the frame frequency of the image captured by the camera 12.

Then, the time code decoder unit 112 associates the K kinds of illumination states A _K (K ≦ N) by dimming with the illumination numbers S ₀ , S ₁ , S ₂ , S ₃ , ..., S _N−1 , respectively. Can be identified.

For example, as in the example illustrated in FIG. 5B, the time code decoder unit 112 associates the lighting state of one lighting device 10 with a lighting number S _i , and S ₀ = light control 100% lighting, S _It is assumed that 4 kinds of ₁ = dimming 50% lighting, S ₂ = dimming 30% lighting, S ₃ = extinguishing are sequentially repeated, and i is calculated by [Equation 1]. When i = 0, illumination number S ₀ controlled to be lit in the illuminating state _{a 0} (dimming 100%) with, when i = 1, controlled to be lit in the illuminating state _{a 1} having an illumination number _{S 1} (dimming 50%), i = When 2, the lighting state A ₂ having the lighting number S ₂ (light control 30%) is controlled to be turned on, and when i = 3, the lighting state A ₃ having the lighting number S ₃ is controlled to be turned off (light off). and, when i = 4, controls so as to light up the illumination number _{S 4} illuminating state _{a 0} with (dimming 100%) May be in accordance with lighting control pattern that repeats so on.

  Then, in the illumination control / imaging device 1 illustrated in FIGS. 2 to 7, the illumination angle and the illumination range of the one or more illumination devices 10 are arranged in consideration of the lighting control pattern to be adopted, and the camera 12 is used. By shooting the subject Sbj, it is possible to create shot images with various lighting effects based on the time code.

(Structure of composite video generation device)
FIG. 8 is a block diagram showing an example of a schematic configuration of the composite video generation device 3 according to the embodiment of the present invention. The composite video generation device 3 inputs the time code (time information capable of specifying the captured video frame number F) and the captured video I output from the illumination control / shooting device 1, and performs the capturing based on the time code. An image processing unit 4 for generating a new time-series composite video H from a video, and an operation unit 5 for operating the generation of the composite video H in the image processing unit 4 are provided.

  The image processing unit 4 and the operation unit 5 can be configured as an integrated device or can be configured as individual devices.

  In describing the configuration of the composite video generation device 3 shown in FIG. 8, an example of an embodiment applied to the illumination control / imaging device 1 shown in FIGS. 2 and 3 will be described as a representative. Similarly, the synthetic image generation device 3 shown in FIG. 8 can be applied to the illumination control / imaging device 1 illustrated in FIG. 7.

  More specifically, the image processing unit 4 includes a branching unit 40, N multiplying units 44 (N is an integer of 2 or more), and an image combining unit 45.

  The branching unit 40 has a time code decoder unit 41, a switching unit 42, and N frame memories 43.

  The time code decoder unit 41 inputs the time code (time information capable of specifying the captured video frame number F) output from the illumination control / imaging device 1, and identifies the captured video frame number F indicated by the time code. Is calculated and the switching unit 42 is controlled by this surplus F% N.

  The switching unit 42 inputs the captured video I (F) at a certain time code (captured video frame number F), and receives the input of N captured video frames according to the remainder F% N among N outputs. It is controlled by the time code decoder unit 41 so as to be distributed to and output.

  Each of the N frame memories 43 temporarily stores each of the N captured video frames input via the switching unit 42. For example, each time a new captured video frame is input, the temporarily stored captured video frames are stored. It is composed of an output FIFO (First In, First Out) memory.

  For example, when N = 5, the frame memory 43 for i = 0 temporarily holds the captured video frame with F% 5 = 0, and the frame memory 43 for i = 1 has F% 5 = 1. Holds the captured video frame temporarily. Similarly, the frame memory 43 for i = 2 temporarily holds the captured video frame with F% 5 = 2. Note that the time temporarily stored in each frame memory 43 is, for example, after the switching unit 42 selects a predetermined captured video frame in the captured video I, and “the correspondingly connected next stage multiplication unit 44 has It suffices if it is the time after the acceptance of the input is completed and before the switching unit 42 selects the next captured video frame.

  However, although not shown in FIG. 8, in the composite video generation device 3 of the present embodiment, the operator operating the operation unit 5 temporarily stops the FIFO operation in each of the N frame memories 43 at an arbitrary time. It is preferable to provide an HMI (Human Machine Interface) for displaying captured video frames stored in each of the N frame memories 43 on a monitor (not shown) at the time of suspension.

  Therefore, the branching unit 40 time-sequentially branches a plurality of captured video frames included in the captured video based on the time code (time information with which the captured video frame number F can be specified), and the corresponding N frames. Each time a new captured video frame is temporarily stored in the memory 43 and is input from the N frame memories 43, the temporarily stored captured video frames are sequentially output to the corresponding N multiplication units 44 in time series. .

  In this way, the branching unit 40 calculates the surplus F% N based on the time code (time information with which the photographed video frame number F can be specified) (N is the illumination control / the illumination control controlled during the photographing in the photographing device 1). The number of types of lighting effects corresponding to the pattern) and the captured video frame I (F) at a certain time code (captured video frame number F) are distributed to N outputs according to the remainder F% N.

  For example, in the case of N = 5, the branching unit 40 uses the captured video frames (I (0), I (5), I (10), ...) When the residual F% N is 0 for i = 0. Of the captured video frames (I (1), I (6), I (11), ...) When the remainder F% N is 1 to the multiplier 44 for i = 1. Output. Similarly, the branching unit 40 transfers each captured video frame (I (2), I (7), I (12), ...) When the remainder F% N is 2 to the multiplier 44 for i = 2. And outputs each captured video frame (I (3), I (8), I (13), ...) When the remainder F% N is 3 to the multiplier 44 for i = 3, and the remainder Each captured video frame (I (4), I (9), I (14), ...) When F% N is 4 is output to the multiplier 44 for i = 4.

  Therefore, the branching unit 40 inputs the time code and the captured video output from the illumination control / imaging device 1, and branches a plurality of captured video frames included in the captured video in time series based on the time code. Then, it functions to classify as a predetermined number of base lighting frames (N base lighting frames corresponding to the number of types of lighting effects (N) corresponding to the lighting control pattern based on the time code).

Since the illumination control in the illumination control / imaging device 1 is controlled by [Equation 1] as described above, a captured image captured while switching the illumination in each frame at a cycle of N captured image frames from the branching unit 40 is used. can get. Therefore, in the specification of the present application, the captured video frame number F corresponding to the numerical value i based on [Equation 1] is selected from among the N captured video frames of the captured video I corresponding to the illumination number L _i obtained from the branching unit 40. The captured video frame that is held is referred to as a base illumination frame J _i, and can be defined as in the following [Equation 2].

By the above operation, the base illumination frame J ₀ is stored in the frame memory 43 for i = 0, the base illumination frame J ₁ is stored in the frame memory 43 for i = ₁ , and the base illumination frame is stored in the frame memory 43 for i = 2. J _2, i = 3 frame memory 43 basal lighting frame _{J 3} in for are respectively temporarily stored, that is, the base lighting frame _{J N-1} are respectively temporarily stored in the frame memory 43 for i = N-1 It The captured video frame temporarily stored in the frame memory 43 for i = N-1 is input every time a new captured video frame is input, except when the above HMI (human machine interface) is provided and suspended. Are sequentially output to N multiplying units 44 corresponding to each other in time series.

The operation unit 5 is a user interface for designating the gain a _i for each base illumination frame J _i . For example, the operator commands the gain a _i desired by the operator via the operation unit 5, and the operation unit 5 outputs an operation signal indicating the command to the corresponding multiplication unit 44. The operation unit 5 can operate any one of zero-fold, equal-magnification, attenuation, and increase as the gain a _{i that} can be individually multiplied with each base illumination frame J _i .

The operation unit 5 has a fader (sliding type, lever type, rotary type, etc.) operation function, and the operation amount can be input by a variable resistor, a sliding resistor, a linear encoder, a rotary encoder, a touch sensor, or the like. Device), or an electronic control device such as a computer, or when the entire composite video generation device 3 is configured as an electronic control device such as a computer, it can be configured by a software functional unit that functions as the operation unit 5. . In the example of the operation unit 5 shown in FIG. 8, N faders 53 corresponding to each of the N multiplication units 44 are provided, and each fader 53 adjusts its gain a _i in multiple stages. The level specifying unit 51 has a gain range unit 51 that indicates a range for the gain, and the level specifying unit 51 has a gain specifying unit 52 for specifying the gain a _i . The operator can operate the control unit 52 to transmit a transmission signal for instructing the desired gain a _i to the corresponding multiplication unit 44 from the operation unit 5.

Each of the N multiplication units 44 receives the operation signal from the operation unit 5, and multiplies the base illumination frame J _i related to the generation of the composite image H by the individual gain a _i based on the operation signal. And outputs it to the image composition section 45.

For example, when N = 5,
multiplication unit 44 for i = 0, executes the multiplication of _{J 0} and _{a 0,} and outputs the result _{K 0.}
i = multiplication section 44 for 1 performs multiplication of _{J 1} and _{a 1,} and outputs the result _{K 1.}
multiplication unit 44 for i = 2 performs multiplication of _{J 2} and _{a 2,} and outputs the result _{K 2.}
i = multiplication section 44 for 3 executes a multiplication of _{J 3} and _{a 3,} and outputs the result _{K 3.}
i = multiplication section 44 for 4 executes a multiplication of _{J 4} and _{a 4,} and outputs the result _{K 4.}

Here, as an example, the “multiplication” in each multiplication unit 44 may be multiplication of the pixel value (signal value) of the image of the base illumination frame J _{i by} the gain a _i as shown in [Equation 3] below. it can. This can provide a virtual lighting effect for each of the base lighting frames J _i .

Further, as another example, the “multiplication” in each multiplication unit 44 is a pixel after the input pixel value in the base illumination frame J _i is converted by a predetermined function φ as shown in the following [Equation 4]. The value (signal value) can be multiplied by the gain a _i .

  When the function φ shown in [Equation 4] is applied, the function φ is preferably a linear or curved monotonically increasing function as an output pixel value with respect to an input pixel value.

For example, the function φ is a so-called OETF ^-1 that converts an input pixel value (signal value) into a scene brightness value and outputs the luminosity value (OETF is an opto-electric transfer function, and OETF ^-1 is an inverse function of OETF). Is preferred.

By using a monotonically increasing function as the function φ in this way, it is possible not only to provide a virtual lighting effect for each of the base lighting frames J _i , but also when the above HMI (human machine interface) is provided. , The lighting effect when the virtual lighting effect is brought is emphasized, and the adjustment by the operator becomes easy.

The image synthesizing unit 45 synthesizes the pixel value of the multiplication result K _i from each of the N multiplying units 44 based on the time code to generate a synthetic image H, and outputs it to the outside.

  The image synthesizing unit 45 may perform a moving-type synthesizing process, which will be described later, at each frame point of the captured image I to generate and output a synthesized image H, or a plurality of frames of the captured image I (preferably N frames: N is It is also possible to generate and output a composite image H by performing a later-described segmental combination processing in units of time corresponding to the number of types of base illumination frames. Hereinafter, the composite video frame at the output time point G of the composite video H will be referred to as H (G).

  For example, as illustrated in FIG. 9A, when the image combining unit 45 outputs the combined image H at each frame point of the captured image I, the images of the Gth to G + Nth frames of the captured image I are used. The moving-type combining process is performed, and the combined video frame H (G) is output at every frame based on the N base illumination frames. In this case, when the base illumination frame has a frame frequency of 60 Hz, for example, the frame frequency of the composite video frame H (G) also becomes 60 Hz.

  Further, for example, as shown in FIG. 9B, when the image synthesizing unit 45 outputs the synthetic video H in units of N frames of the captured video I, the Nth to Gth to the Gth video of the captured video I are output. A segmented synthesizing process is performed using N.G + N frame images, and a synthesized image frame H (G) is output in units of N frames based on N base illumination frames. In this case, a high-speed camera is used as the camera 12, and when the base illumination frame has a frame frequency of 300 Hz, for example, the frame frequency of the composite video frame H (G) is 60 Hz.

  The image composition unit 45 can use, for example, the additive composition shown in [Equation 5] as a pixel value composition method.

  Note that in [Equation 5], F = G + N−1 when using the moving-type combining process as shown in FIG. 9A, and when using the partitioning-type combining process as shown in FIG. 9B. Is F = N · G + N−1.

  Further, the image composition unit 45 may be configured to use addition composition as a combination method of pixel values as shown in [Equation 6] and then apply a predetermined function ψ.

  Note that in [Equation 6], F = G + N−1 when using the moving-type combining process as shown in FIG. 9A, and when using the partitioning-type combining process as shown in FIG. 9B. Is F = N · G + N−1.

  One of the above functions φ and ψ may be omitted, but when the function ψ is applied, the function ψ is preferably used as the output pixel value with respect to the input pixel value, and a linear or curved monotone increasing function. And

  In particular, when the function φ and the function ψ described above are both applied, the function ψ is preferably an inverse function of the function φ. For example, the function ψ is a so-called OETF that converts an input scene brightness value into a pixel value (signal value) and outputs the pixel value (signal value).

  Thus, by using a monotonically increasing function as the function φ and making the function ψ an inverse function of the function φ, a lighting effect when a virtual lighting effect is provided when the above HMI (human machine interface) is provided Is emphasized, adjustment by the operator is facilitated, and the output of the image synthesizing unit 45 can be returned to a signal form that is visually at the same level as the image captured by the camera 12.

  With the above configuration, the captured video captured by the illumination control / capturing device 1 in advance or in real time, and the time code that enables the captured video frame in this captured video to be specified directly or by the transmission device or the storage device. Input to the image processing unit 4 in the composite video generation device 3 and the operator operates the operation unit 5 in the composite video generation device 3 to virtually change the illumination state from the composite video generation device 3. A variable composite image can be obtained with high S / N.

  For example, FIG. 10A is a diagram showing a lighting effect on a subject of a video signal obtained by photographing a picture or a video in a general studio as a conventional technique, and FIG. 10B is a diagram showing an embodiment of the present invention. It is a figure which shows the illumination effect with respect to the to-be-photographed object of the video signal obtained by the illumination control / imaging | photography apparatus 1 and the synthetic | combination image generation apparatus 3.

  As shown in FIG. 10 (a), when photographing a photo or video in a general conventional studio, the irradiation angle, the irradiation range, the irradiation light amount, and the irradiation color of each individual lighting device 10 (for example, three lighting devices 10) are used. As for the above, it is necessary to adjust the balance of all the lighting devices 10 at the time of shooting (before or during shooting), and the lighting effect of each lighting device 10 is limited.

  On the other hand, as shown in FIG. 10B, in the lighting control / imaging device 1 according to the present invention, the irradiation angle / irradiation range / irradiation light amount / irradiation color of each individual lighting device 10 (for example, three lighting devices 10) As a result, the degree of freedom in regard to is increased, and a wide dynamic range of a video image captured by the camera 12 can be achieved.

  Further, as shown in FIG. 10 (b), in the composite video generation device 3 according to the present invention, a composite video is generated from a captured video after shooting by freely operating a virtual lighting effect with high S / N. It will be possible.

The operation unit 5 generates an operation signal for controlling each of the N multiplication units 44, and the gain a _i for a certain specific base illumination frame J _i included in the operation signal is a _i = It is also possible to operate the image synthesizing unit 46 by setting 0 to substantially 2 to N-1 of the N base illumination frames. This is synonymous with the configuration in which the operation signal includes a signal for instructing selection of only the base illumination frame multiplied by the gain a _i of a _i ≠ 0.

  Therefore, in the illumination control / photographing apparatus 1 and the composite video image generating apparatus 3 of the present embodiment, it is possible to further increase the degree of freedom of the lighting effect and improve the S / N for the video image taken in the studio or the like. For example, it is possible to obtain a new image with the lighting effect variably adjusted for the captured image after shooting, change the lighting condition after shooting, change the lighting condition at a remote place, acquire images under multiple different lighting conditions. Is possible.

  Regarding the example of the above-described embodiment, it is possible to suitably use a program that configures a computer that functions as the composite video generation device 3 and causes each unit of these devices to function. Specifically, a control unit for controlling each unit can be composed of a central processing unit (CPU) in a computer, and at least a storage unit for appropriately storing a program necessary for operating each unit. It can be configured with one memory. That is, by causing such a computer to execute the program by the CPU, it is possible to realize the function of each unit described above. Further, a program for realizing the function of each unit can be stored in a predetermined area of the storage unit (memory) described above. Such a storage unit can be configured by a RAM or ROM inside the device, or can be configured by an external storage device (for example, a hard disk). Further, such a program can be configured by a part of software (stored in the ROM or the external storage device) on the OS used in the computer. Furthermore, a program for causing such a computer to function as each unit can be recorded in a computer-readable recording medium. Further, each of the above-mentioned means can be configured as a part of hardware or software and can be realized by combining them.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as limited by the above-described embodiments, but only by the claims.

  According to the present invention, it is possible to increase the degree of freedom of the lighting effect and improve the S / N of a shot image in a studio or the like, which is useful for editing the lighting effect of the shot image.

1 Lighting control / photographing device 2 Transmission device / storage device (transmission device or storage device)
3, 3B Synthetic video generation device 4 Image processing unit 5 Operation unit 10 Illumination device 11 Illumination control unit 12 Camera 40 Branch unit 41 Time code decoder unit 42 Switching unit 43 Frame memory 44 Multiplying unit 45 Image combining unit 51 Gain range unit 52 Gain Designating unit 53 Fader 101 Lighting lamp 102 Lighting power supply 103 Switch 104 Dimming lighting power supply 111 Time code generation unit 112 Time code decoder unit

Claims (10)

One or a plurality of illumination devices for illuminating the subject with illumination light;
Illumination control means for controlling the illumination device by a predetermined illumination control pattern based on time information capable of specifying a captured video frame and outputting the time information to the outside.
A camera that generates a captured image by capturing the subject illuminated by the illumination device and outputs the captured image to the outside.
An illumination control / imaging device comprising:   The illumination control means controls the one or a plurality of illumination devices for any one of an irradiation angle, an irradiation range, an irradiation light amount, and an irradiation color according to the predetermined lighting control pattern. The illumination control / imaging device according to claim 1.   The illumination control means is characterized in that, as the predetermined illumination control pattern, control is performed so as to switch between lighting and dimming and extinguishing of the irradiation light amount of a plurality of the illumination devices at a frame frequency of the captured video. The lighting control / imaging device according to claim 1.   The lighting control means controls, as the predetermined lighting control pattern, to switch one of the lighting devices between dimming of an irradiation light amount and turning off at a frame frequency of the captured image. The lighting control / imaging device according to claim 1. The said time information and the said picked-up image directly or indirectly output from the illumination control and the photographing device as described in any one of Claims 1 to 4 are input, and it is contained in the said picked-up image based on the said time information. A branching unit that branches a plurality of captured video frames in time series and classifies them as a predetermined number of base lighting frames,
Multiplying means for individually multiplying the predetermined number of base illumination frames by gain;
An image combining unit that combines two or more of the predetermined number of base illumination frames obtained through the multiplying unit to generate a time-series combined image,
A composite video generation device comprising:   The operation unit is configured to operate a gain used by the multiplying unit, and is any one of zero times, equal times, attenuation, and increase as a gain that can be individually multiplied with respect to the predetermined number of base illumination frames. The composite video generation device according to claim 5, wherein the composite video generation device is configured to operate.   The multiplying unit is configured to multiply the pixel value after the conversion is performed on the input pixel value in the predetermined number of base illumination frames by a predetermined function indicating a linear or curved monotone increasing function. The composite video generation device according to claim 5 or 6, characterized in that.   The image compositing unit generates the composite video at each frame time point of the captured video by moving type compositing processing, or generates the composite video by unit time of a plurality of frames of the captured video by segmental compositing processing. The combined image generating device according to claim 5, wherein   9. The image synthesizing unit includes means for performing conversion by a predetermined function indicating a monotonically increasing function of a straight line or a curved line after generation of the synthetic video and outputting the transformed image. The composite video generation device according to.   A program for causing a computer to function as the composite video generation device according to claim 5.