JP7212497B2 - Lighting control/photographing device, composite image generation device, and program - Google Patents

Description

The present invention relates to a lighting control/shooting device for performing lighting control when a camera captures an image of an object projected using a lighting device in a studio or the like, and a synthetic video generation that synthesizes video signals shot through the lighting control. It relates to a device and a program.

2. Description of the Related Art Conventionally, a large number of lighting devices are used to generate various lighting effects when taking pictures or videos in a typical studio. Then, after appropriately adjusting the attitude of each lighting device, the range of light projection, luminance, color, etc. (hereinafter referred to as lighting conditions), the subject is photographed. Generally, during photographing, the lighting condition of the subject is changed for some or all of the lighting devices.

By the way, conventionally, there has been known a synthesized image generation system that synthesizes a foreground image shot in a virtual studio and a background image to be synthesized (see, for example, Patent Document 1).

In Patent Document 1, a camera shoots a foreground subject in a virtual studio to generate a foreground image, generates a background image to be synthesized, and synthesizes the background image and the foreground image. to generate a composite image. At this time, based on the difference in the amount of illumination between the background image and the foreground image, a subject-side illumination pattern representing the illumination position and the amount of illumination on the subject side is calculated, and the illumination apparatus projects using the illumination pattern for the illumination apparatus. The foreground subject is photographed by a photographing camera by controlling the direction of light and the amount of illumination.

Patent No. 5227883

First, in taking pictures and videos in a conventional general studio, it is necessary to appropriately set the lighting conditions at the time of taking pictures (before or during taking pictures). In other words, when shooting, it is necessary to adjust the balance of all lighting devices with respect to the irradiation angle, irradiation range, irradiation light amount, and irradiation color of the lighting device, so the lighting effect of each lighting device is limited, and Only images (including still images and moving images) can be obtained in the light conditions of the lighting environment at the time of shooting.

Also, in the generation of captured images in the virtual studio disclosed in Patent Document 1, it is necessary to control the direction and amount of illumination projected by the lighting device based on the difference in the amount of illumination between the background image and the foreground image during shooting. There is Therefore, in this case as well, it is only possible to calculate the subject-side lighting pattern in which the balance of all lighting devices has been adjusted at the time of shooting regarding the irradiation angle, irradiation range, irradiation light amount, and irradiation color of each lighting device. Lighting effects are limited and only images (including still or moving images) are available in the ambient lighting conditions at the time of capture.

SUMMARY OF THE INVENTION In view of the above-mentioned problems, an object of the present invention is to increase the degree of freedom of the lighting effect and improve the S/N ratio, so that the subject illuminated by the lighting device can be captured by the camera. An object of the present invention is to provide a lighting control/shooting device that performs lighting control, a composite image generating device that synthesizes images (including still images and moving images) shot through the lighting control, and a program.

The lighting control/photographing device of the present invention comprises one or more lighting devices for irradiating a subject with illumination light, and a predetermined lighting control pattern based on time information that makes it possible to identify a photographed image frame. lighting control means for controlling a device and outputting the time information to the outside; The control means is characterized in that, as the predetermined lighting control pattern, the one lighting device is controlled to switch between dimming and extinguishing of the irradiation light amount at the frame frequency of the captured image .

Further, in the lighting control/photographing device of the present invention, the lighting control means controls any one of irradiation angle, irradiation range, irradiation light amount, and irradiation color according to the predetermined lighting control pattern. is characterized by controlling the lighting device of

Further, in the lighting control/photographing apparatus of the present invention, the lighting control means controls, as the predetermined lighting control pattern, the lighting of the plurality of lighting devices or dimming and turning off of the amount of light emitted from the plurality of lighting devices at the frame frequency of the photographed image. It is characterized by controlling to switch with.

Further, the composite image generating apparatus of the present invention uses one or a plurality of lighting devices for irradiating a subject with illumination light, and a predetermined lighting control pattern based on time information that enables identification of a captured image frame. Lighting control comprising: lighting control means for controlling a lighting device and outputting the time information to the outside; and a camera for generating a photographed image by photographing the subject illuminated by the lighting device and outputting it to the outside.・Input the time information and the captured image directly or indirectly output from the image capturing device, and based on the time information, branch the plurality of captured image frames included in the captured image in time series and divide a predetermined number of bases. a branching unit for classifying illumination frames; multiplication means for individually multiplying the predetermined number of base illumination frames by gains; and combining two or more of the predetermined number of base illumination frames obtained through the multiplication means. , and an image synthesizing unit that generates a time-series synthetic video.

Furthermore, the composite video generation device of the present invention receives the time information and the captured video output directly or indirectly from the lighting control/shooting device of the present invention, and includes the captured video based on the time information. a branching unit that branches a plurality of captured video frames in time series and classifies them as a predetermined number of base illumination frames; a multiplication unit that individually multiplies the predetermined number of base illumination frames by a gain; and an image synthesizing unit that synthesizes two or more of the predetermined number of base illumination frames obtained to generate a time-series synthesized image.

Further, in the composite image generating apparatus of the present invention, an operation unit configured to operate a gain used in the multiplication means is provided, and the operation unit is capable of individually multiplying the predetermined number of base illumination frames. It is characterized in that it is configured to operate any one of zero-fold, unity-fold, attenuation, and increase as a gain to be applied.

Further, in the composite image generating apparatus of the present invention, the multiplication means converts the input pixel values in the predetermined number of base illumination frames by a predetermined function representing a linear or curved monotonically increasing function to the pixels. It is characterized in that it is configured to multiply the value by the gain.

Further, in the composite video generating apparatus of the present invention, the image synthesizing unit generates the composite video at each frame point of the captured video by movement type combining processing, or generates a plurality of frames of the captured video by segmented combining processing. It is characterized in that the composite image is generated in units of minutes.

Further, in the synthetic video generating apparatus of the present invention, the image synthesizing unit is characterized by having means for performing conversion by a predetermined function indicating a monotonically increasing function of a straight line or a curved line after generating the synthetic video and outputting the result. .

Also, the program of the present invention is configured as a program for causing a computer to function as the composite image generation device of the present invention.

According to the lighting control/photographing device of the present invention, the lighting of one or a plurality of lighting devices is changed from moment to moment, and the photographed image photographed under the lighting of the lighting control pattern is captured at the time associated with the lighting control pattern. Since it can be output so as to be transmitted or recorded together with information, it is possible to increase the degree of freedom of lighting effects and improve the S/N ratio for images shot in a studio or the like.

Further, according to the composite video generation apparatus of the present invention, a captured video frame (base lighting frame) corresponding to the lighting control pattern of the lighting device is obtained from the captured video acquired by the lighting control/photographing device of the present invention, and the gain is manipulated to generate a synthesized image in a virtual illumination distribution, so that the degree of freedom of illumination effects can be increased and the S/N ratio can be improved.

Therefore, according to the present invention, it is possible to increase the degree of freedom of the lighting effect and improve the S/N ratio for images shot in a studio or the like. For example, it is possible to obtain a new image with variable lighting effects after shooting, change the lighting conditions after shooting, change the lighting conditions at a remote location, and acquire images under multiple different lighting conditions. becomes possible.

1 is a block diagram showing an example of a connection configuration of a lighting control/photographing device and a composite image generation device according to one embodiment of the present invention; FIG. 1 is a block diagram showing an example of a schematic configuration of an illumination control/photographing device according to an embodiment of the present invention; FIG. 1(a) is a block diagram showing an example of the schematic configuration of an illumination control unit and an illumination device in an illumination control/photographing device according to one embodiment of the present invention, and FIG. 1(b) shows an example of an illumination control pattern in this case; It is a diagram. (a) is a diagram showing another example of an illumination control pattern by the illumination control unit in the illumination control/imaging apparatus of one embodiment according to the present invention, and (b) is a diagram showing another example of the illumination control pattern. . (a) is a block diagram showing another example of the schematic configuration of the lighting control unit and the lighting device in the lighting control/photographing device of one embodiment according to the present invention, and (b) is an example of the lighting control pattern in this case. FIG. 4 is a diagram showing; FIG. 10 is a block diagram showing another example of the schematic configuration of the illumination control/image capturing device of one embodiment according to the present invention; (a) is a block diagram showing an example of a lighting control unit that controls one lighting device in the lighting control/photographing device of one embodiment according to the present invention, and (b) is an example of a lighting control pattern in this case. It is a figure which shows. 1 is a block diagram showing an example of a schematic configuration of a synthetic video generating device according to one embodiment of the present invention; FIG. (a) is a diagram showing an example of moving type synthesis processing executed when generating a synthesized video in an image synthesizing unit in the synthesized video generating apparatus of one embodiment according to the present invention; FIG. 7 is a diagram showing an example of segmented compositing processing executed when generating a composite video in the image compositing unit in the composite video generating device of the embodiment; (a) is a diagram showing the lighting effect on a subject of a video signal obtained by photographing a photograph or video in a general studio as a conventional technique, and (b) is a lighting control/photographing apparatus according to an embodiment of the present invention; FIG. 10 is a diagram showing lighting effects on a subject of a video signal obtained by the composite video generation device;

A lighting control/photographing device 1 and a composite image generation device 3 according to an embodiment of the present invention will be described below with reference to the drawings.

(overall structure)
FIG. 1 is a block diagram showing an example of a connection form of a lighting control/photographing device 1 and a composite video generation device 3 according to one embodiment of the present invention. As shown in FIG. 1, the lighting control/photographing device 1 and the composite image generating device 3 may be directly connected, or the photographed video generated by the lighting control/photographing device 1 may be transmitted by wire or wirelessly and temporarily stored. The connection may be made via a transmission device, a storage device, or both (illustrated as “transmission device/storage device 2” in FIG. 1).

The detailed configuration of the lighting control/photographing device 1 will be described later with reference to FIGS. N or M lighting devices 10), and a lighting control unit 11 that controls lighting by the lighting devices 10 according to a predetermined lighting control pattern based on a time code (time information) that enables identification of captured image frames. , and a camera 12 for generating a captured image of the subject Sbj illuminated by the one or more lighting devices 10 .

The lighting control/photographing device 1 is configured to output the time code and the photographed image to the outside.

The composite video generation device 3, whose detailed configuration will be described later with reference to FIGS. An image processing unit 4 for generating a new time-series composite image from the captured image based on the captured image, and an operation unit 5 for operating generation of the composite image in the image processing unit 4.

A delay may occur in the case of passing through the transmission device. In the case of passing through the storage device, the lighting control/photographing device 1 writes the shot video and time code (time information) to the storage device, and the synthesized video generation device 3 writes the shot video and time code from the storage device. is configured to read This writing and reading may be executed at different times (however, when reading a certain frame from the storage device, the frame must be written to the storage device before reading). ).

Further, as shown in FIG. 1, another synthetic video generating device 3B having a similar configuration can be connected in parallel with the synthetic video generating device 3. FIG. Although not shown, when the captured video and the time code are transmitted via the transmission device, transmission from the lighting control/photographing device 1 to the composite video generation device 3 and transmission from the lighting control/photographing device 1 to the composite video A different route may be used for transmission to the generation device 3B. Also, in the case of transmitting the shot video and the time code via the storage device, the timing of reading from the storage device by the composite video generation device 3 and the timing of reading from the storage device by the composite video generation device 3B It does not matter if the

(Lighting control/shooting device)
FIG. 2 is a block diagram showing an example of the schematic configuration of the lighting control/photographing device 1 of one embodiment according to the present invention. The lighting control/photographing device 1 shown in FIG. 2 includes N lighting devices 10 (N is an integer equal to or greater than 2) for illuminating a subject Sbj, a lighting control unit 11 and a camera 12 .

The lighting control unit 11 is a device that individually controls lighting by the N lighting devices 10 according to a lighting control pattern based on a time code that makes it possible to specify a captured video frame in a video captured by the camera 12. The time code is Configured to output externally.

The camera 12 is configured to generate a photographed image of the subject Sbj illuminated by the N lighting devices 10 and output it to the outside.

FIG. 3(a) is a block diagram showing an example of a schematic configuration of the lighting control unit 11 and N lighting devices 10 in the lighting control/photographing device 1 of one embodiment according to the present invention, and FIG. ) is a diagram showing an example of an illumination control pattern in this case.

Here, as shown in FIG. 2, the _N lighting devices 10 are identified as lighting numbers L ₀ , L ₁ , L ₂ , L ₃ , . Thus, each of the N lighting devices 10 includes an illumination lamp 101, an illumination power supply 102 that supplies power to the illumination lamp 101, and a switch 103 that controls the power supply.

In the example shown in FIG. 3A, the lighting lamps 101 in each of the N lighting devices 10 are, for example, white, red, green, or blue monochromatic lamps (eg, fluorescent lamps, LED lamps, etc.). By intermittently turning the switch 103 on and off, it is possible to control lighting (or dimming of the amount of irradiation light)/lighting out at a frequency higher than the frame frequency of the image captured by the camera 12 .

Further, as shown in FIG. 3A, the illumination control section 11 includes a time code generation section 111 and a time code decoder section 112. As shown in FIG. The lighting control unit 11 can be configured as one device including the time code generation unit 111 and the time code decoder unit 112, but the time code generation unit 111 and the time code decoder unit 112 are configured as separate devices. You may

The time code generation unit 111 is a functional unit that generates a time code that enables specification of a captured image frame in the image captured by the camera 12, and outputs the time code to the time code decoder unit 112 and to the outside.

Based on the time code input from the time code generating unit 111, the time code decoder unit 112 determines a predetermined lighting for switching and controlling the lighting (on/off of the switch 103) in each of the N lighting devices 10 in time series. It is a functional unit that generates a control signal directed to each of the N lighting devices 10 in a control pattern.

As an example of this lighting control pattern, as shown in FIG. Based on this, it is possible to determine which of the lights in each of the N lighting devices 10 should be turned on.

For example, when the time code is associated with the captured image frame number F starting from the time 00:00:00:00 frame (the 0th frame), the image captured by the camera 12 has the captured image frame number F= _They are expressed as _F0 , F1, F2 _{, F3} , _F4 , _F5 , _F6 , . . .

Then, the time code decoder unit 112 obtains the remainder when the value of the captured image frame number F is divided by the total number N of lighting devices 10 to be controlled, and determines the lighting device having the lighting number designated by the numerical value of the remainder. and turn off the other lighting devices. As a result, the time code decoder unit 112, with respect to the video captured by the camera 12 represented by the captured video frame _number F ₌ F0 _, F1, F2, _F3 , _F4 , _F5 , _F6 , . . . The illumination by the illumination device 10 is switched every N consecutive frames.

Since the time code controls the lighting of the lighting device 10, it is not necessary to time-synchronize with the start (or end) of the image frame captured by the camera 12, and the lighting control is switched according to the frame frequency of the image captured. Any material can be used as long as it can form a pattern. In other words, even if there is a time synchronization shift between the start of lighting of the lighting device 10 and the start (or end) of the captured video frame, if the lighting of the lighting device 10 is switched in successive captured video frames, Anything that can identify the captured video frame is acceptable.

For example, as in the example shown in FIG. 3B, when controlling N lighting devices (for example, N=5), the time code decoder unit 112 calculates i by the following [Equation 1]. , the switch 103 of the lighting device having the lighting number L _i is closed and the switches 103 of the other lighting devices are opened so that only the lighting device having the lighting number L _i is lit.

In [Equation 1], the operator % is defined as a remainder obtained by dividing the front numerical value (F in this example) by the rear numerical value (N in this example).

By the way, in the example shown in FIG. 3B, only one of the N lighting devices 10 is turned on, and the other lighting devices are turned off, but the control is limited to turning on only one. do not have to. For example, FIG. 4A is a diagram showing another example of a lighting control pattern by the lighting control unit 11 in the lighting control/photographing device 1 of one embodiment according to the present invention, and FIG. It is a figure which shows another example.

As in the example shown in FIG. 4A, the time code decoder unit 112 closes only the switches 103 of the lighting devices having the lighting numbers L ₀ and L ₁ , opens the switches 103 of the other lighting devices, and then A lighting control pattern may be followed such that only the switches 103 of the lighting devices with lighting numbers L ₁ and L ₂ are closed and the switches 103 of the other lighting devices are opened repeatedly.

Alternatively, as in the example shown in FIG. 4B, the time code decoder unit 112 closes only the switches 103 of the lighting devices having the lighting numbers L ₀ and L ₃ and opens the switches 103 of the other lighting devices. Next, it is possible to follow a lighting control pattern such that only the switches 103 of the lighting devices having the lighting numbers L ₁ and L ₄ are closed and the switches 103 of the other lighting devices are opened repeatedly.

Also, in the examples shown in FIGS. 3B and 4A and 4B, the time code decoder unit 112 switches ON/OFF of any one of the N lighting devices 10. , a lighting control pattern for controlling various dimming/extinguishing of the irradiation light amount. Various dimming control of the irradiation light amount is realized by intermittently turning on and off the switch 103 provided in each lighting device 10 shown in FIG. The power supply of the lighting power source 102 that supplies power to the lamp 101 may be controlled.

For example, FIG. 5(a) is a block diagram showing another example of the schematic configuration of the lighting control unit 11 and the lighting device 10 in the lighting control/photographing device 1 of one embodiment according to the present invention, and FIG. It is a figure which shows an example of the illumination control pattern in this case. 5, components similar to those shown in FIG. 3 are given the same reference numerals.

The _N lighting devices 10 shown in FIG. 5(a) are identified as lighting numbers L ₀ , L ₁ , L ₂ , L ₃ , . Each lighting lamp 101 and a dimming lighting power supply 104 for supplying power to the lighting lamp 101 are provided.

Also in the example shown in FIG. 5A, the illumination lamps 101 in each of the N lighting devices 10 are, for example, white, red, green, or blue monochromatic lamps (eg, fluorescent lamps, LED lamps, etc.). By adjusting the power supplied by the dimming lighting power supply 104 , it is possible to control dimming/turning off of the amount of irradiation light at a frequency higher than the frame frequency of the image captured by the camera 12 .

For example, as in the example shown in FIG. 5B, the time code decoder unit 112 sets A ₀ = 100% dimming lighting, A ₁ = 50 dimming, where A _j is the lighting state of the lighting device to be turned on. % lighting and A ₂ = dimming 30% lighting are sequentially repeated, i is calculated by [Equation 1], and when i=0, only the lighting device with the lighting number L ₀ is placed in the lighting state A ₀ (dimming 100%) is controlled to turn on, and the other lighting devices are turned off. When i= ₁ , control is performed so that only the lighting device having the lighting number L1 is turned on in the lighting state A1 ( ₅₀ % dimming), and the other lighting devices are turned off. Then, when i=2, control is performed so that only the lighting device having the lighting number L ₂ is turned on in the lighting state A ₂ (30% dimming), and the other lighting devices are turned off. Subsequently, when i= ₃ , only the lighting device having the lighting number L3 is controlled to turn on in the lighting state _A0 (dimming 100%), and the other lighting devices are turned off. may be complied with.

3(b), 4(a), 4(b), and 5(a), 5(b), the time code decoder unit 112 selects one of the N lighting devices 10. Although an example of switching on/off has been described, it is not necessary to be limited to this.

For example, FIG. 6 is a block diagram showing another example of the schematic configuration of the lighting control/photographing device 1 of one embodiment according to the present invention. In the example shown in FIG. 6, the lighting device has red (R), green (G), and blue (B) light emitting sources, and M units of RGB color switching type lighting devices capable of switching the respective light emitting sources. A device 10 (M is an integer of 2 or more) is used. 3 ( b), FIG. 4(b), or FIG. 5(b).

2 and 6 described above, the lighting control unit 11 turns on (or adjusts the amount of light emitted from) any of the N lighting devices 10 as a predetermined lighting control pattern. Although an example in which turning off is switched at the frame frequency of the captured video has been described, it is not necessary to be limited to this. can also

For example, FIG. 7(a) is a block diagram showing an example of the lighting control unit 11 for controlling one lighting device 10 in the lighting control/photographing device 1 according to one embodiment of the present invention, and FIG. 7(b). is a diagram showing an example of a lighting control pattern in this case.

One lighting device 10 shown in FIG. 7(a) includes a lighting lamp 101 and a dimming lighting power supply 104 that supplies power to the lighting lamp 101, similarly to the one shown in FIG. 5(a). shall be Therefore, the illumination lamp 101 in this case can be, for example, a white, red, green, or blue monochromatic lamp (for example, a fluorescent lamp or an LED lamp). , dimming/extinguishing of the irradiation light amount can be controlled at a frequency higher than the frame frequency of the image captured by the camera 12 .

Then, the time code decoder unit 112 associates the K types of illumination states A _K (K≦ _N ) by dimming with the illumination numbers S ₀ , S ₁ , S ₂ , S ₃ , . can be identified by

For example, as in the example shown in FIG. 5B, the time code decoder unit 112 associates the lighting state of one lighting device 10 with the lighting number S _i , S ₀ = dimming 100% lighting, S ₁ = lighting at 50% dimming, S ₂ = lighting at 30% dimming, and S _{3 =} turning off are sequentially repeated. When i= ₁ , the lighting is controlled to be in the lighting state A1 ( ₅₀ % dimming) with the lighting number S1, and i ₌ 2, control is performed to turn on in lighting state A ₂ (dimming 30%) with lighting number S ₂ , and when i = 3, control is performed to turn on in lighting state A ₃ (light off) with lighting number S ₃ . Then, when i=4, the lighting control pattern can be repeated such that the lighting is controlled to be in the lighting state A ₀ (dimming 100%) having the lighting number S ₄ .

2 to 7 , the illumination angles and illumination ranges of one or more illumination devices 10 are arranged in consideration of the lighting control pattern to be adopted, and are arranged by the camera 12. By photographing the subject Sbj, it is possible to create photographed images with various lighting effects based on the time code.

(Structure of Composite Video Generating Device)
FIG. 8 is a block diagram showing an example of a schematic configuration of the composite image generation device 3 of one embodiment according to the present invention. The composite video generation device 3 receives the time code (time information that can specify the captured video frame number F) output from the lighting control/shooting device 1 and the captured video I, and performs the shooting based on the time code. An image processing unit 4 for generating a new time-series synthetic image H from images, and an operation unit 5 for operating the generation of the synthetic image H in the image processing unit 4 are provided.

Note that the image processing unit 4 and the operation unit 5 can be configured as an integrated device or can be configured as individual devices.

In describing the configuration of the composite image generation device 3 shown in FIG. 8, an example of an embodiment applied to the lighting control/photographing device 1 shown in FIGS. 2 and 3 will be described as a representative. Similarly, the synthetic video generation device 3 shown in FIG. 8 can be applied to the illumination control/shooting device 1 shown in FIG. 7 as an example.

More specifically, the image processing unit 4 includes a branching unit 40 , N multiplying units 44 (N is an integer equal to or greater than 2), and an image synthesizing unit 45 .

The branching unit 40 has a time code decoder unit 41 , a switching unit 42 and N frame memories 43 .

The time code decoder unit 41 receives the time code (time information that can specify the captured video frame number F) output from the lighting control/shooting device 1, and specifies the captured video frame number F indicated by the time code. A remainder F%N is calculated for , and the switching unit 42 is controlled by this remainder F%N.

The switching unit 42 receives a captured image I(F) at a certain time code (captured image frame number F), and converts the input into N captured image frames according to the remainder F%N among N outputs. are controlled by the time code decoder section 41 so as to be sorted into and output.

Each of the N frame memories 43 temporarily stores the N shot image frames input via the switching unit 42, and, for example, each time a new shot image frame is input, the temporarily stored shot image frame is stored. It consists of an output FIFO (First In, First Out) memory.

For example, when N=5, the frame memory 43 for i=0 temporarily holds the captured video frame F%5=0, and the frame memory 43 for i=1 holds F%5=1. Temporarily holds the captured image frame. Similarly, the frame memory 43 for i=2 temporarily holds the captured image frame of F%5=2. The time for temporary storage in each frame memory 43 is, for example, after the switching unit 42 selects a predetermined captured video frame in the captured video image I, and "when the correspondingly connected next-stage multiplier 44 It may be the time after the completion of acceptance of the input and before the switching unit 42 selects the next captured image frame.

However, although illustration is omitted in FIG. 8, in the composite image generation device 3 of the present embodiment, the FIFO operation in each of the N frame memories 43 is temporarily stopped at an arbitrary time by the operator who operates the operation unit 5. It is preferable to provide an HMI (Human Machine Interface) for displaying, on a monitor (not shown), each captured video frame stored in each of the N frame memories 43 during the pause.

Therefore, the branching unit 40 branches a plurality of captured video frames included in the captured video in time series based on the time code (time information that can specify the captured video frame number F) to generate corresponding N frames. Each time a new captured video frame is input from the N frame memories 43, the temporarily stored captured video frame is sequentially output to the corresponding N multipliers 44. .

In this way, the branching unit 40 calculates the remainder F%N based on the time code (time information that can specify the captured image frame number F) (N is the lighting control/illumination control that was controlled at the time of shooting in the shooting device 1). number of types of lighting effects corresponding to the pattern), and the captured video frame I(F) at a certain time code (captured video frame number F) is distributed to N outputs according to the remainder F%N.

For example, when N=5, the branching unit 40 selects each captured image frame (I(0), I(5), I(10), . . . ) when the remainder F%N is 0 for i=0. to the multiplication unit 44, and each captured video frame (I(1), I(6), I(11), . . . ) when the remainder F%N is 1 to the multiplier 44 for i=1 Output. Similarly, the branching unit 40 sends each captured video frame (I(2), I(7), I(12), . . . ) when the remainder F%N is 2 to the multiplier 44 for i=2. output, and output each captured video frame (I(3), I(8), I(13), . . . ) when the remainder F%N is 3 to the multiplier 44 for i=3, Each captured image frame (I(4), I(9), I(14), . . . ) when F%N is 4 is output to the multiplier 44 for i=4.

Therefore, the branching unit 40 receives the time code and the shot video output from the lighting control/shooting device 1, and branches a plurality of shot video frames included in the shot video in time series based on the time code. and classify it into a predetermined number of base lighting frames (N base lighting frames corresponding to the number of types of lighting effects (N) corresponding to the lighting control pattern based on the time code).

Illumination control/Since the lighting control in the photographing apparatus 1 performs the control according to [Equation 1] as described above, the photographed image photographed while switching the illumination every frame with a cycle of N photographed image frames from the branch unit 40 is displayed. can get. For this reason, in the specification of the present application, of the N captured image frames of the captured image _I corresponding to the lighting number Li obtained from the branching unit 40, the captured image frame number F corresponding to the numerical value i based on [Equation 1] is The photographed video frame having the following is called a base illumination frame _Ji , and can be defined as in [Equation 2] below.

By the above operation, the frame memory 43 for i=0 stores the base illumination frame J ₀ , the frame memory 43 for i=1 stores the base illumination frame J ₁ , and the frame memory 43 for i=2 stores the base illumination frame. The frame memory 43 for J ₂ , i=3 temporarily stores the base illumination frame J ₃ , that is, the frame memory 43 for i=N−1 temporarily stores the base illumination frame J _N−1 . be. The captured image frames temporarily stored in the frame memory 43 for i=N−1 are changed every time a new captured image frame is input, except when the above HMI (human machine interface) is provided to temporarily stop the captured image frames. are sequentially output to N multipliers 44 corresponding to each other in chronological order.

The operation unit 5 is a user interface for specifying the gain a _i for each base illumination frame J _i . For example, the operator commands the gain _ai desired by the operator through the operating section 5, and the operating section 5 outputs an operation signal indicating the command to the corresponding multiplier 44. FIG. The operation unit 5 can operate any one of zero-fold, one-to-one, attenuation, and increase as a gain a _i that can be individually multiplied for each base illumination frame J _i .

The operation unit 5 has a fader (slide type, lever type, rotary type, etc.) operation function, and the operation amount can be input by a variable resistor, a sliding resistor, a linear encoder, a rotary encoder, a touch sensor, or the like. device), or an electronic control device such as a computer, or when the entirety of the composite image generation device 3 is configured by an electronic control device such as a computer, it can be configured with a functional unit by software that functions as the operation unit 5. . In the example of the operation unit 5 shown in FIG. 8, N faders 53 corresponding to the N multiplication units 44 are provided, and each fader 53 adjusts the respective gains _ai in multiple stages. and a gain designation section 52 for designating the gain _ai in the level designation section 51 . An operator can operate the control section 52 to transmit a transmission signal for commanding a desired gain _ai from the operation section 5 to the corresponding multiplication section 44 .

Each of the N multipliers 44 receives an operation signal from the operation unit 5 and multiplies the base illumination frame J _i for generating the composite image H by an individual gain a _i based on the operation signal. and output to the image synthesizing unit 45 .

For example, when N=5,
The multiplier 44 for i=0 performs the multiplication of J ₀ and a ₀ and outputs K ₀ as a result.
The multiplier 44 for i=1 performs the multiplication of J ₁ and a ₁ and outputs K ₁ as a result.
Multiplier 44 for i=2 performs the multiplication of J ₂ and a ₂ and outputs K ₂ as a result.
The multiplier 44 for i=3 performs the multiplication of J ₃ and a ₃ and outputs K ₃ as a result.
The multiplier 44 for i=4 performs the multiplication of J ₄ and a ₄ and outputs K ₄ as a result.

Here, as an example, the “multiplication” in each multiplier 44 may be multiplication of the pixel value (signal value) of the image of the base illumination frame J _i by the gain a _i as shown in [Equation 3] below. can. This can provide a virtual lighting effect for each of the base lighting frames J _i .

Further, as another example, the “multiplication” in each multiplier ₄₄ is the pixel It can be the multiplication of the gain a _i to the value (signal value).

When the function φ shown in [Equation 4] is applied, the function φ is preferably a linear or curved monotonically increasing function of the output pixel value with respect to the input pixel value.

For example, the function φ converts an input pixel value (signal value) into a scene luminance value and outputs a so-called OETF ⁻¹ (OETF is an Opto-Electrical Transfer Function, and OETF ⁻¹ is an inverse function of OETF). It is preferable that

By using a monotonically increasing function as the function φ in this way, not only can a virtual lighting effect be produced for each of the base lighting frames _Ji , but also when the above-mentioned HMI (human machine interface) is provided, , the lighting effect is emphasized in providing a virtual lighting effect, making it easier for the operator to adjust.

The image synthesizer 45 synthesizes the pixel values of the multiplication results K _i from the N multipliers 44 based on the time code to generate a synthesized image H, and outputs it to the outside.

The image synthesizing unit 45 may generate and output a synthesized image H by performing a moving type synthesizing process, which will be described later, at each frame of the shot image I, or multiple frames of the shot image I (preferably N frames: N is The number of types of base illumination frames) may be used as a time unit to perform the segmented synthesis process described later to generate and output a synthesized image H. FIG. Hereinafter, the synthesized video frame at the output time point G of the synthesized video H is assumed to be H(G).

For example, as shown in FIG. 9A, when outputting a synthesized image H at each frame of the captured image I, the image synthesizing unit 45 uses the images of the G-th to G+N-th frames of the captured image I. A moving type synthesis process is performed, and a synthesized video frame H(G) is output at each frame based on the N base illumination frames. In this case, when the base illumination frame has a frame frequency of, for example, 60 Hz, the composite video frame H(G) also has a frame frequency of 60 Hz.

Further, for example, as shown in FIG. 9B, when the image synthesizing unit 45 outputs the synthetic video H in units of time for N frames of the captured video I, A segmented synthesis process is performed using images of N·G+N frames, and a synthesized image frame H(G) is output in units of time for N frames based on N base illumination frames. In this case, when a high-speed camera is used as the camera 12 and the base illumination frame has a frame frequency of 300 Hz, for example, the frame frequency of the synthesized video frame H(G) is 60 Hz.

The image synthesizing unit 45 can use, for example, additive synthesizing shown in [Equation 5] as a method of synthesizing pixel values.

In [Equation 5], F=G+N−1 when using the moving type combining process as shown in FIG. , F=N·G+N−1.

Further, the image synthesizing unit 45 may be configured to use additive synthesizing as a method of synthesizing pixel values, as shown in [Equation 6], and then apply a predetermined function ψ.

In [Formula 6], F=G+N−1 when using the moving type combining process as shown in FIG. , F=N·G+N−1.

One of the functions φ and ψ described above may be omitted. and

In particular, when the above-described function φ and function φ are both applied, the function φ is preferably the inverse function of the function φ. For example, the function ψ is a so-called OETF that converts an input scene luminance value into a pixel value (signal value) and outputs it.

As a result, by using a monotonically increasing function as the function φ and the function ψ as an inverse function of the function φ, when the above HMI (human machine interface) is provided, the lighting effect when producing a virtual lighting effect is emphasized, and adjustment by the operator is facilitated, and the output of the image synthesizing unit 45 can be returned to a signal form that is visually at the same level as the image captured by the camera 12 .

With the above configuration, the video captured in advance or in real time by the lighting control/photographing device 1 and the time code that makes it possible to identify the captured video frame in this captured video can be transmitted directly or by the transmission device or the storage device. input to the image processing unit 4 in the synthetic video generation device 3 via the, and the operator operates through the operation unit 5 in the synthetic video generation device 3 to virtually change the lighting state from the synthetic video generation device 3 A variable composite image can be obtained with a high S/N ratio.

For example, FIG. 10(a) is a diagram showing the lighting effect of a video signal obtained by taking a picture or video in a general studio as a prior art, and FIG. FIG. 2 is a diagram showing lighting effects on a subject of video signals obtained by the lighting control/shooting device 1 and the composite video generation device 3;

As shown in FIG. 10( a ), when taking pictures or videos in a typical conventional studio, the irradiation angle, irradiation range, irradiation light amount, and irradiation color of each lighting device 10 (for example, three lighting devices 10 ) , it is necessary to adjust the balance of all lighting devices 10 at the time of photographing (before or during photographing), and the lighting effect of each lighting device 10 is limited.

On the other hand, as shown in FIG. 10B, in the lighting control/photographing device 1 according to the present invention, the irradiation angle, irradiation range, irradiation light amount, and irradiation color of each lighting device 10 (for example, three lighting devices 10) The degree of freedom for is increased, and it is possible to widen the dynamic range of the image captured by the camera 12 .

Further, as shown in FIG. 10(b), the synthesized video generating apparatus 3 according to the present invention generates a synthesized video from the captured video after shooting by freely manipulating virtual lighting effects with a high S/N ratio. It becomes possible.

The operation unit 5 generates an operation signal for controlling each of the N multiplication units 44. The gain a _i for a certain base illumination frame J _i included in the operation signal is given by a _i = 0, in effect operating the image synthesizer 46 to selectively synthesize 2 through N−1 of the N base illumination frames. This is synonymous with the operation signal including a signal for selecting only the base illumination frame to be multiplied by the gain a _i of a _i ≠0.

Therefore, with the lighting control/photographing device 1 and the composite video generation device 3 of the present embodiment, it is possible to increase the degree of freedom of the lighting effect and improve the S/N ratio for video captured in a studio or the like. For example, it is possible to obtain a new image with variable lighting effects after shooting, change the lighting conditions after shooting, change the lighting conditions at a remote location, and acquire images under multiple different lighting conditions. becomes possible.

With respect to the above-described example of the embodiment, it is possible to preferably use a program for configuring a computer functioning as the composite video generating device 3 and causing each means of these devices to function. Specifically, a control unit for controlling each means can be configured by a central processing unit (CPU) in a computer, and at least a storage unit for appropriately storing programs required to operate each means It can be configured with one memory. That is, by causing the CPU of such a computer to execute the program, the functions of the above-described means can be realized. Furthermore, a program for realizing the function of each means can be stored in a predetermined area of the aforementioned storage section (memory). Such a storage unit can be configured with a RAM or ROM inside the device, or can be configured with an external storage device (eg, hard disk). Also, such a program can be made up of a part of software (stored in a ROM or an external storage device) on an OS used in a computer. Furthermore, a program for causing such a computer to function as each means can be recorded on a computer-readable recording medium. Further, each means described above can be configured as a part of hardware or software, and can be realized by combining them.

Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions may be made within the spirit and scope of the invention. Accordingly, the present invention should not be construed as limited by the above-described embodiments, but only by the claims.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to increase the degree of freedom of lighting effects and to improve the S/N ratio for images captured in a studio or the like, and is therefore useful for editing the lighting effects of captured images.

1 Lighting control/imaging device 2 Transmission device/storage device (transmission device or storage device)
3, 3B Synthetic video generation device 4 Image processing unit 5 Operation unit 10 Illumination device 11 Illumination control unit 12 Camera 40 Branching unit 41 Time code decoder unit 42 Switching unit 43 Frame memory 44 Multiplication unit 45 Image synthesizing unit 51 Gain range unit 52 Gain Designating unit 53 Fader 101 Illumination lamp 102 Power supply for illumination 103 Switch 104 Power supply for dimming illumination 111 Time code generator 112 Time code decoder

Claims (10)

one or a plurality of lighting devices for irradiating an object with illumination light;
lighting control means for controlling the lighting device according to a predetermined lighting control pattern based on time information that makes it possible to specify a captured image frame, and for outputting the time information to the outside;
a camera that captures the subject illuminated by the lighting device to generate a captured image and outputs the captured image to the outside ;
The lighting control means is characterized in that, as the predetermined lighting control pattern, the one lighting device is controlled so as to switch between dimming and turning off the irradiation light amount at the frame frequency of the captured image. Lighting control and photography equipment. The lighting control means controls the one or more lighting devices with respect to any one of the irradiation angle, irradiation range, irradiation light amount, and irradiation color according to the predetermined lighting control pattern, 2. The lighting control/photographing device according to claim 1. The lighting control means, as the predetermined lighting control pattern, controls to switch between lighting of the plurality of lighting devices or dimming and turning off the irradiation light amount at the frame frequency of the captured image, 3. The lighting control/photographing device according to claim 1 or 2. One or more lighting devices for irradiating illumination light onto a subject, and controlling the lighting devices according to a predetermined lighting control pattern based on time information that makes it possible to specify a captured image frame, and transmitting the time information to an external device. and a camera that generates a shot video by shooting the subject illuminated by the lighting device and outputs it to the outside. a branching unit that inputs the time information and the captured image, branches a plurality of captured image frames included in the captured image in time series based on the time information, and classifies them as a predetermined number of base illumination frames;
multiplication means for multiplying gains individually for the predetermined number of base illumination frames;
an image synthesizing unit for synthesizing two or more of the predetermined number of base illumination frames obtained through the multiplication means to generate a time-series synthesized image;
A composite video generation device comprising: Inputting the time information and the captured video directly or indirectly output from the lighting control/photographing device according to any one of claims 1 to 3 , and included in the captured video based on the time information a branching unit that branches a plurality of captured image frames in time series and classifies them as a predetermined number of base illumination frames;
multiplication means for multiplying gains individually for the predetermined number of base illumination frames;
an image synthesizing unit for synthesizing two or more of the predetermined number of base illumination frames obtained through the multiplication means to generate a time-series synthesized image;
A composite video generation device comprising: An operation unit configured to operate a gain used in the multiplication means, wherein the operation unit selects a gain that can be individually multiplied with respect to the predetermined number of base illumination frames by zero, unity, or attenuated. 6. Apparatus according to claim 4 or 5, characterized in that it is arranged to operate either , increase. The multiplication means is configured to multiply input pixel values in the predetermined number of base illumination frames by a gain after transforming the input pixel values by a predetermined function representing a linear or curved monotonically increasing function. 7. The synthetic image generation device according to any one of claims 4 to 6, characterized in that The image synthesizing unit generates the synthesized image at each frame point of the captured image by moving synthesis processing, or generates the synthesized image in time units corresponding to a plurality of frames of the captured image by segmented synthesis processing. 8. The composite image generation device according to any one of claims 4 to 7, characterized in that: 9. The image synthesizing unit according to any one of claims 4 to 8, characterized in that, after the synthesized image is generated, the synthesized image is converted by a predetermined function representing a monotonically increasing function of a straight line or a curved line, and output. Synthetic image generation device according to . A program for causing a computer to function as the composite image generation device according to any one of claims 4 to 9.

Images