WO2021182214A1

WO2021182214A1 - Photography system, photography method, and program

Info

Publication number: WO2021182214A1
Application number: PCT/JP2021/008096
Authority: WO
Inventors: 拓也池田; 健太郎井田
Original assignee: ソニーグループ株式会社
Priority date: 2020-03-12
Filing date: 2021-03-03
Publication date: 2021-09-16
Also published as: JP2021145233A

Abstract

A photography system (1) that includes a camera (140), a projector (120), and a shutter device (130). The shutter device (130) is synchronized with the camera (140) or the projector (120) and blocks light that is projected from the projector (120) and would enter the camera (140).

Description

Shooting system, shooting method and program

The present invention relates to a photographing system, a photographing method and a program.

In movie and drama rehearsals, the performer's standing position and movement route are marked with tape. The tape is changed for each shooting scene. Therefore, it is being considered to replace the marking with tape by projection mapping. In order to reduce the burden on the performer, it is conceivable to project information such as the movement route not only during rehearsals but also during actual shooting. In this case, it is necessary to prevent the projected light from being reflected in the captured image. The technique of Patent Document 1 is known as a technique for preventing the projected light from being reflected in the captured image. In Patent Document 1, the shooting cycle of the camera and the projection cycle of the projector are synchronized.

Japanese Unexamined Patent Publication No. 2015-186248

However, in the above-mentioned conventional technology, it is necessary to create a dedicated structure for performing synchronization processing in both the camera and the projector. Therefore, the camera and the projector cannot be replaced by a general-purpose machine, and the development cost becomes high.

Therefore, in this disclosure, we propose a shooting system, shooting method, and program that can shoot at low cost while reducing the burden on the performer.

According to the present disclosure, there is provided a photographing system including a camera, a projector, and a shutter device that synchronizes with the camera or the projector and blocks light projected from the projector and incident on the camera. Further, according to the present disclosure, there is provided a photographing method in which the information processing of the photographing system is executed by a computer, and a program for realizing the information processing of the photographing system in the computer.

It is the schematic of the photographing system of 1st Embodiment. It is a flowchart explaining an example of a shooting method. It is a flowchart explaining an example of the process of synchronizing a camera and a shutter device. It is a timing chart which shows an example of the operation of a projector, a shutter device, a camera and a transmitter. It is a timing chart which shows an example of the operation of a projector, a shutter device, a camera and a transmitter. It is a figure which shows the relationship between the brightness of a photographed image, and the brightness of a projected image. It is a flowchart which shows an example of the presentation process of the content to a performer. It is a figure which shows another example of the method of generating a corrected image. It is a figure which shows an example of the correction image projected from the projector. It is a figure which shows the main part of the photographing system of 2nd Embodiment. It is a timing chart which shows an example of the operation of a projector, a shutter device, a camera and a transmitter. It is a flowchart explaining an example of a shooting method. It is the schematic of the photographing system of 3rd Embodiment. It is a figure which shows an example of the projection time setting screen. It is a flowchart which shows an example of a shooting method. It is a timing chart which shows an example of the operation of a projector, a transmitter, a shutter device and a camera. It is a timing chart which shows an example of the operation of a projector, a transmitter, a shutter device and a camera. It is a figure which shows the relationship between the brightness of a photographed image, and the brightness of a projected image. It is a figure which shows the main part of the photographing system of 4th Embodiment. It is a timing chart which shows an example of the operation of a projector, a transmitter, a shutter device and a camera. It is a flowchart explaining an example of a shooting method. It is a figure which shows an example of the measurement result of the light intensity measured through a shutter device.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are designated by the same reference numerals, so that duplicate description will be omitted.

The explanation will be given in the following order.
[1. First Embodiment]
[1-1. Shooting system configuration]
[1-2. Shooting method]
[1-3. effect]
[2. Second Embodiment]
[2-1. Shooting system configuration]
[2-2. Shooting method]
[2-3. effect]
[3. Third Embodiment]
[3-1. Shooting system configuration]
[3-2. Shooting method]
[3-3. effect]
[4. Fourth Embodiment]
[4-1. Shooting system configuration]
[4-2. Shooting method]
[4-3. effect]
[5. Modification example]

[1. First Embodiment]
[1-1. Shooting system configuration]
FIG. 1 is a schematic view of the photographing system 1 of the first embodiment.

The photographing system 1 includes, for example, an information processing device 110, a projector 120, a shutter device 130, a camera 140, and a transmitter 150.

The information processing device 110 generates an image projected by the projector 120. The information processing device 110 includes, for example, a 3D model storage unit 111, a parameter storage unit 112, a display content generation unit 113, and a correction video generation unit 114.

The 3D model storage unit 111 reads and stores, for example, the 3D model of the studio ST from the information input to the information processing device 110. The parameter storage unit 112 reads and stores, for example, the internal parameters of the projector 120 and the position / orientation information of the projector 120 in the 3D model from the input information.

The display content generation unit 113 generates, for example, the content projected on the studio ST. The content includes information that supports the performance of the performer AC. The content includes, for example, the standing position of the performer AC, a conductor indicating the movement path of the performer AC, and a CG character existing around the performer AC.

The corrected video generation unit 114 geometrically corrects the content using, for example, the content edited by the user, the internal parameters of the 3D model and the projector 120, and the position / orientation information of the projector 120 in the 3D model, and generates the corrected video. The corrected video generation unit 114 transmits the video source of the corrected video to the projector 120.

The information processing device 110 is, for example, a computer composed of a processor and a memory. The memory of the information processing apparatus 110 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). The information processing device 110 functions as a display content generation unit 113 and a corrected video generation unit 114 by executing an information processing program (not shown).

The projector 120 has, for example, a video output unit 121 and a projection unit 122.

The video output unit 121 is a processor in the projector 120. The video output unit 121 converts the video source received from the corrected video generation unit 114 into a projected video signal and outputs it to the projection unit 122.

The projection unit 122 projects an image based on the image source to a target location of the studio AT. The image of the content corrected for distortion is projected on the studio ST according to the position / orientation information of the 3D model and the projector 120. The projection unit 122 projects an image at a predetermined refresh rate. As the projection unit 122, known projection methods such as a 3LCD (Liquid Crystal Display) method, an LCOS (Liquid Crystal on Silicon) method, and a DLP (Digital Light Processing; registered trademark) method are adopted.

The video output unit 121 and the projection unit 122 have known structures. A general-purpose machine is used for the projector 120.

Camera 140 captures the performer AC acting in the studio ST. The camera 140 shoots at a predetermined frame rate. The camera 140 includes, for example, an exposure time setting unit 141, a synchronization control unit 142, a synchronization signal output unit 143, an exposure control unit 144, an imaging unit 145, and a recording unit 146.

The exposure time setting unit 141 sets the frame rate of the camera 140 and the exposure time (shutter speed) of the camera 140 within one frame period. The frame rate is set to, for example, a setting value input by the user. The exposure time is set, for example, manually or automatically. In the manual mode, the exposure time is set to a set value entered by the user. In the automatic mode, the exposure time is automatically set based on, for example, the brightness of the shooting environment. The exposure time setting unit 141 synchronously controls information (frame rate and exposure time) regarding the changed exposure timing when the exposure timing is changed by changing at least one of the frame rate and the exposure time, for example. Output to 142.

The synchronization control unit 142 generates a synchronization signal based on the exposure time. The synchronization control unit 142 outputs a synchronization signal to the synchronization signal output unit 143 and the exposure control unit 144.

The synchronization signal output unit 143 causes the transmitter 150 to transmit a signal indicating the exposure timing of the camera 140 based on the synchronization signal. The exposure control unit 144 opens and closes the camera shutter in accordance with the synchronization signal. The camera shutter is, for example, a focal plane shutter or a lens shutter provided inside the camera 140. The camera shutter opens only during the exposure period of the camera 140 so that the image sensor of the camera 140 is exposed to light only during shooting, and shields the light during other periods.

The transmitter 150 is, for example, an IR (Infrared) emitter that transmits an infrared signal. The transmitter 150 is connected to the camera 140 by, for example, a synchronization signal line.

The synchronization signal output unit 143 blinks the transmitter 150 for a predetermined time in accordance with the synchronization signal, for example. In the synchronous signal output unit 143, for example, the transmitter 150 is lit when the camera 140 is exposed (when the camera shutter is open), and the transmitter 150 is turned on when the camera is not exposed (when the camera shutter is closed). The transmitter 150 is blinked in a pattern that turns off the light. The exposure timing of the camera 140 and the lighting timing of the transmitter 150 may be opposite to each other.

The blinking time (time when a signal is transmitted from the transmitter 150) is determined based on the time required to synchronize the exposure timing of the camera 140 and the shielding timing of the shutter device 130. For example, the blinking time is a time of 1 frame period or more and 10 frame period or less.

The imaging unit 145 intermittently captures an image at a predetermined frame rate according to the opening / closing timing of the camera shutter. The recording unit 146 records an image taken by the imaging unit 145.

The shutter device 130 shields the light projected from the projector 120 and incident on the camera 140. The shutter device 130 receives, for example, a signal transmitted from the transmitter 150, and controls the shielding time based on this signal. The shutter device 130 synchronizes with the camera 140 and shields the light projected from the projector 120 in accordance with the exposure timing of the camera 140.

The shutter device 130 includes, for example, a receiving unit 131, a storage unit 132, a shutter control signal generation unit 133, a shutter control unit 134, and a shutter member 135.

The receiving unit 131 receives the signal transmitted from the transmitter 150. For example, the receiving unit 131 binarizes the amount of received light to detect whether the transmitter 150 is lit or not lit. The receiving unit 131 detects the exposure timing based on the received signal. For example, the receiving unit 131 detects the exposure time and the exposure cycle (frame rate) of the camera 140 based on the lighting time and the lighting cycle of the transmitter 150. The receiving unit 131 outputs information indicating the exposure timing of the camera 140 to the shutter control signal generation unit 133.

The shutter control signal generation unit 133 generates a shutter control signal based on the signal received by the reception unit 131. The shutter control signal is, for example, a signal having information on the shielding time and the shielding cycle corresponding to the lighting time and the lighting cycle of the transmitter 150. The shutter control unit 134 controls the shielding timing of the shutter member 135 based on the shutter control signal. As a result, the shutter device 130 dynamically changes the shielding time for shielding the light projected from the projector 120 according to the exposure time of the camera 140.

The shutter member 135 is, for example, a liquid crystal shutter whose visible light transmittance changes depending on the voltage. The shutter member 135 is arranged, for example, on the optical axis of the light projected from the projection unit 122. The shutter member 135 transmits, for example, the light projected from the projection unit 122 when no voltage is applied. For example, the shutter member 135 shields the light projected from the projection unit 122 when a voltage equal to or higher than the threshold value is applied.

The shutter device 130 includes a computer composed of a processor and a memory. The storage unit 132 stores the program PG that causes the computer to execute the processing of the shutter device 130. The shutter device 130 functions as a receiving unit 131, a storage unit 132, a shutter control signal generation unit 133, and a shutter control unit 134 by executing the program PG.

The storage unit 132 includes any non-transient storage medium such as a semiconductor storage medium and a magnetic storage medium. The storage unit 132 includes, for example, an optical disk, a magneto-optical disk, or a flash memory. The program is stored, for example, in a non-transient storage medium that can be read by a computer.

[1-2. Shooting method]
FIG. 2 is a flowchart illustrating an example of the photographing method of the present embodiment.

In step S1, the synchronization control unit 142 determines the blinking pattern of the transmitter 150 based on the exposure time set by the exposure time setting unit 141.

In step S2, the transmitter 150 starts blinking in the blinking pattern determined by the synchronization control unit 142. The transmitter 150 transmits a signal using blinking light as a signal indicating the exposure timing of the camera 140.

In step S3, the receiving unit 131 receives the signal transmitted from the transmitter 150. The shutter control signal generation unit 133 generates a shutter control signal based on the signal received by the reception unit 131. The shutter control unit 134 synchronizes the shielding timing of the shutter member 135 with the exposure timing of the camera 140 based on the shutter control signal.

In step S4, the exposure time setting unit 141 determines whether or not the exposure time has been changed. The exposure time setting unit 141 determines that the exposure time has been changed, for example, when the user manually sets the exposure time or when the exposure time is automatically changed based on the change in the brightness of the shooting environment. do.

If it is determined in step S4 that the exposure time has been changed (step S4: Yes), the process returns to step S1.

If it is determined in step S4 that the exposure time has not been changed (step S4: No), the process proceeds to step S5. In step S5, the camera 140 determines whether or not to end the shooting. The camera 140 determines that the shooting has been completed, for example, when the shooting end button (not shown) provided on the camera 140 is operated.

If it is determined in step S5 that the shooting is finished (step S5: Yes), the camera 140 ends the shooting. If it is not determined in step S5 that the shooting is completed (step S5: No), the process returns to step S4.

FIG. 3 is a flowchart illustrating an example of a process of synchronizing the camera 140 and the shutter device 130.

In step S11, the receiving unit 131 determines whether or not the transmitter 150 has started blinking. When the exposure time is changed, the transmitter 150 blinks for a predetermined time in a blinking pattern having a predetermined lighting time and a lighting cycle according to the exposure time. For example, when the frame rate of the camera 140 is 60 fps (frame per sec) and the exposure time is 4 milliseconds, the transmitter 150 lights up for 4 milliseconds out of 1/60 seconds, and the remaining time (= about 12). .7 milliseconds) repeats turning on and off so that it goes out. The receiving unit 131 determines that the blinking has started when the first lighting of the transmitter 150 is detected. The receiving unit 131 detects the timing at which the first lighting starts to be detected as the blinking start timing.

If it is determined in step S11 that blinking has started (step S11: Yes), the process proceeds to step S12. In step S12, the receiving unit 131 starts measuring the lighting time for one cycle from the lighting start timing. If it is not determined in step S11 that the blinking has started (step S11: No), step S1 is repeated until the blinking starts.

In step S13, the receiving unit 131 determines whether or not the lighting for one cycle has been completed.

When it is determined in step S13 that the lighting for one cycle is completed (step S13: Yes), the receiving unit 131 detects the timing when the lighting for one cycle is completed as the lighting end timing (lighting start timing). Then, in step S14, the measurement of the lighting time is completed. If it is not determined in step S13 that the lighting for one cycle is completed (step S13: No), step S13 is repeated until the lighting is completed.

In step S15, the receiving unit 131 starts measuring the extinguishing time for one cycle. Then, in step S16, the receiving unit 131 determines whether or not the lighting start timing of the next cycle after the elapse of the extinguishing time has been detected. When it is determined in step S16 that the lighting start timing of the next cycle is detected (step S16: Yes), the measurement of the extinguishing time for one cycle is completed. The receiving unit 131 detects the timing at which the lighting of the next cycle starts as the turning-off end timing for one cycle. If it is not determined in step S16 that the lighting start timing of the next cycle is detected (step S16: No), step S16 is repeated until the lighting start timing of the next cycle is detected.

In step S18, the receiving unit 131 detects the lighting time, the lighting time, and the lighting cycle for one cycle based on the detected lighting start timing, lighting end timing (lighting start timing), and lighting end timing for one cycle. do. The shutter control signal generation unit 133 determines, for example, the light shielding timing (shielding start time and shielding time) during one frame period based on the lighting time and the lighting cycle for one cycle. The shutter control signal generation unit 133 generates a shutter control signal based on the determined shielding timing.

4 and 5 are timing charts showing an example of the operation of the projector 120, the shutter device 130, the camera 140, and the transmitter 150.

During the exposure period (camera shutter open), the shutter device 130 is in a shielded state (ON). During the non-exposure period (camera shutter closed), the shutter device 130 is in the non-shielded state (OFF). The projector 120 always projects an image. The shutter device 130 shields the projected light from the projector 120 at the timing when the projection of the projector 120 and the shooting of the camera 140 overlap. The image projected from the projector 120 is intermittently shielded by the shutter device 130 according to the exposure timing of the camera 140. Similarly, the transmitter 150 is turned on during the exposure period and turned off during the non-exposure period.

FIG. 4 is a timing chart when the exposure period (camera shutter open) and the non-exposure period (camera shutter closed) of the camera 140 are equal. In the example of FIG. 4, the exposure time of the camera 140 and the non-shielding time of the shutter device 130 are equal. Therefore, a bright image is taken without significantly impairing the brightness of the projected image. FIG. 5 is a timing chart when the non-exposure time is longer than the exposure time. In the example of FIG. 5, a bright image is projected, but the brightness of the captured image is smaller than that of the example of FIG.

FIG. 6 is a diagram showing the relationship between the brightness of the captured image and the brightness of the projected image.

The brightness of the captured image and the brightness of the projected image change depending on the exposure time of the camera 140. The longer the exposure time, the brighter the image will be. On the other hand, the longer the exposure time, the longer the shielding time by the shutter device 130. Therefore, the projected image of the projector 120 becomes dark. Therefore, there is a trade-off between the brightness of the captured image and the brightness of the projected image. The user can manually set the exposure time based on the shooting phase, the studio ST environment, and the like in consideration of these trade-offs. Further, the exposure timing information detected by the receiving unit 131 can be output to the video output unit 121, and the video output unit 121 can correct the brightness of the projected image.

FIG. 7 is a flowchart showing an example of the content presentation process to the performer AC by the projector 120.

In step S21, the 3D model storage unit 111 reads and stores the 3D model of the studio ST from the information input to the information processing device 110. In step S22, the parameter storage unit 112 reads and stores, for example, the internal parameters of the projector 120 and the position / orientation information of the projector 120 in the 3D model from the input information.

In step S23, the display content generation unit 113 generates projection content including information that supports the performance of the performer AC. The corrected image generation unit 114 geometrically corrects the projection content using the internal parameters of the 3D model and the projector 120 and the position / orientation information of the projector 120 in the 3D model, and generates a corrected image.

In step S24, the projector 120 projects the corrected image onto the studio ST. In step S25, the corrected image is intermittently shielded by the shutter device 130 synchronized with the camera 140 in accordance with the exposure timing of the camera 140.

In step S26, the projector 120 determines whether or not to end the projection. The projector 120 determines that the projection has been completed, for example, when the projection end button (not shown) provided on the projector 120 is operated. If it is determined in step S26 that the projection has been completed (step S26: Yes), the projector 120 ends the projection. If it is not determined in step S26 that the projection has been completed (step S26: No), the process returns to step S23.

The method of generating the corrected video is not limited to the above. FIG. 8 is a diagram showing another example of the method of generating the corrected video PI. FIG. 9 is a diagram showing an example of a corrected image PI projected from the projector 120.

FIG. 8 shows a state in which a corrected video PI that supports the performances of two performers is input on the 3D model. In FIG. 8, as the corrected image PI, the standing position PI1 and the conducting wire PI2 for the performer AC1 and the standing position PI3 and the conducting wire PI4 for the performer AC2 are shown. A virtual camera having the same position and orientation and internal parameters of the current projector is placed on this 3D model, and the input content is photographed. By outputting the captured image as the projected image of the projector 120, the geometrically corrected image as shown in FIG. 9 is displayed on the floor surface of the studio ST.

[1-3. effect]
The photographing system 1 includes a camera 140, a projector 120, and a shutter device 130. The shutter device 130 synchronizes with the camera 140 and shields the light projected from the projector 120 and incident on the camera 140. In the photographing method of the present embodiment, the information processing of the photographing system described above is executed by the computer. The program of the present embodiment makes the computer realize the information processing of the above-mentioned photographing system.

According to this configuration, the light projected from the projector 120 is shielded by the shutter device 130 before being incident on the camera 140. Therefore, the light (image) projected from the projector 120 is not reflected in the image captured by the camera 140. The reflection of the projected image is suppressed by the shutter device 130 synchronized with the camera 140. Therefore, a general-purpose machine can be used for the projector 120. Therefore, a shooting system 1 capable of shooting at low cost while reducing the burden on the performer AC is provided.

For example, the shutter device 130 dynamically changes the shielding time for shielding the light projected from the projector 120 according to the exposure time of the camera 140.

According to this configuration, the projected light is suppressed from being reflected in the image of the camera even if the exposure time of the camera 140 changes.

The photographing system 1 has, for example, a transmitter 150. The transmitter 150 transmits, for example, a signal indicating the exposure timing of the camera 140. The shutter device 130 receives, for example, a signal transmitted from the transmitter 150, and controls the shielding timing based on this signal.

According to this configuration, the degree of freedom in arranging the camera 140 and the shutter device 130 is increased.

[2. Second Embodiment]
[2-1. Shooting system configuration]
FIG. 10 is a diagram showing a main part of the photographing system 2 of the second embodiment.

The difference between the first embodiment and the first embodiment is that the projector 125 is different from the first embodiment during the period from when the exposure timing of the camera 140 is changed until the shielding timing of the shutter device 136 is synchronized with the exposure timing of the camera 140 after the change. This is the point at which the projection of light by the camera is stopped. Hereinafter, the differences from the first embodiment will be mainly described.

The receiving unit 137 generates a projection stop signal of the projector 125 based on the signal transmitted from the transmitter 150, for example. For example, when the fluctuation of the exposure time is larger than the threshold value, the receiving unit 137 outputs the projection stop signal to the projector 125 only during the period of receiving the signal from the transmitter 150. The receiving unit 137 does not output the projection stop signal to the projector 125, for example, when the fluctuation of the exposure time is within the threshold value.

The video output unit 126 stops the output of the projected video signal to the projection unit 122 for a predetermined time after the transmitter 150 starts transmitting the signal, for example, based on the projection stop signal. The time at which the signal is transmitted from the transmitter 150 is set based on the time required to synchronize the camera 140 and the shutter device 136. Therefore, the projection of light by the projector 125 is stopped during the period from when the exposure timing of the camera 140 is changed until the shielding timing of the shutter device 136 is synchronized with the exposure timing of the camera 140 after the change. When the fluctuation of the exposure time is within the threshold value, the projection stop signal is not output to the projector 125. Therefore, the shielding time is not changed.

The shutter control signal generation unit 138 sets, for example, the shielding time of the shutter device 136 to be longer than the exposure time of the camera 140. For example, the shielding period of the shutter device 136 is set so that the shielding start timing is earlier than the exposure start timing and the shielding end timing is later than the exposure end timing in one frame period of the camera 140. The shutter control unit 134 controls the shielding timing of the shutter member 135 based on the shutter control signal in which the shielding time is set longer than the exposure time.

FIG. 11 is a timing chart showing an example of the operation of the projector 125, the shutter device 136, the camera 140, and the transmitter 150.

In the shielding period of the shutter device 136, a buffer period of Δt is provided on the shielding start side and the shielding end side, respectively. The shielding start timing is Δt earlier than the exposure start timing, and the shielding end timing is Δt later than the exposure end timing. The receiving unit 137 uses, for example, the buffer period as a threshold value as a reference for determining the projection stop. The receiving unit 137 generates a projection stop signal, for example, when the exposure time fluctuates beyond the buffer period. The receiving unit 137 does not generate a projection stop signal, for example, when the fluctuation of the exposure time falls within the buffer period.

[2-2. Shooting method]
FIG. 12 is a flowchart illustrating an example of the photographing method of the present embodiment.

In step S31, the projector 125 projects an image supporting the performance of the performer AC on the studio ST.

In step S32, the receiving unit 137 determines whether or not the exposure time of the camera 140 has been changed based on the signal transmitted from the transmitter 150. If it is not determined in step S32 that the exposure time has been changed (step S32: No), step S32 is repeated until the exposure time is changed.

If it is determined in step S32 that the exposure time has been changed (step S32: Yes), the process proceeds to step S33. In step S33, the receiving unit 137 determines whether or not the fluctuation of the exposure time exceeds the threshold value.

If it is determined in step S33 that the fluctuation of the exposure time exceeds the threshold value (step S33: Yes), the process proceeds to step S34. In step S34, the receiving unit 137 outputs a projection stop signal to the projector 125. The projector 125 stops the projection of the image based on the projection stop signal. Then, in step S35, the shutter control signal generation unit 133 generates a shutter control signal based on the exposure timing information detected by the reception unit 137.

The shutter control signal generation unit 138 sets the shielding time of the shutter device 136 to be longer than the exposure time of the camera 140. The shutter control unit 134 controls the shielding timing of the shutter member 135 based on the shutter control signal in which the shielding time is set longer than the exposure time.

If it is not determined in step S33 that the fluctuation of the exposure time exceeds the threshold value (step S33: No), the process proceeds to step S35.

After changing the shielding timing, in step S36, the camera 140 determines whether or not to end the shooting. If it is determined in step S36 that the shooting is finished (step S36: Yes), the camera 140 ends the shooting. If it is not determined in step S36 that the shooting is finished (step S36: No), the process returns to step S31.

[2-3. effect]
In the present embodiment, the projection of light by the projector 120 is stopped during the period from when the exposure timing of the camera 140 is changed until the shielding timing of the shutter device 130 is synchronized with the exposure timing of the camera 140 after the change. ..

According to this configuration, the projected light is suppressed from being reflected in the captured image of the camera 140 during the shielding timing adjustment period.

If the fluctuation of the exposure time is within the threshold value, for example, the shielding time is not changed.

According to this configuration, a dead zone area where the shielding time does not change is provided. Therefore, even if the exposure time is changed frequently, the presentation of information by the projector 120 is unlikely to be hindered. For example, in a configuration in which the projection of light is stopped during the adjustment period of the shielding timing, if the exposure time is changed frequently, the presentation of information by the projector 120 may be hindered. By providing a dead zone region in the exposure time, such a problem can be suppressed.

The shielding time of the shutter device 136 in one frame period of the camera 140 is longer than, for example, the exposure time of the camera 140.

According to this configuration, even if the exposure time fluctuates within the buffer period set outside the exposure period in the shielding period of the shutter device 136, the projected image is not reflected in the captured image. Further, by providing the buffer period in the shielding period, even if there is a lag in the processing of various devices, it is possible to suppress the projection image from being reflected in the captured image due to the lag. For example, the projected light of the projector 120 is reflected in the captured image of the camera 140 during the period from the start of the shutter device 130 to the end of the shielding and the period from the start of closing the camera shutter to the end of closing. Is suppressed.

[3. Third Embodiment]
[3-1. Shooting system configuration]
FIG. 13 is a schematic view of the photographing system 3 of the third embodiment.

The difference between the first embodiment and the first embodiment is that the shutter device 230 synchronizes with the projector 220 and shields the light projected from the projector 220 and incident on the camera 240. Hereinafter, the differences from the first embodiment will be mainly described.

The information processing device 210 has, for example, a projection time setting unit 215 in addition to the configuration of the first embodiment. The projector 220 has, for example, a synchronization control unit 223, a synchronization signal output unit 224, and a projection time control unit 225, in addition to the configuration of the first embodiment.

The projection time setting unit 215 sets, for example, the refresh rate of the projector 220 and the projection time of the projector 220 within one refresh period. The refresh rate and projection time are set, for example, to the set values entered by the user. For example, when at least one of the refresh rate and the projection time is changed, the projection time setting unit 215 outputs the changed refresh rate and the projection time to the synchronization control unit 223.

FIG. 14 is a diagram showing an example of a projection time setting screen.

The information processing device 210 displays the projection time setting screen on the display device 216, for example. The display device 216 is, for example, a display device with a built-in touch panel, and also serves as an input device for inputting to the projection time setting screen.

The projection time setting screen has, for example, an area for setting the refresh rate of the projector 220 and an area for setting the projection time of the projector 220. For example, the refresh rate is set by the user clicking a radio button. The magnitude of the refresh rate presented as an option is higher than, for example, the frame rate (24 fps) used in filming movies and dramas. Whichever option is selected, the refresh rate of the projector 220 is higher than the frame rate of the camera 240.

The settable range of projection time changes according to the refresh rate. For example, when the refresh rate is 60 Hz, the settable range of the projection time is in the range of 0.0 to 16.7 milliseconds. When the refresh rate is 120 Hz, the settable range of the projection time is in the range of 0.0 to 8.3 milliseconds. The projection time is set using, for example, the slide bar SB. For example, when the refresh rate is set to 60 Hz and the projection time is set to t seconds, the period from time 0 seconds to time t seconds within one refresh period is the projection period (the period during which the image is projected), and the rest. The period from time t seconds to time 16.7 seconds is the non-projection period (the period during which the image is projected).

Returning to FIG. 13, the synchronization control unit 223 generates a synchronization signal based on the projection time. The synchronization control unit 223 outputs a synchronization signal to the synchronization signal output unit 224 and the projection time control unit 225.

The synchronization signal output unit 224 causes the transmitter 150 to transmit a signal indicating the projection timing of the projector 220 based on the synchronization signal. The projection time control unit 225 outputs information regarding the projection timing to the video output unit 221. The video output unit 221 generates a projected video signal so that the video is projected for a set projection time during one refresh period, for example. As a result, the projection unit 122 intermittently projects an image in accordance with the synchronization signal.

The transmitter 150 is connected to the projector 120 by, for example, a synchronization signal line. The synchronization signal output unit 224, for example, causes the transmitter 150 to blink for a predetermined time in accordance with the synchronization signal. The synchronous signal output unit 224 blinks the transmitter 150 in a pattern such that the transmitter 150 is turned on when the projector is projected and the transmitter 150 is turned off when the projector 220 is not projected. The projection timing of the projector 220 and the lighting timing of the transmitter 150 may be opposite to each other.

The blinking time (time when a signal is transmitted from the transmitter 150) is determined based on the time required to synchronize the projection timing of the projector 220 with the shielding timing of the shutter device 130. For example, the blinking time is a time of 1 refresh period or more and 10 refresh periods or less.

The camera 240 does not have the synchronization control unit 142 and the synchronization signal output unit 143 as compared with the configuration of the first embodiment. The exposure time setting unit 141, the exposure control unit 144, and the recording unit 146 have known structures. A general-purpose machine is used for the camera 240.

The configuration of the shutter device 130 is the same as that of the first embodiment. The shutter device 130 receives, for example, a signal transmitted from the transmitter 150, and controls the shielding time based on this signal.

The shutter member 135 is arranged, for example, on the optical axis of the light incident on the camera 240. The shutter control signal generation unit 133 generates a shutter control signal based on the signal received by the reception unit 131. The shutter control signal is, for example, a signal having information on the shielding time and the shielding cycle corresponding to the lighting time and the lighting cycle of the transmitter 150. The shutter control unit 134 controls the shielding timing of the shutter member 135 based on the shutter control signal. As a result, the shutter device 130 dynamically changes the shielding time for shielding the light incident on the camera 240 according to the projection time of the projector 220.

[3-2. Shooting method]
FIG. 15 is a flowchart showing an example of the photographing method of the present embodiment.

In step S41, the 3D model storage unit 111 reads and stores the 3D model of the studio ST from the information input to the information processing device 110. In step S42, the parameter storage unit 112 reads and stores, for example, the internal parameters of the projector 220 and the position / orientation information of the projector 220 in the 3D model from the input information.

In step S43, the receiving unit 131 receives the signal transmitted from the transmitter 150. The shutter control signal generation unit 133 generates a shutter control signal based on the signal received by the reception unit 131. The shutter control unit 134 synchronizes the shielding timing of the shutter member 135 with the projection timing of the projector 220 based on the shutter control signal.

In step S44, the display content generation unit 113 generates projection content including information that supports the performance of the performer AC. The corrected image generation unit 114 geometrically corrects the projection content using the internal parameters of the 3D model and the projector 220 and the position / orientation information of the projector 220 in the 3D model, and generates a corrected image.

In step S45, the projector 120 projects the corrected image onto the studio ST. The camera 240 photographs the performer performing in the studio ST. The light incident on the camera 240 is intermittently shielded by the shutter device 130 synchronized with the projector 220 in accordance with the projection timing of the projector 220. As a result, it is possible to prevent the projected image from being reflected in the captured image of the camera 240.

In step S46, the projection time setting unit 215 determines whether or not the projection time has been changed. The projection time setting unit 215 determines that the projection time has been changed, for example, when the user manually sets the projection time.

If it is determined in step S46 that the projection time has been changed (step S46: Yes), the process returns to step S43.

If it is determined in step S46 that the projection time has not been changed (step S46: No), the process proceeds to step S47. In step S47, the projector 220 determines whether or not to end the projection. If it is determined in step S47 that the projection has been completed (step S47: Yes), the projector 220 ends the projection. If it is not determined in step S47 that the projection has been completed (step S47: No), the process returns to step S44.

16 and 17 are timing charts showing an example of the operation of the projector 220, the transmitter 150, the shutter device 130, and the camera 240.

During the projection period, the shutter device 130 is in the shielded state (ON). During the non-projection period, the shutter device 130 is in the non-shielded state (OFF). The projector 220 projects an image at a refresh rate higher than the frame rate of the camera 240, more preferably at a refresh rate that is at least twice the frame rate. The shutter device 130 shields the light incident on the camera 240 at the timing when the projection of the projector 220 and the shooting of the camera 240 overlap. The light incident on the camera 240 is intermittently shielded by the shutter device 130 in accordance with the projection timing of the projector 220. Similarly, the transmitter 150 is turned on during the projection period and turned off during the non-projection period.

FIG. 16 is a timing chart when the projection time and the non-projection time are equal. In the example of FIG. 16, the projection time of the projector 220 and the non-shielding time of the shutter device 130 are equal. Therefore, a bright image is taken without significantly impairing the brightness of the projected image. FIG. 17 is a timing chart when the non-projection time is longer than the projection time. In the example of FIG. 17, the projected image is dark, but the shielding time by the camera shutter is short, so that the brightness of the captured image is brighter than that of the example of FIG.

FIG. 18 is a diagram showing the relationship between the brightness of the captured image and the brightness of the projected image.

The brightness of the captured image and the brightness of the projected image change depending on the projection time of the projector 220. The longer the projection time, the brighter the image will be projected. On the other hand, the longer the projection time, the longer the shielding time by the shutter device 130. Therefore, the captured image becomes dark. Therefore, there is a trade-off between the brightness of the captured image and the brightness of the projected image. The user can manually set the projection time based on the shooting phase, the studio ST environment, and the like, taking these trade-offs into consideration. Further, the projection time information set by the projection time setting unit 215 can be output to the video output unit 221, and the brightness of the projected image can be corrected by the video output unit 221.

[3-3. effect]
In the present embodiment, the shutter device 130 dynamically changes the shielding time for shielding the light incident on the camera 240 according to the projection time of the projector 220.

According to this configuration, the projected light is suppressed from being reflected in the image of the camera 240 even if the projection time of the projector 220 changes.

The refresh rate of the projector 220 is higher than the frame rate of the camera 240, for example.

According to this configuration, the problem that the incident light to the camera 240 is always blocked by the shutter device 130 during one frame period is unlikely to occur.

The photographing system 3 has, for example, a transmitter 150. The transmitter 150 transmits, for example, a signal indicating the projection timing of the projector 220. The shutter device 130 receives, for example, a signal transmitted from the transmitter 150, and controls the shielding time based on this signal.

According to this configuration, the degree of freedom in arranging the projector 220 and the shutter device 130 is increased.

[4. Fourth Embodiment]
[4-1. Shooting system configuration]
FIG. 19 is a diagram showing a main part of the photographing system 4 of the fourth embodiment.

The difference between the third embodiment and the third embodiment is that the projector 226 is different from the third embodiment during the period from the change of the projection timing of the projector 226 to the synchronization of the shielding timing of the shutter device 230 with the projection timing of the changed projector 226. The point where the projection of light by is stopped. Hereinafter, the differences from the third embodiment will be mainly described.

The synchronization control unit 227 generates, for example, a projection stop signal of the projector 226 based on the projection time information output from the projection time setting unit 215. For example, when the fluctuation of the projection time is larger than the threshold value, the synchronization control unit 227 outputs the projection stop signal to the video output unit 228 for a predetermined time after the transmitter 150 starts transmitting the signal. The synchronization control unit 227 does not output a projection stop signal to the video output unit 228, for example, when the fluctuation of the projection time is within the threshold value.

The video output unit 228 stops the output of the projected video signal to the projection unit 122 for a predetermined time after the transmitter 150 starts transmitting the signal, for example, based on the projection stop signal. The time at which the signal is transmitted from the transmitter 150 is set based on the time required to synchronize the projector 226 and the shutter device 230. Therefore, the projection of light by the projector 226 is stopped during the period from when the projection timing of the projector 226 is changed until the shielding timing of the shutter device 230 is synchronized with the projection timing of the changed projector 226. When the fluctuation of the projection time is within the threshold value, the projection stop signal is not output to the video output unit 228. Therefore, the shielding time is not changed.

The shutter control signal generation unit 233 sets, for example, the shielding time of the shutter device 230 in one refresh period of the projector 226 to be longer than the projection time of the projector 226. For example, the shielding period of the shutter device 230 is set so that the shielding start timing is earlier than the projection start timing and the shielding end timing is later than the projection end timing in one refresh period of the projector 226. The shutter control unit 134 controls the shielding timing of the shutter member 135 based on the shutter control signal in which the shielding time is set longer than the projection time.

FIG. 20 is a timing chart showing an example of the operation of the projector 266, the transmitter 150, the shutter device 230, and the camera 240.

In the shielding period of the shutter device 230, a buffer period of Δs is provided on each of the shielding start side and the shielding end side. The shielding start timing is Δs earlier than the projection start timing, and the shielding end timing is Δs later than the projection end timing. The synchronization control unit 227 uses, for example, the buffer period as a threshold value as a reference for determining the projection stop. The synchronization control unit 227 generates a projection stop signal, for example, when the projection time fluctuates beyond the buffer period. The synchronization control unit 227 does not generate a projection stop signal, for example, when the fluctuation of the projection time falls within the buffer period.

[4-2. Shooting method]
FIG. 21 is a flowchart illustrating an example of the photographing method of the present embodiment.

In step S51, the projector 226 projects an image supporting the performance of the performer AC on the studio ST.

In step S52, the synchronization control unit 227 determines whether or not the projection time of the projector 226 has been changed based on the projection time information output from the projection time setting unit 215. If it is not determined in step S52 that the projection time has been changed (step S52: No), step S52 is repeated until the projection time is changed.

If it is determined in step S52 that the projection time has been changed (step S52: Yes), the process proceeds to step S53. In step S53, the synchronization control unit 227 determines whether or not the fluctuation of the projection time exceeds the threshold value.

If it is determined in step S53 that the fluctuation of the projection time exceeds the threshold value (step S53: Yes), the process proceeds to step S54. In step S54, the synchronization control unit 227 outputs a projection stop signal to the video output unit 228. The projector 226 stops the projection of the image based on the projection stop signal. Then, in step S55, the shutter control signal generation unit 233 generates a shutter control signal based on the projection timing information detected by the reception unit 131.

The shutter control signal generation unit 233 sets the shielding time of the shutter device 230 longer than the projection time of the projector 226. The shutter control unit 134 controls the shielding timing of the shutter member 135 based on the shutter control signal in which the shielding time is set longer than the projection time.

If it is not determined in step S53 that the fluctuation of the projection time exceeds the threshold value (step S53: No), the process proceeds to step S55.

After changing the shielding timing, in step S56, the camera 240 determines whether or not to end the shooting. If it is determined in step S56 that the shooting is finished (step S56: Yes), the camera 240 finishes the shooting. If it is not determined in step S56 that the shooting is finished (step S56: No), the process returns to step S51.

[4-3. effect]
In the present embodiment, for example, the projection of light by the projector 226 is stopped during the period from the change of the projection timing of the projector 226 to the synchronization of the shielding timing of the shutter device 230 with the projection timing of the changed projector 226. Will be done.

According to this configuration, the projected light is suppressed from being reflected in the image of the camera 240 during the adjustment period of the shielding timing.

If the fluctuation of the projection time is within the threshold value, for example, the shielding time is not changed.

According to this configuration, a dead zone area where the shielding time does not change is provided. Therefore, even if the projection time is changed frequently, the presentation of information by the projector 226 is unlikely to be hindered.

The shielding time of the shutter device 230 in one refresh period of the projector 226 is longer than, for example, the projection time of the projector 226.

According to this configuration, even if the projection time fluctuates within the buffer period set outside the projection period during the shielding period of the shutter device 230, the projected image is not reflected in the captured image. Further, by providing the buffer period in the shielding period, even if there is a lag in the processing of various devices, it is possible to suppress the projection image from being reflected in the captured image due to the lag. For example, the projected light is reflected in the image of the camera 240 during the period from the start of the shutter device 230 to the end of the shielding and the period from the start of the projector 226 displaying the image on one screen to the end of the display. Is suppressed.

[5. Modification example]
In the above embodiment, an example in which only one projector is installed in the studio ST is shown, but the number of projectors is not limited to one. A plurality of projectors may be installed in the studio ST so that the entire area of the studio ST can be projected. When a plurality of projectors are installed in the studio ST in the first embodiment and the second embodiment, a shutter device is provided for each projector. When a plurality of projectors are installed in the studio ST in the third embodiment and the fourth embodiment, the projectors and the transmitters are synchronized with each other. When projecting the entire area of the studio ST with a small number of projectors, a projector capable of projecting an image in any direction by a drive mechanism may be used.

In the first embodiment, the timing of turning on and off the transmitter 150 and the timing of shielding and transmitting through the shutter device 130 coincide with each other. However, as in the second embodiment, a buffer period may be provided in the shielding period of the shutter device 130. In this case, the shielding time of the shutter device 130 in one frame period of the camera 140 is longer than the exposure time of the camera 140. According to this configuration, even if there is a lag in the processing of various devices, it is possible to prevent the projected image from being reflected in the captured image due to the lag.

The following can be considered as equipment lags.
(A) Shooting camera There is a lag of operation from when the camera shutter starts to close until the light is completely blocked, or vice versa.
(B) Projector When a laser projector is used, there is a lag from image projection by the raster scan method or image projection to non-projection.
(C) Shutter device When using a liquid crystal shutter, there is a lag in operation from the start of shielding until the entire screen is shielded, or vice versa.

For example, when determining the size of the lag of the shutter device 130, the following measurement is performed. In the case of the configuration of the first embodiment, first, a photodiode is installed on the light emitting side of the shutter device 130. Then, the intensity of the projected light of the projector 120 is measured through the shutter device.

FIG. 22 is a diagram showing an example of the measurement result of the light intensity measured through the shutter device. When the shutter device changes from the on state to the off state and when the shutter device changes from the off state to the on state, a slight deviation occurs between the switching timing of the shutter device 130 and the switching timing of the light intensity. By measuring this deviation, the size of the required buffer period can be detected.

In the above embodiment, the liquid crystal shutter is used as the shutter device, but the shutter device is not limited to the liquid crystal shutter. A rotating wheel provided with a light transmitting window may be used to switch between a light transmitting state and a light blocking state. By changing the rotation speed of the rotating wheel, the shielding cycle is changed. The shielding time is changed by providing a mechanism capable of continuously changing the size of the light transmitting window.

In the above embodiment, the IR emitter is used as the transmitter 150, but the transmitter 150 is not limited to the IR emitter. Synchronization may be performed by the shutter device communicating with the camera or projector by wire or wirelessly.

Note that the effects described in this specification are merely examples and are not limited, and other effects may be obtained.

Note that this technology can also take the following configurations.

(1)
With the camera
With a projector
A shutter device that synchronizes with the camera or the projector and shields the light projected from the projector and incident on the camera.
Shooting system with.
(2)
The imaging system according to (1), wherein the shutter device dynamically changes the shielding time for shielding the light projected from the projector according to the exposure time of the camera.
(3)
It has a transmitter that transmits a signal indicating the exposure timing of the camera.
The imaging system according to (2), wherein the shutter device receives the signal transmitted from the transmitter and controls the shielding time based on the signal.
(4)
During the period from when the exposure timing of the camera is changed until the shielding timing of the shutter device is synchronized with the exposure timing of the camera after the change, the projection of light by the projector is stopped in (3). Described shooting system.
(5)
The imaging system according to (4) above, wherein the shielding time is not changed when the fluctuation of the exposure time is within the threshold value.
(6)
The imaging system according to (5), wherein the shielding time of the shutter device in one frame period of the camera is longer than the exposure time of the camera.
(7)
The imaging system according to (1), wherein the shutter device dynamically changes the shielding time for shielding light incident on the camera according to the projection time of the projector.
(8)
The imaging system according to (7), wherein the refresh rate of the projector is larger than the frame rate of the camera.
(9)
It has a transmitter that transmits a signal indicating the projection timing of the projector.
The imaging system according to (7) or (8), wherein the shutter device receives the signal transmitted from the transmitter and controls the shielding time based on the signal.
(10)
During the period from when the projection timing of the projector is changed until the shielding timing of the shutter device is synchronized with the projection timing of the projector after the change, the projection of light by the projector is stopped in (9). Described shooting system.
(11)
The imaging system according to any one of (7) to (10), wherein the shielding time of the shutter device in one refresh period of the projector is longer than the projection time of the projector.
(12)
The imaging system according to (11) above, wherein the shielding time is not changed when the fluctuation of the projection time is within the threshold value.
(13)
A shutter device synchronized with a camera or a projector shields light projected from the projector and incident on the camera.
A method of shooting that is performed by a computer.
(14)
A shutter device synchronized with a camera or a projector shields light projected from the projector and incident on the camera.
A program that makes a computer realize that.

1,2,3,4

Shooting system

120, 125, 220, 226

Projector

130, 136, 230

Shutter device

140, 240 Camera 150 Transmitter PG program

Claims

With the camera
With a projector
A shutter device that synchronizes with the camera or the projector and shields the light projected from the projector and incident on the camera.
Shooting system with.
The imaging system according to claim 1, wherein the shutter device dynamically changes the shielding time for shielding the light projected from the projector according to the exposure time of the camera.
It has a transmitter that transmits a signal indicating the exposure timing of the camera.
The imaging system according to claim 2, wherein the shutter device receives the signal transmitted from the transmitter and controls the shielding time based on the signal.
The third aspect of claim 3, wherein the projection of light by the projector is stopped during the period from when the exposure timing of the camera is changed until the shielding timing of the shutter device is synchronized with the exposure timing of the camera after the change. Shooting system.
The imaging system according to claim 4, wherein the shielding time is not changed when the fluctuation of the exposure time is within the threshold value.
The imaging system according to claim 5, wherein the shielding time of the shutter device in one frame period of the camera is longer than the exposure time of the camera.
The imaging system according to claim 1, wherein the shutter device dynamically changes the shielding time for shielding light incident on the camera according to the projection time of the projector.
The imaging system according to claim 7, wherein the refresh rate of the projector is larger than the frame rate of the camera.
It has a transmitter that transmits a signal indicating the projection timing of the projector.
The imaging system according to claim 7, wherein the shutter device receives the signal transmitted from the transmitter and controls the shielding time based on the signal.
The ninth aspect of claim 9 is that the projection of light by the projector is stopped during the period from when the projection timing of the projector is changed until the shielding timing of the shutter device is synchronized with the projection timing of the projector after the change. Shooting system.
The imaging system according to claim 7, wherein the shielding time of the shutter device in one refresh period of the projector is longer than the projection time of the projector.
The imaging system according to claim 11, wherein the shielding time is not changed when the fluctuation of the projection time is within the threshold value.
A shutter device synchronized with a camera or a projector shields light projected from the projector and incident on the camera.
A method of shooting that is performed by a computer.
A shutter device synchronized with a camera or a projector shields light projected from the projector and incident on the camera.
A program that makes a computer realize that.