WO2020237268A1

WO2020237268A1 - Image capturing device and illumination setup

Info

Publication number: WO2020237268A1
Application number: PCT/AT2020/000012
Authority: WO
Inventors: Friedrich Reining
Original assignee: Friedrich Reining
Priority date: 2019-05-31
Filing date: 2020-05-26
Publication date: 2020-12-03
Also published as: AT522631A1

Abstract

The present invention relates to an image capturing device (1) together with an illumination setup such as a flashlight system (2,3,4) or a set of smartphones that allow to adjust the direction and intenstity of exposition after the picture is taken.

Description

IMAGE CAPTURING DEVICE AND ILLUMINATION SETUP

Field of the Invention

The present invention relates to an image capturing device together with an illumination setup such as flashlight system or a set of smartphones.

Background Art

When taking a picture with a camera the light setup is one of the most important things to consider. There is no right or wrong in the light setup, but most setups use a combination of different light sources to create the naturalness of an everyday illumination. Light sources be point sources which create very distinct shadows. This type of light is called hard light and does not reflect the typical daylight but can be found within direct sun exposure. If clouds screen the sun, the sunlight is being scattered into millions of light sources that lead to soft light and therefore soft shadows.

In portrait photography a quite common technique uses the so-called soft box comparable to the above-mentioned cloud situation. This setup transforms the light from a flashlight into many light sources evenly spaced on a large screen, which gives very soft shadows but still a main illumination direction, which is sometimes complemented by a reflector or a second (softer) flashlight to maintain spatiality of the scenery.

Today's flash technology offers a variety of flashlights and flashlight control units. All of them are dedicated to optimizing the light exposure with respect to other settings of the camera as there for example expo sure time or ISO settings of the sensor.

Sophisticated flashlight systems use additional pre-light bursts to calcu late the correct power emitted by all flashlights to even further optimize correct scenery illumination.

Once the image is taken a 2D version of the three-dimensional scenery is represented by millions of color and light values with respect to the position of each pixel on the sensor chip. At this stage post processing can alter illumination, color, and many other properties of the 2D im age, known also by the term image processing.

To alter the illumination after the picture is taken a state-of-the-art method makes use of the fact, that the image is stored in 3 black white images dedicated to the colors red, blue and green.

If the light exposure uses colored light from different directions, it is possible to use a post processing that weights each of the 3 black white channels of the image to emphasize a specific light direction accord ingly. This method results in a final black white image with adjustable spatial illumination feature. Since the differentiation between the light directions is based on color, the resulting black white image is not a true black white image, for following reason:

If say the object of interest is green, the red light will hardly be re flected on that object and hence the red channel will not show too much of that object. If the object is grey, this approach will give reasonable results for any color is represented in grey equally. SUMMARY OF THE INVENTION

It is an object of the present invention to provide an image capturing device that allows to adjust the direction of lights after the picture is taken. Any combination of light sources can be adjusted in a post pro cessing step. In contrast to the state-of-the-art technique the present invention can be applied to any object color and color images.

It is a further object to provide the ability to decide in a post processing step, which lightening direction applies to which part of the image. This allows to get rid of unwanted shadows in some areas of the picture.

The above objects are achieved by means of a flash lightening device comprising :

A photo-camera or movie camera, typically of digital model, referred to the "camera", a control module connected to the camera that is trig gered by the cameras X-sync signal, which is the simplest to explain and implement: the flash is fired at the instant the shutter is fully open.

One or more flashlights are connected to the control module and are triggered by a defined electrical signal on the sensing connection of the flashlight.

Once the camera has adjusted the focus a set of pictures is taken at a possibly high rate. The switching module recognizes each sync signal and multiplexes the cameras sync signal to subsequent flashlights. With this approach a set of pictures is taken where every picture is illuminated by only one of the flashlights. If the geometrical relation object to camera is static during the period when the set of pictures is taken, it is easy to combine these single pictures, referred to a "frame" by means of a post processing program like Photoshop into a final image. In this case the camera has not to be driven at high repetition rates, there is even no need to have more than one flashlight. This setup allows to take picture after picture and change the flashlight position from picture to picture.

A completely different situation applies to a non-static scenery as can be found in portrait photography. The subject in this case may be instructed to keep still but even then, the above-mentioned method of stacking the frames and combining them into a final image will fail. There is a tradeoff between speed of the repetition rate and the image resolution. Speaking of portrait photography todays cameras can easily achieve a point accuracy that allows to identify whiskers on a face.

Current repetition rates are in the area of a movie say 24 frames per second, which allows to take some images within a timeframe of 300ms in which a subject will move more than few tens of micrometers and hence a shift of the image from frame to frame will be unavoidable. A shift of the image is somehow the best case, practically speaking there are changes in the 3D scenery that cannot even modeled by a set of 2D frames representing the 3D scenery e.g. if the subject turns the head.

A shift of the pictured subject between the frames is one of the basic features of video tracking and video coding techniques, which are state of the art and can be used to compensate these artefacts. The naming of these techniques does furthermore disclose their limitations: They are optimized to find correlations between frames where the illumination is supposed to be the hardly changed. If the above-mentioned setup uses e.g. 2 flashlights that illuminate a face from the left side for frame 1 and from the right side for frame 2, these video tracking algorithms will fail for there is no correlation be tween these two images.

The utility model DE202017007065U1 describes very comprehensively the implementation of a complex lighting setup to simulate effects such as flashing blue lights from an emergency service, fire and the like in filming.

The elements described, such as controls, lighting means and the DMX protocol mentioned are state of the art and enable lighting effects to be realized on a stage or in film recordings.

The essential difference from the present invention is found in the lack of relationship of subsequent recorded images, since this is not the aim of the lighting setup described.

The terms "master" and "slave" are mentioned in both writings, but have completely different meanings:

In the present invention, the master frame is a captured image that is used for the later calculation of the movement-induced differences and their compensation.

In the utility model, "master" [0027] is defined with respect to a light source with which secondary light sources "slaves" are synchronized. There is no need to compensate for movement-induced differences between the individual image frames in the utility model since no individ ual image is to be synthesized from the recorded sequence. Rather, the complex lighting should create the most believable ambience of the film scene in focus.

Thus, the present invention cannot be derived from the utility model since a less complex wiring of lighting fixtures does not solve the problem of detecting movement-induced differences between individual frames.

The patent specification US2003165335A1 "slave flash controlling device and slave flash device" solves the problem of exposure with the aid of an external flash.

Specifically, the efficient use of the flash energy of the secondary flash is focus of this invention. One feature of the improvement relates to the evaluation of the relationship between the properties of the pre-flash of the master flash unit and the secondary flash unit.

All these properties relate to the illumination of a photo, the optimization of which is based on a pre-flash for light estimation.

The present invention already presupposes optimal lighting using these methods and furthermore requires an adaptation of the so-called master frames' illuminance during the image sequence-generation. The characteristic of the master frame is the exposure of the image scene through all the light sources involved. To prevent overexposure of that master frame, the exposure levels must be reduced accordingly. Furthermore, the patent specification "slave flash controlling device and slave flash device" describes the implementation via a circuit technology that works without memory.

Thus, the present invention cannot be derived from the patent specifi cation "slave flash controlling device and slave flash device" since transferring the pre-flash and main flash properties to a secondary flash does not avoid a possibly overexposure of the master frame, embedded in an image sequence.

The US2013120636A1 "Illumination system" describes a similar exposure setup as in the present invention, but differs in the basic application and the problems to be solved:

The setup described allows a photographer to adjust the light control of the lighting fixtures until an image with the desired exposure has been taken.

The present invention attempts to place this step of the adjustability of the exposure AFTER the image acquisition, which enables a multitude of further applications, such as partial exposure control as described in claim 1. The disturbing drop shadow of a body part, e.g. the nose, can be mentioned here as to be solved by partial exposure control. While in the prior art these shadow areas can only be influenced by partial "brightening", the present invention offers the possibility of replacing these problematic image parts from one or more images of the se quence which do not show any shadows at this point. Thus, the present invention cannot be derived from the "Illumination system" patent, since the exposure setup alone is not sufficient to make the exposure setting subsequently adjustable.

It is subject to this invention to overcome this problem by pasting a master frame in the set of taken frames which has following properties:

The master frame is characterized by the fact, that all flashlights are fired at the same time and that certain modifications on the optical setup are applied in order not to overexpose the master frame. These modifications are e.g. applying less optical power by the flashlights or reducing the sensitivity of the sensor for that specific frame. While the first modification can be handled by the control module, the second ap proach requires a completely different setup, where the control mecha nism of the control module is transferred to the camera itself.

The timewise positioning of the master frame in between the image se quence can be chosen arbitrarily but will be set advantageously in the middle of the image sequence to have least changes between the first and the last frame.

After the set of pictures is taken, all frames are processed with refer ence to the master frame which defines the target position of the subject containing all light directions and hence can be used to correlate each of the frames with it.

The compensating techniques are not subject to this invention, sufficient to say, that there a simple technique that simply shift the whole image by some pixels as well very sophisticated techniques which esti mate the 3D scenery and apply the estimated trajectories to each pixel of each frame.

By now the camera is expected to have focused before the set of pic tures is taken which requires for most cameras some light to gain optimal performance. If distinct light directions are of high interest, the am bient light must be kept at a very low level. To provide sufficient light for focusing and reduce the ambient light during the flashlight impulse train, following modification must be implemented in the control mod ule.

The switching module is expanded by another electrical output which is switched once the first flashlight signal has been recognized by the con trol module. If this output is connected to the ambient light, the ambi ent light is switched off once the first frame is taken by the camera and kept off until the acquisition process of the whole sequence is finished.

This means that the first frame taken by the camera is not illuminated by the first flashlight but illuminated by the ambient light which was used to ease focusing the subject or object to be pictured.

The basic principle of time multiplexing implicates some restrictions to the setup which on the one hand do not allow complex movements dur ing the shooting, but can provide on the other hand additional values by augmented post processing features if limitations are met within the scenery setup.

Today's smartphones offer already an image quality that comes remark ably close to that of a typical SLR or reflex camera. The invention is therefore not only related to the classical studio setup with a camera and external flashlights but can be easily adapted to one or more smartphones.

While the camera module of the smartphone can be easily identified as the image acquisition module of the invention, the control module is distributed over many modules defined by hard and software.

A possible implementation of the invention uses the smartphone as the image acquisition and a simple software application to convert the image acquisition moments into electrical signals available at any of the given interfaces (e.g. phone plug, or USB connector). All other components remain the same as described above. The above-mentioned solu tion with ambient light to solve for the right focus can be also solved easily with that smartphone setup. The said software application can be used further on to find the optimal optical parameters by either using the ambient light setup mentioned above. Due to the fact, that a flash is usually available in a smartphone this can be used to acquire a first image with that built in flash of the camera to base optimal camera-set- tings as there are correct shutter and focus on that image. These pa rameters are kept constant for the subsequent image sequence during the acquisition process.

Another implementation goes one step further and uses a more complex software application to take over the control function and dedicated distribution of trigger to each of the available flashlights via wireless communication as there are e.g. NFC, Wi-Fi or Bluetooth referred to as standard wireless communication protocols. In this implementation the flashlights need to be augmented by a receiving function for the chosen wireless communication protocol. Another implementation goes again one step further and uses other smartphones as the flashlights where the built-in standard wireless communication protocols of said other smartphones need to be routed to the flashlight module via a software application running on each of the smartphones.

Another implementation uses above said setup with several

smartphones, with the difference, that all smartphones are using their camera-module at every trigger pulse to acquire an image. This leads to a multiple set of images, each of which can be used to synthesize an ar bitrary illumination setup per viewpoint of a specific smartphone in volved in the shooting.

Yet another implementation uses a different method of communication between the involved devices, referred as the time delay approach. To this point all described setups require a very precise synchronization between the moment of image acquisition and triggering of the flashlights. While this is quite easy to achieve in a wired setup, most of the availa ble wireless protocols are not supporting low latency behavior because they are dedicated to data transfer at high data rate whereas latency of a single data package is of minor significance and has only to be within a certain period of time. To avoid any latency issues the time-delay ap proach splits up the information about the moment of trigger and which flashlight to address. This works as follows:

Each smartphone runs an application, connected via the standard wire less communication protocols. In the initialization process these applica tions are negotiating the role of each device, e.g. which of the

smartphones to act as the master, and which flashlight to be addressed by which ID - an approach already found within several networkable de vices. Furthermore, the individual time-delay for each ID is distributed.

Once the master-device starts the image acquisition process, the master smartphone flashes its flashlight to take the first frame in order to adjust the optical system (exposure time, ISO sensitivity, etc.). At the same time all involved smartphones receive this flash impulse via their camera system and trigger a counter in their application. The afore communicated individual time delay for each involved device between this received trigger and the individual flashlight activation only relies on a synchronized time-base among all smartphones, which is expected to be far more accurate than the overall latency found within standard wireless communication protocols.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 shows a schematic illustration of an embodiment of the photo camera flash device and their electrical connections;

Figure 2 shows a schematic illustration of a studio setup using the photo camera flash device of Figure 1 including the scenery;

Figure 3 shows a schematic illustration of the different states of the signal levels while taking the set of images;

Figure 4 shows a schematic illustration of a studio setup using

smartphones only; Figure 5 shows a schematic illustration of a studio setup using

smartphones only, where each smartphone is in addition to acting as il lumination source acquiring an image sequence accordingly to its view point;

Figure 6 shows a schematic illustration of the different states of the sig nal levels while taking the set of images using smartphones;

Figure 7 shows a schematic illustration of the different states of the sig nal levels while taking the set of images if a above mentioned time de lay approach is used to synchronize the images.

Figure 8 shows a schematic illustration of a hybrid studio setup using ambient light, external flashlights and smartphones;

DETAILED DESCRIPTION OF EMBODIMENTS

Figure 1 shows a schematic representation of an embodiment of the photo camera flash device and their electrical interconnections in case of an external switching module. This means, that the signal flow from the camera to the switching module is unidirectional by the X-sync sig nal or any other signal that indicates a frame being taken. This means that the camera is taking frame by frame without controlling the illumi nation at each frame.

Power supply for the camera (1), the switching module (2), the ambient light (4) and the flashlights (3a, 3b, 3c, 3d) are not drawn. For safety reasons all parts are electrically isolated using separate power supplies, signal interconnections are implemented via optocouplers to prevent damage due to a possible failure in the high voltage circuitry inside of the flashlights.

Figure 2 shows a schematic representation of a portrait studio scenery including the setup for the photo camera flash device. All light sources are aligned around the subject including the ambient light. To minimize the differences between the frames by means of the portrait position a camera stand is required.

Figure 3 shows a set of diagrams showing the different states of the signal levels at the input of the component, e.g. one of the flashlights, while taking the set of pictures.

The first diagram shows the pulse train originated from the camera (1) while keeping the release button pressed for approximately Is. Every pulse indicates a frame taken by the camera, which reads in this example 6 frames.

The first pulse of the pulse-train (6) triggers the ambient light (4) to switch off (9) and remain dark until the set of pictures is taken. A simple timer in the switching box ensures, that the camera triggers all the flashlights while the ambient light remains switched off.

The second pulse of pulse-train (6) originated by the camera (1) trig gers the first flashlight(3a) to flash, where the duration of the pulse is altered by the switching module to the required light power accordingly. There are so many ways to realize an optimized illumination by fast switching flash circuitries, which are not part of the present invention and hence are not described in detail here. The pulse duration in this example stands for any kind of light power management and can be ad justed even manually before the actual picture set is taken.

The third camera pulse of the pulse train (6) triggers the second flash- light(3b), whereas the 4^th pulse (8) triggers all flashlights(3a,3b,3c,3d) at once in order to illuminate the master frame. Note that the duration of the 4^th pulse (8) delivered to the flashlights is significantly shorter, for the picture is being illuminated by all flashlights at once which re leases a lot more light energy compared to the single flashlight case. The light power management as mentioned above, it is not part of this invention for there are many ways for realizing equivalent illumination on the master frame compared to the other frames. The illustration in Figure 3 simply shows a shorter pulse (8) which can be used to reduce the emitted power per flashlight by switching the flashlights off before the whole energy, stored in the capacitor, is being discharged.

The fifth and sixth camera pulse found in the pulse train triggers (7) the third (3c) and fourth flashlight (3d) accordingly.

This above described setup describes an embodiment where the control mechanism is found in the control module, the camera itself is expected to take frame by frame with the focus settings gained from the ambient light situation and the aperture and shutter time as adjusted manually by the photographer.

A completely different situation arises if the camera takes over the control management and triggers the flashlights via a parallel- or serial data interface. In this case the correct illumination can be calculated by the camera in real time without any manual pre-settings. Figure 4 shows a schematic representation of a portrait studio scenery including the setup for use with smartphones (100-104) only. All smartphones are aligned around the subject - again to minimize the differences between the frames by means of the portrait position a fixture for the smartphone acquiring images (100) is recommendable. The control unit (2a-2e) is distributed over all involved smartphones the master smartphone (100) is using the camera module (1) to acquire the images and controls the flashlights (3a-3d) via the wireless connected control units (2a-2e).

Figure 5 shows a schematic illustration of a studio setup using

smartphones(100-104) only, where each smartphone is in addition to acting as illumination (3a-3d) source acquiring an image sequence accordingly to its viewpoint by using the built in camera module (la-le). As mentioned above the distributed control units (2a-2e) take care of a proper timewise alignment of image acquisition and flashlight activation.

Figure 6 shows a schematic illustration of the different states of the signal levels while taking the set of images using smartphones (100-104) together with ambient lights (4a, 4b) as can be found in the schematic illustration of figure 8. The distributed control units (2a-2e) are not only found within the involved smartphones but also found in an external control unit(2b) connected to the ambient lights (4a, 4b).

Figure 7 shows a schematic illustration of the different states of the signal levels while taking the set of images if an above-mentioned time de lay approach is used to synchronize the image acquisition process. Each smartphone (100-104) runs an application which are connected via the standard wireless communication protocols. In the initialization process the master smartphone (100) negotiates the ID or address of each device. Furthermore two timing parameters are distributed to each slave smartphone (101-104): The time delay (20) between the initial flashlight of the master smartphone (100) and the trigger to illuminate the master-frame is the same for all involved devices. The second timing parameter is different for each smartphone in order to generate the expected illumination scenario. This means smartphone (101) is set to the timing parameter (20) and (21) during the initialization phase, whereas smartphone (102) is set to timing parameter (20) and (22), smartphone 103 is set to (20) and (23), smartphone 104 is set to (20) and (24).

Figure 8 shows a schematic illustration of a hybrid studio setup using ambient light (4a, 4b) and an external flashlight (3b) controlled by an external control unit (2b) and smartphones (100-104) together with the camera unit of the master smartphone (100) and the distributed control units (2a, 2c, 2d,2e) among the slave smartphones (101-104). Furthermore, all slave smartphones are using their flashlight units (3b, 3c, 3d) to illuminate the scenario from different viewpoints.

Reference throughout the specification to "various embodiments," "some embodiments," "one embodiment," or "an embodiment," or the like, means that a particular feature, structure, or characteristic de scribed in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases "in various embodi ments," "in some embodiments," "in one embodiment," or "in an em bodiment," or the like, in places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Thus, the particular features, structures, or characteristics illustrated or described in connection with one embodiment may be combined, in whole or in part, with the features, structures, or characteristics of one or more other embodiments without limitation given that such combination is not illogical or non-functional.

It must be noted that, as used in this specification and the appended claims, the singular forms "a," "an" and "the" include plural referents unless the content clearly dictates otherwise.

The terms "first," "second," and the like in the description and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological or der. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in se quences other than those illustrated or otherwise described herein. Furthermore, the terms "include," "have," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or ap paratus.

All directional references (e.g., "plus", "minus", "upper", "lower", "upward", "downward", "left", "right", "leftward", "rightward", "front", "rear", "top", "bottom", "over", "under", "above", "below", "vertical", "horizontal",'' clockwise", and "counterclockwise") are only used for identification purposes to aid the reader's understanding of the present disclosure, and do not create limitations, particularly as to the position, orientation, or use of the any aspect of the disclosure. It is to be under stood that the terms so used are interchangeable under appropriate cir cumstances such that the embodiments of the invention described herein are, for example, capable of operation in other orientations than those illustrated or otherwise described herein.

As used herein, the phrased "configured to," "configured for," and simi lar phrases indicate that the subject device, apparatus, or system is de signed and/or constructed (e.g., through appropriate hardware, software, and/or components) to fulfill one or more specific object purposes, not that the subject device, apparatus, or system is merely capable of performing the object purpose.

Joinder references (e.g., "attached", "coupled", "connected", and the like) are to be construed broadly and may include intermediate mem bers between a connection of elements and relative movement between elements. As such, joinder references do not necessarily infer that two elements are directly connected and in fixed relation to each other. It is intended that all matter contained in the above description or shown in the accompanying drawings shall be interpreted as illustrative only and not limiting. Changes in detail or structure may be made without de parting from the spirit of the invention as defined in the appended claims.

All numbers expressing measurements and so forth used in the specifi cation and claims are to be understood as being modified in all in stances by the term "about."

Claims

1. A digital image capture device for the acquisition of a scene comprising :

an optical imaging system adapted for capture of plurality of images sequentially,

a plurality of illumination sources for illumination said scenery, means for controlling said illumination sources and said optical imaging system enabling different illumination during capturing said plurality of images by means of illumination direction or a combination of illumination directions and weighting of said combination by means of illumination power or illumination duration or a combination of illumination of power and duration, characterised by the fact, that the system is augmented by:

means for controlling said illumination sources to illuminate at least one of the captured images referred to as the master frames during the image sequence acquisition by powering all available illumination sources,

means for compensation of motion induced differences between the images captured and one or more of the said master frames by image post processing,

means for combine the processed images or areas of the pro cessed images or a subset of the processed images or areas of a subset of the processed images into a final image by a weighted sum of corresponding pixels of said processed images or areas of said processed images.

2. The image capture device according to claim 1 wherein one or more persons are subject for the shooting.

3. The image capture device according to claim 1 wherein one or more objects are subject to the shooting.

4. The image capture device according to claim 1 wherein the optical imaging system is a digital photo camera.

5. The image capture device according to claim 1 wherein the optical imaging system is a digital photo camera module in a mobile de vice such as a smartphone or a tablet computer.

6. The image capture device according to claim 1 wherein the illumi nation sources are flash units.

7. The image capture device according to claim 1 wherein the illumi nation sources are flash units found within a mobile device such as a smartphone or a table computer.

8. The image capture device according to claim 1 wherein means for controlling said illumination sources is a module receiving trigger signals from the image capturing device and sending trigger signals to the flash units denoted by routing each received trigger signal to one of the flash units except for said master frames where all flash units receive a trigger signal.

9. The image capture device according to claim 8 wherein one of the illumination sources is ambient light which will be switched off be fore the first flash unit is being activated and switched on after completion of the image acquisition sequence.

10. The image capture device according to claim 9 which is connected to the master device via standard wireless communication protocols.

11. The image capture device according to claim 8 wherein said illumination sources are controlled by a software application in a mobile device referred to as the master device connected to the camera module of said mobile device as well as to external flash units via standard wireless communication protocols.

12. The image capture device according to claim 8 wherein said illumination sources are controlled by a software application in a mobile device referred to as the master device connected to the camera module of said mobile device as well as to external flash units via a wired connection.

13. The image capture device according to claim 11 wherein the external flash units are flash units found within mobile devices like a smartphone or a tablet computer controlled by a software application which receives the flash trigger signals from the master device via standard wireless communication protocols.

14. The image capture device according to claim 1 wherein

means for controlling said illumination sources to illuminate at least one of the captured images referred to as the master frames during the image sequence acquisition by powering all available illumination sources equally.

15. The image capture device according to claim 11 wherein means for controlling said illumination sources is a software application running on the master smartphone connected to the slave smartphones software application by standard wireless communi cation protocols, distributing in an initial phase the timing infor mation with respect to the very first flashlight pulse originated by the master smartphone during the acquisition phase for each in volved slave device individually, starting the image acquisition process by said first flashlight pulse and hence synchronize the moment of each single image acquisition with the moment of flashlight trigger of each involved slave device according to the timing schedule deployed during the initial phase.