HK1220512B - System and method for controlling an equipment related to image capture - Google Patents

Description

System and method for controlling devices related to image capture

Technical Field

The present invention relates to the field of motion tracking in camera environments. More particularly, the present invention relates to systems and methods for controlling the scenery of cameras or related equipment.

Background Art

In a camera environment (e.g., film, television, live entertainment, sporting events), there are a variety of devices that operate the functions of the camera, lighting, and sound. The control and interrelationship of these functions determine the quality of the final image and sound perceived by the audience. One such function is camera focus. "Focus" or "rack focusing" refers to the act of changing the focal length setting of a lens in response to the physical distance of a moving object from the focal plane. For example, if an actor moves from 8 meters to 3 meters from the focal plane in one shot, the focus puller will change the distance setting on the lens in the shot to precisely correspond to the actor's changed position. In addition, the focus puller may move the focus from one object to another within the frame as dictated by the specific aesthetic requirements of the composition.

This process of adjusting the focus is performed manually by the "First Assistant Camera" (first AC) or "Focus Puller".

Depending on the parameters of a given lens, there's often very little room for error. As such, the role of the focus puller in filmmaking is extremely important; in most cases, since such errors can't be corrected in post-production, a "soft" image is often considered unusable. One must also consider that an actor may not be able to replicate their performance in subsequent takes, so the focus puller is expected to perform perfectly on each take. Due to these factors, some production crews consider the focus puller to have the toughest job on set.

Even though the focus puller may be very skilled, current processes still slow down production due to the complexity and difficulty of the task.

Current film productions begin with blocking rehearsals, where the positions of the various actors are established. During rehearsals, camera assistants place tape markers on the floor at all the points where the actors pause in motion. The actors then leave the set to get their hair and makeup done, and stand-ins come in to replace them at these various positions for lighting, framing, and focus mark setting purposes.

Once the camera positions are established by the Director of Photography and the Camera Operator, the First AC begins measuring the various distances between the actors' marks and the focal plane of the camera. These distances are recorded with a series of crayon/drawing pen marks on the lens's focal barrel, and/or with a marking disc on the follow focus device. Using a stand-in, these marks are checked through the viewfinder and/or on the onboard monitor to ensure accuracy. If the marks are repositioned in order to provide a specific desired composition, the First AC must remeasure/reset their marks accordingly. In addition, the First AC can place specific distance markers on the floor - which will be referenced as the actors move between their marks during filming - to help accurately adjust the focus to the correct intermediate distance.

When the actors return to set, there is usually a rehearsal for the camera where the focus puller and camera operator will practice the shot and make sure everything is set up properly. During the shoot, the focus puller modifies the focus based on the actors' or subject's dialogue, movement, camera motion, and compensates for drift or any unforeseen movement of the actors missing their marks. In the event that an obstruction prevents the focus puller from seeing all of his marks, he can request a second AC to call out marks for him via a two-channel radio during the shoot. In some cases, such as with long lenses, wide apertures, very close distances, or any combination of the three, even a few millimeters of subject movement may require immediate and very precise focus correction.

After a shot, if the focus puller feels that he or she has made a mistake - perhaps a timing error, a missed mark, or any other problem that may make parts of the shot appear "soft" - he or she will usually report the problem to the camera operator (who is most likely to notice the error in the viewfinder) or the director of photography, and may request a retake if one is not already planned.

Besides keen eyesight, reflexes, and intuition, the focus puller's primary tools are a cloth or fiberglass measuring tape, a steel measuring tape, a laser rangefinder, and in some cases an on-camera ultrasonic rangefinder that provides real-time distance readings and is mounted on the matte box or the side of the camera body. In arrangements where the focus puller cannot access the camera, such as on a steadicam or crane shot, he or she will use a remote follow focus system, although some focus pullers prefer to use a remote system at all times. In any of these situations, the focus puller is still required to manually adjust focus during the course of the shot.

The current method is time-consuming, difficult, and very error-prone. This has long been a technical hurdle in feature film animation, placing significant creative constraints on directors and increasing production costs due to unavailable shoots, slow set times, and the need for very experienced and highly paid focus pullers.

Known to the applicant are semi-automatic focus systems that rely on laser, sonar, and facial/object recognition tracking.

These methods are essentially variations of the same approach, as they each sense a "two-dimensional plane" of an image and capture depth or distance information for any given area or pixel on that plane. With the most advanced systems, the system's photographer can then select a point on the two-dimensional image, at which point the distance data for that point is then input into a motor that controls focus adjustment in real time.

These known methods exhibit several limitations. More specifically, these systems are all "line of sight." They cannot focus on objects that are not currently visible in the "two-dimensional image plane." Laser systems require an additional camera operator to aim the laser at the desired object. If an object turns quickly, leaves the frame, or disappears behind another object or objects, the facial recognition system will lose track of it.

Perhaps more importantly, none of these systems can truly achieve the superb accuracy required for the most challenging focusing tasks: long focal lengths with wide apertures when the subject is moving rapidly, and the focus on a very specific subject, such as the eye, is very specific, because for both LIDaR (Light Detection and Ranging) and laser systems, the human photographer must track the eye in real time, either by moving an on-screen cursor or by aiming an actual laser. It should also be noted that projecting a laser into a person's eye may not be desirable. While facial recognition systems can theoretically track the eye, there is a need to provide a higher level of precision and accuracy.

The applicant is known to U.S. Patent Nos. 5,930,740 (MATHISEN), 8,448,056 (PULSIPHER) and 8,562,433 (LARSEN); U.S. patent applications having publication Nos. 2008/0312866 (SHIMOMURA), 2010/0194879 (PASVEER), 2013/0188067 (KOIVUKANGAS), 2013/0222565 (GUERIN), 2013/0229528 (TAYLOR) and 2013/0324254 (HUANG); and Japanese patent application having publication No. JP 2008/011212 (KONDO).

Therefore, in view of the foregoing, a need exists for an improved system, which by virtue of its design and components will be able to overcome some of the above-discussed problems of the prior art.

Summary of the Invention

It is an object of the present invention to provide a system which, by virtue of its design and components, satisfies some of the above needs and thereby represents an improvement over other related systems and/or methods known in the prior art.

The present invention provides a system and method for controlling settings of a device related to image capture. Such a device may include a camera, and the settings may be, for example, focus settings, zoom settings, aperture settings, interocular lens angle settings, and/or controlling pan settings, tilt settings, camera rotation settings, and/or camera position settings, and/or lighting device settings and/or sound device settings and/or the like.

According to aspects of the present application, there is provided a method for controlling settings of a device related to image capture, comprising:

a) capturing position data and orientation data at a sensing device;

b) determining, by a processor, position information of an area of interest to be processed by the device from the already captured position data and orientation data; and

c) outputting a control signal directed to the device via an output port of the processor, so as to control the settings of the device in real time based on the position information of the area of interest.

A "device" may include an image capture device, such as a camera that captures images (photographs or video images) of an object and/or it may include devices that cooperate with the image capture device - such as lighting devices, sound capture devices and/or the like.

According to another aspect of the present application, there is provided a system for controlling settings of a device related to image capture, comprising:

- a sensing device configured to capture position data and orientation data;

a processor in communication with the sensing means, the processor being configured to determine, from the position data and orientation data, position information of an area of interest to be processed by the device; and

- an output port, integrated in the processor, configured to output a control signal directed to the device, so as to control the settings of the device in real time based on the position information of the area of interest.

According to another aspect of the present application, a non-transitory computer-readable storage device is provided, having stored thereon data and instructions executed by a computer, the data and instructions comprising:

- a code module for receiving position data and orientation data of a sensing device;

- a code module for determining, from the position and orientation data, position information of a region of interest to be processed by the device; and

- A code module for outputting a control signal directed to the device so as to control the settings of the device in real time based on the position information of the area of interest.

According to another aspect of the present application, there is provided a method for controlling settings of a device related to image capture, comprising:

a) storing in a memory one or more identifiers, each identifier being associated with a predetermined area of interest to be processed by the device, and storing corresponding location information;

b) receiving, at a processor, a selection of the one or more identifiers; and

c) outputting a control signal directed to the device via an output port of the processor to facilitate real-time control of settings of the device based on the position information of the selected one of the one or more predetermined regions of interest.

According to another aspect of the present application, there is provided a system for controlling settings of a device related to image capture, comprising:

- a memory configured to store one or more identifiers and corresponding position information of predetermined regions of interest to be processed by the device;

- a processor in communication with the memory and configured to receive a selection of the one or more identifiers; and

an output port, integrated with the processor, configured to output a control signal directed to the device, so as to control settings of the device in real time based on position information of a selected one of the one or more predetermined regions of interest.

According to an embodiment, the components of the above-described system are provided in a central device (e.g., a computer), and the system further includes one or more user devices (e.g., computers, which may be tablet computers with touch screens) for receiving user commands, wherein the user devices communicate with the central device. More specifically, the user devices may be configured to present one or more predetermined regions of interest to a user via a graphical user interface, receive a selection of the one or more regions of interest from the user, and transmit a reference to the one or more regions of interest to the central device.

According to another aspect of the present application, a non-transitory computer-readable storage is provided, on which one or more identifiers of predetermined regions of interest to be processed by the device and corresponding location information are stored, the computer-readable storage further comprising data and instructions executed by a processor, the data and instructions comprising:

- a code module for receiving a selection of said one or more identifiers; and

- a code module for outputting a control signal directed to the device so as to control settings of the device in real time based on the position information of the selected one of the one or more predetermined regions of interest.

According to another aspect of the present application, there is provided a method for controlling settings of a device related to image capture, comprising:

a) capturing position data at a sensing device by a visibility-independent sensing device;

b) determining, by a processor, from the position data, position information of an area of interest to be processed by the device; and

c) outputting a control signal directed to the device via an output port of the processor, so as to control the settings of the device in real time based on the position information of the area of interest.

According to another aspect of the present application, there is provided a system for controlling settings of a device related to image capture, comprising:

- a visibility-independent sensing device configured to capture position data;

a processor in communication with the sensing means, the processor being configured to determine position information of a region of interest to be processed by the device based on the position and orientation data; and

an output port, integrated with the processor, configured to output a control signal directed to the device, so as to control settings of the device in real time based on the position information of the area of interest.

According to an embodiment, the system further comprises a controller in communication with the output port and configured to control a setting of the device using the control signal.

According to an embodiment, the settings may include: a focal length setting of the camera, a zoom setting of the camera, an aperture setting of the camera, an interocular lens angle setting of the camera, a pan setting of the camera, a pitch setting of the camera, a rotation setting of the camera, a position setting of the camera, a lighting device control setting and/or a sound device setting.

According to another aspect of the present application, there is provided a non-transitory computer-readable storage device having stored thereon data and instructions executed by a computer, the storage device having an input port for receiving position data from a visibility-independent sensing device, the data and instructions comprising:

- a code module for determining position information of an area of interest to be processed by the device based on the position data and the orientation data; and

- A code module for outputting a control signal directed to the device so as to control the settings of the device in real time based on the position information of the area of interest.

According to yet another aspect of the present application, there is provided a system for controlling settings of a device related to image capture, comprising:

a) a sensor, mounted on the object to be captured by the camera, adapted to capture three-dimensional position data;

b) a processor adapted to communicate with the sensor for receiving position data and generating a control signal based on the position data; and

c) A controller adapted to communicate with the processor to control settings of the device in response to the control signal.

In certain embodiments, the settings may include: focal length settings, zoom settings, aperture settings, interocular lens angle settings, and/or control pan settings, tilt settings, camera rotation settings, camera position settings, lighting equipment settings, sound equipment settings and/or any combination thereof.

In certain embodiments, the sensor device captures orientation data in all three degrees of freedom, for example, in Euler angles of azimuth, altitude, and rotation (A, E, R). In such embodiments, the processor is adapted to calculate the position of a focal point or "node" relative to the position and orientation data representing the location of the sensor device. The processor is thereby adapted to generate a control signal based on the position of the node.

By "focal point" or "node" is meant a specific point or area of interest on an object, based on which the settings of a device (e.g., focus, zoom, aperture, lighting, sound, etc.) are controlled. This "node" is sometimes referred to as a "tip offset" in motion tracking systems, which provide both position and orientation, for example in some cases where the node does not have the same coordinates as the sensor but is located at a fixed distance from the sensor. For example, a node may correspond to one of a person's eyes, while the position and orientation data corresponds to the back of the person's head where the sensor is located. Thus, the focus, zoom, aperture, interocular angle of a camera can be set, pan, tilt, roll, the position of the camera, lighting, and/or sound can be controlled by calculating from the position and orientation of the sensor the positioning of the eyes depending on the particular person.

In a particular embodiment, the system further comprises a sensor mounted on the camera, ie in case the camera is moved with respect to the object to be captured.

According to yet another aspect of the present application, there is provided a method for controlling settings of a device related to image capture, comprising:

- capturing three-dimensional position data related to the object to be captured by the camera;

- generating a control signal based on the position data; and

- Controlling settings of the device in response to the control signal.

According to yet another aspect of the present application, a non-transitory processor-readable storage medium for controlling settings of a device related to image capture is provided, the storage medium containing data and instructions executed by a processor to:

- receiving three-dimensional position data relating to an object to be captured by the camera;

- generating a control signal based on the position data; and

-Send control signals to the controller for controlling the settings of the device.

According to yet another aspect of the present application, there is provided a system for controlling settings of a device related to image capture, comprising:

- sensors and transmitters to be mounted on the object to be captured by the camera, adapted to capture position and/or orientation data;

- the processor is adapted to communicate with the transmitter of the sensor for receiving position data and for sending control signals based on said position and/or orientation data; and

- a controller adapted to communicate with the processor so as to receive the control signal and to control a setting of the device in response to the control signal.

According to a further aspect, a method associated with the above system is provided.

According to a further aspect, a non-transitory processor-readable storage medium containing data and instructions for executing the method associated with the above-described system is provided.

An advantage of embodiments of the present invention is that the use of motion tracking data having very specific properties to create multiple predetermined positional and directional 'nodes' in three-dimensional space allows for improved levels of device control and automation in a variety of mobile and still photography environments.

An advantage of embodiments of the present invention is that they allow tracking and/or selection from a plurality of predetermined stationary or moving points (nodes) in three-dimensional space in real time, with or without user interaction, and without requiring any additional manual intervention; any of these nodes can be selected at any time using a software interface or a mechanical dial or other mechanical input device. In the example of focus control, once the user selects the desired node, the system automatically adjusts the focus to that node and maintains focus on that node even as the node and camera move. It will also enable focusing on nodes that are not in the current field of view, allowing objects to be in focus as soon as they enter the composition or appear from behind other objects (doorways, walls, vehicles, etc.).

The objects, advantages and features of the present invention will become more apparent upon reading the following non-limiting description of preferred embodiments with reference to the accompanying drawings, which description is given for purposes of illustration only.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a system for controlling camera settings according to an embodiment of the present application.

FIG. 1B is a flow chart illustrating steps of a method performed by the system shown in FIG. 1A , according to an embodiment.

FIG. 1C is a sequence diagram illustrating a method performed by the system shown in FIG. 1A according to an embodiment.

2A and 2B illustrate block diagrams of a system for simultaneously controlling multiple camera settings and camera controls according to another embodiment of the present invention.

3 is a schematic diagram illustrating a single or double boom pole source mount to be used with the system shown in FIG. 1A , according to an embodiment.

4 is a schematic diagram illustrating a camera arm source mount to be used with the system shown in FIG. 1A , according to an embodiment.

5 is a schematic diagram illustrating a camera sensor base to be used with the system shown in FIG. 1A , the camera sensor base comprising a wand and a source housing mounted at each end of the wand, according to an embodiment.

5A is a perspective view of a source housing of the camera sensor base shown in FIG. 5 .

5B is a side plan view of a portion of the wand shown in FIG. 5 , showing one end of the wand having a mounting shaft extending therefrom.

5C is an outline view of a mounting hole of the source housing shown in FIG. 5A , the mounting hole being configured to receive the end of the rod shown in FIG. 5B .

6 is a schematic diagram illustrating a modular source installation system to be used with the system of FIG. 1A , according to an embodiment.

FIG. 7 illustrates a home screen displayed on a graphical user interface (GUI) of a user device in the system illustrated in FIG. 1A .

FIG. 8 shows a node creation/modification window of the GUI shown in FIG. 7 .

FIG. 9 shows a portion of the home screen shown in FIG. 7 , namely, a node array defining various nodes.

FIG. 10 illustrates specific node buttons of the node array shown in FIG. 9 .

FIG. 11 shows a selected node button of the node array shown in FIG. 9 .

FIG. 12 shows a portion of the home screen shown in FIG. 7 , namely, showing a sequencer component.

FIG. 13 shows another portion of the home screen shown in FIG. 7 , namely showing the corner dial control interface.

FIG. 14 shows another portion of the home screen shown in FIG. 7 , namely showing another corner dial control interface.

FIG. 15 illustrates a display screen that may be displayed on a user device of the system shown in FIG. 1A for defining a camera to be controlled, according to an embodiment.

FIG. 16 illustrates another display screen to be displayed on a user device of the system shown in FIG. 1A for calibrating a lens of a camera to be controlled, according to an embodiment.

FIG. 17 illustrates another display screen that may be displayed on a user device of the system shown in FIG. 1A for selecting a configuration of a sensor device, according to an embodiment.

FIG. 18 illustrates another display screen that may be displayed on a user device of the system shown in FIG. 1A for recording the configuration of a node array and the configuration of a sequencer in a memory, according to an embodiment.

19 illustrates a portion of a display screen as would be displayed on a user device of the system shown in FIG. 1A , including corner controls for adjusting the amount of delay/lag compensation to be applied to node data, according to an embodiment.

20 illustrates an alternative control display screen that would be displayed on a user device of the system shown in FIG. 1A , including interactive graphical representations related to linear sequencer functionality, according to an embodiment.

21 illustrates an alternative control display screen that would be displayed on a user device of the system shown in FIG. 1A , including interactive graphical representations related to customized sequencer functionality, according to an embodiment.

22 illustrates an alternative control display screen that may be displayed on a user device of the system shown in FIG. 1A , including interactive graphical representations related to free sequencing functionality, according to an embodiment.

23 illustrates an additional control display screen that may be displayed on a user device of the system shown in FIG. 1A , including interactive graphical representations related to free sequencing functionality, according to an embodiment.

24 illustrates a portion of a home screen that would be displayed on a graphical user interface (GUI) of a user device in the system shown in FIG. 1A , namely a 4-node geometry controller feature, according to an embodiment.

25 illustrates a portion of a home screen that would be displayed on a graphical user interface (GUI) of a user device in the system shown in FIG. 1A , namely a 3-node geometry controller feature, according to an embodiment.

DETAILED DESCRIPTION

In the following description, like reference numerals refer to similar elements.The embodiments and/or geometric configurations and dimensions shown in the drawings or described in this specification are embodiments of the present invention given for illustrative purposes only.

Broadly described, systems and methods for controlling camera settings, according to certain embodiments, use motion capture or a global (or local) positioning system to generate three-dimensional position and orientation data. This data is processed by software that calculates in real time the position and orientation in three-dimensional space, as well as other metrics including relative distance data between a desired object and the camera. This data is then used to control all of the camera-related equipment, such as servo motors used to operate the servo motors, in real time, in real time.

More specifically, according to certain embodiments, the present disclosure relates to controlling focus and composition, and includes generating predetermined points in three-dimensional space, which are referred to as "nodes" hereinafter. A node can be a node that is fixed in space, i.e., a vase of flowers. Or it can be a moving node, i.e., a person or an animal. A fixed node does not require a sensor if the camera does not move or if the camera has a sensor. A moving node, such as a moving camera, does require a sensor. Since the motion tracking system essentially creates the possibility of drawing an infinite number of defined points in a given three-dimensional space, interfacing with this data allows for much greater complexity and unlocks creative and practical possibilities. An important feature of a "node" as defined and used in this system is that it has both position and orientation data: this allows for intelligent operations such as automatically focusing between the left and right eye - see "AutoProfiling" later in this document.

Thus, when referring to FIG1 , a system 10 for controlling settings of an image capture-related device 112, such as a camera 12, is provided. The system 10 includes one or more sensing devices 114, such as sensors 14, configured to capture position data and orientation data at the sensing devices. The system 10 also includes a processor 16 embedded in a data processing device 28 (also referred to herein as a "data processing unit"). The processor 16 is in communication with the sensing devices 114 and is configured to determine position information x of an area of interest to be processed by the device 112 based on the position and orientation data. The processor 16 also includes an output port 43 configured to output a control signal directed to the device 112 to facilitate real-time control of the settings of the device 112 based on the position information of the area of interest.

The system 10 also includes a controller 118 in communication with the output port 43 and configured to control the settings of the device 112 using control signals. The system 10 also includes a memory 132, such as RAM 32, for storing position data and orientation data. The system 10 also includes the device 112. According to this embodiment, the sensing device 114 is a visibility independent (i.e., a non-line-of-sight sensor) and includes a transmitter 22. The system 10 also includes a receiver 26 in communication between the transmitter 22 and the processor 16. The system 10 also includes a user device 40, the user device 40 including a user interface 42, and the user interface 42 is in communication with the data processing device 28 via the wireless communication network 39.

More specifically, FIG1 illustrates a system 10 for controlling the settings of a camera 12. The system 10 includes sensors 14, each mounted on an object to be captured by the camera 12, and each adapted to capture three-dimensional position data based on the location of each sensor 14. The system 10 also includes a processor 16 adapted to communicate with the sensors 14 for receiving position data and sending control signals based on the position data. The system 10 also includes a controller 18 adapted to communicate with the processor 16 for controlling the settings of the camera 12 in response to the control signals.

1 , the sensors 14 are each hardwired 20 to a hub/transmitter 22. The hub/transmitter 22 communicates via wireless radio frequency (RF link) communication means 24 to a universal serial bus (USB) receiver 26, which in turn connects via a USB connection 27 to a data processing device 28 having the processor 16 embedded therein.

The data processing device 28 also includes a power supply 30 and DDR3 random access memory (RAM) 32, and embedded flash non-volatile computer storage 34. The data processing device 28 also includes a WiFi communication module 36 and a Zigbee ^™ wireless communication module 38 for communicating with a user device 40 via a wireless data network 39, wherein the user device 40 is an iPad ^™ in this example, and the data processing device 28 includes a user interface 42. It should be understood that the iPad ^™ can be replaced or combined with any other suitable computer device, such as, for example, an Android ^™ tablet computer.

Controller 18 is connected to data processing device 28 via hardwire 44. Controller 18 is attached to the area within camera 12 and includes a Cypress PSOC ^™ 5LP microcontroller unit (MCU) 46, and a power supply 48. H-bridges 50, 52, 54 connect controller 18 to various servo motors 56, 58, 60 that automatically operate specific settings of camera 12, namely focus, iris, and zoom, respectively.

It will be appreciated that, according to alternative embodiments, the above-described components may be interconnected in any suitable manner and via any suitable communication means.

2A and 2B , a plurality of cameras 12 are controlled by the system 10 ′. Each camera 12 is connected to a “slave” data processing device 28 b, which is operable via a corresponding user interface of a user device 40. The “slave” data processing device 28 b communicates with the “master” data processing device 28 a.

The remaining components of Figures 2A and 2B refer to similar components shown in Figure 1.

In the embodiment shown in Figures 1 and 2, the sensor system is provided by a magnetic motion tracking system. More specifically, the sensor 14 is provided by an inductive coil and the system 10, 10' also includes an alternating current (AC) magnetic source generator (see Figure 3). The hub 22 powers the sensor 14, interprets the data, and transmits the position data over a radio frequency 24. Preferably, the magnetic source is mounted on a custom, retractable pole base along with an onboard power supply.

Optionally, a radio frequency repeater can be provided to extend the range of data transmission from the motion capture system. A USB RF receiver is required to obtain data from the sensor and send it to the camera. If the distance between the camera and the sensor is very large (for example, when using a 2000mm or 200mm lens for commercial vehicles, etc.), the range may need to be increased. Also optionally, a USB repeater can be provided to extend the range of data transmission from the motion capture system.

The user interface 42 of each user device 40 , ie, an iPad ^™ , includes a touch screen, and the user device 40 is adapted to execute interface software that communicates with one or more central controllers 28 , 28 a , 28 b .

Alternatively, a mechanical input device (e.g., a focus control dial or slider) may be provided to act as an analog/digital interface to add additional control features to the software. For example, as shown in Figures 2A and 2B, one of the user devices 40 has a user interface 42 including a focus knob 62.

The central data processing unit 28 operates with the Linux ^™ operating system and performs most of the processing to control the one or more servo motors 56 , 58 , 60 .

As previously described, the servo motors 56, 58, 60 mechanically adjust camera settings such as, for example, focus, zoom, iris, and/or control pan, tilt, roll, and/or the like.

It should be understood that, depending on the specific embodiment, the settings may include any one or a combination of the following: focal length setting of the camera, zoom setting of the camera, aperture setting of the camera, interocular lens angle setting of the camera, pan setting of the camera, tilt setting of the camera, rotation setting of the camera, position setting of the camera, lighting device control setting, sound device setting, and the like.

In the context of this specification, the term "processor" refers to an electronic circuit configured to execute computer instructions, such as a central processing unit (CPU), a microprocessor, a controller, and/or the like. According to embodiments of the present invention, as will be appreciated by those skilled in the art, a plurality of such processors may be provided. The processor may, for example, be provided in one or more general-purpose computers and/or any other suitable computing device.

Still in the context of this specification, the term "storage" refers to any computer data storage device or assembly including, for example: a temporary storage unit such as random access memory (RAM) or dynamic RAM; a permanent storage such as a hard disk; an optical storage device such as a CD or DVD (rewritable or write-once/read-only); flash memory; and/or the like. As will be appreciated by those skilled in the art, a plurality of such storage devices may be provided.

Furthermore, "computer-readable storage" refers to any suitable non-transitory processor-readable storage medium or computer product.

Other components that may be used with the above-described systems 10, 10' include:

- A custom modular system of non-metallic rod bases for source placement, ie carbon fiber scaffolding apparatus with pre-defined sizes, allowing for quick and easy assembly when more than two sources are used.

- Various clips and brackets for mounting sensors and magnetic field sources to cameras, subjects, and objects; and

- Various instruments to facilitate convenient measurement of node offsets and placements and source locations.

Specifically, FIG3 illustrates a single or double boom source base to be used with the system, according to an embodiment. Furthermore, FIG4 illustrates a camera arm source base to be used with the system, according to an embodiment. Furthermore, FIG5 illustrates a camera sensor base to be used with the system, portions of which are shown in FIG5A-5C , according to an embodiment. Furthermore, FIG6 illustrates a modular source mounting system to be used with the system, according to an embodiment.

System Operation

As previously described, embodiments of the present application allow for control of focus and composition and involve creating predetermined points in three-dimensional space, referred to herein as "nodes," which have both position and orientation data. A node can be a fixed node in a room, such as a vase of flowers. Or a node can be a mobile node, such as a person or an animal. Fixed nodes do not require sensors if the camera does not move or if the camera has sensors. Mobile nodes, such as a moving camera, do require sensors.

In operation, referring to Figure 1, the sensor 14 generates coordinates representing its physical location, such as X, Y, and Z coordinates of a Cartesian coordinate system and/or information representing the orientation, elevation, and rotation (A, E, and R) of the sensor. For example, if the sensor 14 is placed on the back of a person's head being captured by the camera 12, the information generated by the sensor will indicate the location of the sensor and whether the person's head is facing forward, backward, etc.

Processor 16 receives the position and orientation information and calculates the location of a "node." For example, if sensor 14 is placed at the back of a person's head, the "node" may correspond to one of the person's eyes. Therefore, processor 16 locates the predetermined position of the person's eye relative to sensor 14 and, based on the received location and orientation information, calculates the position of the eye, i.e., the focal point. The processor then calculates the distance between camera 12 and the focal point. Based on the calculated distance, processor 16 outputs a control signal to control the settings of camera 12.

Therefore, as better illustrated in FIG. 1B with further reference to FIG. 1A , a method 200 of controlling the settings of a device 112 is provided. The method 200 comprises capturing 210, by means of the sensing device 114, three-dimensional position data and orientation data of the sensing device 114; and storing 212 the position data and orientation data in a memory 132. The captured position data and orientation data of the sensing device 114 are generated by generating coordinates representing a physical location and characteristics representing the orientation of the sensing device. The method 200 also comprises determining 214, by means of the processor 16, position information of an area of interest to be processed by the device, i.e., a "node", based on the three-dimensional position data and orientation data. The node and the sensor device 114 are typically located at different locations. The processor 16 then determines 216 the position information of the node and further calculates 218 the distance between the device 112 and the node.

The method further comprises outputting 220 a control signal directed towards the device 112 via the output port 43 based on the calculated distance.

More specifically, the "distance formula" is derived from the Pythagorean theorem and calculates the distance between two points ( _x1 , _y1 , _z1 ) and ( _x2 , _y2 , _z2 ) in three-dimensional Euclidean space. Once the exact locations of two nodes are determined, the distance formula can be used to calculate the distance between these nodes. For the example of focusing a camera, if one of the nodes is at the center of the focal plane on the camera, the lens's external focus ring or internal electronic focus mechanism can be set to that distance to facilitate focusing on the object.

More specifically, the position information of each node in the calculation step 216 includes the Euclidean space coordinates of the node (x ₁ , y ₁ , z ₁ ), and the calculation step 218 includes:

- receiving 222 position information of the device in Euclidean space coordinates (x ₂ , y ₂ , z ₂ ); and

- Calculate 224 the distance between the device's location information and the node's location information using the following Pythagorean theorem:

For motion tracking sensors that measure both position and orientation, vector math can be used to apply a "tip offset" to the location of the sensor. For example, if an actor places the sensor on the back of his/her skull, the tip offset can project the location of the sensor onto the surface of the actor's left eye, in effect creating a virtual sensor on the actor's eye. For rigid objects/subjects, applying a tip offset allows nodes to be defined anywhere within or on the surface of the object/subject. Likewise, tip offsets (nodes) can be created anywhere in 3D space, i.e., they can exist outside of the object at location coordinates representing the position and orientation relative to the sensor. Thus, determining step 216 includes applying 226 the tip offset from the position data and orientation data of the sensing device 114 of capturing step 210 in order to calculate the location information of the node.

One method of performing the tip offset (nodal point) projection utilizes X, Y, and Z offsets measured from the origin of the sensor to the eye with respect to an axis system defined by the sensor. For the example of an eye, the offsets may be 10 cm in the X-direction, 0 cm in the Y-direction, and 8 cm in the Z-direction with respect to the sensor's local coordinate system. With these offsets, rotation matrices and/or quaternions can be used to calculate the absolute position (X, Y, Z) and orientation (yaw, roll, pitch) of the actor's eye in the motion tracking system's coordinate system. The following equations use the standard rotation matrix approach to solve for the tip offset problem (see http://www.flipcode.com/documents/matrfaq.html#Q36).

Thus, in this embodiment, the step 226 of applying a tip offset (see FIG. 1B ) includes obtaining relative coordinates of the node's three-dimensional position data and orientation data relative to the sensing device 114 in an axis system defined by the sensing device 114. In this case, the step 216 of determining includes estimating the node's absolute position with respect to the apparatus 112.

The absolute position of the node is estimated as follows:

Use the rotation matrix M=X.Y.Z, where M is the final rotation matrix and X, Y, Z are the individual rotation matrices.

in:

A and B are the X-axis rotation axes, i.e., the cosine and sine of the rotation;

C and D are the cosine and sine of the Y-axis rotation, i.e., pitch;

E and F are the cosine and sine of the Z-axis rotation, i.e., panning;

X _f =X _s +X _t *M (1, 1) + Y _t *M (2, 1) + Z _t *M (3, 1);

Y _f =Y _s +X _t *M (1, 2) + Y _t *M (2, 2) + Z _t *M (3, 2);

Z _f =Z _s +X _t *M (1, 3) + Y _t *M (2, 3) + Z _t *M (3, 3);

in:

_Xf , _Yf , _Zf are the absolute (or "final") coordinates of the node;

_Xs , _Ys , _Zs are the coordinates of the center of the sensing device;

_Xt , _Yt , _Zt correspond to the coordinates of the tip offset relative to the center of the sensing device;

M(row, col) are the elements of the rotation matrix in terms of rows and columns, respectively, where the element "row" represents the number of rows in the matrix and the element "col" represents the number of columns in the matrix.

The measurement of "tip offset" can be facilitated by other methods. For example, there is an initial orientation at the back of the actor's skull that can be expressed in Euler angles or through quaternions. The user wishes to define a sensor with a node on the actor's left eye. Additional motion tracking sensors can be placed against the actor's eyes to calculate the X, Y, and Z offsets (rather than, for example, trying to use a tape measure). One solution is to measure the "tip offset" and orientation at this initial time. Given a base sensor at position P1, and a sensor at the desired node P2, the "tip offset", V1, is P2-P1. The initial orientation can be defined as a quaternion Q1 with X, Y, Z, and W attributes. At any other time, a new orientation, Q2, will be generated.

Thus, in this embodiment, the step 226 of applying a tip offset comprises obtaining a tip offset that has been pre-calculated by the position of the node of the sensing device at the location of the node with respect to the position and orientation of the base sensing device at the location of the sensing device. As described above, the initial orientation is defined as a quaternion Q1 having X, Y, Z and W attributes, and the orientation data of the capture step is defined as Q2. The position information of the node is determined according to the following:

P _n +(q _i q _n )P _i (q _i q _n )

in:

_Pi is the offset from the sensor at orientation q;

_Pn is the current position of the sensor;

q _i is the orientation of the sensor when calculating the time of Pi;

q _n is the current orientation of the sensor; and

q _i and q _n are unit quaternions.

Various other means and/or methods can be implemented to facilitate various advanced system functions using position and/or orientation data. An example might be using quaternions to calculate the position and orientation of a motion capture "magnetic field source" relative to the origin of the motion capture coordinate system. If a member of a film crew places a source at a random position and orientation, using a motion sensor within the range of the random source, along with data from a sensor or source with a known position and orientation, and data from a measurement device such as a laser encoder, the exact position and orientation of the source and the random source can be determined. Simple assembly tools and software can make this exemplary process very quick and easy to perform.

Referring back to the embodiment shown in FIG. 1A and FIG. 1B , the method 200 further includes controlling 228 , by way of the controller 118 (which is embedded in the device 112 ), a setting of the device 112 using the control signal.

Given that a node is offset from the sensor, the offset position rotates with the sensor, so the heading data advantageously allows locating the node even if the sensor is rotated. For example, the sensor can be mounted on the handle of a knife, and the focus can be fixed to the tip of the knife and track the tip with high accuracy regardless of how the knife moves and rotates.

A further advantage of using heading data involves a "calibration offset" feature. Using the heading data, a second sensor can be used to instantly calculate the desired offset position of the focus node. For example, placing a sensor behind a performer's neck and then placing a second "calibration sensor" over the performer's eyes is a fast and powerful way to create a node. This feature will be further explained below.

A further advantage of using heading data relates to a "quick setup" function, which is a special case of the calibration offset feature. The quick setup function is beneficial when both the camera and the object have sensors mounted thereon, and the camera is pointed at the object with the sensor out of sight (e.g., on its back). The camera focus is then adjusted until the desired part of the object, such as their eyes, is in focus. Using both heading data from the object and the camera, and then using the distance data indicated by the lens, a quick and reasonably accurate setup of the node of interest can also be achieved.

Various functional features and aspects according to specific embodiments of the present invention will now be described.

According to the embodiment shown in FIG1C , with further reference to FIG1A , a method 300 for controlling settings of a device related to image capture is shown. The method 300 includes storing 314 in the memory 132 one or more identifiers and corresponding position information (i.e., three-dimensional coordinates relative to the device) for predetermined regions of interest (i.e., "nodes") to be processed by the device 112. The position information is obtained by: capturing 310 position data and orientation data at the sensing device 114; and determining 312 position information for the regions of interest to be processed by the device 112 from the position and orientation data of the sensing device 114. The method 300 also includes receiving 316 at the processor 16 a selection of one or more identifiers. The method 300 also includes outputting 318, via the output port 43, a control signal directed to the device 112 to facilitate real-time control 320 of the settings of the device 112 based on the position information of the selected regions of interest.

Node array:

By pre-defining nodes (stationary or moving), an array of desired nodes can be created in the interface. Simply by selecting a node, the lens will instantly focus and/or the camera will be pointed at and framed at that node in the field of view. This allows for extremely fast zooming between a large number of subjects/objects and accurate adjustment between two moving subjects/objects for live improvisation without requiring any manual measurement or manual adjustment of the focus dial - or in the case of camera operation, any manual adjustment of the camera itself. Therefore, in this case, the receiving step 316 of the method 300 described in Figure 1C includes receiving a predetermined order of selected nodes; and the method repeats the output step 318 for each node selected, so as to automatically control 320 the settings of the device 112 for multiple nodes in sequence according to the sequential node selection.

Node Sequencer:

It is also possible to create a pre-determined sequence of nodes, which fits the current paradigm of film production, where the director knows the order of objects in advance. In this way, by pre-loading the desired nodes, the user can simply shift from one object/object to the next by simply clicking a "next" button or turning a dial (real or virtual) back and forth. This allows the user to not only switch between two objects at any desired moment, but also to dictate the speed at which the focus is adjusted between objects (the speed of focusing). Thus, the aforementioned repetition of steps 318 and 320 shown in FIG. 1C (with reference to FIG. 1A ) is prompted upon receiving a user input command via input port 41. Alternatively, steps 318 and 320 may be repeated based on a schedule stored in memory 132.

Geometry Slider:

A graphical representation of nodes (or an array of nodes) could also be arranged on a touch screen device in a geometric (triangles and squares) or random pattern (zigzag lines, curves, etc.), and by sliding a finger between each node the user would "focus" between objects, again with control over the speed of focusing, and again regardless of object or camera movement without the need to measure or adjust the actual focus distance.

Therefore, the method 300 shown in FIG1C (with reference to FIG1A ) further includes receiving a user input command via input port 41 by a sliding motion on the touch screen corresponding to a displacement between two adjacent nodes, wherein the selection of receiving step 316 includes identifiers of the adjacent nodes. The method 300 further includes associating intermediate positions between the adjacent nodes based on the displacement, and repeating output step 318 for each of the intermediate positions.

Interface mode:

Using the node array, sequencer, geometry slider, and hardware dial or other input device, you can choose between two basic modes of focus.

One mode is "tap to focus," where the user simply taps a button (virtual or on a physical input device) to select a node or move forward in a sequence of nodes to the next predetermined node. In this mode, it should also be noted that the speed at which the focus is adjusted when the next node is selected can be predetermined by pre-defining a preference or by adjusting a virtual "speed dial" or analog input device.

The second mode is "slide to focus," where the user not only selects the next node, but can select the next node and adjust the speed of focus in real time by using a geometric slider, a virtual dial, or an analog input device. This emulates the current focus paradigm, where the focuser controls the speed of adjustment without introducing any risk of losing focus on the desired object.

Tip offset and multiple nodes from a single sensor:

By using a sensor that provides real-time position and orientation data, multiple nodes can be created using the same sensor. This is achieved by inputting an "offset value" using X, Y, and Z position coordinates, as well as relative azimuth, altitude, and rotation coordinates. Thus, a sensor attached to the back of a subject's head can have several nodes associated with the head, as it is a rigid object. Eyes, the tip of the nose, ears, and so on can all be defined as nodes from a single sensor using this technology.

Fine adjustment of tip offset:

In situations where it is difficult to measure the exact offset in 3D space, two automated techniques are provided:

- Assuming the sensor is placed at the back of the actor's neck and the desired node is actually the eye, a second sensor can be temporarily placed over the eye. Using the data from the second sensor "tip offset" data can be automatically calculated and applied to the node.

- The tip offset can be adjusted manually by bringing the subject into line of sight to the camera, the focuser can then adjust the focus until the desired nodal point is in focus (usually the eye). The system is able to roughly calibrate its own tip offset because it knows the orientation of the sensor and it will know how much to adjust the focus relative to the sensor data.

Automatic profiling:

If the user defines a node as an eye using a sensor hidden somewhere on the performer's body, the system can be informed that the node is actually "two nodes", a left eye and a right eye. Since the system knows at all times where the camera is and where the object is and how the object is oriented relative to the camera, it can, for example, focus on the left eye when the left side of the face is facing the camera and on the right eye when the right side of the face is facing the camera. Therefore, the method 300 shown in Figure 1C (with reference to Figure 1A) also includes determining at step 316 the nodes (or one or more regions of interest) that meet the given condition among the selection of nodes received. The signal of step 318 is thereby generated based on the nodes that meet the given condition.

Likewise, any rotating subject or object may have several "auto-analyze" nodes associated with it that may be triggered as the subject or object turns.

Zoom control:

Similar to focus, position and orientation data can also be used to adjust zoom. For example, if the desire is to keep an object at exactly the same size within the frame regardless of its distance, by inputting lens parameters, the system can automatically zoom in and out as the object or subject moves. NB: This effect is sometimes called "Dolly Zoom" or "Triple Reverse Zoom" and currently requires very stable camera motion and multiple rehearsals to achieve. This system makes it possible to create this effect in handheld footage with random performer and camera movements.

Mirror Mode:

The functionality can also be extended to calculate virtual distances and or angles needed to capture, for example, a reflection in a mirror. Where the focal length between the camera and the object reflected in the mirror is equal to the distance from the camera to the mirror plus the distance from the mirror to the object, by placing sensors on the mirror and the object (and on the camera if the camera is moving) the system can quickly calculate the correct virtual distance to focus on the reflection when desired.

The focal length based on the optimal focal plane between two nodes or two offset nodes:

For example, it may be desirable to focus on two subjects per sensor. One can thus select the midpoint so that the selected lens will simultaneously allow the subjects to be in focus, as the focal plane will be midway between each subject, and will also allow for optimal focus of both subjects, as the focal plane will be approximately midway between the depth of field. The photographer can also select any point between the two subjects, particularly if they wish to ensure that one subject is prioritized and in focus if the other subject falls outside the depth of field.

Interocular angle adjustment for 3D production:

Some 3D photography setups require real-time adjustment of the interocular angle. The system can automate this adjustment by tethering the angle to a selected subject/object.

Aperture Control:

In some situations, it may be desirable to "pull the iris" to adjust the amount of light entering the lens, such as when moving from a brighter outdoor location to a darker indoor location in a single shot. By tying the camera position to the iris adjustment, automatic iris adjustment can be performed for a range of predetermined locations. Furthermore, because heading data is available to the camera, the iris can be adjusted based solely on the camera's orientation, allowing for currently impossible scenarios where a set or location can be lit to more than one "key light," and the iris will always adjust smoothly between these exposure values.

Save settings:

The system can be used to pre-plan very complex shots or scenes, and all the required data about the "nodes" and any sequence can be entered into a file on the interface software. This storage of "scenes" greatly improves the efficiency of setup overall, and also gives the creator the ability to plan and prepare highly complex shots that are impossible with current technology.

Distance display:

The system can calculate the relative distance between an object and the camera at any given time and display this distance data on any desired reading at any given time. For example, the distance data for a selected "node" can always be displayed on the main control screen of the software interface. Furthermore, "satellites" can be associated with this distance data, and the user can select any node to determine the data at any time.

For example, a focus puller may be focusing on actor A during rehearsal, but the cinematographer may want to know how far away actor B is to assess the lighting levels needed to create the depth of field the director desires. Using a handheld device like an iPod Touch ^™ or smartphone, the cinematographer can obtain distance data for actor B in real time, even while actor A is in focus.

Multi-camera support:

The system allows the user to set up one or more cameras, with no definable upper limit, and to direct multiple cameras at the same object or to direct each camera at a separate object.

Other real-time data shows:

Retrieving real-time data also allows for other real-time calculations and metrics:

-Depth of field for any given node at any given time.

- Minimum focus distance warning - for example: the distance can be displayed in orange when a predetermined close distance is reached and marked in red when the object reaches the actual minimum focus distance.

Manual override and automatic switching:

Since any focus puller or camera operator may wish to manually control focus at any time, regardless of the efficiency of the system, the system enables full-time manual or automatic switching between automatic and manual. These are the methods available in the current system:

- A digital fine adjustment "dial" is always available to the focuser. Simply by adjusting this fine adjustment, the focuser can override the automatic focus setting by any amount.

- "Slate Mode". By selecting a button, the automatic system switches instantly to full manual mode.

- "Auto Switch". This mode allows the user to pre-define a point at which a node, object, or subject switches from automatic to manual or vice versa. This may be useful for very long lenses where the subject travels long distances, and/or may be a method for avoiding undesirable changes in the data.

Source for boom installation:

Since the film industry is accustomed to mounting microphones on long, retractable poles called "booms," a unique implementation of this system is to mount a magnetic source on the boom, which can then be placed above the performance area in the same secure manner that microphones are placed at the nearest convenient location above the performance area. If both the subject and the camera are equipped with sensors, ideal focus data can still be collected for multiple nodes. However, this method does not allow for camera operation or the use of fixed nodes that are not associated with sensors.

Dual (and multiple) source booms:

Extending the basic idea of mounting a single source on a boom, it is also possible to mount two sources, one at each end of the boom, to extend the range. Similarly, other handheld configurations—a triangle or square, for example—can extend the range, and since the relative positions of the sources can be pre-configured in the setup software, this allows for quick setup without requiring on-set calibration.

Camera installation source:

Mounting the source directly on the camera and using software to calibrate the camera's relative position to the source allows the system to operate without the need for an on-camera sensor. This allows for quick setup of a "single-source system" that provides great accuracy at close ranges where sharp focus is most needed.

Modular system:

Multiple sources (with no theoretical upper limit) can be arranged in a predetermined configuration or randomly. Predetermined configurations can enable quick setup (such as an equilateral triangle with 10 ft sides) and cover a larger area. Random configurations require some manual setup in the software, but allow for great flexibility in the shapes and areas to be covered by the system.

Stationary magnetic source (or optical sensor) calibration:

Since the system uses multiple magnetic sources, (or in the case of infrared, multiple cameras) and the X,Y,Z and A,E,R of each source needs to be entered into the system, a simple interface for entering this data is included in the system.

Predictive (or Kalman) filtering:

Since any automated system is viewing data in real time, it is always looking back. While the system will be extremely fast, even a lag of one millionth of a second can have a visible effect in extremely challenging situations (i.e., very long shots with rapidly moving subjects in low light). Currently, filmmakers and cinematographers avoid these challenging situations and, in fact, spend a great deal of money trying to overcome them, especially in renting very expensive lighting kits to maintain an average F/stop of 5.6. By adding a predictive algorithm to the system, any slight lag in the data can be easily overcome by compensating for any delay in focus position by adjusting the focus position at a fixed ratio relative to the speed of the subject's motion toward or away from the camera. By adding this feature, even achieving focus in even the most challenging situations is relatively simple.

As with all features in this system, the user can adjust the automation as much or as little as desired. For example, a very aggressive setting will create tight focus even on very fast-moving objects. A less aggressive setting will create a more natural delay, which may be more suitable for some creative goals.

Data Records:

As previously mentioned, position and orientation data in the system can be recorded in real time (ie, stored in memory 132 - see FIG. 1A ) and later used in other post-production scenarios.

Enhanced camera controls:

Using position and orientation data, camera operation and the movement of dollies and/or camera crane jibs or carts can be fully automated. However, camera operators and cinematographers desire complete control over the subtleties of the final composition. A unique feature of this system is that it completely automates the complexities of camera control while allowing the operator to adjust the composition simply by moving his finger through a touchscreen-enabled video playback screen. For example, an automated system might keep a performer in the exact center of the frame, but the operator wishes to position the performer to the left of the frame. By simply dragging a finger from any point on the video image to the left, the system will compensate and adjust the performer's position within the frame to the desired composition. In this way, framing rapidly moving objects is as simple as framing stationary ones. This same adjustment can be accomplished using joystick control, currently used for standard remote camera operation and a significant improvement over current technology. However, the touchscreen drag feature is more intuitive and requires no training.

Infrared LED:

The system described above uses an AC magnetic motion capture system. However, an equally viable alternative, applicable to larger studio configurations, is to use an infrared LED motion tracking system to capture the same data. While infrared is line-of-sight to the sensor camera, it does not require line-of-sight between the camera and the subject. Small infrared LEDs can be hidden in clothing, hair, and other objects that are invisible to the camera. It is also possible to create "smart fabrics" with infrared patterns sewn into them that can provide the same data.

Differential Global (and Local) Positioning Systems:

Differential GPS provides nearly all the relative position data needed to operate the system. Augmenting GPS to provide heading data by speeding up processing time, "tethering," and adding additional sensing capabilities will make the system fully functional in virtually any outdoor location in the world. Indoor studio applications can be enhanced by developing and using a "local positioning system" that operates on the same principles as differential GPS but on a much smaller scale and can achieve much greater accuracy because the "satellites" can be stationary.

Lighting and other equipment control:

Once the nodes are defined, the data can be made available to any number of auxiliary control systems requiring precise designation, following or locking onto targets, as well as other qualitative adjustments such as beam width and the like.

Physical training:

Adapting this system to sports training is a relatively simple matter. For example, by tethering a tennis machine to a software interface that knows the player's exact position, the machine can be programmed to always hit the ball to the player's weak spot (backhand) and/or create a more challenging virtual opponent with the machine's ability to serve at any speed or angle.

Applications for visually impaired environments:

Another application of the system can be used in low-light situations or for visually impaired people. For example, the environment can be mapped as nodes and the visually impaired person can receive various types of feedback about their position and orientation as well as the position and orientation of objects and people in the room. Another example is in low-light situations—such as an extremely dark room—where a person cannot see their surroundings.

7 to 25, there will be described components of a graphical user interface (GUI) 64. The GUI 64 is displayed by the user interface device 42 for the device 40 to allow a user to operate the system 10 (see Figs. 1, 2A and 2B).

FIG. 7 shows a home screen 66 of the GUI 64 .

FIG8 shows a node create/modify window 68 .

FIG9 shows a portion of the home screen 66 of FIG7 , namely a node array 70 , in which a user has created various nodes 72 in the array 70 .

FIG10 illustrates a portion of node array 70 of FIG9 , and more specifically, an example of node 72 .

Figure 11 shows another portion of the node array 70 of Figure 9, and more specifically, the highlighted node 72 indicates that it has been selected by the user by tapping the node. A node can indicate various information to the user (such as whether it is associated with a sensor, whether the sensor is online, etc.).

Figure 12 shows a portion of the home screen 66 of Figure 7, namely the sequencer 74. The user has recorded various nodes into the sequencer 74 in a specified order.

Figure 13 shows another portion of the home screen 66 of Figure 7, illustrating the corner dial control interface 76. In this embodiment, the dial is used to fine-tune the focal length distance of the lens.

Figure 14 shows yet another portion of the home screen 66 of Figure 7, illustrating another corner dial control interface 78. In this embodiment, the dial is used to control the speed at which the lens pulls focus from one node to another.

FIG. 15 shows a window 80 of the GUI 64 for defining a camera.

FIG. 16 shows a window 82 of the GUI 64 for calibrating the lenses and selecting which lens is on the camera.

FIG. 17 shows a window 84 of the GUI 64 for selecting settings for the motion tracking system.

FIG. 18 shows a window 86 of the GUI 64 for saving the current state of the application in memory, including the node array 70 and the sequencer 74 .

FIG19 shows a portion of the GUI window 64, including a corner control 88 that allows the user to adjust the amount of delay/lag compensation that the system applies to the nodal data.

FIG20 shows an alternative control window 90 to the GUI 64 ("Full Function Geometry") that allows for an interactive graphical representation of the sequencer's functionality. The user can focus (or make other automatic adjustments) simply by sliding a finger from one point (each point representing a node) to the next. The speed at which the user moves their finger from one point to another controls the speed at which focus (or other) adjustments are made.

FIG21 shows an alternative control window 92 to the GUI 64 ("Full Function Geometry") that allows for an interactive graphical representation of the sequencer's functionality. The user can determine the exact number and location of points on the screen (each point representing a node) and then focus (or make other automatic adjustments) simply by sliding a finger from one node to the next. The speed at which the user moves their finger from one point to another controls the speed at which focus (or other) adjustments are made.

FIG22 shows an alternative control window 94 to the GUI 64 ("Fully Functional Geometry 6-Node") that allows interactive adjustment between any six points, each representing a node. This configuration has the advantage that no predetermined sequence is required. The speed at which the user moves their finger from one point to another controls the speed at which focus (or other) adjustments will be made.

FIG23 shows an alternative control window 96 ("Fully Functional Geometry 5-Node") to the GUI 64 that allows interactive adjustment between any five points, each representing a node. This configuration has the advantage that no predetermined sequence is required. The speed at which the user moves their finger from one point to another controls the speed at which focus (or other) adjustments are made.

FIG24 shows a detail 98 of the corner controller 88 of FIG19 in a main control window on a GUI 64 having multiple functions ("Corner Geometry 4 Nodes"). This function shows how it can be used as a graphical representation for easy control when using four nodes. It allows interactive adjustment between four points. The advantage of this configuration is that no predetermined sequence is required and it can be easily operated by the right (or left) thumb in the main GUI 64 window. The speed at which the user moves his finger from one point to another controls the speed at which the focus (or other) adjustment will be made.

FIG25 shows a detail 100 of the corner controller 88 in the main control window of the GUI 64 having multiple functions ("Corner Geometry 3 Nodes"). This function shows how it can be used as a graphical representation for easy control when using three nodes. It allows interactive adjustment between three points. The advantage of this configuration is that it does not require a predetermined sequence and can be easily operated by the right (or left) thumb in the main GUI 64 window. The speed at which the user moves the finger from one point to another controls the speed at which the focus (or other) adjustment will be made.

The following list provides additional features, components, uses, etc. according to embodiments of the present invention:

The data flow and features of the system facilitate use in post-production. All data and video feeds can be stored and instantly replayed (e.g. for each 'take' on a film set) and/or stored for use in post-production (e.g. for CGI). This includes camera movement/orientation, node movement/orientation, and device control.

- The system's data flow and features facilitate its use in virtual and augmented reality environments. All data and video feeds can be transmitted, stored, and replayed instantly.

The system's data flows and features facilitate interoperability across a wide range of hardware. For example, aperture and lighting adjustments can be linked and pre-programmed so that as the aperture is adjusted to change depth of field, the lights automatically dim or brighten simultaneously, allowing the audience to perceive the change in depth of field without experiencing a change in lighting. This interoperability extends across all devices, without limitation.

The system design according to an embodiment facilitates interoperability of multiple camera interface devices (e.g., iPads, iPhones, iPod touches) running applications and controlling all device types. Along with this interoperability, each interface device can send and receive data with the other. For example, if a camera operator taps a node to focus their camera on an object, that focus decision can be immediately reflected on the device of the other focus operator controlling another camera, as well as on the devices of various other crew members, including the director and producer.

- The system design according to an embodiment is suitable for extremely flexible multi-camera functionality. In the example of focusing, one iPad can control multiple cameras, and multiple iPads can control multiple cameras simultaneously. One iPad can control multiple cameras simultaneously by tapping on a node, or cameras can be selected for individual control. A second copy of the node array can also temporarily replace the sequencer graphic for simultaneously controlling one or more secondary cameras to the permanent node array. The video feedback portion of the application can be made to switch to a split screen (for example, a split screen for 2 cameras, or a quad split screen for 4 cameras) to facilitate monitoring of all focusing activity.

- Advanced hardware and software design aims to minimize system latency to milliseconds (e.g., interrupts, multiple cores, multi-threaded software, etc.).

Due to the system's low latency and responsiveness, a feature allows photographers to actually slow down the autofocus's responsiveness so it doesn't look too "robotic."

- A mechanical input device (such as a digital follow focus dial attached to an iPad) can be linked to any element of the software's graphical user interface (such as a sequencer).

A 'stretchable' touchscreen that can create the feel of texture, grooves, etc. through electrical charges on the screen surface could aid in this system. For example, the lines and nodes of a graph in a 'geometry slider' function could become grooves for improved operability, including limiting the photographer's reliance on looking at the touchscreen.

- Recording and playback of the built-in video feedback display is extremely useful for focus pullers, DOPs, directors, etc. For example, a focus puller can assess the quality of focus at the end of the last 'take' or 'take' or at the end of the day.

- Touching an area of the video feedback allows you to select a node for focus and/or control other device functions such as remote head pointing, lighting, etc.

- Sensors and transmitters can be placed inside free-standing objects. For example, sensors and transmitters can be placed inside a custom basketball in a way that does not affect the ball's mass or center of gravity, making it easier to focus on the ball during a basketball game.

- Together with the 'scene save' feature that saves the state of the application, the node manager can allow photographers to save similar groups of nodes (e.g. all parts of a car can be defined as nodes and reloaded at any time in the future to reuse the same car or facilitate the creation of nodes for a new car).

- Device control events can be triggered based on the node's coordinate position (hardware and/or software triggering).

Many 'intelligent' uses of node data are possible. For example, an indication could alert the camera operator when a node approaches or enters the camera's field of view (frame). In this example, the node could be pre-programmed to automatically come into focus when it enters the frame.

The motion tracking data stream can be filtered using a variety of mathematical methods. For example, the noise in the data stream can be quantified to determine when the data becomes suspect or unusable. This data can be fed into the "manual override and automatic switching" software functions. A variety of filters can also be applied to the data stream to control the level of vibration reduction, etc.

- When the node sequencer is 'neutral', 2 (line), 3 (triangle) or 4 (square) geometry nodes are all set to green. This way when the sequencer is set to 'forward' or 'reverse', the next node will be outside the 2, 3 or 4 group, and the next logical node in the sequence will become the only green node.

A software feature can allow the photographer to quickly correct slight errors in the tip offset of the node by observing the node from the camera and then manipulating the focus fine adjustment function until the node is accurately focused. At this point, the photographer can trigger the system to automatically recalculate the tip offset of the node (via quaternion calculations).

- Pre-recorded motion tracking data (e.g. earthquake motion) can be fed into the system to move the camera and equipment to simulate the pre-recorded motion. This technology can enhance the audience's "natural experience" (e.g. earthquake motion, vehicles on uneven terrain, etc.).

- Specific (and difficult) pre-defined device actions can be automated and/or facilitated (e.g., a Hitchcock zoom using a handheld camera, camera rotation synchronized with a trapeze artist, etc.).

- Effects related to the musical content are possible including feedback loops (e.g., focusing and defocusing in time with the beat of the song or positioning/specifying the camera in relation to the beat, including live performances).

- The entire system can be ‘scriptable’ so that any user interaction with the software can be recorded and automated.

- Various accessories are available for sensor placement on objects. For example, sensors can be placed in a strap to be placed on an actor, or can snap into various mounts for easy placement/attachment.

The source setup feature may include a 3D modular source building function for setups that use modular poles to connect source system accessories. This feature allows photographers to quickly build a 3D representation of their manually constructed modular setup. Because the pole lengths and source angles are predefined by the physical design of the modular source system accessories, the software can then instantly calculate the positions and orientations of all sources.

- For modular source systems, the connecting rod can be removed after setup without moving the source. This allows for rapid, untethered source placement without the need to measure source position or orientation, as calculated in the iPad app's 3D modular source building function.

- Together with servo motor control of the lens iris, it is possible to access the internal electronics of some camera lenses to directly control focus, iris, and zoom, removing the requirement for servo motors.

- System software allows full control over the configuration of the motion tracking system.

- One accessory is a sensor calibration 'body cap' tool that will fit onto the lens mount of the camera for precise measurement. This will allow very accurate measurement of the centre of the focal plane, which is important for visual effects work as it 'nodes' the camera data.

Embodiments of the present invention are advantageous in that using a real-time stream of three-dimensional position and orientation data to adjust shot function, composition, camera positioning, lighting, and sound greatly facilitates and expands the functionality available to filmmakers and animation and/or still image content designers.

According to embodiments of the present invention, the use of nodes in the context of movie control exhibits many advantages, including:

1) The node system allows for pre-definition of multiple mobile nodes (almost all other camera/focus systems cannot, but Pictorvision Eclipse uses GPS for coarse applications http://www.pictorvision.com/aerial-products/eclipse/).

2) The node system allows true auto-tracking of multiple mobile nodes (which perhaps all other camera/focus systems cannot; some try by having people do the tracking; the Pictorvision Eclipse may only have one mobile node; an example of a "true auto-tracker" for lighting might be: http://www.tfwm.com/news-0310precision).

3) The nodal system provides three-dimensional position data (versus distance which is far less useful, unlike almost all other systems).

4) The nodes used are characterized by position and orientation, allowing them to be defined on subjects/objects rather than general 'regions' (unlike probably all other camera/focus systems; without this, other systems cannot apply offsets to define nodes arbitrarily somewhere on the object, such as focusing on the eyes).

5) Position and orientation allows attempts to control the angles of subjects/objects, such as switching from an actor's right eye to their left eye when their head is at certain angles to the camera (no other system can do this).

6) The nodal system provides extremely high accuracy (less than 1 cm in many cases), unlike likely all other automatic tracking systems (due to the increased level of control/focus provided by heading and offset).

7) The nodal system also offers extremely high frequencies (120Hz), unlike probably all other automatic tracking systems (eg gps systems, active face detection tend not to have this).

8) The node system also provides low latency (10ms). This level of latency does not hinder 'cinematic' control in most situations (again, many systems lack this).

9) The node system provides predictive/corrective functions, which significantly reduces latency.

10) Node systems don't require 'line of sight'; that is, nodes use sensors placed on actors/objects, so lasers or sounds won't bounce off actors. Facial recognition obviously requires line of sight as well. Another benefit of sensors in this area is constant node data. For example, if an actor pops out from behind a bush, he or she is already 'instantly' in focus, compared to a line-of-sight system that needs to react to the actor's new appearance.

11) The nodal system continues to operate in mobile environments. For example, if the source is mounted to a handheld camera system (or used with a source boom accessory), the system continues to operate near the camera operator no matter where he/she moves. Similarly, the system operates in a moving vehicle, such as on a moving train.

12) Furthermore, the node system is a portable system.

The above-described embodiments are intended to be illustrative and not restrictive in all respects, and this application is intended to cover any modification or variation that would be apparent to those skilled in the art. Of course, as would be apparent to those skilled in the art, various other modifications may be made to the above-described embodiments without departing from the scope of the present invention.

Claims (38)

1. A method for controlling settings of a device related to image capture, comprising: a) capturing three-dimensional position data and orientation data at a sensing device; b) determining, by a processor, position information of an area of interest to be processed by the device from the already captured position data and orientation data; and c) outputting a control signal directed to the device via an output port of the processor so as to control the settings of the device in real time based on the position information of the area of interest, Wherein the region of interest in the determining step (b) includes one or more nodes, the determining step (b) comprises, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein said calculating step (b)(i) comprises applying the position data and the tip offset of the orientation data from the sensing device of said capturing step (a) so as to calculate the position information of said node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Wherein said determining step (b)(i) further comprises estimating an absolute position of said node relative to said device. 2. The method according to claim 1, further comprising: d) controlling, by a controller, the settings of the device using the control signal. 3. The method according to claim 1, further comprising: - Storing said position data and orientation data in a memory. 4. The method according to claim 1, wherein the settings include at least one of the following: a focal length setting of a camera, a zoom setting of the camera, an aperture setting of the camera, an interocular lens angle setting of the camera, a pan setting of the camera, a tilt setting of the camera, a rotation setting of the camera, a position setting of the camera, a position setting of the camera, a lighting device control setting, and a sound device setting. 5 . The method of claim 1 , wherein the capturing comprises generating coordinates representing a physical location and a characteristic representing an orientation of the sensing device. 6. The method of claim 1, wherein the control signal of outputting step (c) is generated based on the distance calculated at step (b). 7. The method according to claim 6, wherein the position information of each node in determining step (b)(i) comprises the Euclidean space coordinates of the node ( _x1 , _y1 , _z1 ), and wherein the calculating step (b)(ii) comprises: - receiving position information of said device in Euclidean space coordinates (x ₂ , y ₂ , z ₂ ); and - Calculate the distance between the device's location information and the node's location information using the following Pythagorean theorem: 8. The method of claim 6, wherein the absolute position of the node is estimated as follows: in: Rotation matrix M = X.Y.Z, where M is the final rotation; Matrix, and X, Y, Z are separate rotation matrices; A and B are the X-axis rotation axes, i.e., the cosine and sine of the rotation; C and D are the cosine and sine of the Y-axis rotation, i.e., pitch; E and F are the cosine and sine of the Z-axis rotation, i.e., panning; X _f =X _s +X _t *M (1, 1) + Y _t *M (2, 1) + Z _t *M (3, 1); Y _f =Y _s +X _t *M (1, 2) + Y _t *M (2, 2) + Z _t *M (3, 2); Z _f =Z _s +X _t *M (1, 3) + Y _t *M (2, 3) + Z _t *M (3, 3); in: _Xf , _Yf , _Zf are the absolute coordinates of the node; _Xs , _Ys , _Zs are the coordinates of the center of the sensing device; _Xt , _Yt , _Zt correspond to the coordinates of the tip offset relative to the center of the sensing device; M(row, column) are the elements of the rotation matrix in terms of rows and columns respectively. 9. The method of claim 6, wherein the absolute position of the node is estimated as follows: in: Rotation matrix M = X.Y.Z, where M is the final rotation; Matrix, and X, Y, Z are separate rotation matrices; A and B are the X-axis rotation axes, i.e., the cosine and sine of the rotation; C and D are the cosine and sine of the Y-axis rotation, i.e., pitch; E and F are the cosine and sine of the Z-axis rotation, i.e., panning; X _f =X _s +X _t *M (1, 1) + Y _t *M (2, 1) + Z _t *M (3, 1); Y _f =Y _s +X _t *M (1, 2) + Y _t *M (2, 2) + Z _t *M (3, 2); Z _f =Z _s +X _t *M (1, 3) + Y _t *M (2, 3) + Z _t *M (3, 3); in: _Xf , _Yf , _Zf are the final coordinates of the node; _Xs , _Ys , _Zs are the coordinates of the center of the sensing device; _Xt , _Yt , _Zt correspond to the coordinates of the tip offset relative to the center of the sensing device; M(row, column) are the elements of the rotation matrix in terms of rows and columns respectively. 10. The method of claim 6, wherein applying the tip offset comprises obtaining a tip offset that has been pre-calculated by measuring the position of a node sensing device at the location of the node relative to the position and orientation of a base sensing device at the location of the sensing device. 11. The method according to claim 10, wherein the initial orientation is defined as a quaternion _Q1 having X, Y, Z and W attributes, the orientation data of the capturing step is defined as _Q2 , and wherein the position information of the node is determined according to: P _n +(q _i q _n )P _i (q _i q _n ) in: _Pi is the offset from the sensor in orientation q; _Pn is the current position of the sensor; q _i is the orientation of the sensor when calculating the time of Pi; q _n is the current orientation of the sensor; and q _i and q _n are unit quaternions. 12. The method according to any one of claims 1 to 11, wherein the region of interest and the sensing device are located at different locations. 13. A system for controlling settings of a device related to image capture, comprising: - a sensing device configured to capture position data and orientation data; a processor in communication with the sensing means, the processor being configured to determine, from the position data and orientation data, position information of an area of interest to be processed by the device; and - an output port, integrated with the processor, configured to output a control signal directed to the device, so as to control the settings of the device in real time based on the position information of the area of interest, Wherein the area of interest includes one or more nodes, the processor is further configured to, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein determining the position information of the node comprises applying a tip offset to the position data and orientation data from the sensing device to facilitate calculating the position information of the node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Determining the location information of the node further comprises estimating the absolute location of the node relative to the device. 14. The system of claim 13, further comprising: - a controller in communication with the output port and configured to control a setting of the device using the control signal. 15. The system of claim 13, further comprising: - a memory for storing the position data and the orientation data. 16. The system according to claim 13 further comprises the device, wherein the setting includes at least one of the following: a focal length setting of the camera, a zoom setting of the camera, an aperture setting of the camera, an interocular lens angle setting of the camera, a pan setting of the camera, a tilt setting of the camera, a rotation setting of the camera, a position setting of the camera, a lighting device control setting, and a sound device setting. 17. The system according to any one of claims 13 to 16, wherein the sensing device is a visibility independent sensing device. 18. The system of claim 13, wherein the sensing device comprises a transmitter, the system further comprising a receiver in communication between the transmitter and the processor. 19. The system according to claim 13, further comprising a data processing unit embedded in the processor and a user device in communication with the data processing unit, the user device comprising a user interface. 20. The system of claim 19, wherein the user device communicates with the data processing unit via a wireless communication network. 21. An apparatus for controlling settings of a device related to image capture, comprising: processor; A memory, coupled to the processor, stores data and instructions for performing the following steps when executed by the processor: - receiving position data and orientation data of the sensing device; - determining position information of an area of interest to be processed by the device from the position data and the orientation data; and - outputting a control signal directed to the device so as to control the settings of the device in real time based on the position information of the area of interest, The area of interest includes one or more nodes, and determining the location information further includes, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein determining the position information of the node comprises applying a tip offset to the position data and orientation data from the sensing device to facilitate calculating the position information of the node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Determining the location information of the node further comprises estimating the absolute location of the node relative to the device. 22. A method for controlling settings of a device related to image capture, comprising: a) storing in a memory one or more identifiers, each identifier being associated with one or more predetermined regions of interest to be processed by the device, and storing corresponding location information; b) receiving, at a processor, a selection of the one or more identifiers; and c) outputting a control signal directed to the device via an output port of the processor so as to control settings of the device in real time based on the position information of the selected one of the one or more predetermined areas of interest, Each of the predetermined areas of interest corresponds to a node, and the location information is determined by the following steps, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein determining the position information of the node comprises applying position data and a tip offset to the position data from a sensing device capturing the position information to facilitate calculating the position information of the node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Determining the location information of the node further comprises estimating the absolute location of the node relative to the device. 23. The method according to claim 22, further comprising: d) controlling, by a controller, the settings of the device using the control signal. 24. The method according to claim 22, wherein the position information stored in step (a) is obtained by: - capturing position data and orientation data at the sensing device; and - determining, by a processor, position information of a region of interest to be processed by the device from the position and orientation data of the sensing means. The method of claim 24 , wherein the corresponding position information comprises three-dimensional coordinates relative to the device. 26. The method of claim 25, wherein the receiving step (b) comprises - selection of a predetermined order of receiving nodes; The method further comprises repeating the outputting step (c) for each selected node according to the sequential node selection so as to automatically control the settings of the device for the plurality of nodes sequentially. 27. The method of claim 26, wherein repeating step (c) is performed based on a predetermined schedule stored in a memory. 28. The method of claim 26, wherein repeating step (c) is activated upon receiving a user input command via the input port. 29. The method of claim 25, further comprising receiving a user input command corresponding to a displacement between two adjacent nodes via an input port, wherein the selection of step (a) includes identifiers of the adjacent nodes, the method further comprising: - associating intermediate positions between said adjacent nodes according to said displacement; and The outputting step (c) is repeated for each of the intermediate positions. 30. The method of claim 29, wherein the user input is input via a touch screen by a sliding motion. 31. The method according to any one of claims 22 to 30, further comprising: - determining one or more regions of interest satisfying given conditions from the selections received in step (b); and The control signal of step (c) is generated according to one or more regions of interest that meet the given condition. 32. A system for controlling settings of a device related to image capture, comprising: - a memory configured to store one or more identifiers and corresponding position information of one or more predetermined regions of interest to be processed by the device; - a processor in communication with the memory and configured to receive a selection of the one or more identifiers; and an output port, integrated with the processor, configured to output a control signal directed to the device so as to control settings of the device in real time based on position information of a selected one of the one or more predetermined regions of interest, Each of the predetermined regions of interest corresponds to a node, and the processor is further configured to, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein determining the position information of the node comprises applying position data and a tip offset to the position data from a sensing device capturing the position information to facilitate calculating the position information of the node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Determining the location information of the node further comprises estimating the absolute location of the node relative to the device. 33. An apparatus for controlling settings of a device related to image capture, comprising: processor; a memory, coupled to the processor, storing thereon one or more identifiers of one or more predetermined regions of interest to be processed by a device associated with image capture and corresponding location information, the memory further storing data and instructions for execution by the processor to perform the following steps: - receiving a selection of said one or more identifiers; and - outputting a control signal directed to the device so as to control settings of the device in real time based on the position information of the selected one of the one or more predetermined areas of interest, Each of the predetermined regions of interest corresponds to a node, and the steps further include, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein determining the position information of the node comprises applying position data and a tip offset to the position data from a sensing device capturing the position information to facilitate calculating the position information of the node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Determining the location information of the node further comprises estimating the absolute location of the node relative to the device. 34. A method for controlling settings of a device related to image capture, comprising: a) capturing position data at said sensing device by a visibility-independent sensing device; b) determining, by a processor, from the position data, position information of an area of interest to be processed by the device; and c) outputting a control signal directed to the device via an output port of the processor so as to control the settings of the device in real time based on the position information of the area of interest, Wherein the region of interest in the determining step (b) includes one or more nodes, the determining step (b) comprises, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein said calculating step (b)(i) comprises applying the position data and the tip offset of the orientation data from the sensing device of said capturing step (a) so as to calculate the position information of said node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Wherein said determining step (b)(i) further comprises estimating an absolute position of said node relative to said device. 35. A system for controlling settings of a device related to image capture, comprising: - a visibility-independent sensing device configured to capture position data; a processor in communication with the sensing means, the processor being configured to determine position information of a region of interest to be processed by the device based on the position and orientation data; and - an output port, integrated with the processor, configured to output a control signal directed to the device, so as to control the settings of the device in real time based on the position information of the area of interest, Wherein the area of interest includes one or more nodes, the processor is further configured to, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein said determining the position information of the node comprises applying a tip offset to position data and orientation data from the sensing device to facilitate calculating the position information of the node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Wherein determining the location information of the node further comprises estimating the absolute location of the node relative to the device. 36. The system of claim 35, further comprising: - a controller in communication with the output port and configured to control a setting of the device using the control signal. 37. The system according to claim 35 or 36 further comprises the device, wherein the settings include at least one of the following: a focal length setting of the camera, a zoom setting of the camera, an aperture setting of the camera, an interocular lens angle setting of the camera, a pan setting of the camera, a tilt setting of the camera, a rotation setting of the camera, a position setting of the camera, a lighting device control setting, and a sound device setting. 38. An apparatus for controlling settings of a device related to image capture, comprising: processor; A memory, coupled to the processor, stores data and instructions for execution by the processor to perform the following steps: - receiving position data from a visibility-independent sensing device; - determining position information of an area of interest to be processed by the device based on the position data and the orientation data; and - outputting a control signal directed to the device so as to control the settings of the device in real time based on the position information of the area of interest, Wherein the area of interest includes one or more nodes, the processor is further configured to, for each node: i) determining the location information of the node; and ii) calculating the distance between the device and the node, wherein said determining the position information of the node comprises applying a tip offset to position data and orientation data from the sensing device to facilitate calculating the position information of the node, wherein applying the tip deflection comprises: - obtaining relative coordinates of the position data and orientation data of the node relative to the sensing device in an axis system defined by the sensing device; and Wherein determining the location information of the node further comprises estimating the absolute location of the node relative to the device.