WO2001006316A1

WO2001006316A1 - Apparatus and method for programming a color light projector system

Info

Publication number: WO2001006316A1
Application number: PCT/US2000/016331
Authority: WO
Inventors: Daniel R. Gosselin; R. Emerson Washburn
Original assignee: Chromasync, Ltd.
Priority date: 1999-07-14
Filing date: 2000-07-14
Publication date: 2001-01-25
Also published as: AU6050400A

Abstract

A method and apparatus for programming a color light projector system (100) capable of producing light of various colors in concert with a displayed image and projecting the light of various colors on surfaces surrounding the displayed image. The method includes reading an image frame, selecting colors for plural channels corresponding to the image frame, and recording selected colors. Colors can be selected for any image frame in which a lighting event of the light projector system (100) is desired. An editing type user interface facilitates the programming method.

Description

APPARATUS AND METHOD FOR PROGRAMMING A COLOR LIGHT PROJECTOR SYSTEM

BACKGROUND OF THE INVENTION

1. Field of the Invention:

The invention relates to a color light projector system capable of projecting light of various colors and intensities in accordance with a displayed image and more particularly to a method and apparatus for developing a control program for the light projector system.

2. Description of the Related Art:

It is known to control a light source based on various external conditions. U.S. patent 5,739,928, U.S. patent 5,488,434, and U.S. patent 3,813,686, are representative of patents disclosing the control of image parameters, such as brightness, based on ambient lighting conditions. U.S. patent 5,111,113, U.S. patent 5,056,399, and U.S. patent 4,205,585 are representative of patents disclosing the control of light based on sound. U.S. patent 4,378,466 discloses an apparatus for converting spoken words to a displayed color. Recently the "home theater" or "home cinema" industries have expanded greatly.

Products such as large screen and wide screen televisions, video cassette recorders, CD- ROM drives, DVD players, DIVX players, DOLBY PROLOGIC SURROUND SOUND devices, and DOLBY DIGITAL SURROUND SOUND devices have permitted the theater atmosphere to be reproduced in the home. The object of these products is to make the viewer feel enveloped by the movie or other program being watched. However, existing image displays, while relatively large, do not create the effect that the viewer is in the middle of the action because the viewer still has surrounding objects, the furniture in the room for example, in his peripheral vision. Darkening the room merely creates a sort of black frame around the image and is harsh on the viewer's eyes. To some extent, this problem is addressed by IMAX projectors and screens that provide a 180° viewing angle, i.e. field of vision. However, IMAX movies require very expensive specialized cameras, projectors, and screens and thus have limited practicality in commercial theaters and even less practicality in home theater situations. Accordingly, there is no practical solution for extending the feel of a movie or other program beyond the edges of a display screen to a viewer's peripheral field of view. As a result, the feeling of envelopment achieved by home theater systems is limited. Further, existing lighting systems have limited value in enhancing a home theater experience.

BRIEF DESCRIPTION OF THE INVENTION It is an object of the invention to extend a sense of location of a displayed image beyond the confines of a viewing screen.

It is another object of the invention to enhance a theater experience by coordinating colors in the peripheral field of view of a viewer with objects in the focal area of the viewer;

It is another object of the invention to surround a viewer with an apparent image by varying the color and intensity of light in the viewer's peripheral field of view.

It is another object of the invention to coordinate the color and intensity of light surrounding a viewing screen with the image being viewed on the screen in a dynamic manner.

It is another object of the invention to create three-dimensional synchronous representations of perceived on-screen light functions.

It is another object of the invention to enhance the affect of motion of an onscreen image.

It is another object of the invention to permit a user to develop a control program for controlling a light projector system in accordance with a displayed image in a dynamic manner.

These and other objects of the invention are achieved by a light projector programming apparatus and method of the invention. A first aspect of the invention is an apparatus for developing a control program for controlling a light projector system in accordance with a displayed image. The apparatus comprises means for displaying an image frame of a video recording, means for selecting at least one color and intensity to be displayed by the light projector in association with the image frame, and means for recording data relating to the selected color and intensity in accordance with the image frame.

A second aspect of the invention is a method for creating a control program for a light projector system for a video recording comprising the steps of, selecting a frame of a video recording, selecting at least one color and intensity to be displayed by a light projector in association with the selected frame, and recording data indicative of the selected color and intensity.

BRIEF DESCRIPTION OF THE DRAWING The invention will be described through preferred embodiments and the attached drawing in which:

Fig. 1 is a schematic view of a color light projector system that can be used with the preferred embodiment of the invention;

Fig. 2 is a top view of optical components of the light projectors of Fig. 1 ; Fig. 3 is a perspective view of the optical components of the light projectors of Fig. 1;

Fig. 4 is a schematic illustration of the preferred embodiment in use with a typical home theater system; and

Fig. 5 is a flow chart of a programming routine in accordance with the preferred embodiment of the invention;

Fig. 6 is a schematic block diagram of a programming terminal for use with the preferred embodiment; Fig. 7 is an illustration of a main window and toolbar of a user interface for creating a control program according to the preferred embodiment;

Fig. 8 is an illustration of a color palette window of the preferred embodiment; Fig. 9 is an illustration of an editing window of the preferred embodiment; Fig. 10 is a block diagram illustrating the steps in selecting colors for a video/movie frame(s) in accordance with the preferred embodiment;.

Fig. 11 is a graphical view of the multi-track editing window of the preferred embodiment; Fig. 12 is a graphical view of a pop-up window; and

Fig. 13 is a list view of the multi-track editing window of the preferred embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT To fully appreciate the operation and affect of the invention, it is helpful to consider the manner in which light is perceived by various organisms, including the human eye. Even the simplest organic photo receptors, in single cell animals for example, can be used to interpret relatively complex situations. For example, a simple organism that can only detect the presence or absence of light can use the light detection to move toward or away from the light. Further, a sudden termination of perceived light could be interpreted to mean that something has passed between the source of light and the photo receptor. In simple organisms, this can be used to initiate a "fight or flight" response, for example.

In humans, the subconscious response to changes in light intensity is quite remarkable. The human retina has a central portion known as the "macula lutea" that has closely packed photo receptors. There is a central depression in the macula lutea known as the "fovia centralis" on which light is focused. Accordingly, the fovia is responsible for our acute vision. However, the retina outside of t e fovia is used also. The outer portions of the retina are responsible for our peripheral vision with which we may be unable to focus on objects but are aware of their existence. For example, when we are watching television, we may not be consciously aware of the furniture in the room. However, if someone were to enter the room into our peripheral field of view, we would become aware of their presence prior to being able to focus on the person. Further, our immediate emotional reaction to the presence might be more extreme if we are watching a horror movie than if we are watching a comedy.

We cannot make out details in our peripheral vision but we can detect colors, change in intensity, motion, and shapes. Our subconscious mind processes the peripheral vision signals and we react in a way that is related to the activities going on in our conscious mind. Therefore, the level of intensity and color of light in our peripheral vision can greatly enhance the emotional effect of images on which we are focused.

Applicants have discovered that by synchronizing light in the peripheral field of view of an observer with an image on which the observer is focused, a context or relative location of an on-screen object or event can be established and the viewing experience can be enhanced without the need to expand the viewing screen into the peripheral field of vision of the observer. A home theater viewing experience can be enhanced by controlling light in the peripheral view of the viewer in synchronism with the image viewed on a screen, thus creating a three-dimensional synchronous representation of perceived on-screen light functions over time to envelope the viewer in the on-screen image.

Color light projector system 100 that can be used with the preferred embodiment is schematically illustrated in Fig. 1 and includes one or more light projectors 200 (9 in the preferred embodiment), and controller 300. Controller 300 can be any type of device capable of controlling light projectors 200 in a desired manner. In the preferred embodiment, controller 300 is a microprocessor based device, such as a personal computer, and includes central processing unit (CPU) 340, random access memory (RAM) 360, and memory device 320. Memory device 320 stores a control program developed by the preferred embodiment disclosed below, an operating system, and any other software required for operation of controller 300. Memory device 320 can be a magnetic device, such as a hard disk or a removable diskette drive, an optical device, such as a CDROM drive, or any other known memory device. Memory device 300 can be integral to controller 300 or can be external. RAM 360 serves as a workspace for CPU 340 in a known manner and can include known types of memory chips, such as dynamic RAM (DRAM) chips, magnetic core memory, or the like.

Controller 300 also includes input/output device (I/O) 380 which includes any interfacing circuitry for coupling other portions of controller 300 to light projectors 200 or other peripherals, such as an input device or display (not illustrated). For example, I O 380 can include standard RS232 serial, Centronics parallel, or universal serial bus (USB) physical ports and any required digital to analog conversion, filtering, or other circuitry. All elements of controller 300 can communicate using known protocols over data bus 310. For example, data bus 310 can be an Industry Standard Architecture (ISA) bus or the like. I/O 380 of controller 300 is coupled to light projectors 200 through communication links 390 which can be any appropriate type of electrical conductor, such as a serial, parallel, or USB cable, or an optical conductor, such as an optical fiber bundle. Alternatively, links 390 can be accomplished in a wireless manner using radio frequency, infra-red, or other techniques. Further, links 390 can be twisted pair or coaxial cables using a network protocol, such as 10 base T, 100 base T, or TCPIP communication.

Also, standard telephone cable and RJ11 jacks can be used. In such a case, each light projector 200 will have the requisite interface such as a network card or a modem. Links 390 can be any mechanism for transferring control signals from I/O 380 to light projectors 200 in a local or remote manner. While controller 300 is illustrated as being coupled to light projectors 200 in a hub-like manner, the connection can be daisy-chained from one light projector 200 to the next depending on the hardware and protocol used for links 390.

Figs. 2 and 3 schematically illustrate the optical components of one of projectors 200 in accordance with the preferred embodiment. Each projector 200 can be the same or similar and thus only one projector 200 is discussed in detail. Light source 200 is a white light source, such as a high intensity tungsten halogen lamp. Other white light sources can be used or sources of other colors can be used and corrected to eliminate or attenuate unwanted color components or add or increase necessary color components. Lens system 204 is provided to collimate light from light source 202 to produce input light beam 206 of substantially white light. Lens system 204 can optionally include a polarization filter to polarize light beam 206. Dichroic filters 220, 221, 222, and 223 reflect a certain bandwidth, i.e. color of light and let other wavelengths, i.e. colors, of light pass therethrough. Such dichroic filters are readily available and the provision thereof will be apparent to one of skill in the art in light of the disclosure herein. For example, filters sold under part numbers D43454 and D43455 by Edmunds Scientific, can be used as the dichroic filters 220, 221, 222, and 223. Dichroic filter 220 is a blue filter and is arranged in the path of light beam 206, at a 45° angle for example, to reflect component light beam 240 which consists of primarily light of wavelengths corresponding to the color blue.

Remaining wavelengths of light beam 206 pass through dichroic filter 220 as light beam 250.

Dichroic filter 222 is a red filter and is arranged in the path of light beam 250, at a 45° angle for example, to reflect component light beam 242 which consists of primarily light of wavelengths corresponding to the color red. Remaining wavelengths of light beam 250 pass through dichroic filter 222 as component light beam 252. Since component light beam 252 has had red and blue components removed therefrom, by dichroic filters 222 and 220 respectively, light beam 252 consists primarily of wavelengths corresponding to the color green. Light beam 252 is reflected 90° by mirror 224. Accordingly, light beam 206 is divided into three component light beams 240, 242, and 252 that are of wavelengths corresponding to blue, red, and green respectively.

Component light beams 240, 242, and 252 are directed through LCD panels 230, 232, and 234 respectively, which preferably are of a twisted nematic field effect type or any other type capable of attenuating a light beam passing therethrough. For example, LCD panels sold under part numbers 738MA by LXD, Inc. can be used. Twisted nematic field effect type LCD panels are preferred because of a high contrast ratio (ordinarily on the order of 2000:1) and extinction ability (roughly 98%). Twisted nematic field effect LCD panels are effective only on polarized light. Therefore, if a polarization filter is not included in lens system 204, or at another point along the path of light, such a filter can be incorporated onto a surface of LCD panels 230, 232, and 234. LCD panels 230, 232, and 234 serve as light attenuators for the respective component beams 240, 242, and 252, i.e. colors, of light. Component light beam 240 passes through LCD panel 230 and is selectively attenuated thereby in accordance with a blue color control signal generated by controller 300 based on a control program executed by CPU 340. Component light beam 242 passes through LCD panel 232 and is selectively attenuated thereby in accordance with a red color control signal generated by controller 300 based on the control program. Similarly, component light beam 252 passes through LCD panel 234 and is selectively attenuated thereby in accordance with a green color control signal generated by controller 300 based on a control program executed by CPU 340. Each LCD panel 230, 232, and 234 can be driven by the respective control signals at plural discrete voltage levels applied to a single pin thereof. For example, LCD panels 230, 232, and 234 can be driven independently at 20 voltage levels from 1.6v to 6.5v in roughly 245mv increments.

Such an arrangement will provide over 7000 possible colors when the component light beams 240, 242, and 252 are recombined in the manner described below thus providing a perceived linearity of color change to the viewer. The control signals can be a voltage modulated DC square wave, a pulse width modulated signal approximating a square, wave, a linear voltage modulated signal, or any other form. Note that filters, or color correctors, 231 (blue), 233 (red), and 235 (green) can be placed anywhere in the component light paths to provide color saturation of the respective bandwidth of each component light beam by further attenuating unselected wavelengths. For example, the filters can be disposed immediately before or after the respective LCD panels. Filters sold under part numbers D43456, D43457, and D43458 by Edmunds Scientific can be used as filters 231, 233, and 235 respectively.

After passing through LCD panel 230, component light beam 240 is reflected by blue dichroic filter 221 into light beam 258. Similarly, after passing through LCD panel 232, component light beam 242 is reflected by red dichroic filter 223 into light beam 256. After passing through LCD panel 234, component light beam 252 is reflected by mirror 225 into light beam 254. As illustrated in Fig. 2, the light beams 258, 256, and 254 are superposed on one another. Accordingly, the three component light beams are attenuated in accordance with a respective color control signal and recombined, i.e. superposed on one another, into output light beam 260 of a desired color. Keep in mind that dichroic filters 221 and 223 allow light of colors other than blue and red, respectively, to pass therethrough. Output light beam 260 is reflected by mirror 226 and directed through LCD panel 236 which is similar to LCD panels 230, 232, and 234 and which serves to attenuate output light beam 260 to render the desired output light intensity. LCD 236 can be driven by an intensity control signal generated by controller 300 at plural discrete voltage levels. For example, LCD panel 236 can be driven at 40 voltage levels in roughly 122.5 mv increments. The intensity control signal can be of the same type as the color control signals. Of course, mirror 226 can be omitted, and LCD panel 236 can be repositioned, to direct output light beam 260 in a direction parallel to input light beam

206. Also, the various angles and positions of each element of light projector 200 can be varied in accordance with the dimensions of a housing of light projector 200 and other practical variables.

As noted above, the color and intensity of output light beam 260 is adjusted based on a control signal or signals generated by controller 300 in accordance with a control program stored in memory device 320 and executed by CPU 340. The electrical connections and associated circuitry of light projectors 200 are not illustrated in detail but will be apparent in view of the disclosure herein. For example, light projectors 200 can each include a dedicated power supply or power can be supplied from a central power supply located in controller 300 or elsewhere. In the preferred embodiment, the color and intensity of light outputted by plural light projectors 200 are controlled in accordance with an image displayed on a viewing screen to enhance the viewing experience by simulating a relative location of an on-screen object or event or otherwise being driven in synchronism with the image displayed on the viewing screen to enhance the viewing experience. While particular light projectors are disclosed above, any light source capable of providing the desired colors and intensities can be used.

Fig. 4 is a schematic illustration of the preferred embodiment used in connection with a typical home theater system. The home theater system includes screen 400 on which an image is displayed. Screen 400 can be CRT, LCD, projection system, or the like. The image signal can be from a broadcast television program, a video cassette, a DVD, a CDROM, photographic film or from any other source. The home theater system further includes speakers 410 distributed throughout the area surrounding screen 400 to provide realistic multi-channel sound, such as SURROUND SOUND™, to a viewer in viewing area 430 which can be a couch, chair, or the like. Additionally, subwoofer 420 can be provided to simulate low frequency sounds to add to the realistic effect.

Light projectors 200 are distributed throughout the area surrounding screen 400, preferably close to walls of the room in which the home theater system is contained to enhance the effect of the displayed image. Preferably, the output light beams 260 of projectors 200 are directed to fall incident on the walls and/or ceiling of the room. The use of nine light projectors 200 permits nine independent channels of light color and intensity. However, any number or arrangement of light projectors 200 can be used. For example, the light projector can be centrally located and directed outward to the appropriate surface. Note that in the preferred embodiment three of the light projectors

200 are directed toward the ceiling above and in front of the viewer. Applicant has found that this creates a "tunnel effect". Controller 300 (not illustrated in Fig. 4) controls light projectors 200, in the manner described above, to be operated in synchronism with the image displayed on screen 400. For example, if a lightning strike is displayed on-screen or indicated off-screen, one or more appropriate light projectors can be driven to provide a flash of white light (with no color components attenuated). Other examples of onscreen images that can be enhanced by proper control of light projectors 200 are sunrises, moonlit scenes, police beacons, vehicle headlights, shadow events, and explosions. Further, light projectors 200 can be controlled to take advantage of the unfocused peripheral vision to suggest movement and images that surround the viewer. The effects are accomplished by controlling the light projectors independently in accordance with the on-screen image, and even the soundtrack, of the displayed program or movie. Fig. 5 is a flow chart of a basic programming routine for creating a control program in accordance with the preferred embodiment for permitting controller 300 to output the control signals described above. The programming sequence can be stored in storage device 320 of controller 300 and run when a control program is to be created. Alternatively, the control program can be developed on a separate programming device, such as a personal computer, running the programming routine as disclosed below. The control program includes a "lighting track" coordinated with the video and audio tracks of a movie or other program. The phrase "lighting track" refers to data or signals that represent the status, e.g. color and intensity, of the light projector. Accordingly, the phrase "control program" as used herein, refers to any data, information, signal, or signals which controls the lighting system and can include a lighting track alone, or other data, information, signal, or signals for operating the light projector system.

In step A, parameters of an image frame of an event recorded on film, tape, CD, or the like, are read. For example, the average color and relative intensity of the entire frame, or particular zones of the frame, can be read. Any device for reading the image signal can be used depending on the format on which the image signal is recorded. For example, a VHS cassette player/editing device can be used to read the signal. If the image is on film, it may be necessary to convert the image to an electrical image signal in a known manner.

In step B, the color for each light projector 200 to be displayed in accordance with the frame is determined manually or based on a desired algorithm. For example, the average color and intensity values read from the frame can be matched by the output of light projectors 200 or a lookup table of corresponding color values can be used. The color and intensity at the edges of the image corresponding to the frame can be used for respective light projectors 200 located proximate the edge. For more complex effects, the parameters of plural frames can be used to determine the color of light projectors 200 to simulate or enhance shadowing or movement. For example, an on-screen event, such as moving clouds, can be enhanced by simulating the movement off-screen by synchronizing appropriately matched colors of white and sky blue. If colors are determined and selected manually, the routine can display the image of the frame onscreen for the operator to select an appropriate color from a displayed pallette of colors. In step C, the intensity of the light output from each projector 200 is determined in a manner similar to the manner for determining the color in step B. Note that the color and intensity of each light projector 200 can be the same at any given time, or each light projector 200 can display different and independent colors and intensities.

In step D, the color and intensity for each light projector 200 for the frame read in step A is recorded on tape, CDROM, a magnetic hard disk, or any other appropriate recording medium that can be read by processor 300 for playback. Step E determines if the frame read in step A is the last frame to be read, i.e. the end of the movie or the end of a particular sequence. If so, the routine ends and the recorded data is incorporated into a control program for controller 300. This can include conventional compiling or other manipulation to render the control program. If not, the frame is incremented, by one for example, and the entire process is repeated. When the routine ends and the data is incorporated into a control program, a light track has been recorded and can be played back, i.e. executed, by controller 300 in synchronism with the video and sound tracks of the movie, or other program, to produce the enhanced peripheral lighting effects disclosed above. The light track is converted to discreet voltage levels for each channel by I/O 380 to drive the four LCD panels in each light projector 200 thereby producing the colors and intensities for achieving the desired effects. Time synchronization can be accomplished by using a prerecorded time track on the video medium or merely by starting playback of the light track at the same time as playback of the video and sound tracks. Alternatively, the light track can be added to the recording medium on which the sound and video tracks are recorded in an integral manner. Also, the light track can be produced in real time with the video and sound tracks. For example, the controller can control the light projection directly based upon the output signal of a video and/or audio device. In either of those cases, the playback hardware can be modified to output the necessary light track channel signals and can be coupled directly to light projectors 200. In other words, the playback device, such as a VHS player, a CDROM player, a DVD player, or the like, can serve as the controller.

Fig. 6 illustrates a programming, i.e. development, terminal which can be used in connection with the preferred embodiment. Terminal 450 is a digital computer, such as a general purpose personal computer. Terminal 450 includes a microprocessor based processing unit (CPU) 452, random access memory 454, input memory device 456, and output memory device 458. Input memory device 456 can be a conventional hard disk, removable diskette drive, tape drive, CD ROM drive, DVD drive, or any type of memory device capable of storing and reading a video track (and optionally a sound track). Input memory device 456 can also store and read identifying data, SMPTE timing data, and any other data necessary for accomplishing the function described below. Output memory device 458 can be a hard disk, a removable diskette drive, a recordable CD ROM drive, or any other device capable of writing and storing data. A development program, including the programming routine discussed above, is stored on input memory device 456, or on another memory device, and is executed by CPU 452 to record a control program on output memory device 458 in the manner described below.

Terminal 450 also includes display 460 (such as a conventional CRT, LCD or the like), display interface 462 (such as a standard video card), speakers 464, audio interface 466, keyboard 468, pointing device 470 (such as a mouse, trackball, or stylus), and input device interface 472. All elements of terminal 450 communicate over data bus 474, which can be any type of physical bus using any appropriate protocols. For example, an ISA bus, a VME bus, or a PCI bus can be used.

Fig. 7 illustrates a user interface of the programming system of the preferred embodiment. Upon initialization of terminal 450 running the development software of the preferred embodiment, the user is presented with main window 500 and toolbar 612. Main window 500 is displayed on display 460 and provides an editing type of environment including five sub-windows; information display window 502, time code window 504, editing window 506, color palette window 508 and multi-track event window 510 (all of which are only schematically illustrated in Fig. 7). Multi-track event window 510 includes two different views from which the user can select, a graphical view shown in Fig. 11 and a list view shown in Fig. 13.

Information display window 502 displays information read from one or both of input memory device 456 and output memory device 458 such as the movie name, the user's (i.e. the programmer's) name, the date of last modification, the file size, the total number of events or frames, the optimum number of tracks, the version number of the movie track and any other information relevant to identifying, describing, or cataloging the file being created. Time code window 504 operates on standards developed and promulgated by the Society of Motion Picture and Television Engineers (SMPTE) which provides a standardized sequencing time base for conventional editing purposes. Time code window 504 dynamically displays the time within the movie recorded on input memory device 456, i.e., the frame time, which corresponds to both editing window 506 and multi-track event window 510. For any given displayed time, the movie frame for that particular time will be displayed in edit window 506. When in the graphical view, the cursor will be positioned to the proper event time in multi-track event window 510 in the manner described below. In the list view, the event closest to the displayed time code will be shown at the top of multi-track event window 510 also in the manner described below. Figure 8 illustrates color palette window 508 used to choose a particular color to be displayed by a light projector at a particular time. When a user left clicks (with mouse 470) the cursor on any portion of color band 512 the associated RGB (red, green, blue) value is displayed in the respective color value boxes 514, 516 and 518. Color band 512, can be of any appropriate shape and can display colors in any desirable geometry. For example, color band 512 can be circular, triangular, square, or of any shape. Display box 520 at the top of window 508 is then filled with the chosen color. If the user right clicks on color band 512 the associated RGB values are shown in color value boxes 514, 516 and 518 and display box 522 is filled with that color. Alternatively, the user may specify a specific set of RGB values by clicking and typing the values in boxes 514, 516 and 518. The supported values of the preferred embodiment are 1 through 16 for each R, G and B value, yielding a total color palette of at least 4096 colors. However, any level of color resolution can be supported. Fig. 9 illustrates editing window 506 which is used to provide the user with a virtual view of the editing process in action. For any given time code displayed in time code window 504 the corresponding frame from the movie or video is read from input memory device 456 and displayed in movie frame window 530. The image can be displayed frame by frame, at normal viewing speed, at a slow speed, or at double speed. Frame 530 has an aspect ratio of, for example, 16 x 9. Therefore a 4 x 3 frame will normally show black bars on the left and right. With cinemascope films having an aspect ratio of 2.35 x 1, as shown in Fig. 9, small black bars will appear on the top and bottom of frame 530. Only regular widescreen films having a 16 x 9 aspect ratio will entirely fill frame 530. Alternatively, the size and aspect ratio of frame 530 can be adjusted in accordance with the aspect ratio of the movie recorded on input memory device 456.

Surrounding movie frame window 530 are nine channel bars 532, 534, 536, 538, 540, 542, 544, 546 and 548 that correspond to nine light channels used in the control program to control corresponding ones of the nine light projectors 200 illustrated in Fig. 4. The bars represent the light projector positions of rear left (channel bar 532), middle left (channel bar 534), front left (channel bar 536), rear center (channel bar 538), middle center (channel bar 540), front center (channel bar 542), front right (channel bar 544), middle right (channel bar 546) and rear right (channel bar 548). For any given frame, up to nine different colors may be associated with the nine different channels respectively. As noted above, there can be any number of light projectors 200 and thus any number of channels. Colors can be assigned by clicking on the respective bars. When this is done with either the left or right button (in the case of a mouse), the color from the associated display box 520 (for a left click) or display box 522 (for a right click) of color palette window 504 is copied to the respective bar and recorded for the respective channel. Also by left or right clicking on any part of frame 530, the color under the cursor is copied to the color palettes left or right display box 520, 522. This allows the user to choose colors directly from the movie frame to be used in one or more of the channel bars 532-548 and to be recorded in connection with the movie frame on output memory device 458. In the case of color selection from the movie palette or the movie frame, the closest color supported by the software and projectors will be chosen and displayed. Since there are 4096 color supported in the preferred embodiment, the user will not be able to discern any difference between the colors seen in the palette or movie frame versus the color used to fill in the left and right color boxes and channel bars. Referring to Fig. 10, the color determination step (Step B) referred to in Fig. 5 will be described in greater detail using the preferred user interface illustrated in Fig. 7. The frame of a movie recorded on input memory device 456 and read in step A of Fig. 5 is displayed in Step Bl in window 530 and the corresponding frame time is displayed in time code window 504. The user right or left clicks on a portion of color bar 512 of color pallette window 508 using pointing device 470 in Step B2. Depending on which cursor button is used, the respective red, green and blue values are displayed in color value boxes 514, 516 and 518. Alternatively, the user can directly input the color values into the color value boxes 514, 516, and 518 using keyboard 458 in Step B2. Either way, the respective display box 520 or 522 will display the chosen color in Step B3. Alternatively, the user may, as previously described, choose the color directly from the image displayed in frame 530 in Step B2 ^". By left or right clicking on any portion of frame 530, block 568, the chosen color will be displayed in the respective display box 520 or 522.

Next, working from editing window 506, the user can select any of the nine channel bars 532-548 by clicking directly thereon in Step B4. This results in the color selected in Step B2, B2 , or B2 , and displayed in the respective display box 520 or 522 in Step B3, being copied into the selected channel bar in Step B5. This is repeated for each channel bar of the frame by determining if the color for the last channel bar has been selected. Once all the colors for the nine channel bars have been chosen, a new frame is chosen and the process repeats until the control program is completed. Data is periodically saved to output memory device 458.

Most known computer displays have constant brightness, with only minor adjustments being able to be made with the brightness control. Projectors 200 support sixteen different levels of intensity, seventeen including OFF. Therefore, display 460 may be unable to display a selected intensity. In order for the user to visualize the intensity of each projector in the user interface, slider 550 (only some of which are marked in Fig. 9) is provided in each of the channel bars 532-548. Sliders 550 can be moved up or down, or left to right for center channels 538-542. Sliders 550 represent the intensity level for the projectors for a given channel for a given frame. By dragging slider 550 along slider index 552 (only some of which are indicated in Fig. 9) the user can set the intensity level for each channel in each frame.

The color selection process described above in connection with Fig. 10 associates colors to be projected by light projectors 200 with an image frame one frame at a time. While this provides a high degree of creative control, it can be time consuming and tedious when a large amount of lighting events are to be recorded. Therefore, the user interface also permits real time recording of a light track, i.e. programming. First the user selects a starting point of the movie using time code window 504. The user then clicks on real time record toggle button on toolbar 612 (see Fig. 7 and the description below). Then the user selects one of play button 616, slow motion forward button 630, or double speed forward button 632. Prior to playback of the film or video, one of channel bars 532-548 is selected in visual surround edit window 506 and the cursor is positioned over a desired portion of movie frame window 530. At this time, pressing the space bar of keyboard 468 will alternately start and stop playback of the video or film and recording of color data for the control program at the selected speed. The color under the cursor will automatically be assigned to the selected channel bar 532-548 during playback, i.e. recorded in the control program in correspondence to the channel of the selected channel bar. The cursor can be moved over the displayed image during playback to dynamically change the recorded color for the selected channel bar 532-548. This procedure is repeated for each channel bar 532-548. However, plural channel bars 532-548 can be selected if colors for plural channels are desired to be the same.

When the user uses the toolbar buttons described below, to play the movie in any number of speeds and in either direction, the nine channel bars 532-538 will display the programmed color events associated with each frame of the movie and the channel sliders will indicate the chosen intensity. This allows the user to see the overall effect of light projector system 100 on a particular segment of film over time using the programming operator interface. Since light projectors 200 will continue displaying the same color and intensity until another light event is received, edit window 506 mimics this by continuing to show the same colors and intensity levels in respective channel bars 532-538 from a past light event until the occurrence of a new event. Accordingly, a new light event will only have to be entered upon a change in lighting, not for every single channel in every single frame. Each channel receives an individual light event. For example, if all nine channels are given an event at time x and only one channel changes once per second for the next five seconds, then a total of just fourteen light events occur. In fact, data will only be recorded for a lighting event when the lighting even changes. In other words, redundant data is not recorded to minimize the required memory space of a control program. As previously discussed the multi-track event window 510 has two views, the graphical view 576 shown in Fig. 11 and the list view 578 shown in Fig. 13. Graphical view 576 is similar to edit window 506, in that graphical view 576 shows the colors for all nine channels at one time. However, graphical view 576 represents these colors simultaneously over a span of time. Graphical view 576 also differs in that it does not directly display any intensity level data for any of the channels 532-548. The span of time displayed in Fig. 11 varies according to the interval chosen on the toolbar 612 (Fig. 7), as described below. As shown in Fig. 11, the interval of one frame is chosen, meaning that each box of color 580, from left to right, represents l/24th or l/30th of a second of movie run time, depending on whether film or video is being edited. With fifteen frames being shown in total, Fig. 11 represents just over '/_ second of film or exactly V₂ second of video.

By double-clicking on any one of channel bars 532-548 shown in the leftmost column 582, a pop-up window 584, as shown in Fig. 12, appears and shows intensity over time. Thus, the user can monitor changes of intensity over time with pop up window 548. As with graphical view 576 of the multi-track event window (which shows color changes over time), pop-up channel intensity window 586 shows an amount of data which is dependent upon the time interval selected on toolbar 612 as discussed below. In both graphical view 576 of Fig. 11 and channel intensity pop-up window 584, selection of light events is possible by dragging the cursor over the grid area where the colors or intensities are displayed. By selecting a region over time for one or more channels, users may apply any number of color functions that are available on toolbar 612.

List view 577 of multi-track event window 510 is shown in Fig. 13. In list view 577, events are displayed in text mode in a tabular form in accordance with column headings 579 of the top of list view 577. This allows the users to see the colors and intensities simultaneously in a numerical view along with the exact event time for when the particular event is triggered. Like graphical view 576 of multi -track event window 510 and edit window 506, list view 577 can be monitored during playback. During monitoring the events can scroll from bottom to top over time.

Fig. 7 illustrates toolbar 612 which allows the user to control the "flow" of the action. Toolbar 612 is displayed on display 460 just above main window 500. Toolbar 612 can be located or sized as desired. Toolbar 612 preferably is displayed at all times with main window 500 if the size of display 460 is adequate. The user can advance the current movie or video one frame at a time or in normal time using standard VCR controls. Toolbar 612 includes stop button 614, record button 615, play button 616, pause button 618, full rewind button 620, advance one frame button 628, reverse one frame button 622, slow motion forward button 630, slow motion reverse button 624, regular speed reverse button 625, double speed forward button 632, full advance button 633, and double speed reverse button 626. Toolbar 612 also includes drop-down list control 634 to select the time scale for graphical view 576 of multi-track event window 510. The default time scale corresponds to one frame for each horizontal grid increment, but other time scales may be chosen producing a different look to the graphical view.

Changing the timescale will not affect the look of playback in edit window 506. In addition, toolbar 612 also supports standard new, open and save file functionality in buttons 635, 636 and 637. The user will be prompted to save the file each time the play button on toolbar 612 is pushed or the user can save the file by pressing the save button on the pull down file menu.

The program development sequence also includes built-in color functions to aid the user during control program creation. The color functions can be accessible from toolbar 612 or by right clicking on multi-track event window 510 after a selection is made. The color functions can include fade, transition, shadow, color burst or any other effect. For example, fade button 638 only affects intensity. The color remains the same but the intensity either drops off or increases by a certain percentage over a given time span. Transition button 640 affects only color. A change from one color to another occurs using the shortest path possible on the color palette over a predetermined period of time while the intensity remains constant. Shadow button 642 also affects intensity. With this function, intensity drops off for a brief instant and then returns to the original selected value. Color burst button 644 affects both color and intensity. The colors get richer (darker) and the intensity increases to produce a rich, bright color before returning to the original selected color and intensity. Each preprogrammed color function modifies the colors and/or intensity chosen by the user as described above. Pop up dialog boxes can be used to specify details of the color functions.

Function libraries of stored lighting sequences can be created. For example canned sequences for lightning, headlights, beacons, and the like can be stored and edited for time duration and the like. Various methods of color and intensity calibration can be used to ensure that the proper colors and intensities are displayed.

As described above, the purpose of the programming routine is to create a file which controller 300 can read and whose data can be passed to light projectors 200. A user can synchronize tracks of data with video or film events on a frame by frame basis or based on portions of the video or film. The programming routine can be effected by computer software and can be a standalone product with the capability to read in movie tracks or a "plug in" to an industry standard movie-editing tool such as those produced by AVID™ or ADOBE™. Moreover, the programming routine can be run under, for example, Microsoft Windows™ 95/98/NT, on a variety of different computer systems. In most circumstances the control program will be small enough to fit on a floppy disk.

An example of a file format which can be saved on output memory device 458 and read as a control program by controller 300 to control light projectors 200 is described below. The first 192 bytes of the file are static making up a header for the file and contain information from information display window 502: Movie name 64 bytes

Track artist's name 64 bytes

Date/time of last modification 14 bytes (date: 8, time: 6, both ASCII decimal values)

Optimum number of tracks 1 byte (1-256)

Version number 1 byte (1-128.1-128)

For Future Use 48 bytes After the header, the file includes an undetermined number of light events in a continuous stream. Each light event will be 8 bytes containing:

SMPTE time code 3 bytes (6 bits each for hour, minute, second and frame) Channel number 1 byte (1-256, currently 9)

Color number 3 bytes (1-16,777,216, currently 1-4096

Intensity level 1 byte (0-255, currently 0-16)

For example, for a two hour film having events for all nine channels at every frame, the maximum amount of data would be:

192 + [(((8 x 9) x 30) x 60) x 120] = 15,552,192 bytes or 14.84 MB Most files will be significantly smaller because lighting events need not change for every frame.

After each frame has respective color and intensity values assigned to it, the lighting event is compiled and another event is selected for the process. When the entire film or a portion thereof is completed, the track contains the entire sequence of lighting events, i.e. a light track, that is inserted into a control program to be read by controller 300 to control light projectors 200 as a movie is displayed on screen 400.

Time delay functions can be incorporated into the program development routine for multiple light projectors 200. Functions can be thus executed by plural light projectors 200 in sequence. For example, if an event such as a sunset occurs on-screen in a time lapse fashion, the color, intensity, and shadow cast would be subject to gradual but observable change. The change in color and intensity in one area of the on-screen image is enhanced by a commensurate change in color and intensity of light in portions of the surrounding room. This is accomplished by changing those same characteristics in the light output by light projectors 200 in positions corresponding with the location of the area of the on-screen image. The light projectors 200 change color and intensity over time in sequence with one another to execute a movement function. Also, the function of the light projectors 200 can be shifted to be slightly in advance of the on-screen image to set a context in the viewers peripheral view and to "prepare" the viewer for the on-screen image.

The invention permits a control program to be developed which includes signals or data indicative of the color and intensity of the light projectors over time during playback of film or a video image. The development software can be run on any machine (i.e. terminal). For example, the controller and the terminal can be the same computer. The various memory devices need not be distinct devices. For example, the input memory device and the output memory device can be the same physical and/or logical drive of a computer. A standard programming interface can be used to create the development software and user interface. For example, if the software is to be run on Windows 98™, the Windows API can be used. One of skill in the art will readily be able to program the novel functions described above in view of the disclosure herein.

The controller can take any form depending on the application of the light projector or plural light projectors. The controller can be contained in the same housing as the light projector to provide an integral unit. The controller can include various manual or programmable controls for changing colors and intensity in a desired manner. The invention can be applied to home theater, television sets, video players, film projectors, commercial theater, or in any application where variable color and intensity lighting is desired. Any type of light projector can be used. The invention has been described in connection with video recordings or movies. However, the term "video", as used herein encompasses all visual recordings including video tape, CDROM, movies on film, DVD, and the like. The invention has been described through a preferred embodiment. However, various modifications can be made without departing from the scope of the invention as defined by the appended claims.

Claims

What is claimed is:

1. An apparatus for developing a control program for controlling a light projector system in accordance with a displayed image, said apparatus comprising: means for reading an image frame of a video recording; means for selecting at least one color corresponding to a light channel to be displayed by the light projector system in association with the image frame; and means for recording data relating to the selected at least one color in accordance with the image frame.

2. The apparatus as recited in claim 1, wherein said means for reading a frame of the video recording comprises means for displaying the frame in an editing window.

3. The apparatus as recited in claim 2, wherein the editing window includes an image frame window surrounded by a plurality of channel bars, each channel bar corresponding to a light channel.

4. The apparatus as recited in claim 3, comprising nine channel bars.

5. The apparatus as recited in claim 3, wherein each of said plurality of channel bars displays a particular selected color to be displayed by a corresponding light projector in accordance with an image frame displayed in said image frame window when said means for selecting has selected a color for the light projector for the image displayed in said image frame window.

6. The apparatus as recited in claim 1, wherein said means for selecting a color palette window.

7. The apparatus as recited in claim 6, wherein said color palette window comprises a color band representing a plurality of different colors.

8. The apparatus as recited in claim 7, wherein said color palette window further comprises three color value boxes, such that when a color is selected from said color band a red, green and blue numerical color value corresponding to the selected color is displayed in one of said color value boxes.

9. The apparatus as recited in claim 8, wherein the color palette window includes a plurality of color display boxes for displaying the color selected from the color band.

10. The apparatus as recited in claim 5, further comprising a multi-track event window displaying colors over time corresponding to colors selected by said means for selecting to be displayed by the light projectors.

11. The apparatus as recited in claim 10, wherein said multi-track event window comprises a graphical window wherein the colors are corresponding to plural light channels are displayed over time.

12. The apparatus as recited in claim 11, wherein said multi -track event window comprises a list window which displays the colors corresponding to plural light channels in tabular form.

13. The apparatus as recited in claim 12, further comprising means for adjusting the intensity of the at least one color.

14. The apparatus as recited in claim 13, wherein said means for adjusting the intensity comprises a slider bar on each of said channel bars.

15. The apparatus as recited in claim 1, further comprising means for selecting an image frame to be read by said means for reading.

16. The apparatus as recited in claim 1, wherein said means for reading, said means for selecting, and said means for recording comprise a computer executing instructions recorded on a computer readable medium.

17. A method for creating a control program for a light projector system for a video recording comprising the steps of: reading an image frame of a video recording; selecting at least one color corresponding to a light channel to be displayed by a light projector system in association with the selected frame; and recording data relating to the selected color in association with the image frame.

18. The method as recited in claim 17, wherein said step of selecting comprises selecting a color from a color band of a color palette.

19. The method as recited in claim 15, wherein said step of selecting comprises inputting R, G, and B values of a desired color.

20. The method as recited in claim 17, wherein said step of selecting comprises selecting a color from a portion of the image frame.

21. The method as recited in claim 17, further comprising the steps of: displaying an image corresponding to the frame; selecting a channel bar corresponding to a light channel; and copying the at least one color to the selected channel bar.

22. The method as recited in claim 20, wherein said step of selecting comprises placing a cursor over the portion of the image frame during playback of the video recording.

23. The method as recited in claim 22, wherein said steps of reading, selecting, and recording are accomplished for plural image frames in seriatim during playback of the video recording.

AMENDED CLAIMS

[received by the international Bureau on 27 October 2000 (27.10.00); original claims 3, 4, 6 and 10 amended; new claims 24-25 added; remaining claims unchanged (3 pages)]

3. The apparatus as recited in claim 2, wherein the editing window includes an image frame window surrounded by a plurality of channel bars, each of said channel bars corresponding to a light channel.

4. The apparatus as recited in claim 3, wherein there are nine of said channel bars.

6. The apparatus as recited in claim 1, wherein said means for selecting comprises a color palette window.

28

10. The apparatus as recited in claim 5, further comprising a multi-track event window displaying colors over time corresponding to the at least one color selected by said means for selecting to be displayed by the light projectors.

11. The apparatus as recited in claim 10, wherein said multi-track event window comprises a graphical window wherein the colors are coαesponding to plural light channels are displayed over time.

12. The apparatus as recited in claim 11 , wherein said multi-track event window comprises a list window which displays the colors corresponding to plural light channels in tabular form.

29

16. The apparatus as recited in claim 1 , wherein said means for reading, said means for selecting, and said means for recording comprise a computer executing instructions recorded on a computer readable medium.

30