A METHOD AND SYSTEM FOR EXPANDING THE MEDIA TYPES SUPPORTED IN A DIGITAL CAMERA
FIELD OF THE INVENTION
The present invention relates generally to digital cameras, and more particularly to a method and system for expanding the media types supported in a digital camera.
BACKGROUND OF THE INVENTION
Modern digital cameras typically include an imaging device which is controlled by a computer system. The computer system accesses raw image data captured by the imaging device and then processes and compresses the data before storing the compressed data into an internal memory. Efficient operation of the computer is therefore an important consideration for camera designers and manufacturers. The memory architecture of a particular computer system determines data storage techniques and can thus significantly effect the operational efficiency of the entire digital camera system. The user interface in conventional digital cameras typically includes a view finder in which small versions of the captured images may be displayed to allow a user to review several images at once. By selecting one of the images, the user may then display the full-sized version of the images in the view finder.
Due to architectural limitations of conventional digital cameras, there are several drawbacks in creating multimedia objects from captured images. As used conventionally, the term multimedia implies that the object includes some combination of graphics, sound, and text. Some conventional digital cameras, however, do not have sound recording capability, not to mention the capability of attaching a sound clip to a captured image in order to create a multimedia object.
Conventional cameras that have sound recording capability, however, typically only allow the user to annotate the last image captured. This is done by capturing the image, recording a sound clip to annotate the image, and then attaching the sound clip to the captured image.
Although the recording of sound to annotate a captured image is an improvement over cameras that have no sound recording capability, the method has several disadvantages. One disadvantage is that only the last image captured by the camera may be annotated by sound. Thus, if the user captures an image and forgets to annotate it before capturing another image, the previously captured image can never be annotated.
Another disadvantage in annotating a single image is that after annotating the image, the user must download the captured image and the attached sound to a personal computer in order to view the image and hear the sound annotation. Thus, an annotated image cannot be played back on the digital camera, even if the digital camera includes a view finder.
Therefore, conventional digital cameras are incapable of supporting objects that include combinations of several media types, such as an object that includes several images combined with sound, for instance. And those conventional digital cameras that allow limited sound annotation do not provide a way to increase the multimedia capability to the digital camera.
Accordingly, what is needed is an improved method and system for expanding the media types supported in a digital camera. The present invention addresses such a need.
SUMMARY OF THE INVENTION
The present invention provides a method and system for expanding the media types supported in a software controlled digital camera, which includes a view finder for displaying a plurality of image cells corresponding to previously captured images. The method and system includes providing the digital camera with a first media type module that defines a first media type originally supported by the digital camera. The digital camera is also provided with an application program interface that enables the software to access the first media type module. The method and system further includes loading a second media type module into the digital camera that defines a second media type, wherein the second media type module is accessible by the software through the application
program interface. After capturing an object with the digital camera, the second media type is associated with the object.
According to the system and method disclosed herein, basic media types in the digital camera are written in software in a modular fashion using an application program interface (API). The API of the present invention enables camera manufacturers and developers of digital camera applications to create new media types for the digital camera. These new media types may then be added to the camera by the user to expand the media types supported by the camera, and therefore allow the creation of objects in the digital camera that have varying multimedia content.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of a digital camera that operates in accordance with the present invention. FIG. 2 is a block diagram of the preferred embodiment for the imaging device of FIG. 1.
FIG. 3 is a block diagram of the preferred embodiment for the computer of FIG. 1.
FIG. 4 is a block diagram depicting a user interface for the digital camera. FIG. 5 is a block diagram illustrating the format of an image cell.
FIG. 6 is a table listing example media types and corresponding icons that may be associated with a captured image.
FIG. 7 is a block diagram illustrating the software organization in the digital camera in accordance with the present invention. FIG. 8 is a block diagram showing the format of an image file in accordance with the present invention.
FIG. 9 is a block diagram illustrating a doubly-linked list of cells which is managed by a cell manager software module.
FIG. 10 is a block diagram illustrating the organization of an application program interface (API) for a plug-in media type module in accordance with the present invention.
FIGS. 11A and 11B are a block diagrams showing example plug-in media types displayed in the view finder of the digital camera.
FIG. 12 is a block diagram depicting one method for capturing data for text-based media types on the digital camera.
DESCRIPTION OF THE INVENTION
The present invention relates to an improvement in digital cameras. The following description is presented to enable one of ordinary skill in the art to make and use the invention and is provided in the context of a patent application and its requirements. Various modifications to the preferred embodiment will be readily apparent to those skilled in the art and the generic principles herein may be applied to other embodiments. Thus, the present invention is not intended to be limited to the embodiment shown but is to be accorded the widest scope consistent with the principles and features described herein. The present invention is a digital camera that includes a method and system for expanding the media types supported in a digital camera. A digital camera architecture has been disclosed in co-pending U.S. Patent Application
Serial No. , entitled "A System And Method For Using A Unified
Memory Architecture To Implement A Digital Camera Device.," filed on , 1996, and assigned to the Assignee of the present application. The Applicant hereby incorporates the co-pending application by reference, and reproduces portions of that application herein with reference to FIGS. 1-3 for convenience.
Referring now to FIG. 1, a block diagram of a camera 110 is shown according to the present invention. Camera 110 preferably comprises an imaging device 114, a system bus 116 and a computer 118. Imaging device 114 is optically coupled to an object 112 and electrically coupled via system bus 116 to computer 118. Once a photographer has focused imaging device 114 on object 112 and, using a capture button or some other means, instructed camera 110 to capture an image of object 112, computer 118 commands imaging device 114 via system bus 116 to capture raw image data representing object 112. The captured raw image data is transferred over system bus 116 to computer 118 which performs various
image processing functions on the image data before storing it in its internal memory. System bus 116 also passes various status and control signals between imaging device 114 and computer 118.
Referring now to FIG. 2, a block diagram of the preferred embodiment of imaging device 114 is shown. Imaging device 114 preferably comprises a lens 220 having an iris, a filter 222, an image sensor 224, a timing generator 226, an analog signal processor (ASP) 228, an analog-to-digital (A/D) converter 230, an interface 232, and one or more motors 234.
U.S. Patent Application Serial No. 08/355,031, entitled "A System and Method For Generating a Contrast Overlay as a Focus Assist for an Imaging
Device," filed on December 13, 1994, is incorporated herein by reference and provides a detailed discussion of the preferred elements of imaging device 114. Briefly, imaging device 114 captures an image of object 112 via reflected light impacting image sensor 224 along optical path 236. Image sensor 224 responsively generates a set of raw image data representing the captured image
112. The raw image data is then routed through ASP 228, A/D converter 230 and interface 232. Interface 232 has outputs for controlling ASP 228, motors 234 and timing generator 226. From interface 232, the raw image data passes over system bus 116 to computer 118. Referring now to FIG. 3, a block diagram of the preferred embodiment for computer 118 is shown. System bus 116 provides connection paths between imaging device 114, power manager 342, central processing unit (CPU) 344, dynamic random-access memory (DRAM) 346, input/output interface (I/O) 348, read-only memory (ROM) 350, and buffers/connector 352. Removable memory 354 connects to system bus 116 via buffers/connector 352. Alternately, camera
110 may be implemented without removable memory 354 or buffers/connector 352.
Power manager 342 communicates via line 366 with power supply 356 and coordinates power management operations for camera 110. CPU 344 typically includes a conventional processor device for controlling the operation of camera
110. In the preferred embodiment, CPU 344 is capable of concurrently running
multiple software routines to control the various processes of camera 110 within a multi-threading environment. DRAM 346 is a contiguous block of dynamic memory which may be selectively allocated to various storage functions.
I/O 348 is an interface device allowing communications to and from computer 118. For example, I/O 348 permits an external host computer (not shown) to connect to and communicate with computer 118. I/O 348 also permits a camera 110 user to communicate with camera 110 via an external user interface and via an external display panel, referred to as a view finder.
ROM 350 typically comprises a conventional nonvolatile read-only memory which stores a set of computer-readable program instructions to control the operation of camera 110. Removable memory 354 serves as an additional image data storage area and is preferably a non-volatile device, readily removable and replaceable by a camera 110 user via buffers/connector 352. Thus, a user who possesses several removable memories 354 may replace a full removable memory 354 with an empty removable memory 354 to effectively expand the picture-taking capacity of camera 110. In the preferred embodiment of the present invention, removable memory 354 is typically implemented using a flash disk.
Power supply 356 supplies operating power to the various components of camera 110. In the preferred embodiment, power supply 356 provides operating power to a main power bus 362 and also to a secondary power bus 364. The main power bus 362 provides power to imaging device 114, I/O 348, ROM 350 and removable memory 354. The secondary power bus 364 provides power to power manager 342, CPU 344 and DRAM 346. Power supply 356 is connected to main batteries 358 and also to backup batteries 360. In the preferred embodiment, a camera 110 user may also connect power supply 356 to an external power source. During normal operation of power supply 356, the main batteries 358 provide operating power to power supply 356 which then provides the operating power to camera 110 via both main power bus 362 and secondary power bus 364.
During a power failure mode in which the main batteries 358 have failed
(when their output voltage has fallen below a minimum operational voltage level) the backup batteries 360 provide operating power to power supply 356 which then provides the operating power only to the secondary power bus 364 of camera 110. Selected components of camera 110 (including DRAM 346) are thus protected against a power failure in main batteries 358.
Power supply 356 preferably also includes a flywheel capacitor connected to the power line coming from the main batteries 358. If the main batteries 358 suddenly fail, the flywheel capacitor temporarily maintains the voltage from the main batteries 358 at a sufficient level, so that computer 118 can protect any image data currently being processed by camera 110 before shutdown occurs.
FIG. 4 is a block diagram depicting a user interface 400 for the digital camera as described in co-pending U.S. Patent Application Serial No. , entitled "A Method and System For Displaying Images In The Interface of a Digital Camera," which is assigned to the Assignee of the present application and incorporated herein by reference. In one preferred embodiment, the user interface includes a view finder 402, an image capture button called a photo button 404, a four-way navigation control button 406, a menu button 408, a menu area 410 within the view finder 402, and function keys 412. The user interface 400 may also include an optional sound button 414, and a mode button 416. Referring again to FIGS. 1 and 4, the user interface 400 operates in two modes: live view finder mode and review mode. In a preferred embodiment, the photo button 404 is a two position button. The live view finder mode begins when a user aims the camera at an object 112 and presses the photo button 404 into the first position. Once this occurs, the view finder 402 displays the image of the object 112 as shown through the camera's imaging device 114. The user may then press the photo button 404 into the second position to capture the image shown in the view finder 402. Review mode begins by pressing any other button on the interface 400.
Referring again to FIG. 4, once the digital camera 110 is placed in the review mode, the view finder 402 displays a series of cells 420 that represent the digital images that have been captured in the digital camera. The view finder 402
is shown here as displaying nine image cells 420. Each cell 420 displays a small- sized image corresponding to one of the captured images. The user may navigate through the series of displayed cells 420 in the view finder 402 using the four-way navigation control button 406. As the user navigates through the cells 420, the old image cells 402 are scrolled-off the view finder 402 and replaced by new image cells 420 representing other images stored in the camera.
The cell 420 currently selected by the four-way navigation control 406 is encircled with a highlighted area 419, which is shown as a selection rectangle. Other shapes for the highlighted area are also suitable. Once a cell 420 has been selected, the user may depress one of the function buttons 412 to in order to display a full-sized version of the image in the view finder 402.
FIG. 5 is a block diagram illustrating the format of a cell 420. Each cell 420 includes an image area 422 and an icon/information area 424. The icon/information area 424 is for displaying one or more graphical icons, and/or for displaying text information. The icons and text information displayed in the icon/information area 424 indicate to the user what media types have been associated with the image displayed in the image area 422. Indicating what types of media have been associated with an object means identifying what combination of specific types of media are included in that object. Examples of specific types of media include graphic images, text, and sound.
FIG. 6 is a table listing example media types and corresponding icons that may be associated with a captured image. The media type of a captured image may represent a single image, a time lapse or burst image, a movie clip, or a panorama. The media type may also represent sound, where the corresponding icon indicates that a sound clip is attached to the image that is displayed in the image area 422.
Other media types may also be associated with a captured image that are not shown in FIG. 7. Examples include a slide show media type and a folder media type, for example. A slide show comprises several images stored sequentially in the same file with sound, while a folder is one or more images stored in the directory or folder. With media types comprising groups of images,
the image area 422 of the corresponding cell 420 would display the first image in the group, or a representative image from the group, and the icon/information area 424 would display a graphical icon representing the specific media type (e.g., a slide show icon or a folder icon). The present invention is a method and system for expanding the number of media types supported by the digital camera. In the present invention, basic media types in the digital camera are written in software in a modular fashion so that new media types may be added by the manufacturer of the camera or developers of software applications for the camera. Referring now to FIG. 7, a block diagram illustrating the software organization in the digital camera is shown in accordance with the present invention. The software 430 comprises a control application 432, a tool box 434, a set of drivers 436, a kernel 438, and a startup/configuration module 440. The control application 432 is the main program that controls the functions of the digital camera and is responsible for interfacing with functions in the tool box 434. The tool box 434, in turn, controls how the digital camera captures and manipulates images. The drivers 436 help control the I/O 348 interface for external communication; the kernel 438 controls basic operating system functions; and the startup/configuration module 440 controls the camera's boot-up process.
Referring to both Figs. 4 and 7, the tool box 434 includes a cell manager 442, and a media type manager 444. The cell manager 442 controls the display of image cells 420 in the view finder 402 of the digital camera as well as the capture process in view finder mode, and the retrieval process when presenting the data. The media type manager 444 interfaces with a built-in media types module 446, and optional plug-in media type modules 448, as explained further below.
The built-in media types module 446 contains the graphical icons corresponding to the media types originally supported by the digital camera 110 at the time of manufacture. The media type manager 444 controls the display of media type icons in the icon/information area 424 using the built-in media types module 446 and the plug-in media type modules 448.
According to the present invention, the plug-in media type modules 448 enable the camera manufacturer or the developer of digital camera applications to create new media types for the digital camera. These new media types may then be added to the camera by the user to expand the media types supported by the camera.
More specifically, the present invention defines an application program interface (API) between the media type manager 444 and both the built-in media types 446 and the plug-in media type modules 448. Using the API, the media type manager 444 accesses the media types originally supported by the camera contained in the built-in media types module 446. The API also allows the media type manager 44 to access new media types that are added to the camera via the plug-in media type modules 448.
Referring to both FIGS. 3 and 7, the operation of the present invention will now be described. When the camera 110 is first turned on and booted up, the startup/configuration 440 component begins to execute and loads the drivers
436 and the kernel 438 into DRAM 346. Included in the ROM 350 is a system file that contains configuration information for the digital camera 110, such as which built-in media types 446 are supported by the camera 110.
Software applications written for the camera 110 may be installed in the camera 110 from the removable memory 354 (e.g., from a flash disk) or by downloading the applications from a host computer. The software applications may include their own system folder in order to configure the digital camera 110 for the new application. If the software application requires new media types, then the new media types are defined in the system folder in the form of plug-in media type modules 448.
After the kernel 438 is loaded, the startup/configuration module 440 searches for the system folder in both the DRAM 346 and the removable memory 354. If a system folder exists in both the DRAM 346 and the removable memory 354, then in a preferred embodiment, the contents of both system files are used. Alternatively, the system file from the removable memory 354 can be used to overwrite the system file in the DRAM 346.
After the built-in media types module 446 are retrieved from the ROM 350 and the plug-in media type modules 448 (if any) are retrieved from the system folder, the plug-in media type modules 448 are linked with the built-in media types module. All of the media types present in the camera are then reported to the media type manager 444 so that the media type manager 444 is made aware what set of media types are supported by the digital camera 110.
After the start-up process, the control application 432 begins to execute. As the user operates the camera, the control program 432 makes calls to various components of the tool box 434 in order to perform tasks, such as displaying an image cell 420 in the view finder 402 when the camera is in review mode. To display an image cell 420, the control application 432 makes a call to the media type manager 444.
In response to the call, the media type manager 444 calls the cell manager 442 to retrieve the necessary information. After the information regarding the image cell 420 is returned, the media type manager 444 displays both the image area 422 of the particular cell 420 and the necessary media icons in the icon area 424 of the cell 420. To do this, the media type manager 444 must first determine whether each of the media types associated with the image are built-in or plug-in types. If the media type is built-in, then the media type manager 444 makes an API call to the built-in media type module 446 to access the appropriate icon. If the media type is plug-in, then the media type manager 444 makes an API call to the plug-in media type module 448.
Although the API for expanding the number of the media types in a digital camera of the present invention may be implemented using a variety of file organizations, a preferred file organization is explained below.
In a preferred embodiment, each object captured by the camera is stored in a media type file in memory, and objects comprising more than one media type are stored in either a file or a folder. In order to support objects comprising multiple media types, each cell references a file or folder that includes more than just image data. An example media type file is a file used for storing a still image, as described below.
FIG. 8 is a block diagram showing a format of a still image file in accordance with the present invention. The file format 460 includes a header 462, image data 464, a thumb nail image 466, and an image tag field 468.
The header 462 identifies the particular image file. The image data 464 is the actual data comprising the captured image in compressed form. The thumb nail image 466 is a smaller version of the captured image. Including the thumb nail image 466 enables the digital camera to directly display the image in the image area 422 of a cell 420 without processing and decompressing the image data 462. The image tag field 468 includes information, preferably in the form of tags, regarding the image represented by the image data 464. Media type tags, for instance, indicate all the media types associated with the image, such as whether the image is a single image or a panorama image, for example. The media type tags are used to select the type of icon that is displayed in the icon/information area 424 of a cell 420 along side the thumb nail image 466.
Besides media tags, the image tag field 468 may also include other types of tags for storing additional information regarding the image and/or the camera 110 itself. For example, a tag could be used to indicate the settings of the camera 110 at the time the image was captured, or indicate the identity of the camera manufacturer, for instance. The information in these tags may be accessed through the buttons on the camera interface 400. The additional information may then be displayed either as text in the icon/information area 424, or displayed in a dialog box that is displayed in the view finder 402.
The formats for files and folders corresponding to other media type files are similar to the file format for still images. However, the content of the image data 464 may differ. For example, the image data 464 for a panorama may contain data for several images, rather than data for just one image, and the image data 464 for other types of media may not contain image data at all, only text. As a user captures images with the digital camera, the captured images are stored in the file system in various locations in memory, but not necessarily in the
order that the images were captured. The cell manager 442 keeps track of the stored image files 460 by generating and maintaining a list of cells, where each cell corresponds to each image.
FIG. 9 is a block diagram illustrating a list 500 of cells managed by the cell manager 422. According to the present invention, each cell is a data structure containing various pointers. In a preferred embodiment, the cell list 500 is implemented as a doubly-linked list. Therefore, two of the pointers in each cell are used to point to a preceding and following cell. The beginning of the list 500 is called a header 502 and the end of the list is called the footer 504. Referring to both FIGS. 8 and 9, address pointers in each cell point to locations in the file system 506 where the image data 464, the thumb nail 466, and the image tags 468 of the corresponding image file 430 are stored.
Referring now to both FIGS. 4 and 9, as a user navigates through the view finder 402, new image cells 420 may be scrolled across the view finder 402. As an image cell 420 nears the view finder 402 for display, the pointers in the corresponding cell in the cell list 500 are used to copy the image's thumb nail 466 and image tags 468 from the file system into DRAM 346. When its time to display the image cell 420, the cell manager 442 displays the thumb nail 468, and makes a call to the media type manager 444 to display the media icons associated with the image.
If the image tags 468 indicate the image includes built-in media types, then the media type manager 444 calls the built-in media types module 446 using the API. If the image tags 468 indicate the image includes plug-in media types, then the media type manager 444 calls the appropriate plug-in media type modules 448 using the API.
FIG. 10 is a block diagram illustrating the organization of a plug-in media type module 448 API in accordance with the present invention. The API 520 in each of the plug-in media type modules 448 includes a capture section 522, a view section 524, and an edit/play section 526. The capture section 522 includes capture process code and view finder control code for controlling how the specific media type is captured and stored by
the camera. The view section 524 includes cell display code for controlling how the image cell representing that media type is reviewed. And the edit/play section 526 of the API 520 includes full screen display code for the playing and editing of the specific media type in the view finder 402. These three APIs define the behavior for each media type so that the control application 432 recognizes how to have the specific media type captured and stored by the camera. Each of the sections 522, 524, and 526 also includes a respective menu code section 522A, 524A, and 526A, and an icon section 522B,
524B, and 526B. The menu code sections 522A, 524A, and 526A contain the menu items that are displayed in the view finder 402 during the corresponding mode (capture, edit/play, view) of the media type. The icons sections 522B,
524B, and 526B contain the icons that are displayed in the image cells of objects that include the corresponding media type.
The API 520 allows new media types of any type to be added to the digital camera 110. Also, the media types supported by the camera are not required to be image-based, they may also be text base. Examples of text-base media types include a to-do list and a phone list, for example.
Assuming such media types were not built-in, a software developer could develop plug-in media types 448 for the to-do list and a phone list application using the API 520. The new media types could then be loaded into the camera via the removable memory 354 or downloaded from a host computer connected to the camera.
After a new media type is loaded, the media type manager 444 calls the view section 524 of the API to control the display of a single image cell 420 representing the new media type application; or the media type manager 444 calls the edit/play section 526 of the new media type API to control how the new media type application is played in the view finder 402.
FIGS. 11A and 11B are a block diagrams showing example plug-in media types being displayed in the view finder of the digital camera. FIG. 11 A depicts a to- do list plug-in media type that has been added to enable to a user to record voice notes in the camera. Each voice note is shown with a time and date that
the note was recorded, and the length of the note. Rather than an image-based media type, the to-do list is a sound-based media type.
FIG. 11B depicts a phone list plug-in media type that has been added to enable a user to record names and telephone numbers in the camera. A phone list is an example of a text-based media type.
After a media type, such as the phone list, has been displayed in the view finder 402 and the user places the camera into capture mode, the capture menu code 522A of the API 520 is responsible for displaying a menu for the types of possible user operations. FIG. 12 is a block diagram depicting one method for capturing data for text-based media types on the digital camera. To enable a user to enter information, such as names and telephone numbers, the capture section 522 of the API 520 may display an alphabetical menu selection box in the view finder 420. The user may then repeatedly navigate to desired letters and press an "Accept" button to enter the letters.
Another example of a capture method for capturing data for a text-based media type is to couple a scanner to the camera and then scan-in the data. To capture data for a phone list, for example, a user may scan a client's business card. The information from the scan would then be captured by the capture section 522 of the API 520 and preferably converted to text via an OCR module, and then stored in the file system 506 of the digital camera.
Referring again to FIG. 4, after the user has captured data for the new media type, an image cell 420 is displayed in the view finder 420 identifying the new instance of the media type. FIG. 4 shows, for example, image cells 420 corresponding to the to-do list and the phone list of FIGS. 11A and 11B.
A method and system for expanding the media types supported in a digital camera has been disclosed. Although the present invention has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and those variations would be within the spirit and scope of the present invention.
Accordingly, many modifications may be made by one of ordinary skill in the art without departing from the spirit and scope of the appended claims.