TW201737199A - Multiple camera computing system having camera-to-camera communications link - Google Patents

Abstract

An apparatus is described. The apparatus includes a first camera system having a processor and a memory. The first camera system includes an interface to receive images from a second camera system. The first camera system includes a processor and memory. The processor and memory are to execute image processing program code for first images that are captured by the first camera system and second images that are captured by the second camera system and that are received at the interface.

Description

Multi-camera computing system with camera-to-camera communication link

One problem in conventional computing systems with one or more integrated cameras is that the excess image data is streamed to a processing core of a computing system (eg, one or more application processors) to cause the processing core to process the image. Data and make intelligent decisions based on the content of the image data. Unfortunately, many of the data streamed to the processor is unrelated or unattended. Thus, a significant amount of power and resources are consumed, and substantially no meaningful data is transmitted through the system.

The present invention describes an apparatus. The device includes a first camera system having a processor and a memory. The first camera system includes an interface for receiving images from a second camera system. The first camera system includes a processor and a memory. The processor and the memory perform a first image captured by the first camera system and an image processing code of the second image captured by the second camera system and received at the interface. The present invention describes an apparatus. The apparatus includes means for processing an image received by the first camera system at a first camera system. The apparatus also includes means for processing, at the first camera system, an image received by a second camera system and transmitted to the first camera system via a communication link coupled to the first camera system and the second camera system . The device also includes means for notifying the first camera system of an event associated with an application processor and either or both of the first camera system and the second camera system.

1 shows a first prior art computing system having a dual camera configuration in which two different cameras 101, 102 have separate respective hardware 105, 106 channels to an application processor 103. In accordance with the operation of the system of FIG. 1, the two cameras 101, 102 substantially independently direct their own dedicated video streams and other forms of communication to their respective channels through their respective channels 105, 106 of the hardware platform 104 of the system. One of the problems associated with the method of Figure 1 is that the additional burden and wiring of the single camera solution resides with the computer system. For example, if the first camera 101 desires to communicate to the processor 103, one or more signals are transmitted along the channel 105; and if the second camera 102 desires to communicate to the processor 103, one or more signals are transmitted along the channel 106. Therefore, the processor 103 needs to be able to service two different communications at two different processor inputs 107, 108 (e.g., processor interrupt inputs). The consumption of the two different processor inputs 107, 108 is invalid, in this regard, the processor 103 has only a limited number of inputs and the two such inputs 107, 108 are consumed by the dual camera system. Therefore, it may be difficult to feed other boot channels from other components in the system (possibly with a large number of components), which can be especially troublesome if it is not designed to directly reach any component of the processor. Another problem associated with the method of Figure 1 is the complex wiring density and associated power consumption. Here, consider a situation in which two cameras simultaneously stream to the processor 103 along their respective channels 105,106. Therefore, both data streams are separately transmitted to the processor through the hardware platform 104. In addition to the inherent wiring complexity inherently from the two separate dedicated hardware channels 105, 106 of the design to hardware platform 104, there is also the problem of inefficient power consumption, especially when the original data or slightly processed image data Boot to processor 104 (i.e., when the processor performs a rather complex function on the data streamed from cameras 101, 102). In this case, two separate streams of bulk data may need to be transmitted within the platform 104 over a potentially large distance, which would require a significant amount of power to implement. Another problem with the method of Figure 1 is that the interfaces 109, 110 of the dual camera system are relatively inflexible. Here, the two cameras must be connected to the physical interface 109, 110 pairs provided for them. That is, one of the hardware platforms 104 refuses to integrate the opportunity to integrate cameras that do not support interfaces 109 and 110 and, likewise, the camera vendor refuses to integrate their cameras into the designer's platform 104. An improved method known in the art is observed in Figure 2. According to the method of FIG. 2, a bridge function 212 is placed between the dual camera systems 201, 202 and the processor 203. The bridging function 212 is substantially enhanced and/or multiplexed from two cameras 201, 202 (e.g., dual image streams, etc.) to a single channel 213 that is fed to the processor 203. The introduction of the bridging function 212 helps alleviate some of the inefficiencies discussed above with respect to FIG. In particular, only one input 207 is consumed at the processor to "release" an input 208 (as compared to the method of FIG. 1) such that, for example, some other system components other than a camera can communicate directly with the processor 203. . However, power consumption remains the focus. Here, the bridging function 212 is limited to multiplex and/or interleaving and does not perform a substantial data reduction procedure (such as data compression). Thus, if a large amount of data is streamed to the processor 203, then the hardware platform 204 will consume a significant amount of power to transfer large amounts of data over the long distance within the platform 104. In addition, the bridging function 212 does not address any of the types between the types of interfaces 209, 210 that the platform 204 provides to connect to a camera and the types of interfaces that can be included for integration into one of the options available to the system. Does not match the problem. Referring to Figure 3, the power consumption problem can be mitigated at least to some extent by introducing processing intelligence into one of the cameras. Here, FIG. 3 shows another prior art method in which one of the cameras in one of the dual camera systems ("primary" camera 301) has a local processor 314 and local memory 315. Processor 314 executes the code from memory 315 and can perform a particular data size reduction function, such as data compression, to effectively reduce the amount of data that needs to be transferred to host processor 303. The hardware platform 304 will consume less power as very little data is sent to the main processor 303 (e.g., ideally only the main processor 303 needs to perform information related to the image-related application sent from the main camera 301 to the main processor 303). The main processor 303 is expected to provide any loss of functionality. Note, however, that the method of FIG. 3 includes only one processor solution 314 in one of the cameras 301. Here, the dual camera system typically has a primary camera 301 and a primary camera 302 (eg, the secondary camera can be one of the "back side" cameras facing the user of a handheld device and the primary camera can be used for a handheld device. One of the "front side" cameras (or alternatively the secondary camera can be the primary camera and the front side camera can be the secondary camera)). The secondary function of the secondary camera 302 typically does not result in additional costs for the processor 314 and memory 315 resident in the primary camera 301. Thus, the power consumption reduction improvement of transmitting less data through the platform 304 to the main processor 303 is achieved only for the transfer from the primary camera 301 to the main processor 303 and not from the secondary camera 302 to the main processor 303. Additionally, like the methods of FIGS. 1 and 2, the hardware platform 304 of FIG. 3 provides a pair of fixed interfaces 309, 310 for a dual camera system. Thus, there is still a mismatch between the interfaces 309, 310 supported by the hardware platform 304 and the design-to-camera interface that may otherwise be considered as candidates for integration into the platform 304. In addition, the method of FIG. 3 consumes two processor inputs 307, 308, as discussed with respect to FIG. 1, which may exclude other important components within the computing system from communicating directly with the main processor 303. 4 shows a novel method that overcomes the aforementioned problems over any of the prior art solutions discussed above with reference to FIGS. 1-3. The method of FIG. 4 includes a communication channel 416 between the secondary camera 402 and the primary camera 401. Additionally, a bridge function 417 is included in the primary camera 401 to, for example, multiplex and/or combine image streams from the two cameras 401, 402 via a single channel 405 that exists between the primary camera 401 and the main processor 403. As discussed in further detail below, channel 405 can be a direct fixed channel or a physical channel through one of a plurality of components of hardware platform 404. In the method of FIG. 4, image data from the second camera 402 is transmitted to the primary camera 401 via a communication channel 416 that exists between the two cameras 401, 402. The bridge function 417 embedded in the main camera 401 (for example, as one of the executable software programs executed by the processor 414) enables the main camera 401 to transmit the image data of the secondary camera and the image data of the main camera to the main processor 403 along the channel 405. . Thus, as with the method of FIG. 2, the improved method of FIG. 4 consumes only one input 407 at the main processor 403 which "releases" a processor input 408 to make it available for direct communication with some other components in the system. In addition, as with the method of FIG. 3, power savings can be achieved because the data size reduction routine (such as data compression) can be performed by the primary camera 401 (which reduces the amount of data that needs to be transmitted to the main processor 403 via the platform 404). However, although the method of FIG. 3 can only reduce the power consumption of the main camera 301 (ie, only reduce the size of the image data of the main camera), the method of FIG. 4 can reduce the transmission of information from the two cameras 401, 402 to the main processing. The associated power consumption of 403. Here, the data reduction procedure (e.g., data compression) performed by the primary camera 401 on its own video material may also be performed on the video material it receives from the secondary camera 402 via channel 416. Thus, a smaller size stream from the two cameras 401, 402 can be sent to the main processor 403. Still further, the secondary camera 402 does not care at least for the particular type of camera interface 409 that has been implemented on the host hardware platform 404. Therefore, only the primary camera 401 needs to be compatible with one interface 409 of one of the platforms 404. To implement this solution, the secondary camera interface 419 only needs to be compatible with the second interface 418 of the primary camera. Thus, the presence of channel 416 provides a more liberal choice between the primary and secondary cameras 401, 402 for the system designer to integrate with the camera 404. For example, as just one example, the channel 416 resident between the cameras 401, 402 can be a dedicated channel for one of the camera manufacturers that manufacture one of the primary and secondary cameras 401, 402. Although secondary camera 402 may not have one interface that is compatible with host platform 404, secondary camera 402 can stream its data to main processor 403 via camera-to-camera channel 416 and bridge function 417 of primary camera 401. Additionally, the method of FIG. 4 is substantially more efficient for applications in which images from two cameras 401, 402 are combined or otherwise processed together to achieve a combined singular information set. An example is an embodiment in which two cameras 401, 402 act as a stereo pair and their respective images are combined to determine a three-dimensional depth profile ("depth map") of one of the objects on which both cameras 401, 402 are focused. . The depth profile can be used by the main processor 403 to perform some image depth functions (such as hand/finger motion detection, facial recognition, etc.). Here, the software executing on the main camera 401 can process its own image stream data and image stream data from the secondary camera 402 to calculate a depth map. The depth map can be sent from the primary camera 401 to the main processor 403. Here, the previously known solution requires both video streams to be sent to the main processor 403. Next, the main processor 403 performs calculation to determine a depth map. In the improved method described above (where the depth map is computed within the primary camera 401), only one depth map is transmitted across the platform 404 to the main processor 403 and (potentially large) image stream data remains localized to the dual camera. The systems 401, 402, thus achieving substantial power savings. Here, the depth map is understood to be the amount of data that is much smaller than the data of the depth map from the image stream it calculates. Another example is autofocus. Here, the depth profile information calculated from the image streams of the two cameras 401, 402 by the software executing on the main camera 401 can be used to control one of the one or two cameras 401, 402. For example, software executing on primary camera 401 can process image streams from two cameras 401, 402 to provide control signals to voice coils, actuators, or other electromechanical devices within one or more cameras 401, 402. The focus position of the lens system(s) of the cameras 401, 402 is adjusted. As a comparison point, the conventional system streams the image data to the main processor and the main processor determines the auto focus adjustment. In an improved method that can be performed by the improved system of FIG. 4, main processor 403 receives only focused image data (ie, main processor 403 does not have to perform various autofocus tasks). The reduced amount of data sent to the main processor 403 again corresponds to a power reduction improvement. Other functions may also be performed by software executing on the primary camera 401 to reduce the amount of information sent from the dual cameras 401, 402 system to the main processor 403. Significantly, in conventional systems, much of the information streamed to the main processor 403 is less valuable. For example, in the case of the old image recognition function, a large amount of data that does not need to be searched for images is costly streamed to the main processor 403, and once the main processor 403 realizes that the image sought is not present, the data will be discarded. A better method would perform image recognition within the primary camera 401 and notify the host processor 403 only if the sought image has recognized that the desired or sought image is currently in view of the camera(s). After the item being sought (or the item of interest) is recognized by the primary camera 401, then the image material can be streamed to the main processor 403 such that the processor can perform any function to be performed after identifying the desired image (eg, tracking objects, Record the characteristics around the object, etc.). Thus, ideally, only relevant or focused information (or information with a high probability of having one of the relevant or focused information) is actually delivered across the platform 404 to the main processor 403. Ideally, other information that does not contain related items is abandoned by the primary camera 401. Here, note: so the search item of interest can be found in the image stream of the main camera and the image stream of the secondary camera, because the main camera can process the image stream of the main camera and the image stream of the secondary camera. Depending on the implementation, the criteria that triggers attention to the main processor 403 that the item of interest has been found can be configured to identify an item in either stream or only one of the streams. The associated feature search program executed by the primary camera on the video stream of either or both cameras 401, 402 may include, for example, face detection (detecting the appearance of any face), face recognition (detecting one) The appearance of a particular face), facial expression recognition (detecting a specific facial expression), object detection or recognition (detecting a general or specific object), motion detection or recognition (detecting a general or specific motion), event detection Measure or identify (detect a general or specific event), image quality detection or identification (detecting one of the image quality's general or specific levels). After the primary camera has detected a search item in an image stream, it may then perform any of a number of related "subsequent" tasks to further limit the amount of information ultimately directed to the host processor 403. Some examples of additional actions that may be performed by a primary camera include any one or more of the following: 1) identifying an area of interest within an image (eg, surrounding an intermediate region of one or more of the sought features in the image); 2) parsing an area of interest within an image and delivering it to other (eg, higher performance) processing components within the system; 3) discarding an unattended area within an image; 4) delivering an image to Other components in the system previously compress the image or portions of the image; 5) capture a particular image (eg, a snapshot, a series of snapshots, a video stream); and 6) change one or more camera settings (eg, change coupling) The setting on the servo motor of the optical device to zoom in, zoom out or otherwise adjust the focus/optical device of the camera; change an exposure setting; trigger a flash). Note that although FIG. 4 shows a direct channel 405 between the primary camera 401 and the primary processor 403, the full end-to-end path between the primary camera 401 and the primary processor 403 may terminate at the primary processor 403 and/or The hard channel can be accessed directly through one of several system function blocks before reaching the camera. In one embodiment, a direct hardware path exists in one of the interrupt inputs from the primary camera 401 to the main processor 403 to notify the host processor 403 of an incident detected at the primary camera. Additionally, the actual data can be delivered to the system memory of the platform 404 (not shown), where the actual data is then read by the main processor 403. In an embodiment, the interface into which the primary camera is actually inserted may be provided by a peripheral control hub (not shown). The data from the primary camera can then be directed from the peripheral control hub to the processor or stored in memory. The software/firmware executed by the primary camera 401 can be stored in non-volatile memory resident in the camera 401 or other location on the platform 404. In the latter case, the software/firmware is loaded from the platform to the primary camera 401 during system startup. Likewise, camera processor 414 and/or memory 415 can be integrated into one of the main cameras 401 or can be physically located external to camera 401 itself but, for example, placed in close proximity to camera 401 such that it operates effectively as one of local cameras 401 Processing system. Thus, the application is more generally related to camera systems than to specific cameras. Note that any of the cameras 401, 401 can be a visible light camera, a depth information camera (such as a time-of-flight ranging camera that radiates infrared light and effectively measures the time it takes for the radiated light to return to the camera after reflection) or Integrate one of the visible light detection and depth information captures in a single camera solution. Although the above discussion focuses on the execution of code (software/firmware) by a camera system, some or all of the above functions may be performed entirely in hardware (eg, as an application specific integrated circuit or programmed to perform) One of these functions can be a programmatic logic device) or a combination of hardware and code. The interface between the primary camera 401 and the hardware platform 404 can be an industry standard interface such as an MIPI interface. The interface and/or channel between the two cameras can be an industry standard interface (such as an MIPI interface) or can be a proprietary interface. Figure 5 illustrates one of the above methods that may be performed by a system having one of a plurality of cameras, wherein a communication link is present between the cameras. According to Figure 5, the method includes processing an image 501 received by a first camera system at a first camera system. The method also includes processing, at the first camera system, an image 502 received by a second camera system and transmitted to the first camera system via a communication link coupled to the first camera system and the second camera system. The method also includes an event 503 from the first camera system to notify an application processor of any one or both of the first camera system and the second camera system. Figure 6 provides an illustrative depiction of one of the computing systems. Many of the components of the computing system described below are applicable to a computing system having an integrated camera and associated image processor (eg, a handheld device such as a smart phone or tablet). The average technician will be able to easily separate the two. As seen in Figure 6, the basic computing system can include a central processing unit 601 (which can include, for example, a plurality of general purpose processor cores 615_1 through 615-N and one of the multi-core processors or application processors) Main memory controller 617), system memory 602, a display 603 (eg, a touch screen or tablet), a local wired point-to-point link (eg, USB) interface 604, and respective network I/O functions 605 (such as An Ethernet interface and/or a cellular modem subsystem, a wireless local area network (e.g., WiFi) interface 606, a wireless point-to-point link (e.g., Bluetooth) interface 607, and a global positioning system interface 608 Various sensors 609_1 to 609_N, one or more cameras 610, a battery 611, a power management control unit 612, a speaker and microphone 613, and an audio encoder/decoder 614. An application processor or multi-core processor 650 can include one or more general purpose processor cores 615, one or more graphics processing units 616, and a memory management function 617 (eg, a memory controller) within its CPU 601. An I/O control function (such as the aforementioned peripheral control hub) 618. The general purpose processor core 615 typically executes application software for the operating system and computing system. Graphics processing unit 616 typically performs graphics intensive functions to, for example, generate graphical information that appears on display 603. The memory control function 617 interfaces with the system memory 602 to write data to/from the system memory 602. Power management control unit 612 generally controls the power consumption of system 600. Each of the touch screen display 603, the communication interfaces 604 to 607, the GPS interface 608, the sensor 609, the camera 610, and the speaker/microphone codecs 613, 614 can be viewed as various forms of I/ relative to the overall computing system. O (input and / or output), the overall computing system also includes an integrated peripheral device (eg, the one or more cameras 610) as appropriate. Depending on the implementation, each of these I/O components can be integrated on the application processor/multi-core processor 650 or can be external to the die or package of the application processor/multi-core processor 650. In one embodiment, at least two of the cameras 610 have a communication channel between the cameras and one of the cameras has one or all of the features and functions of one or all of the features discussed above with respect to FIG. body. Embodiments of the invention may include various programs as set forth above. Programs can be embodied in machine executable instructions. Instructions can be used to cause a general purpose or special purpose processor to execute a particular program. Alternatively, such programs may be executed by specific hardware components containing fixed-line logic for executing programs or by any combination of stylized computer components and custom hardware components. The elements of the present invention may also be provided as a machine readable medium for storing machine executable instructions. A machine-readable medium can include, but is not limited to, a floppy disk, a compact disc, a CD-ROM and a magneto-optical disc, a flash memory, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a broadcast medium, or a storage device suitable for storing electronic instructions. Other types of media/machine readable media. For example, the present invention can be downloaded via a communication link (e.g., a modem or network connection) to be transferred from a remote computer (e.g., a server) to a data signal embodied in a carrier or other communication medium. A computer program that requests a computer (such as a client). In the previous specification, the invention has been described with reference to the specific exemplary embodiments of the invention. However, it will be apparent that various modifications and changes can be made in the present invention without departing from the spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded as

101‧‧‧ Camera / First Camera
102‧‧‧Camera/second camera
103‧‧‧Application Processor
104‧‧‧ hardware platform
105‧‧‧ Hardware/Channel
106‧‧‧ Hardware/Channel
107‧‧‧Processor input
108‧‧‧Processor input
109‧‧‧ interface
110‧‧‧ interface
201‧‧‧Two camera system
202‧‧‧Double camera system
203‧‧‧ processor
204‧‧‧ hardware platform
207‧‧‧ input
208‧‧‧Enter
209‧‧‧ interface
210‧‧‧ interface
212‧‧‧Bridge function
213‧‧‧ single channel
301‧‧‧ "main" camera
302‧‧‧ secondary camera
303‧‧‧Main processor
304‧‧‧ hardware platform
307‧‧‧Processor input
308‧‧‧Processor input
309‧‧‧Fixed interface
310‧‧‧Fixed interface
314‧‧‧Local Processor/Processor Solutions
315‧‧‧Local memory
401‧‧‧Main camera/dual camera system
402‧‧‧Secondary camera/second camera/dual camera system
403‧‧‧Main processor
404‧‧‧Hard Platform/Host Hardware Platform/Host Platform
405‧‧‧ channel
407‧‧‧Enter
408‧‧‧Process input
409‧‧‧ camera interface
414‧‧‧ camera processor
415‧‧‧ memory
416‧‧‧Communication channel/camera to camera channel
417‧‧‧Bridge function
418‧‧‧Second interface of the main camera
419‧‧‧ secondary camera interface
601‧‧‧Central Processing Unit (CPU)
602‧‧‧ system memory
603‧‧‧Display/Touch Screen Display
604‧‧‧Local wired point-to-point link interface/communication interface
610‧‧‧ camera
611‧‧‧Battery
612‧‧‧Power Management Control Unit
615_1 to 615-N‧‧‧General Processor Core
616‧‧‧Graphic Processing Unit (GPU)
617‧‧‧Main memory controller
618‧‧‧I/O control function
650‧‧‧Multicore processor

The following description and the drawings are used to illustrate embodiments of the invention. In the drawings: Figure 1 shows a first prior art dual camera configuration; Figure 2 shows a second prior art dual camera configuration; Figure 3 shows a third prior art dual camera configuration; Figure 4 shows an improved dual camera configuration Figure 5 shows one of the methods performed by one of the camera configurations of Figure 4; Figure 6 shows a computing system.

401‧‧‧Main camera/dual camera system

402‧‧‧Secondary camera/second camera/dual camera system

403‧‧‧Main processor

404‧‧‧Hard Platform/Host Hardware Platform/Host Platform

405‧‧‧ channel

407‧‧‧Enter

408‧‧‧Process input

409‧‧‧ camera interface

414‧‧‧ camera processor

415‧‧‧ memory

416‧‧‧Communication channel/camera to camera channel

417‧‧‧Bridge function

418‧‧‧Second interface of the main camera

419‧‧‧ secondary camera interface

Claims (20)

An apparatus comprising: a first camera system including a processor and a memory, the first camera system including an interface for receiving images from a second camera system, the first camera system including a process And a memory for executing an image processing program for the first image captured by the first camera system and the second image captured by the second camera system and received at the interface code. The device of claim 1, wherein the image processing code determines a depth map according to the first image and the second image. The device of claim 1, wherein the image processing code identifies one of the items of interest. The device of claim 3, wherein the image processing code discards an image that does not contain the item of interest. The device of claim 1, wherein the image processing code performs data compression. The device of claim 1, wherein both the first camera system and the second camera system are capable of capturing a visible image. The device of claim 6, wherein at least one of the first camera system and the second camera system is capable of capturing depth profile information from a time-of-flight ranging technique. A computing system comprising: at least one application processor; a memory controller coupled to the at least one application processor; a system memory coupled to the memory controller; a first camera system a processor and a memory; a second camera system; a communication link between the first camera system and the second camera system, wherein the processor and the memory of the first camera system are executed by The first image captured by the first camera system and the image processing code captured by the second camera system and transmitted to the second image of the first camera system via the communication link. The computing system of claim 8, wherein the first camera system transmits information from both the first camera system and the second camera system to the at least one application processor. The computing system of claim 8, wherein the first camera system is capable of transmitting an interrupt to the at least one application processor based on the processing of the first image and the processing of the second images. The computing system of claim 8, wherein the image processing code determines a depth map based on the first image and the second image. The computing system of claim 8, wherein the image processing code identifies one of the items of interest. The computing system of claim 12, wherein the image processing code discards an image that does not contain the item of interest. The computing system of claim 8, wherein the image processing code performs data compression. A machine-readable storage medium containing image processing code for causing execution of a method when executed by a first camera system deployed in a computing system, comprising: processing an image received by the first camera system; An image received by the second camera system and transmitted to the first camera system via a communication link coupled to the first camera system and the second camera system; and an application processor and the first camera system are notified An event related to either or both of the second camera systems. The machine readable storage medium of claim 15, wherein the image processing code determines a depth map from the first image and the second image. The machine readable storage medium of claim 15, wherein the image processing code identifies one of the items of interest. The machine readable storage medium of claim 17, wherein the image processing code discards an image that does not contain the item of interest. The machine readable storage medium of claim 15, wherein the image processing code performs data compression. The machine readable storage medium of claim 15, wherein the computing system is a handheld device.