Background technology
Existing mobile communication terminal has the ARM core Applied Digital signal processor (aDSP) of camera, utilizing camera interface and process image data etc., and image preview (image preview), picture catching (image capture) are provided, especially in visual telephone (Video Telephony) service of W-CDMA etc. etc.Because the increase that above-mentioned multi-medium data is handled and the exploitation of various application programs etc., most mobile communication terminal need be handled more data.Especially, in order to handle camera and the LCD pixel (Pixel) that improves day by day along with as 1/4 public intermediate-resolution pixel format (QCIF:Quarter Common Intermediate Format), QVGA (Quarter VGA), video and graphic display (VGA:Video Graphics Array), extending video figure display forms such as (XVGA:Extended VGA), mobile communication terminal need be handled more multidata quickly.
As shown in Figure 1, it is the frame assumption diagram of mobile communication terminal with camera of above-mentioned functions.The camera module of filmed image (camera module) 1 is directly connected to processor 2 in the mobile communication terminal with camera module 1 of prior art.And the processor 2 of all functions of control mobile communication terminal connects microphones (mic) 6, audio output part (speaker/receiver) 10, LCD4, external memory storage 3, comprise keypad etc. user interface (user interface) 7, with outside RF module (RF module) 5 of carrying out the RF signal communication.
As shown in Figure 1 to Figure 3, Fig. 2 is the processing procedure schematic diagram of image preview in the mobile communication terminal of Fig. 1 (imagepreview) and picture catching (image capture), and Fig. 3 is a schematic flow sheet of realizing the data of VT service in the mobile communication terminal of Fig. 1.The ARM of the input and output of installation and control image data on the above-mentioned processor 2 (advanced RISC machine) core 2ac, the Applied Digital signal processor of process image data (aDSP:application digital signalprocessor) 2ad, modem digital signal processor (the mDSP:modem digital signal processor) 2md of the reception and registration of control WCDMA signal processing and speech data, directmemoryaccess (to call ' DMA (direct memory access) ' in the following text), utilizing camera interface (camera interface) 2c, LCD interface (LCD interface), the audio codec of audio signal (send or receive and use Audio codec) 8,9 (with reference to Fig. 3), the RF interface of processing RF signals (RF interface), control the event controller (eventcontroller) of the variety of event that comprises user interface event (event) etc.
External memory storage 3 is installed random access memory ram 3R and flash memory (FLASH MEMORY) 3F.
As Fig. 2 or shown in Figure 3, above-mentioned aDSP 2ad handles that the compression, MP3 signal processing, the midi signal that convert the view data of yuv format the images such as image transitions, JPEG of rgb format to are handled, mpeg data is handled, the i.e. H.223 agreement control of the multiplexing agreement of packet unit, the control of the multiple bit rate of adaptability (adaptive multi rate) vocoder (vocoder) function.
Above-mentioned mDSP 2md (with reference to Fig. 3) carries out image and data-signal is handled, and Figure 3 shows that to carry out the WCDMA signal processing.
With reference to Fig. 2 the image preview (imagepreview) of the mobile communication terminal of prior art with said structure and the processing procedure of picture catching (image capture) are described as follows.
After receiving, the utilizing camera interface 2c that the image data of being taken and converted to the signal of telecommunication as shown in Figure 2 by camera module 1 is installed in processor 2 inside is stored in the random asccess memory 3R of external memory storage 3.And, the image data that is stored in RAM 3R read by the aDSP 2ad of processor 2 and convert the image data of yuv format to the rgb format image after (most LCD shows RGB data pixel) be stored in RAM 3R again.The image data of rgb format reads back demonstration one frame (frame) on LCD4 by ARM (Advanced RISCmachine) the core 2ac of processor 2 again.
And then the picture with the aid of pictures processing procedure of catching, the utilizing camera interface 2c that is installed in processor 2 inside receives from the image data of camera module 1 input and is stored in the RAM 3R of external memory storage 3.ADSP2ad will be stored in RAM 3R yuv format image data and convert the image data of rgb format to, store RAM 3R into after carrying out the coding (Encoding) of JPEG etc.Then, be stored in flash memory (Flash memory) 3F after the ARM core 2ac image data that will be stored in RAM 3R again reads.
In the processing procedure when carrying out above-mentioned image preview and picture catching, ARM core 2ac should concurrently, periodically or momentaryly handle other incidents with above-mentioned processing, therefore external memory storage 3 is carried out the continuation visit.And as shown in Figure 3, under the situation of receiver, video phone (VT) service, processor is carried out and will be stored in the random asccess memory 3R of external memory storage 3 and the processing procedure that reads from the image data of camera module 1 input as shown in Figure 2.
As can finding out in the illustrated processing procedure of Fig. 2 and Fig. 3, the visit of external memory storage 3 accounts for a sizable part in the whole load of mobile communication terminal.That is to say, all image processing data are repeatedly via the data/address bus of the random asccess memory 3R of the mDSP 2md of connection processing device 2 and aDSP 2ad and external memory storage 3 in the mobile communication terminal, thereby the random asccess memory 3R of the mDSP 2md of processor 2 and external memory storage 3 takes place between the aDSP 2ad the excessive data of data bus are occupied also because the ARM core of process image data causes producing on the processor overload.
Aforesaid excessive data for the data/address bus data of handling image are occupied and are handled the reason that the overload of the processor 2 that causes takes place by the image data of ARM core2ac and are processor (Processor: comprise Modem﹠amp; Utilizing camera interface directly has been installed DSP etc.), will be after random asccess memory from the image store of camera module input, when image datas such as image preview, picture catching were handled, the various DSP of processor were in order to derive image data and to carry out accessing step from random asccess memory.
That is, in possessing the mobile communication terminal of camera, as in the prior art, when utilizing camera interface is installed in processor inside, overload takes place on the processor of the mobile communication terminal of process image data be inevitable.And, as mentioned above, if cause taking place on the processor overload for image processing, processor 2 just can't be handled the incident of VT service and the loss of image data take place and cause picture to disconnect, simultaneously because the overload of processor 2 has produced the problem that other additional programs also can't be installed immediately.
Embodiment
Below, illustrate in greater detail the present invention with reference to accompanying drawing.
At first, the meaning of the English mark shown in each accompanying drawing of Fig. 4 to Fig. 8 is as follows:
Camera module (Camera Module) 1 with reference to Fig. 5:
CSP: camera signals processor (Camera signal processor);
Color Filter: filter;
AFE: AFE (analog front end) (Analog Front End).
Double Port Random Memory (DPRAM (Dual Port SRAM)) 120 with reference to Fig. 6:
CS_A (B): dual-port A (B) chip is selected (DPRAM Port A (B) Chip Select);
OE_A (B): dual-port A (B) allows output (DPRAM Port A (B) Output Enable);
WE_A (B): dual-port A (B) allows to write (DPRAM Port A (B) Write Enable);
INT_A (B): dual-port A (B) interrupts (DPRAM Port A (B) Interrupt);
Addr_A (B) [0:29]: dual-port A (B) address (DPRAM Port A (B) Address);
D_A (B) [0:15]: dual-port A (B) data (DPRAM Port A (B) Data).
Camera signals (Camera Signal) with reference to Fig. 6:
CAM_MCLK: from main frame (processor 2) input master clock (Master clock for CSP fromhost (or processor2);
CAM_reset: camera signals restart processor signal (reset signal for CSP);
CAM_PCLK: data bit synchronised clock (Data bit synchronizing clock);
CAM_VSYNC: the vertical synchronizing signal of camera image (Vertical Synchronizingsignal);
CAM_HSYNC: the horizontal-drive signal of camera image (Horizontal Synchronizingsignal);
CAM_DATA: camera data/address bus (Camera data bus);
I2C_CLK:I2C main frame (processor 2) clock signal (inter integrated circuit bushost clock signal);
I2C_DATA:I2C data wire (inter integrated circuit bus data line).
Following as follows with reference to the description of drawings the present invention who shows one embodiment of the invention.
The exterior storage portion that has dual-port among Fig. 4 to Fig. 6 is double-port RAM (DPRAM) (hereinafter to be referred as " DPRAM "), is illustrated as dual-port synchronous DRAM (dual portSDRAM) especially.In addition, utilizing camera interface portion is illustrated as utilizing camera interface module 110.
As shown in Figure 4, it is the frame assumption diagram that the present invention has the mobile communication terminal of pick-up head interface device.Mobile communication terminal with pick-up head interface device 100 of the present invention shown in Figure 4 comprises: the processor 2 of all functions of control mobile communication terminal, the microphone (mic) 6 that is connected with described processor, audio output part (speaker/receiver) 10, LCD 4, external memory storage 3, comprise the user interface (user interface) 7 of keypad, can carry out the RF module of communicating by letter (RF module) 5 with the outside according to the RF signal.
Be provided with ARM (advanced RISC machine) core 2ac in the above-mentioned processor 2, Applied Digital signal processor (aDSP:application digital signal processor) 2ad, modem digital signal processor (mDSP:modem digital signal processor) 2md, LCD interface (LCD interface), carry out audio codec (send or receive and use Audio codec) 8,9 (with reference to Fig. 3) of Audio Signal Processing, carry out the RF interface RF interface of RF signal processing, control the event controller (eventcontroller) of the variety of event that is included in user interface event (event) etc.In addition, be equivalent to the pick-up head interface device 100 of utilizing camera interface portion for being made of exterior storage DPRAM of portion 120 of the present invention and utilizing camera interface module 110 of utilizing camera interface (camera interface) 2c of Fig. 1, it is located between processor 2 and the camera module 1.That is, in the present invention, the utilizing camera interface portion that is installed in the inside of processor 2 in the prior art is located at the outside of processor, thereby reduces the load of processor 2.Therefore the outside here (processor outside) memory 3 and aDSP 2ad and mDSP 2md and Fig. 1 omit its detailed description to illustrated in fig. 3 identical.
In said structure, pick-up head interface device 100 of the present invention receives image data from camera module 1, come work from processor 2 reception control commands and synchronizing signal (MCLK), and directly carry out data communication with external memory storage 3 with RAM 3R and flash memory 3F.
As shown in Figure 5, it is the syndeton schematic diagram of the pick-up head interface device of Fig. 4.Above-mentioned pick-up head interface device 100 is made up of utilizing camera interface module 110 (of the present invention utilizing camera interface portion) and DPRAM120 (of the present invention exterior storage portion) institute.Utilizing camera interface module 110 is connected to the signal of video signal output of camera module 1, makes it can receive signal of video signal.The data/address bus that constitutes between utilizing camera interface module 110 and the camera module 1 has the I2C interface (inter-integrated circuit bus) (or I2C data/address bus) that is connected with processor 2, and processor 2 provides the control command of handling signal of video signal and the synchronous master clock (MCLK) of giving image data by I2C interface (inter-integrated circuit bus) to camera module 1 and pick-up head interface device 100.In addition, first port of DPRAM120 (DPRAM Port A) connects the utilizing camera interface module, causes to store the image data of directly importing from camera module 1; Second port of DPRAM 120 (DPRAM Port B) is connected with external memory storage (RAM and FLASHMEMORY) 3 with processor 2 by memory interface (memory interface).
Fig. 6 is the detailed structure block diagram of the utilizing camera interface module 110 of Fig. 5.
As shown in Figure 6, utilizing camera interface module 110 is by image data buffer 111, DPRAM interface 113, controller 112 is formed, wherein, image data buffer 111 receives from 8 bit image datas of camera module 1 input and after converting 32 bit image datas to and exports successively by the order of input, DPRAM interface 113 appends to 16 bit addresses from 32 bit image datas of image data buffer 111 output and to DPRAM 120 outputs, and controller 112 provides image data control and synchronizing signal and gives the address to image data to image data buffer 111 and DPRAM interface 113.In addition, controller 112 is connected with processor 2 by command interpreter (the command interpreter) 114 with I2C interface (inter-integrated circuit bus interface), from processor 2 receive clocks (I2C_CLK), the control command by I2C data wire (I2C_DATA), master clock (MCLK) etc. output to image data buffer 111 and DPRAM interface 113.Image data buffer 111 and DPRAM interface 113 are carried out the multiplexed of image data according to control command and clock that slave controller 112 receives, and the image data that is stored in the DPRAM 120 is given address (address) and is stored in DPRAM 120.
Above-mentioned image data buffer 111 is by the first buffer 111a that receives 8 bit image datas from camera module 1,8 Bit datas that receive from the first buffer 111a are carried out four multiplexed channel multiplexer 111b, store successively from four channels of 8 Bit datas of multiplexer 111b input according to switching successively of multiplexer 111b and to latch buffer (latch buffer) 111c, the shift register 111d that will carry out first-in first-out FIFO after will receiving with 32 bit image datas from four 8 bit images that latch buffer 111c input forms.
At this, multiplexer 111b, latch buffer 111c, shift register 111d data synchronization according to giving by controller 112 from the master clock MCLK and the control command of above-mentioned processor 2 inputs.
Fig. 7 is the schematic diagram with data handling procedure of the image preview (image preview) of mobile communication terminal of Fig. 4 of pick-up head interface device 100 of above-mentioned Fig. 5 and Fig. 6 and picture catching (image capture).
At first the processing procedure of image preview (image preview) is as follows:
The imaged image of being taken by camera module 1 is stored in DPRAM120 by utilizing camera interface module 110.The image data of being imported by camera module 1 in this process is assumed to be rgb format, but is not to only limit to this, also can be yuv format.The image data that the ARM core 2ac of processor 2 will be stored in the RGB picture format of DPRAM120 directly reads and outputs to LCD4 with frame unit from DPRAM120.Thereby, make processor access external memory storage (External Memory) (with reference to Fig. 3 owing to reducing for imaged image being outputed to LCD4, time Fig. 4), compare to prior art shown in Figure 3 with Fig. 1, can significantly reduce the occupation rate of data/address bus and the load of processor 2.
Processing procedure during picture catching in addition, (image capture) is as follows:
The image data that aDSP 2ad will be stored in DPRAM120 reads, it is carried out be stored in behind the codings (Encoding) such as JPEG compression the RAM 3R of external memory storage 3, ARM core 2ac reads it and is stored in flash memory (Flash Memory) 3F again, therefore with prior art shown in Figure 2, after promptly being installed in utilizing camera interfaces (omission artwork symbols) in the processor 2 and will being stored in RAM 3R from the image data of camera module 1 input, the image data that aDSP 2ad will be stored in RAM again reads and it is handled, again be stored in RAM 3R afterwards, then ARM core 2ac compares from the technology that RAM 3R reads image data and is stored in flash memory 3F again, simplified the data access step that processor 2 is connected to RAM 3R, thereby reduced the occupation rate of data access time and data/address bus, and can reduce the load of processor 2.
Below the data access time in the data handling procedure of the present invention's image preview under the following conditions of the prior art by comparative descriptions Fig. 1 to Fig. 3 and Fig. 4 to Fig. 7, the present invention that relative prior art is described is in the effect of improving aspect the load factor of processor.
Condition:
QCIF (Quarter common intermediate format) preview (Preview): 176 * 144 pixels (Pixel)/1 frames (Frame);
YUV view data (Image Data): 8bit, rgb image data (Image Data): 16bit;
Camera module (Camera Module) (YUV, RGB data): ordinary circumstance (normal case);
LCD module (Module) RGB data show (Data Display): ordinary circumstance (normalcase);
Memory access time (Memory Access Time): average (average) 30ns of Page Read benchmark: ordinary circumstance (normal case);
Utilizing camera interface (Camera Interface) adopts double buffering (Dual Buffer) (access time 125ns): ordinary circumstance (normal case).
During the image preview of the mobile communication terminal of Fig. 1 (Image Preview processing) according to processing time of Fig. 2 step:
Time D 1:6.33msec to utilizing camera interface module (CAM Interface and DMA) 2C (with reference to Fig. 2) input YUV view data;
Store RAM 3R institute elapsed time (DMA to RAM) D2:2.02msec (DMA and DATA BUS) into from utilizing camera interface module 2C;
ADSP 2ad reads the time D 3:4.04msec (Data bus and aDSP) of image data from RAM;
YUV RGB picture format conversion time D4:8.25msec aDSP;
The RAM 3R access time D5:1.01msec of aDSP 2ad (aDSP and Data bus);
ARM core 2ac reads the time D 6:2.02msec (Data busand ARM core) of image data from RAM 3R;
ARM core 2ac is to the time D 7:2.53msec of LCD4 image output image (ARM core andLCD).
During the mobile communication terminal image preview of Fig. 4 according to processing time of the step of Fig. 7:
From camera module 1 to the rgb image data delivery time of utilizing camera interface module 110 N1:12.66msec (RGB data:16bit/Pixel);
Utilizing camera interface module 110 is stored in rgb image data the time N2:2.02msec of DPRAM;
ARM core 2ac with rgb image data from time N3:2.02msec (Data bus and ARM core) that DPRAM120 reads;
ARM core 2ac outputs to rgb image data the time N4:2.53msec (ARMcore and LCD) of LCD 4;
Shown in above-mentioned comparison, the time that processing procedure consumed (the time consuming summation of D1~D7) of entire image preview is 28.38msec during the image preview of the prior art of Fig. 2, and the data/address bus holding time (D2+D3+D5+D6) of this moment is 9.09msec.On the contrary, according to processing time (summation of N1~N4) of entire image preview among Fig. 7 of the present invention be 19.23msec, and the data/address bus holding time (Data bus Latency) of this moment is 4.04msec (N3+N4).That is, the present invention has compared with prior art obviously reduced the processing time and the data/address bus occupation rate of the view data of utilizing camera, thereby can reduce the load of the processor of handling the imaged image data.
Fig. 8 is the processing procedure schematic diagram of video telephone (VT) service of the present invention with mobile communication terminal of pick-up head interface device 100.Promptly, the situation of Fig. 8 is the same with Fig. 7's, can reduce processor 2 is connected to the RAM 3R of external memory storage 3 for process image data step, and then the number of times of minimizing processor 2 processing events (Event), can mate the more processing time to the image processing work of video telephone service, thereby can prevent the loss of the image from the video telephone service (VT service) of outside input.In addition, performance when making the VT service can be handled more data according to pixel/frame (Pixel/Frame) number of utilizing camera interface module (Camera Interfacemodule), thereby supports CIF (commonintermediate format) rank or its above VT image quality that current QCIF (Quarter common intermediate format) 15 frames are above.
Fig. 9 is the structural representation of DPRAM120 of the present invention, and shown in it is the functional block diagram of general DPRAM, therefore will omit its detailed description.
Above specific embodiment only is used to illustrate the present invention, but not is used to limit the present invention.