KR20130116796A - Apparatus and mehod for processing a image in camera device - Google Patents

Abstract

PURPOSE: A camera image processing device and method are provided to enable a camera device or a terminal device equipped with a camera to capture an image at the point of image-taking without a shutter delay. CONSTITUTION: A buffer (240) buffers a camera image. A multiplexing part (250) multiplexes the output of an image scaler and the butter. An image processing part processes an output image of the multiplexing part. A demultiplexing part (255) outputs a display image, where the output of the image processing part is demultiplexed, to a display part. A codec outputs a captured image by encoding the camera image outputted from the demultiplexing part. [Reference numerals] (210) Preprocessing unit; (220) Image scaler; (230) Post processing unit; (240) Buffer; (250) Multiplexing part; (255) Demultiplexing part; (260) Stop image codec; (270) Video codec; (AA) Display unit; (BB,CC) Storage unit

Description

Image processing apparatus and method of camera {APPARATUS AND MEHOD FOR PROCESSING A IMAGE IN CAMERA DEVICE}

The present invention relates to an image processing apparatus and method of a camera, and more particularly, to an image processing apparatus and method capable of compensating for a shutter delay.

A mobile terminal device including a camera device and a camera provides high quality images and also provides various user convenience functions. An image sensor (also referred to as an image sensor or a camera sensor) of a current camera device may sense images having a full high definition resolution.

In this case, the camera device displays a preview image as an image detected by the camera in the preview mode, and stores the image acquired by the camera sensor when the user presses the shutter button. At this time, when the user presses the shutter button (shutter on), there is a delay time (shutter delay or shutter lag) until the camera senses and captures the image. That is, there is a time difference from the user's shutter-on time point to the time when the camera sensor photographs the subject, and this time difference may be generated by the photographing environment of the subject and the processing time of the image processing apparatus. Therefore, the user was unable to acquire an image at a desired time point due to the shutter delay (shutter lag).

In addition, at the time of capturing the image, when an unnecessary motion such as a camera shake or a person of the subject is closed is captured, there is a limit to correcting the captured image. Since it could not be reproduced, there was a problem with the limitation of camera photography.

Accordingly, the present invention proposes an apparatus and method capable of capturing an image of a photographing time point without a shutter delay in a camera apparatus or a terminal apparatus including a camera.

To this end, a camera apparatus or a terminal apparatus including a camera according to an embodiment of the present invention may process an image obtained from a camera to generate a plurality of images in each frame, and use the generated images as a preview image and a captured image, respectively. Propose a device and method.

In addition, a camera apparatus or a terminal apparatus including a camera according to an embodiment of the present invention generates an image obtained from the camera in every frame as a preview image and buffers it as an image for capturing, displays a preview image in preview mode, and captures a captured image. The present invention proposes an apparatus and method for buffering and storing a captured image compensated for shutter delay by time-dividing the display image and the buffered captured image upon capture request.

An image processing apparatus according to an embodiment of the present invention, an image scaler for scaling a camera image into a display image, a buffer for buffering the camera image, a multiplexer for multiplexing the output of the image scaler and the buffer, An image processor which processes the output image of the multiplexer, a demultiplexer which demultiplexes the output of the image processor, and outputs a display image to the display unit, and a codec which encodes a camera image output from the demultiplexer and outputs it as a captured image The control unit is configured to control the image processing unit to sequentially process the display image and the camera image in the frame section by time-division control of the multiplexer and the demultiplexer in the capture mode.

In addition, in the method of processing a camera image according to an embodiment of the present invention, a display image processing process of controlling the image processing unit to a display image processing path in the preview mode to buffer the camera image, converting the image into a display image, and processing and displaying the image. And time divisionally controlling the image processor in the frame mode in the capture mode to sequentially set the display image processing path and the capture image processing path, and perform the display image processing process in the display image processing path. If set, the capture image processing process of selecting and processing the camera image of the set frame from the buffered camera image is performed.

In addition, the camera device according to an embodiment of the present invention, the camera is operated when driving the camera to generate a full resolution image, the display unit for displaying the display image, the input unit for generating the preview mode and capture mode signal, and stores the captured image A storage unit configured to control the display image processing path setting when the preview mode signal is detected, and to sequentially set the display image processing path and the captured image processing path by time-dividing the frame section when the capture mode signal is detected; Under the control of the control unit, a display image processing path is set in the preview mode, and the camera image is converted into a buffer and a display image, and the image is output to the display unit. In the capture mode, the display image processing path is set in the frame section. Buffer and display images to convert and process images After outputting to the display unit, the image processing unit is configured to access and process the camera of the frame set in the buffered camera image by setting the captured image path, compression-encoded into a captured image and output to the storage unit.

In addition, the portable terminal device according to an embodiment of the present invention, the camera is operated when the camera generates a full resolution image, the display unit for displaying the display image, the input unit for generating the preview mode and capture mode signal, and capture the image And a storage unit to store and process the applications of the portable terminal device, to control the setting of the display image processing path when the preview mode signal is detected by the camera, and to time-division control the frame section when the capture mode signal is detected. And an application processor configured to sequentially set a capture image processing path, a communication unit to perform a communication function of the device, and a terminal controller to communicate with the application processor, and a display image processing path in the preview mode under the control of the application processor. Buffer the camera image by setting And converting and processing the display image into the display unit, and setting the display image processing path in the frame section in the capture mode, buffering and processing the camera image, converting and processing the display image, and outputting the captured image to the display unit. And an image processor configured to access and process a camera of a frame set in a buffered camera image by setting a path, compressing and encoding the captured image into a captured image, and outputting the compressed image to the storage unit.

The camera device or the terminal device including the camera according to an embodiment of the present invention processes the image obtained from the camera sensor in every frame to generate a preview and a captured image, and captures the captured image at a desired time point to generate a zero shutter rack. In addition, there is an advantage in that the desired image can be selected and stored among the photographed images. Also, by time-division processing of the captured image and the preview image in the camera apparatus and the terminal apparatus including the camera, the configuration of processing the image can be relatively simple. There is an advantage to save power consumption.

1 is a view showing a configuration of a camera device according to an embodiment of the present invention;
2A to 2D are diagrams showing the configuration of an image processing unit according to the first embodiment of the present invention.
3A and 3B are diagrams for describing an operation of an averaging unit when the image scaler of the image processing unit is configured as an averaging unit.
4A to 4C are diagrams showing the configuration of an image processing unit according to the second embodiment of the present invention.
FIG. 5 is a diagram for describing an operation of processing an image according to an operation clock selected by an image processing unit having the configuration as illustrated in FIGS. 4A to 4C.
6A to 6C are diagrams showing the configuration of an image processing unit according to the third embodiment of the present invention.
7A to 7C are diagrams for describing an operation of an image processing unit having a configuration as illustrated in FIGS. 6A to 6C.
8A to 8C are diagrams showing the configuration of an image processing unit according to a fourth embodiment of the present invention.
9 is a view for explaining an operation of an image processing unit having the configuration as shown in FIGS. 8A to 8C.
10A and 10B show an example of a specific configuration of an image processing unit according to an embodiment of the present invention.
11 is a flowchart illustrating an image processing method of a camera apparatus or a terminal apparatus having a camera according to an embodiment of the present invention.
12A to 12D are flowcharts illustrating a procedure of processing a display image in a camera device or a portable terminal having a camera according to an embodiment of the present invention.
FIG. 13 is a flowchart illustrating a scaling process of generating a display image and a process of post-processing and displaying a display image in FIGS. 12A to 12D;
14 to 17 are flowcharts illustrating a procedure of processing a captured image in a camera device or a portable terminal including a camera according to an embodiment of the present invention.
18 is a diagram showing the configuration of a portable terminal apparatus having a camera according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the same configurations of the drawings denote the same reference numerals as possible whenever possible.

In addition, in the following description, specific details such as the number of pixels of the camera, the frame rate, etc. are shown, which are provided to help a more general understanding of the present invention, and the present invention may be practiced without these specific matters. It is self-evident to those of ordinary knowledge in Esau. In the following description of the present invention, detailed descriptions of related well-known functions or configurations will be omitted when it is determined that the detailed descriptions may unnecessarily obscure the subject matter of the present invention.

A camera device or a terminal device having a camera according to an embodiment of the present invention processes an image of every frame obtained from a camera sensor when the camera is driven (for example, a preview mode) to display a preview image or an image for storing a video. And an image (hereinafter referred to as a capture image) for storage as a still image. The display image has a smaller size than the captured image. In this case, when driving the camera, the camera device acquires and buffers an image from the camera at every frame period. In the preview mode, the camera apparatus or the terminal apparatus including the camera converts and displays an image acquired from the camera into a display image, and in the capture mode, selects an image that compensates for a shutter delay from among the buffered images and selects a captured image. To be processed. In general, the camera device has a delay time (shutter delay, shutter lag) from the shutter-on time until the camera acquires an image. Therefore, the camera apparatus according to the embodiment of the present invention includes a buffer for temporarily storing a full resolution image of the camera independently of the display image, and buffers the images photographed by the camera in the buffer when the camera is driven. When the capture request is performed, a zero shutter lag may be realized by selecting and processing a frame image having a capture request time from among the frame images being buffered. In an embodiment of the present invention, the buffer may be configured to a size capable of storing a predetermined number of frame images, and may be set to have a frame size such that the shutter can be eliminated. For example, in a camera apparatus or a terminal apparatus having a camera, it is assumed that the shutter rack is about 2 frames, and the buffer may be set to a size capable of storing a frame image of about 5 frames. In the following description, the term camera image refers to an image acquired by the camera, and the size (pixel count) of the camera image is determined by the image sensor (pixel count acquired) of the camera 110. In addition, the term display image refers to an image displayed on a display unit in a preview mode or an image for storing a video, and means an image obtained by scaling an image acquired by the camera to a predetermined size (or resolution). In the case of the display image, the resolution (size) of the preview image and the moving image image may be the same, and may have different sizes (resolution). The term capture image is an image for storing as a still image and may be an image for scaling the camera image or an image acquired by the camera to a predetermined size. In the embodiment of the present invention will be described on the assumption that the captured image is the camera image.

In addition, the term image scaling refers to adjusting the camera image to a predetermined size (or resolution). In an embodiment of the present invention, the image scaling may be implemented by a resizing or adding and averaging method. Here, the resizing means a method of adjusting the image size by decimating, interpolating, and cropping the image. The adding and averaging method is a method of adjusting the number of pixels by generating adjacent pixels as one pixel, and may perform the decimation, interpolation and cropping after performing the averaging operation.

In the following description, the preprocessor and the post processor will be used in parallel with the term image processor. 1 is a diagram showing a configuration of a camera apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the camera 110 performs a function of acquiring an image through an internal sensor when the camera is driven. The camera 110 may include an optical unit, an image sensor, a signal processor, and the like. The optical unit is driven by a mecha-shutter, a motor and an actuator, and performs operations such as zooming and focusing by the actuator. The optical unit captures an image of a surrounding area, and the image sensor senses an image captured by the optical unit and converts the sensed image into an electrical signal. The image sensor may be a CMOS or CCD sensor, and may be a high resolution image sensor. The camera's image sensor may have a built-in global shutter. The global shutter performs a function similar to the mechanical shutter built in the sensor. The camera 110 may be provided with a viewfinder. In an embodiment of the present invention, the image sensor may be a sensor capable of detecting an image of a UHD (Ultra High Definition) level or higher. The image output from the image sensor is converted into a digital image through a signal processor and output. Output data from the camera 110 may be Bayer data (raw data).

The image processor 120 buffers the frame images output from the camera 110, scales and processes the frame images output from the camera 110 to a display image size in the preview mode, and sets a frame image among the buffered frame images in the capture mode. Select and compress and output the code. In this case, the compressed coded image may be an image of a frame having zero shutter.

The controller 100 controls the overall operation of the camera apparatus according to the embodiment of the present invention. The controller 100 controls the operation of the camera apparatus according to a control command input through the input unit 150. When the camera driving command is generated through the input unit 150, the camera 110 and the image processing apparatus 120 are controlled to perform the preview mode operation. When a capture request is generated through the input unit 150, under the control of the control unit 100, the image processing unit 120 selects and compressively encodes a camera image at the time when the capture request is generated, and the compressed encoded image is stored in the storage unit 130. In particular, in the embodiment of the present invention, the control unit 100 controls the image processing unit 120 to process the display image in the preview mode, and displays and displays the display image and the captured image by time-division control of the processing of the image processing unit 120 in the capture mode. Control to save.

The storage unit 130 is a memory for storing images photographed by the camera device. In the embodiment of the present invention, the storage unit 130 will be described with an example of storing a still image. However, the storage unit 130 may store moving image images. The display unit 140 may be a display device such as an LCD or an OLED, and performs a function of displaying images captured by the camera device and information during shooting. The input unit 150 may include buttons for generating commands for setting and operating respective functions of the camera apparatus. Here, the button of the input unit 150 may be set in the form of a button on the outside of the camera device, some buttons may be configured as a touch panel. In this case, some buttons of the display unit 140 and the input unit 150 may be configured in the form of a touch screen.

In the camera device having the configuration as shown in FIG. To this end, the image processor 120 according to an embodiment of the present invention time-divids the display image and the captured image in one frame section. The image processor which processes the image acquired from the camera 110 in the image processor 120 may be configured as a preprocessor and a post processor. An image scaler may be provided to generate the display image, and a buffer may be configured to buffer a set number of frame images to perform zero shutter. In this case, the image processing unit 120 may configure the above components in various forms to process the display image and the captured image by time division.

2A to 2D are diagrams showing the configuration of the image processor 120 according to the first embodiment of the present invention. FIG. 2A illustrates an example in which an image obtained from the camera 110 is preprocessed first and then processed into a display image and buffered into a capture image, and then time-division processing of the preprocessed image and the buffered preprocessed image into a display image and a captured image, respectively. It is a figure which shows. FIG. 2B is a diagram illustrating an example of a time division processing by buffering an image acquired from the camera 110 and scaling to a display image. 2C and 2D are diagrams illustrating an example of processing an image obtained from the camera 110 into a display image and a captured image by time-dividing the buffered image.

First, referring to FIG. 2A, the preprocessor 210 performs a function of preprocessing an image output from the camera 110. The preprocessing function may include 3A (auto white balance), auto exposure (AE), auto focusing (AF) extraction and processing, lens shading correction, dead pixel correction, and knee correction. And so on. Here, the preprocessor 210 should be able to preprocess the display image or the captured image under the control of the controller 100. That is, the preprocessor 210 preprocesses an image acquired by the camera 110 (a full resolution image of the camera sensed by the camera's image sensor), and the preprocessed image is applied to the scaler 220 and simultaneously applied to the buffer 240. Is buffered.

The scaler 220 scales the full resolution image of the camera 110 output from the preprocessor 210 to a size for displaying on the display 140. The image scaling may be implemented using one or two or more of various methods such as resize, decimation, interpolation, crop, addition, and average. The image scaling may reduce the number of pixels of the full resolution image acquired by the camera 110 to scale to an appropriate size that can be displayed on the display unit 140, and also scale the image to match the aspect ratio of the display unit 140. The image scaling may be scaled at various ratios. For example, assuming that the camera 110 acquires an image of 8 Mbytes and the scaling ratio is 4: 1, the scaler 220 may scale to the display image size of 2 Mbytes.

The buffer 240 buffers the image output from the preprocessor 210, and outputs a camera image of a frame set in the capture mode under the control of the controller 100. In this case, the buffer 240 may buffer a set number of frame images. The buffer 240 may have N ring buffer structures capable of storing N frame image data. In this case, the buffer 240 sequentially buffers the images output from the camera 110 from the first buffer to the last buffer, and overwrites the frame image data in the first buffer when the frame image data is buffered up to the last buffer. In an embodiment of the present invention, N may be 5, and in this case, the ring buffer of the buffer 240 has a structure capable of storing image data of 5 frames. Here, the set image may be a frame image that compensates for the shutter rack of the camera device.

The multiplexer 250 inputs outputs of the buffer 240 and the scaler 220, and selects and outputs a display image output from the scaler 220 and / or a buffered camera image buffered from the buffer 240 under the control of the controller 100. That is, the multiplexer 250 selectively outputs the output of the scaler 220 under the control of the controller 100 in the preview mode. In the capture mode, the multiplexer 250 outputs an image of the camera 110 output from the buffer 240 in a corresponding frame section (frame section processing a capture image) under the control of the controller 100, or a display image output from the scaler 220. And time-dividing the captured image output from the buffer 240. That is, the multiplexer 100 may select and output only the captured image under the control of the controller 100 in the capture mode, and may select and output the display image and the captured image by time division.

The post processor 230 post-processes the image output from the multiplexer 250. Here, the post-processing refers to an operation of generating YUV data by color interpolation, noise reduction, color correction, and image conversion of a preprocessed image. In addition, the post-processing unit 230 post-processes the image of the display image size in the preview mode, and post-processes the camera image in the capture mode. In this case, the post-processing unit 230 may sequentially post-process the display image and the captured image that are time-divided and output from the multiplexer 230 in one frame section in the capture mode.

The demultiplexer 255 demultiplexes the output of the post processor 230 under the control of the controller 100 and outputs the demultiplexer. In this case, the demultiplexer 255 outputs the display image output from the post processor 230 in the preview mode to the display unit 140, and outputs the captured image output from the post processor 230 to the still image codec 260 in the capture mode. In the capture mode, the demultiplexer 255 may time division demultiplex the display image and the captured image processed by the post-processing unit 230 under the control of the controller 100 and output the demultiplexed image to the display unit 140 and the still image codec 260, respectively. The demultiplexer 255 may be implemented as a demultiplexer.

The still image codec 260 compresses and encodes the captured image output from the demultiplexer 255 and stores the compressed image in the storage 130. The still image codec may be a JPEG codec. In addition, the display image may be used as an image for storing a video. In this case, when a video shooting request is generated, the video codec 270 encodes the display image output from the demultiplexer 255 and stores the encoded image in the storage 130. The display image may have the same size as the display image in the preview mode. The video image may be a display image composed of pixels larger than the preview display image. In this case, the scaler 220 may be configured as a scaler in the preview mode and a scaler in the video mode, or may be configured to set different scaling ratios according to mode control. If video storage is required, the controller 100 controls the scaler 220 to be scaled to the size of the video display image. Accordingly, the scaler 220 scales the full resolution image output from the preprocessor 210 to the video display image size. . The video encoding may be performed in various forms such as H.264.

Referring to the operation of the image processor 120 having the configuration as shown in FIG. 2A, when a user generates a camera driving command through the input unit 150, the controller 100 drives the camera 110 and performs a preview mode. In the preview mode, the controller 100 controls the multiplexer 250 to apply the output of the scaler 220 to the post processor 230, and controls the demultiplexer 255 to apply the output of the post processor 230 to the display 140. Control as possible. In addition, the control unit 100 controls the post processing unit 230 to post-process an image having a display image size in the preview mode.

The image output from the camera 110 may be a Bayer image, and the full resolution image is preprocessed by the preprocessor 210. In this case, the preprocessor 210 extracts 3A (auto white balance), auto exposure (AE), auto focusing (AF) from the camera image, and performs lens shading compensation, dead pixel, knee correction, etc. of the camera image. do. The preprocessed camera image output from the preprocessor 210 is input to the buffer 240 and the scaler 220. That is, in the preview mode, the camera image photographed by the camera 110 is pre-processed and then buffered in the buffer 240 and input to the scaler 220 to be scaled into the display image. In the preview mode, the buffer 240 buffers N frames of the image output from the camera 110. Thus, the buffer 240 buffers images from the current frame to the previous N frames. Throw it away. This is to enable capturing the camera image of the frame set in the capture mode (here, the image of the zero shutter rack).

The scaler 220 scales the input full resolution image to a size for displaying on the display unit 140. The image scaling may be implemented using one or more of various methods such as resize, decimation, interpolation, crop, addition and averaging. The image scaling may reduce the number of pixels of the full resolution image acquired by the camera 110 to scale to an appropriate size that can be displayed on the display unit 140, and also scale the image to match the aspect ratio of the display unit 140. The scaler 220 may include a resizer and / or a summing and averaging part. When the scaler 220 is implemented as a resizer, the resizer resizes the image output from the camera 110 to an image size for displaying on the display unit 140. In this case, the resizing may be performed by decimation, interpolation, and / or crop. In addition, when the scaler 220 is implemented as an averaging unit, the averaging unit adds and averages adjacent pixels of the image to reduce the number of pixels of the image. 3A and 3B are diagrams for describing an operation of the averaging unit of the scaler 220.

3A to 3B, the averaging unit reduces the number of pixels of the display image by averaging adjacent pixels of the image of the camera 110. In this case, the averaging unit may achieve signal-to-noise and resolution enhancement using a pseudo-Foveon method. The pixels of the image output from the camera 110 have a structure as shown in FIG. 3A. In the configuration of FIG. 3A, four adjacent pixels may include one R pixel, one B pixel, and two G pixels. Therefore, it can be seen that by averaging four pixels to one pixel, the image can be scaled to 4: 1 pixel. In the above-described embodiment of the present invention, the summing and averaging will be described on the assumption that 4 pixels are added and averaged and reduced to 1 pixel. To this end, in the embodiment of the present invention, two G pixels are added, an average value thereof is obtained, and one G pixel is generated, and the averaged G, R, and B pixels are generated as one pixel. For example, in FIG. 3A, when the pixels G11, B12, R21, and G22 are generated as one pixel, R = R21, B = B12, and G = (G11 + G22) / 2 to one pixel having an RGB value. Create When the averaging operation is performed as described above, four pixels are generated as one pixel as shown in FIG. 3B, and the color interpolation operation is also performed. When the 16 pixels having the configuration as shown in FIG. 3A are averaged in the same manner as described above, the pixels are scaled to 4 pixels as shown in FIG. 3B, and the pixels are also subjected to color interpolation. Pixels scaled by the averaging unit are applied to the post-processing unit 230 through the multiplexer 250, and then interpolated, IPC processed, and image-converted by the post-processor 230 to generate a YUV image. In this case, when performing the image scaling operation using the averaging unit in the scaler 220, the color interpolator 231 of the post-processing 250 skips the operation of converting each pixel data to full color and performs only the color interpolation function of another function. You may.

The scaler 220 is configured using both a resizer for resizing an image of the camera 110 and converting it into a display image size, and an averaging unit for adding and averaging pixels of the image of the pre-processed camera 110 to convert the number of pixels of the display image. You may. In this case, the scaler 220 may be implemented by averaging after resizing, resizing after averaging, or selecting and outputting any one of the outputs of the resizer and the averaging unit.

The image scaled as the display image size is applied to the multiplexer 250, and the multiplexer 250 selects the display image output from the scaler 220 and outputs the image to the post processor 230. The post processor 230 may set sizes of the images to be post-processed to a size for processing the display image under the control of the controller 100 in the preview mode. In this case, the post-processing unit 230 post-processes the pre-processed display image output from the image scaler 220, wherein the post-processing may be color interpolation, noise reduction, gamma correction, and color conversion.

The post processor 230 may include a color interpolator, an image processing chain (IPC), a color converter, and the like. First, the color interpolator performs a color interpolation function to convert the Bayer data into a color image. As described above, the image sensor of the camera 110 may be a CCD or a CMOS image sensor. In this case, the CCD / CMOS image sensor uses a color filter array, which causes each pixel sensor to have only one of three color channels for producing a color image. Color interpolation performs a function of converting an image output from the camera 110 into a color in which pixels of three colors include RGB of three colors (full color conversion). The color interpolator performs color interpolation by using correlation between adjacent pixels. In general, in the image processing apparatus, image processing before color interpolation is called preprocessing, and image processing after color interpolation is called postprocessing. Secondly, IPC performs noise reduction, gamma correction, luminance correction, and the like on the color interpolated images. Third, the color conversion unit converts the post-processed image (raw data, Bayer data) into a YUV image. That is, the post processor 230 converts the preprocessed image into a post-processing such as color interpolation and a YUV image. The post-processed display image is applied to the display unit 140 through the demultiplexer 255 and displayed.

In the camera apparatus according to the embodiment of the present invention, the display image processing path is set by the multiplexer 250 and the demultiplexer 255 in the preview mode, and thus the camera 110 is controlled by the preprocessor 210, the image scaler 220, and the post processor 230. Each frame image obtained is processed as a display image for display on the display unit 140. In addition, the camera image preprocessed by the preprocessor 210 is buffered in the buffer 240. That is, the image processing unit 120 generates and displays a display image, but does not perform a processing operation for storing the image. In this case, since the display image is scaled to a small image size, and the high resolution image for capturing is only buffering, the current and heat generated by the camera device are not large.

In general, the camera device does not capture an image confirmed by the viewfinder or the display unit 140 by the shutter delay when the shutter is on, and acquires an image that is delayed by a certain frame. That is, when the shutter button is pressed, the camera device generates a time lag from the time when the user turns on the shutter until the shutter of the actual camera device operates. This time delay may vary depending on the camera device, and the control unit 100 should be aware of this by measuring the delay time experimentally. The camera device according to the embodiment of the present invention buffers the image of the camera 110 in consideration of the shutter delay time, and selects and processes the frame image of the shutter-on time point from among the buffered camera images when capturing is requested. That is, the camera apparatus according to the embodiment of the present invention implements a zero shutter by buffering the images acquired by the camera 110 in the preview mode.

When the capture request is generated through the input unit 150 while performing the preview mode as described above, the controller 100 detects the shutter-on time, and the frame image of the shutter-on time from among the images buffered in the buffer 240 (that is, zero). Image with a shutter rec) as a captured image. In addition, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to form a path for processing the captured image. That is, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a path for processing the captured image, and controls the post processor 230 to post-process an image of the camera 110. Herein, the control unit 100 may process only the captured image in one frame period by controlling the multiplexer 250 and the demultiplexer 255, and may process both the display image and the captured image by time division control in one frame period.

When processing the captured image, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to select a processing path of the captured image, and a frame image (that is, a shutter having zero shutter rec) in which the shutter delay is compensated in the buffer 240. The frame image at the time of on) is accessed. The multiplexer 250 selects an image output from the buffer 240 and outputs the image to the post processor 230, and the post processor 230 post-processes a captured image (that is, an image of the camera 110) accessed from the buffer 240. The demultiplexer 255 outputs the post-processed camera image as a captured image to the still image codec 260, and the still image codec 260 encodes the captured image and stores the captured image in the storage 130. In this case, the still image codec 260 may be a JPEG encoder as a still image encoder. In the exemplary embodiment of the present invention, the still image is described as a case of a Joint Photographic Expert Group (JPEG). However, the still image may use another still coding method (for example, a tagged-image file format (TIFF)). have. The still image compression-coded as described above is stored in the storage 130.

Although the captured image is described as using an image of the camera 110, the camera image may be scaled as necessary. That is, a user of the camera device may capture a still image having a smaller size (or lower resolution) than the full resolution image. In this case, the user of the camera device may take a picture with a smaller size (or number of pixels) of the captured image. Although an embodiment of the present invention has a configuration of capturing a camera image by having a buffer for buffering a camera image, the scaler 220 is further provided with a scaler different from the scaler 220 at the rear end of the buffer 240 and further has an output of the buffer 240. The captured image processing unit may be configured by transferring the data to the scaler. In this case, when the user generates a capture request command that specifies the size (or number of pixels) of the image through the input unit 150, the control unit 100 transmits the output of the buffer 240 to a scaler (different from the scaler 220) located at the rear end. In addition, the scaler may be controlled to scale the camera image to an image size (or resolution) desired by the user and then to post-process the image. In the embodiment of the present invention described below, it is assumed that the captured image is a camera image.

As described above, when the image processor 120 can simultaneously process the display image and the captured image in one frame section, the controller 100 controls the time division of the multiplexer 250 and the demultiplexer 255 to display the display image and the image in one frame section. Capture images can be processed sequentially. In this case, the display image processing period is set relatively short in the capture image processing period. In addition, the post-processing unit 230 may be configured to process the display image and the captured image in different image sizes under the control of the control unit 100.

In this case, when the capture request is generated, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to time-division the display image processing path and the captured image processing path in one frame section. When the display image processing path is formed, the image is processed by displaying the scaled display image, and when the captured image path is formed, an image of a frame corresponding to the shutter-on time is accessed from the buffer 240 to implement zero shutter. Post-process, encode, and save the image.

As described above, the camera apparatus according to the embodiment of the present invention may implement zero shutter. The image obtained by the camera 110 when the shutter is turned on in the camera device is a frame image obtained after the captured image desired by the user by a shutter lag, time lag, and shutter delay. That is, the image to be captured becomes a previous frame of the frame image acquired by the camera 110. Therefore, in order to eliminate the shutter delay, the embodiment of the present invention selects the image of the previous frame buffered in the buffer 240 and processes the captured image in the capture request.

In addition, when a capture request is generated by the user, the picture to be captured may be a picture that is not normally taken. For example, when a portrait is taken, a picture in which the person closes his eyes or shakes may be taken. In this case, the user preferably captures an image of another frame. Since the camera device according to the embodiment of the present invention buffers the number of frame images set in the buffer 240, another frame image may be selected from the buffered images if necessary. That is, since the camera apparatus according to the embodiment of the present invention buffers a plurality of frame images, there is an advantage that it may be replaced with another frame image when the captured image is not satisfied.

When the user requests continuous shooting (burst shot mode), the controller 100 continuously accesses the buffered frame images at the time when the continuous shooting is requested, and the post-processing unit 230 displays the continuously accessed frame images. After processing, the still image codec 260 may continuously encode the same and store the same in the storage 130. That is, in the continuous shooting mode, the images photographed by the camera 110 are buffered in the buffer 240 every frame period, and the controller 100 may continuously access the buffered full resolution images to process and encode and store the images.

In the above state, when still image reproduction is requested through the input unit 150, the controller 100 accesses an image stored in the storage unit 130, and the still image codec 260 decodes the accessed image and displays the same on the display unit 140. In addition, when a video is played through the input unit 150, the control unit 100 accesses the video requested by the storage unit 130 and applies it to the video codec 270, and the video codec 270 decodes the video and displays it on the display unit 140.

2B is a diagram illustrating another configuration of the image processing unit 120. The image processor 120 shown in FIG. 2B buffers and scales an image output from the camera 110, and has a configuration of pre-processing and post-processing the buffered image or the scaled display image. Referring to FIG. 2B, the scaler 220 scales the full resolution image output from the camera 110 to a size for displaying on the display unit 140. The buffer 240 buffers an image output from the camera 110 and outputs a camera image of a frame set in a capture mode under the control of the controller 100.

The multiplexer 250 inputs outputs of the buffer 240 and the scaler 220, and selects and outputs a display image output from the scaler 220 and / or a buffered camera image buffered from the buffer 240 under the control of the controller 100. That is, the multiplexer 250 selectively outputs the output of the scaler 220 under the control of the controller 100 in the preview mode. In the capture mode, the multiplexer 250 outputs an image of the camera 110 output from the buffer 240 in a corresponding frame section (frame section processing a capture image) under the control of the controller 100, or a display image output from the scaler 220. And time-dividing the captured image output from the buffer 240.

The preprocessor 210 preprocesses the image output from the multiplexer 250, and the postprocessor 230 postprocesses the image output from the multiplexer 250. The preprocessor 210 and the postprocessor 230 control to process an image having a display image size or a captured image size under the control of the controller 100. That is, the preprocessor 210 and the post processor 230 process the display image in the preview mode, and process the capture image or the time-divided display image and the captured image in the capture mode.

The demultiplexer 255 demultiplexes the output of the post processor 230 under the control of the controller 100 and outputs the demultiplexer. In this case, the demultiplexer 255 outputs the display image output from the post processor 230 in the preview mode to the display unit 140, and outputs the captured image output from the post processor 230 to the still image codec 260 in the capture mode. In the capture mode, the demultiplexer 255 may time division demultiplex the display image and the captured image processed by the post-processing unit 230 under the control of the controller 100 and output the demultiplexed image to the display unit 140 and the still image codec 260, respectively. The demultiplexer 255 may be implemented as a demultiplexer.

The still image codec 260 compresses and encodes the captured image output from the demultiplexer 255 and stores the compressed image in the storage 130. In addition, the video codec 270 encodes the display image output from the demultiplexer 255 and stores the encoded image in the storage 130.

Referring to the operation of the image processor 120 having the configuration as shown in FIG. 2B, when the user generates a camera driving command through the input unit 150, the controller 100 drives the camera 110, and the images output from the camera 110 are each image scaler 220. And buffer 240. The buffer 240 then buffers the image output from the camera 110.

The image scaler 220 scales the input full resolution image to a size for displaying on the display unit 140. In the preview mode, the multiplexer 250 selects the output of the image scaler 220 under the control of the controller 100, and the demultiplexer 255 selects the output of the post processor 230 and outputs the output to the display 140. That is, in the preview mode, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a path of the preview image, and does not form a path of the captured image. In this state, the display image output from the image scaler 220 is applied to the preprocessor 210 through the multiplexer 250, and the preprocessor 210 preprocesses the display image, and the postprocessor 230 postprocesses the preprocessed display image. . That is, the controller 100 sets the display image processing path by controlling the multiplexer 250 and the demultiplexer 255 in the preview mode. As a result, the image scaler 220, the preprocessor 210, and the post processor 230 are driven to obtain the camera image. The frame image is processed into an image for display on the display unit 140, and the buffer 240 buffers camera images. When a capture request is generated through the input unit 150, the controller 100 accesses a corresponding frame image as a capture image in consideration of a shutter delay time among the buffered images of the buffer 240. In addition, the control unit 100 controls the time division of the multiplexer 250 and the demultiplexer 255 to form a path for processing the display image and the captured image. That is, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a path for processing a display image (or a captured image) first, and then set a path for processing a captured image (or a display image). do.

First, the controller 100 selects a processing path of the display image by controlling the multiplexer 250 and the demultiplexer 255. Then, the display image output from the image scaler 220 is pre-processed and post-processed by the pre-processing unit 210 and the post-processing unit 230 to be converted into a YUV image, and the display image converted into the YUV image is applied to the display unit 140 and displayed. Thereafter, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to select a processing path of the captured image, and the frame image at which the shutter delay is compensated for in the buffer 240 (that is, the frame image at the shutter-on time having zero shutter rack). )). The captured image accessed from the buffer 240 (that is, the image of the camera 110) is also pre-processed and post-processed by the pre-processor 210 and the post-processor 230, respectively, and converted into a YUV image, and the still image codec 260 through the demultiplexer 255. Is applied to. The still image codec 260 encodes the captured image and stores the captured image in the storage 130.

In this case, when the capture request is made, the control unit 100 must time-control the multiplexer 250 and the demultiplexer 255 to set an image processing path so that the display image and the captured image can be processed in one frame section. In this case, the display image processing period is set relatively short in the capture image processing period. The preprocessor 210 and the post processor 230 may be configured to process the display image and the captured image in different image sizes under the control of the controller 100.

In the image processor 120 having the configuration as shown in FIG. 2B, the image scaler 220 and the buffer 240 input an image of the camera 110, respectively, and the preprocessor 210 is connected between the multiplexer 250 and the post processor 230. In addition, operations of the respective components (preprocessor, post processor, image scaler, buffer, multiplexer, and demultiplexer) of the image processor 120 having the configuration as shown in FIG. 2B perform the same functions as those described with reference to FIG. 2A. In this case, the preprocessor 210 of FIG. 2A preprocesses only a camera image, whereas the preprocessor 210 of FIG. 2B may process both the display image size and the captured image size.

2C is a diagram illustrating still another configuration of the image processing unit 120. The image processor 120 illustrated in FIG. 2C buffers an image output from the camera 110, scales the buffered image, and the multiplexer is configured to multiplex the scaled image or the buffered image to be pre-processed and post-processed.

Referring to FIG. 2C, the buffer 240 buffers an image output from the camera 110 at every frame period, and outputs an image set in a capture mode under the control of the controller 100. In this case, the buffer 240 may buffer a set number of frame images. The image scaler 220 scales the full resolution image of the camera 110 buffered in the buffer 240 to a size for displaying on the display 140.

The multiplexer 250 inputs the outputs of the buffer 240 and the image scaler 220, and selects and outputs a display image output from the image scaler 220 and / or a buffered camera image buffered from the buffer 240 under the control of the controller 100. . The preprocessor 210 performs a function of preprocessing the image output from the multiplexer 250. The post processor 230 post-processes the image output from the preprocessor 210. The demultiplexer 255 demultiplexes the output of the post processor 230 under the control of the controller 100 and outputs the demultiplexer. In this case, the demultiplexer 255 outputs the display image output from the post processor 230 in the preview mode to the display unit 140, and outputs the captured image output from the post processor 230 to the still image codec 260 in the capture mode. The still image codec 260 compresses and encodes the captured image output from the demultiplexer 255 and stores the compressed image in the storage 130.

The image processor 120 having the configuration as shown in FIG. 2C buffers the image output from the camera 110 in the buffer 240, and the image scaler 220 scales the image of the camera 110 buffered in the buffer 240. The subsequent configuration has the same configuration as that of the image processor of FIG. 2B, and the operation may be performed in the same manner. In this case, the image scaler 220 may input an image of the current frame (that is, the most recent frame image photographed by the camera 110) among the images captured by the camera 110 and buffered in the buffer 240 to be scaled into the display image.

2D is a diagram illustrating still another configuration of the image processing unit 120. The image processor 120 illustrated in FIG. 2D buffers an image output from the camera 110, preprocesses and scales the buffered image, and the multiplexer multiplexes the post-processed scaled image or the buffered image.

Referring to FIG. 2D, the buffer 240 buffers an image output from the camera 110. The preprocessor 210 preprocesses the camera image output from the buffer 240. The image scaler 220 scales the camera image output from the preprocessor 220 to the display image. The multiplexer 250 selects and outputs the output of the image scaler 220 in the preview mode under the control of the controller 100, and selectively outputs the camera image preprocessed by the preprocessor 210 in the capture mode. The post processor 230 inputs the output of the multiplexer 250, and post-processes the pre-processed display image in the preview mode under the control of the controller 100, and post-processes the pre-processed image in the capture mode. The demultiplexer 255 outputs the display image output from the post-processing unit 230 to the display unit 140 in the preview mode under the control of the control unit 100, and outputs the image output from the post-processing unit 230 to the still image codec 260 in the capture mode. The configuration shown in FIG. 2D is connected to the preprocessor 210 at the output terminal of the buffer 240 in the configuration of FIG. 2C, the image scaler 220 scales the preprocessed image to the display image size, and the multiplexer 250 controls the controller 100. The configuration other than the configuration of multiplexing the pre-processed display image and / or the captured image under the same configuration is the same as that of the image processing unit as shown in FIG. 2C, and the operation thereof is also the same.

4A to 4C are diagrams illustrating still another configuration of the image processor 120 according to the second embodiment of the present invention. 4A to 4C illustrate a configuration in which an image processing speed is changed by changing an operation clock when an image processing unit 120 processes an image. FIG. 5 is a diagram for describing an operation of processing an image according to an operation clock selected by the image processor having the configuration as illustrated in FIGS. 4A to 4C.

4A and 5, the preprocessor 210 preprocesses an image obtained from the camera 110. The preprocessed image is transferred to the image scaler 220 and scaled by the size and / or number of pixels of the display image. In this case, the image scaler 220 may include a resizer and / or an averager as described above. The camera image output from the preprocessor 210 is applied to the buffer 240 and buffered.

The frame rate of the image processed by the image processing apparatus 200 may have a frame rate of 30 fps or more. Herein, if the frame rate of the image processing apparatus 200 is 30 fps, the display image is generated as an image of 30 frames per second. In this case, the display image is scaled as described above, but the captured image is processed as a full resolution image of the camera 110. In this case, it is possible to use a method of processing an image at a low speed in the preview mode, and at a high speed to process a display image and a captured image in the capture mode. In addition, both the preview mode for processing the display image and the capture mode for time-dividing the display image and the captured image can be processed at high speed.

The clock generator 400 includes a high speed clock generator 410 for generating a high speed clock, a low speed clock generator 420 for generating a low speed clock lower than the high speed clock, and the high speed clock or a low speed clock under the control of the controller 100 to capture the capturing signal. And a selector 430 for supplying the operation clock during image processing. The buffer 240 buffers an image of the camera 110. First, as shown in 511 of FIG. 5, the camera 110 generates and outputs an image of one frame in units of the vertical synchronization signal Vsync. That is, the vertical synchronization signal Vsync is one frame period. The operation clock for processing the display image in the preview mode may be set to a low speed clock or a high speed clock. First, when the high speed clock is selected, the selector 430 selects the output of the high speed clock generator 410, and the selected high speed clock is supplied to the operation clocks of the buffer 240 and the post processor 230. At this time, in the preview mode, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a path for processing the display image. In this case, the access operation of the buffer 240 is not performed. Then, the multiplexer 250 selects the output of the image scaler 220, and the demultiplexer 255 selects the output of the post processor 230 and outputs the output to the display 140, thereby setting the display image processing path. The image output from the camera 110 at the same period as 511 of FIG. 5 is preprocessed by the preprocessor 210, and then scaled to the size of the display image by the image scaler 220, and the display image is transmitted to the postprocessor 230 through the multiplexer 250. . Then, the post-processing unit 230 processes the display image by the fast clock as shown in 513 of FIG. 5 and outputs the demultiplexer 255. The demultiplexer 255 transmits the post-processed display image to the display unit 140. To be displayed.

When the capture request is generated in the above state, the controller 100 first controls the multiplexer 250 and the demultiplexer 255 at the beginning of the frame period (521 frame section in FIG. 5) to display the display image as shown by reference numeral 551. A path for processing is set, and the clock generator 400 is controlled to supply a high speed clock to the buffer 240 and the post processor 230 as an operation clock. Then, as described above, the multiplexer 250 and the demultiplexer 255 each select a display image and select a display image processing path for outputting the processed display image to the display 140. Then, the preprocessed image in the preprocessor 210 is scaled to the size of the display image in the image scaler 200, and the display image is post-processed in the postprocessor 230 and displayed on the display 140.

After processing the display image, the controller 100 controls the multiplexer 250 and the demultiplexer 255 in the remaining frame period (521 frame section of FIG. 5) to form a capture image processing path as shown by reference numeral 553, and a clock generator. The high speed clock is selected at 400 and supplied to the operation clocks of the buffer 240 and the post processor 230. In addition, the controller 100 accesses an image set in the image of the camera 110 buffered in the buffer 240. At this time, the image accessed from the buffer 240 may be an image (that is, zero shutterlag image) in consideration of the shutter delay. The image accessed from the buffer 240 is applied to the post-processing unit 230 through the multiplexer 250 and post-processed in the section 553 of FIG. 5, and the post-processed image is applied to the still image codec 260 from the demultiplexer 255. The still image codec 260 encodes the post-processed image, and the storage unit 130 stores the on-screen image output from the still image codec 260.

Therefore, if image capture is required, the image processor 120 first forms a display image processing path in one frame section to process the display image, and then forms a capture image processing path in the other frame section to process the captured image.

 Secondly, when the low speed clock is selected, the selector 430 selects the output of the low speed clock generator 420, and the selected low speed clock is supplied to the operation clock of the post processor 230. At this time, in the preview mode for processing the display image, the controller 100 sets the multiplexer 250 and the demultiplexer 255 as the display image processing path, and the access operation of the buffer 240 is not performed. Then, the frame image generated by the camera 110 is converted into a preprocessing and display image through the preprocessor 210 and the image scaler 220 at the same period as 511 of FIG. 5, and the display image is transmitted to the postprocessor 230 through the multiplexer 250. . The post processor 230 is processed by the low speed clock as shown in 515 of FIG. 5 and applied to the demultiplexer 255. The demultiplexer 255 outputs the post-processed display image to the display 140. That is, when the low clock is selected, the image processor 120 forms a display image processing path and processes the display image of the corresponding frame at a low speed as shown in 515 of FIG. 5. In this case, power consumption may be reduced when the display image is processed.

When an image capture request is generated, the controller 100 controls the selector 430 to select an output of the fast clock generator 410, and the selected fast clock is supplied to the operation clocks of the buffer 240 and the post processor 230. That is, in the frame period of the section 521 of FIG. 5, the controller 100 controls the selector 430 to supply the fast clock to the image processor 120. The controller 100 first controls the multiplexer 250 and the demultiplexer 255 in the frame period in the start section of the frame 521 to set a path for processing the display image as shown in 551, and the display image as described above. Is processed at high speed as shown in 551 of FIG. Thereafter, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to form a path for processing the captured image in the remaining period of the corresponding frame (that is, the frame section 521 to 553 in FIG. 5), and the buffer The 240 accesses and outputs a frame image of the zero shutter rack considering the shutter delay. Then, the image accessed from the buffer 240 is applied to the post-processing unit 230 through the multiplexing unit 250 and post-processed as 553 in the section 521 of FIG. 5, and the post-processed image is applied to the still image codec 260 from the demultiplexing unit 255. do. The still image codec 260 encodes the post-processed image and stores the captured image in the storage 130 as a captured image. Therefore, when the capture request is made, the operation clock of the image processing apparatus 200 is converted from a low speed clock to a high speed clock, and a display image processing path is first formed in a frame section for processing the captured image to process the display image at high speed, and the remaining frames. The capture image processing path is formed in the section to process the captured image at high speed.

4B and 5, an image of the camera 110 is input to the image scaler 220 and the buffer 240, respectively. Then, the buffer 240 buffers the image of the camera 110, and the image scaler 220 scales the image of the camera 110 to the display image size. The image scaler 220 may include a resizer and / or an averaging unit as described above. In addition, the clock generator 400 generates a low speed clock or a high speed clock to supply the operation clocks of the buffer 240, the preprocessor 210, and the post processor 230.

In this case, in the preview mode for processing the display image, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set the display image processing path. When the display image processing path is set, the multiplexer 250 selects the display image generated by the image scaler 220 and outputs it to the preprocessor 210. The demultiplexer 255 selects the output of the postprocessor 230 and displays the output on the display 140. Output In this case, when the high-speed clock is supplied to the operation clocks of the preprocessor 210 and the postprocessor 230, the preprocessor 210 and the postprocessor 230 are displayed on the image scaler 220 in some sections of the corresponding frame section as shown in 513 of FIG. And pre- and post-processing, the processed display image is displayed on the display unit 140. At this time, when the low-speed clock is supplied to the operation clocks of the preprocessor 210 and the postprocessor 230, the preprocessor 210 and the postprocessor 230 preprocess and display the display image generated by the image scaler 220 in the frame section as shown in 515 of FIG. After processing, the processed display image is displayed on the display unit 140.

In addition, when an image capture request is generated in the 521 frame section of FIG. 5, the controller 100 controls the clock generator 400 to supply a fast clock to the operation clocks of the buffer 240, the preprocessor 210, and the postprocessor 230. The multiplexer 250 and the demultiplexer 255 are controlled at the beginning of the frame section 521 to set the display image processing path. Then, the multiplexer 250 selects the display image of the current frame processed by the image scaler 200 and outputs it to the preprocessor 210. The preprocessor 210 and the post processor 230 process the display image in the section 551 of FIG. The display image is displayed on the display unit 140 through the demultiplexer 255. Thereafter, the control unit 100 sets a capture image processing path in the remaining section 553 of the frame 521 and accesses and outputs the frame image (for example, an image having zero shutter rack) set in the buffer 240. Then, the multiplexer 250 selects the output of the buffer 240 and outputs it to the preprocessor 210, and the captured image processed by the preprocessor 210 and the postprocessor 230 is output to the still image codec 260 through the demultiplexer 255. The still image codec compresses and compresses the captured image and stores the captured image in the storage 130. As described above, when processing the captured image, the image processor 120 time-divisions the display image of the current frame and the set frame capture image (eg, a camera image having zero shutter rack) in the frame section. Therefore, when the image capture request is made, the image processing unit 120 generates a display image of the current frame in the camera 110 and displays it through the display unit 140, and has an image having zero shutter rec set among the camera images buffered in the buffer 240 or an image selected by the user. The compression code is stored in the storage unit 130.

Referring to FIG. 4C, the clock generator 400 generates the high speed clock 410 and the low speed clock 420, and controls the high speed clock or the low speed clock as the operation clocks of the buffer 240, the preprocessor 210, and the post processor 230 under the control of the controller 100. Supply. The buffer 240 buffers the image output from the camera 110. The image scaler 220 scales the camera image of the current frame output from the buffer 240 to the display image. The multiplexer 250 selects and outputs the output of the image scaler 220 in the preview mode to process the display image under the control of the controller 100, and outputs the image scaler 220 in one frame section in the capture mode to capture the image of the camera 110. The display image and the captured image output from the buffer 240 are time-divisionally selected and sequentially output. The preprocessor 210 preprocesses the display image and the captured image which are time-divided and output by the multiplexer 250 under the control of the controller 100. The post processor 230 post processes the preprocessed display image and the captured image under the control of the controller 100. The demultiplexer 255 outputs the display image output from the post-processing unit 230 to the display unit 140 in the preview mode under the control of the control unit 100, and time-divisionally selects the display image and the captured image output from the post-processing unit 230 in the capture mode. The display unit 140 and the still image codec 260 are sequentially output.

The image processor 120 may process the display image at a low speed in the preview mode, and may process the display image and the captured image at high speed in the capture mode. In addition, the image processor 120 may process the camera image at high speed in the preview mode and the capture mode.

In the preview mode for processing a display image, the buffer 240 buffers the camera image under the control of the controller 100, and the image scaler 220 scales and outputs the camera image of the current frame in the buffer 240 as the display image. In addition, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a display image processing path. When the display image processing path is set, the multiplexer 250 selects the display image generated by the image scaler 220 and outputs it to the preprocessor 210. The demultiplexer 255 selects the output of the postprocessor 230 and displays the output on the display 140. Output In this case, when the high-speed clock is supplied to the operation clocks of the preprocessor 210 and the postprocessor 230, the preprocessor 210 and the postprocessor 230 are displayed on the image scaler 220 in some sections of the corresponding frame section as shown in 513 of FIG. And pre- and post-processing, the processed display image is displayed on the display unit 140. In addition, when the low-speed clock is supplied to the operation clocks of the preprocessor 210 and the postprocessor 230, the preprocessor 210 and the postprocessor 230 preprocess and display the display image generated by the image scaler 220 in the frame section as shown in 515 of FIG. After processing, the processed display image is displayed on the display unit 140.

In addition, when an image capture request is generated in the 521 frame section of FIG. 5, the controller 100 controls the clock generator 400 to supply a fast clock to the operation clocks of the buffer 240, the preprocessor 210, and the postprocessor 230. The multiplexer 250 and the demultiplexer 255 are controlled at the beginning of the frame section 521 to set the display image processing path. Then, the multiplexer 250 selects the display image of the current frame processed by the image scaler 240 and outputs it to the preprocessor 210. The preprocessor 210 and the post processor 230 process the display image in the section 551 of FIG. The display image is displayed on the display unit 140 through the demultiplexer 255. Thereafter, the control unit 100 sets a capture image processing path in the remaining section 553 of the frame 521, and accesses and outputs a camera image (eg, an image having zero shutter rack) of the frame set in the buffer 240. Then, the multiplexer 250 selects the output of the buffer 240 and outputs it to the preprocessor 210, and the captured image processed by the preprocessor 210 and the postprocessor 230 is output to the still image codec 260 through the demultiplexer 255. The still image codec compresses and compresses the captured image and stores the captured image in the storage 130.

In addition, although not shown, in the image processor 120, the output of the buffer 240 is connected to the preprocessor 210, the outputs of the preprocessor 240 and the buffer 240 are connected to the multiplexer 250, and the output of the multiplexer 240 is an image scaler. It is connected to the 220, the output of the image scaler 240 may have a configuration that is connected to the post-processing unit 230.

6A to 6C are diagrams illustrating still another configuration of the image processor 120 according to the embodiment of the present invention. 6A to 6C illustrate embodiments in which a frame image is divided and processed when the captured image is generated by the image processor 120, and then the divided images are combined to generate a captured image. 7A to 7C are diagrams for describing an image processing operation of FIGS. 6A to 6C.

6A and 7A to 7C, the preprocessor 210 preprocesses an image obtained from the camera 110. The preprocessed image is transferred to the image scaler 220 and scaled by the size and / or number of pixels of the display image. In this case, the image scaler 220 may include a resizer and / or an averager. In the preview mode for processing the display image, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a display image path through which the output of the image scaler 220 is transmitted to the display unit 140. Then, the multiplexer 250 transfers the display image output from the image scaler 220 to the post processor 230. The post processor 230 performs color interpolation, IPC processing, and image conversion on the image scaled by the display image size 220 (or the number of pixels). The demultiplexer 255 transmits the post-processed display image to the display unit 140, and the display unit 140 displays the display image. In the preview mode, the camera image output from the preprocessor 210 is applied to the buffer 240 and buffered. At this time, in the preview state, the controller 100 can save the operation power in the preview mode without performing an image processing operation buffered in the buffer 240.

When the capture request is generated in the above state, the control unit 100 performs time division control on the multiplexer 250 and the demultiplexer 255 to process the display image and the captured image in the captured image frame section. Then, the multiplexer 250 and the demultiplexer 255 first select the display image by time division control of the controller 100 and then select the captured image in the remaining sections of the frame. The captured image processing section may be set longer than the display image processing section. In the display image processing section, the display image passing through the image scaler 220, the multiplexer 250, the post processor 230, and the demultiplexer 255 is displayed on the display unit 140, and in the capture image processing section, the buffer 240, the multiplexer 250, The captured image via the processor 230 and the demultiplexer 255 is applied to the adder 600.

Referring to the image processing operation of the captured image frame, the controller 100 performs time division control on the image processor 120 to process the display image and the captured image. First, the control unit 100 forms a display image processing path, whereby the camera image preprocessed by the preprocessor 210 is buffered in the buffer 240 and scaled to the display image in the image scaler 220. The display image is applied to the post processor 230 by the multiplexer 250, and the display image post-processed by the post processor 230 is applied to the display unit 140 through the demultiplexer 255 and displayed. The display image processing procedure as described above is performed in the same manner as the procedure in the preview mode. Thereafter, the controller 100 processes the captured image in the remaining section of the capture frame.

In the capture image processing section, the controller 100 accesses a camera image of a frame set in the buffer 240 as a capture image. Here, the captured image may be a camera image that compensates for the shutter delay among frame images buffered in the buffer 240 in consideration of the shutter. In this case, the selected camera image may have a configuration as shown in FIG. 7A. FIG. 7A illustrates an example of a frame image having an m * y size as an example of a camera image, and assumes m = 2n and y = 2x. When the captured image is processed, the controller 100 divides and reads the captured image stored in the buffer 240 into a set block size, and the divided image is applied to the post processor 230 through the multiplexer 250. . The post-processing unit 230 post-processes the split image, and the demultiplexer 255 outputs the post-processed split image to the adder 600. Then, the adder 600 adds the tile images and reproduces the image as a frame image. The reason for splitting and post-processing the camera image is to reduce the number of gates of the post-processor 250 to reduce hardware configuration and power consumption. In addition, the reason for processing the image by dividing the camera image as described above may be useful even when the display image and the camera image cannot be simultaneously processed in one frame section because the size of the camera image is large (that is, a high-pixel image). .

First, an operation of dividing the camera image as shown in FIG. 7A by dividing the frame image into 2 and post-processing will be described. In the case of dividing a camera image having the same size as that of FIG. 7A, the two divided images each have an size of n * y, wherein the divided images are (1)-(n) and (n + 1)-(m ) Can be split into lines. However, since the interpolation and IPC processing functions are performed using adjacent pixels when post-processing the frame image, when the frame image is divided in the buffer 240 as shown in (710) and (720) of FIG. + a) and (n-a + 1)-(m), and the value of a varies depending on the post-processing type. Here, a is the number of overlapping lines, and in the embodiment of the present invention, it is assumed that a = 1. The controller 100 reads the first divided image as shown in 710 of FIG. 7B and applies it to the post-processing unit 230 through the multiplexing unit 250. The post-processing unit 230 post-processes the divided image as described above. The frame image is applied to the adder 600 through the demultiplexer 255. Thereafter, the controller 100 reads the second divided image as shown in 720 of FIG. 7B and transmits the second divided image to the post processor 250 through the multiplexer 250. Is transmitted to the adder 600 through the demultiplexer 255. When transmitting the camera image as shown in FIG. 7A into two divided images as shown in 710 and 720 of FIG. 7B, the divided images may be transmitted in one frame period, and may be divided and transmitted as many as the number of frames of the divided image. . In the latter case, the two divided images are processed in two frame sections. Therefore, the adder 600 adds two divided images received in two frame sections to generate one frame image.

In this case, the divided image transmitted through the demultiplexer 255 further includes adjacent pixels in order to use information of the adjacent pixels in post-processing (in the case of 710 of FIG. 7B, pixels of n + 1 line, In the case of 720, the pixels of the n line), and the pixels of the line must be removed and added. Therefore, the adder 600 removes the pixels of n + 1 lines when receiving the post-processed divided images as shown in 710 of FIG. 7B and n when receiving the post-processed divided images as shown in 720 of FIG. 7B. After removing the pixels of the line, these divided images are added to recreate the frame image. Thereafter, the frame image output from the adder 600 is applied to the still image codec 260 and compressed and encoded into a still image, and the image output from the still image codec 260 is stored as a capture image in the storage 130.

Secondly, an operation of post-processing by dividing the frame image into four will be described. In the case of dividing the frame image having the same size as that of FIG. 7A, the quadrant divided images have n * x sizes, respectively. x); {(n + 1)-(m), (1)-(x); (1)-(n), (x + 1)-(y); can be {(n + 1)-(m), (x + 1)-(y)}

 However, since the interpolation and IPC processing functions are performed using adjacent pixels when post-processing the frame image, when the frame image is divided in the buffer 240 as shown in 750-780 of FIG. 7C, {(1)-( n + 1), (1)-(x + 1); {(n)-(m), (1)-(x + 1); (1)-(n + 1), (x)-(y); Split into {(n)-(m), (x)-(y)}. Thereafter, the controller 100 reads the divided image as shown in 750 of FIG. 7C and applies the divided image to the post processor 250 through the multiplexer 250, and the post processor 250 post-processes the divided image. The divided frame image is applied to the adder 600 through the demultiplexer 255. In the same manner as described above, the controller 100 sequentially reads the divided images, such as 760-780 of FIG. 7C, and delivers the divided images to the post-processing unit 250. Deliver on part 600. When the 1 frame image as shown in FIG. 7A is processed into a 4 segmented image as shown in FIG. 7C, the divided images are processed in 4 frame sections. Therefore, the adder 600 adds four divided frame images received in four frame periods to generate one frame image.

In this case, the transmitted divided image further includes adjacent pixels in order to use information of the adjacent pixels in post-processing (n + 1, x + 1 in the case of 750 of FIG. 7B, pixels in the line, and in the case of 760). Pixels of n, x + 1 lines, n of 770, pixels of x + 1 lines, n of x780 pixels, and pixels of x lines) must be removed and added. Therefore, when the adder 600 receives post-processed divided images such as 750-780 of FIG. 7C, the adder 600 removes pixels of adjacent pixel lines of the divided image area, and then adds the divided images to regenerate the frame image. . Thereafter, the frame image output from the adder 600 is applied to the still image codec 260 and compressed and encoded into a still image, and the image output from the still image codec 260 is stored as a capture image in the storage 130.

As described above, when the frame image is divided into two processes as shown in FIG. 7B, the post-processing unit 230 may process the camera images having 1/2 size. The post processor 230 may process camera images having a size of 1/4. Therefore, since the image of the camera 110 is divided into a predetermined size and processed, the number of logic gates of the post-processing unit 230 for processing the captured image can be greatly reduced, thereby reducing power consumption.

FIG. 6B illustrates an example in which a camera image is divided into pre-processing and post-processing when the image processing unit 120 generates the captured image, and then the captured image is combined to generate a captured image.

Referring to the configuration and operation of the image processor 120 having the configuration as shown in FIG. 6B, the images of the camera 110 are input to the image scaler 220 and the buffer 240, respectively. Then, the buffer 240 buffers the image of the camera 110, and the image scaler 220 scales the image of the camera 110 to the display image size. The image scaler 240 may include a resizer and / or an averaging unit as described above.

In this case, in the preview mode for processing the display image, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set the display image processing path. When the display image processing path is set, the multiplexer 250 selects the display image generated by the image scaler 220 and outputs it to the preprocessor 210. The demultiplexer 255 selects the output of the postprocessor 230 and displays the output on the display 140. Output The preprocessor 210 and the post processor 230 preprocess and postprocess the display image, and the processed display image is displayed on the display 140.

When the capture request is generated in the above state, the control unit 100 performs time division control on the multiplexer 250 and the demultiplexer 255 to process the display image and the captured image in the captured image frame section. Then, the multiplexer 250 and the demultiplexer 255 first select the display image by time division control of the controller 100 and then select the captured image in the remaining sections of the frame. The captured image processing section may be set longer than the display image processing section. In the display image processing section, the display image output from the image scaler 220 is output to the display unit 140 through a multiplexer 250, a preprocessor 210, a post processor 230, and a demultiplexer 255. In the capture image processing section, the divided image that is divided and accessed in the buffer 240 is applied to the adder 600 through the multiplexer 250, the preprocessor 210, the post processor 230, and the demultiplexer 255. That is, as described in FIG. 6A, when capturing a captured image, a camera image such as 7a is divided and accessed as shown in FIG. 7B or 7C in the buffer 240. Pre-treatment and post-treatment are applied to the adder 600, respectively. Then, the adder 600 adds the divided images to generate a frame image, and the still image codec 260 compresses and encodes the one-frame camera image added by the adder 600, and stores the encoded camera image. It is stored in the unit 130. In this case, the operation of dividing and adding a camera image when processing the captured image by the image processor 120 may be performed by the method described with reference to FIG. 6A.

Referring to FIG. 6C, the buffer 240 buffers an image output from the camera 110. In addition, the control unit 100 divides and outputs a camera image of a frame selected from the buffer 240 in a set size in a capture mode. The image scaler 220 scales the camera image of the current frame output from the buffer 240 to the display image. The multiplexer 250 selects and outputs the output of the image scaler 220 in the preview mode to process the display image under the control of the controller 100, and outputs the image scaler 220 in one frame section in the capture mode to capture the image of the camera 110. Time division and selection of the display image and the divided image of the camera image output from the buffer 240 are sequentially performed. The preprocessor 210 preprocesses the display image and the split image which are time-divided and output by the multiplexer 250 under the control of the controller 100. The post-processing unit 230 post-processes the pre-processed display image and the divided image under the control of the controller 100. The demultiplexer 255 outputs the display image output from the post-processing unit 230 to the display unit 140 in the preview mode under the control of the control unit 100, and time-divisionally selects the display image and the split image output from the post-processing unit 230 in the capture mode. Output is sequentially performed to the display unit 140 and the adder 600. The adder 600 adds the split image to generate a camera image of one frame, and the still image codec 260 compresses and encodes the camera image output from the adder 600 in the storage 130.

In the preview mode for processing a display image, the buffer 240 buffers the camera image under the control of the controller 100, and the image scaler 220 scales and outputs the camera image of the current frame in the buffer 240 as the display image. In addition, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a display image processing path. When the display image processing path is set, the multiplexer 250 selects the display image generated by the image scaler 220 and outputs it to the preprocessor 210. The demultiplexer 255 selects the output of the postprocessor 230 and displays the output on the display 140. Output

When a camera image capture request is generated, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a display image processing path. Then, the multiplexer 250 selects a display image of the current frame processed by the image scaler 240 and outputs it to the preprocessor 210. The preprocessor 210 and the post processor 230 process the display image, and the display image is demultiplexer 255. Through the display unit 140 is displayed. Thereafter, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a capture image processing path, and displays a split image having a size set in a camera image of a frame (for example, an image having zero shutter rec) set in the buffer 240. Access and print Then, the multiplexer 250 outputs the divided image output from the buffer 240 to the preprocessor 210, and the divided images processed by the preprocessor 210 and the postprocessor 230 are applied to the adder 600 through the demultiplexer 255. The processing of the captured image is repeated as many times as the number of images divided from the camera image of one frame (that is, the number of divided images), and when the processing of the divided images is completed, the adder 600 processes the camera image of one frame. It generates and outputs to the still image codec 260. The still image codec 260 compresses and encodes the added capture camera image in the storage 130. In this case, the operation of dividing the camera image by adding the same after processing the image is the same as the method described with reference to FIGS. 6A and 7A to 7C. Although not shown, the image processor 120 has an output of the buffer 240 connected to the preprocessor 210, an output of the preprocessor 210 and the buffer 240 connected to the multiplexer 250, and the output of the multiplexer 240 is an image scaler. It may be connected to 220, the output of the image scaler 240 may be configured to be connected to the post-processing unit 230.

The image processing unit 120 having the configuration as shown in FIGS. 6A to 6C processes the camera image of the frame set when the capture is requested. This is to simplify the hardware configuration of the image processor 120 (ie, reduce the number of gates of the preprocessor 210 and / or the post processor 230), and to reduce power consumption of the image processor 120. The reason why the camera image is divided and processed (pre-processed and / or post-processed) as described above is that the size of the camera image is large (that is, a high-pixel image) to simultaneously process the display image and the camera image in one frame section. It can be useful even when there is no. That is, by dividing and processing the camera image for capture, it is possible to stably capture even when the camera image is a large-capacity and high-pixel image.

8A to 8C are diagrams illustrating still another configuration of the image processor 120 according to the embodiment of the present invention. 8A to 8C illustrate embodiments in which the image processor 120 uses a plurality of operation clocks, divides and processes a frame image when generating a captured image, and then combines the divided images to generate a captured image. have. 9 is a diagram for describing an operation of the image processor 120 having the configuration as illustrated in FIGS. 8A to 8C. In addition, as illustrated in FIGS. 7A to 7C, when capturing a captured image, a camera image of one frame is divided into an image having a predetermined size, the divided image is processed, and the processed divided images are added to process the captured image.

8A, 7A, 7C, and 9, the preprocessor 210 preprocesses an image obtained from the camera 110. The preprocessed image is transferred to the image scaler 220 and scaled by the size and / or number of pixels of the display image. In this case, the image scaler 220 may include a resizer and / or an averager. In the preview mode, the image output from the preprocessor 210 is applied to the buffer 240 and buffered. At this time, the post-processing unit 230 and the still image codec 260 are not driven in the preview state, thus saving power consumption in the preview mode.

The clock generator 400 selects an output of the high speed clock generator 410, the low speed clock generator 420 generating a low speed clock lower than the high speed clock, and the high speed clock generator 410 or the low speed clock generator 420 under the control of the controller 100. And a selector 430 for supplying to the operation clock of the image processor 120. First, as shown in 911 of FIG. 9, the camera 110 generates and outputs an image of one frame in units of vertical synchronization signals Vsync. That is, the vertical synchronization signal Vsync is one frame period. In this case, in the preview mode, the image processor 120 may use a low speed clock or a high speed clock as an operation clock. First, when the high speed clock is selected, the selector 430 selects the output of the high speed clock generator 410, and the selected high speed clock is supplied to the operation clocks of the buffer 240 and the post processor 230. In this preview mode, the controller 100 sets the multiplexer 250 and the demultiplexer 255 as a display image processing path, and the access operation of the buffer 240 is not performed. Then, the frame image generated by the camera 110 is converted into a preprocessing and display image through the preprocessor 210 and the image scaler 220 at the same cycle as 911 of FIG. 9, and the display image is transmitted to the postprocessor 230 through the multiplexer 250. . Then, the post processor 230 is processed by the high-speed clock as shown in 913 of FIG. 9 and applied to the demultiplexer 255. The demultiplexer 255 transmits the post-processed display image to the display 140 to be displayed. Secondly, when the low speed clock is selected, the selector 430 supplies the low speed clock to the operation clocks of the buffer 240 and the post processor 230. Then, the post processor 230 is processed by the high-speed clock as shown in 915 of FIG. 9 and applied to the demultiplexer 255, and the demultiplexer 255 transmits the post-processed display image to the display unit 140 to be displayed.

When the capture request is generated in the above state, the controller 100 first controls the multiplexer 250 and the demultiplexer 255 in the capture frame period to set a path for processing the display image as shown by reference numerals 951 and 953. To be processed. Thereafter, the controller 100 controls the multiplexer 250 and the demultiplexer 255 in the capture frame period (921 and 923 frame periods in FIG. 9) to form a capture image processing path such as reference numbers 953 and 963, and the buffer 240. Accesses and outputs the camera image of the frame set in, for example, the frame image of the zero shutter rack considering the shutter delay. In this case, when the frame image is read from the buffer 240, the controller 100 accesses the divided image having the size set in the camera image of one frame in the manner shown in FIGS. 7A to 7C. That is, upon capture request, the controller 100 divides and reads the frame image stored in the buffer 240 into a set block size, and the divided image is applied to the post processor 230 through the multiplexer 250. The post-processing unit 230 post-processes the split image, and the demultiplexer 255 outputs the post-processed split image to the adder 600. Then, the adder 600 adds the divided images and reproduces the image as a frame camera.

In this case, when accessing by dividing the camera image as shown in FIG. 7A into two, the control unit 100 reads as shown in FIGS. Then, the divided frame image, such as 710 of FIG. 7B, accessed from the buffer 240 in the frame 921 of FIG. 9, is applied to the post-processing unit 230 as shown in 953 of FIG. 9 through the multiplexer 250, and is post-processed. The split image is applied from the demultiplexer 255 to the adder 600. Subsequently, the divided image, such as 720 of FIG. 7B, accessed from the buffer 240 in the next frame section (923 frame section of FIG. 9) is applied to the post-processing unit 230 as shown in 963 of FIG. 9 through the multiplexer 250, and then processed. The post-processed split image is applied from the demultiplexer 255 to the adder 600.

In this case, the divided image transmitted through the demultiplexer 255 further includes adjacent pixels in order to use information of the adjacent pixels in post-processing (in the case of 710 of FIG. 7B, pixels of n + 1 line, FIG. 7B). In the case of 720, n pixels of the line), pixels of the corresponding line should be removed and added. Therefore, the adder 600 removes the pixels of n + 1 lines when receiving the post-processed divided images as shown in 710 of FIG. 7B and n when receiving the post-processed divided images as shown in 720 of FIG. 7B. After removing the pixels of the line, these divided images are added to recreate a camera image of one frame. Thereafter, the camera image output from the adder 600 is applied to the still image codec 260 to be compressed and encoded, and the image output from the still image codec 260 is stored as a captured image in the storage 130. In the preview mode as described above, the image processing apparatus processes the display image as shown in 913 or 915 of FIG. 9 according to the selection clock. When the capture request is made, the controller 100 selects a high-speed clock and controls the multiplexer 250 and the demultiplexer 255 to process the display image and the captured image in one frame period. The camera image is divided, and the divided images are processed at high speed in frame sections according to the number of divided images. In this case, if the number of divided images is three, the divided images may be processed in three frame sections, and if four, the divided images may be processed in four frame sections. Referring to FIG. 8B, images of the camera 110 are input to the image scaler 220 and the buffer 240, respectively. Then, the buffer 240 buffers the image of the camera 110, and the image scaler 220 scales the image of the camera 110 to the display image size. The image scaler 220 may include a resizer and / or an averaging unit as described above. In addition, the clock generator 400 supplies the selected clock (low speed clock or high speed clock) to the operation clock of the image processor 120.

In the preview mode for processing the display image, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set the display image processing path. In addition, the controller 100 controls the clock generator 400 to supply a low speed clock or a high speed clock to the operation clock of the image processor 120. In this case, when the display image processing path is set, the multiplexer 250 selects the display image generated by the image scaler 220 and outputs it to the preprocessor 210. The demultiplexer 255 selects the output of the post processor 230 to display the display 140. Output to The preprocessor 210 and the post processor 230 preprocess and postprocess the display image, and the processed display image is displayed on the display 140. In this case, the image processor 120 processes the display image as shown at 915 of FIG. 9 when the low speed clock is supplied to the operation clock, and processes the display image as shown at 913 of FIG. 9 when the high speed clock is supplied to the operation clock.

When the capture request is generated in the above state, the control unit 100 performs time division control on the multiplexer 250 and the demultiplexer 255 to process the display image and the captured image in the captured image frame section. In addition, the controller 100 controls the clock generator 400 to supply a high speed clock to the operation clock of the image processor 120. Then, the multiplexer 250 and the demultiplexer 255 first select the display image by time division control of the controller 100 and then select the captured image in the remaining sections of the frame. In this case, when the camera image is processed by dividing into two, the display image output from the image scaler 220 in the display image processing section such as 951 and 961 of FIG. 9 is the multiplexer 250, the preprocessor 210, the post processor 230 and the demultiplexer 255 Through the display unit 140 is displayed and displayed. Subsequently, the divided images that are divided and accessed in the buffer 240 in the capture image processing intervals such as 953 and 963 of FIG. 9 are applied to the adder 600 through the multiplexer 250, the preprocessor 210, the post processor 230, and the demultiplexer 255. That is, as described in FIG. 8A, a camera image such as 7a is divided and accessed as shown in FIG. 7B or 7C in the buffer 240 and the divided image is accessed in the preprocessor 210 and the post processor 230. Pre-treatment and post-treatment are applied to the adder 600, respectively. Then, the adder 600 adds the divided images to generate a frame image, and the still image codec 260 compresses and encodes the one-frame camera image added by the adder 600, and stores the encoded camera image. It is stored in the unit 130. In this case, the operation of dividing and adding a camera image when processing the captured image by the image processor 120 may be performed by the method described with reference to FIGS. 6A and 8A.

Referring to FIG. 8C, the buffer 240 buffers an image output from the camera 110. In addition, the control unit 100 divides and outputs a camera image of a frame selected from the buffer 240 in a set size in a capture mode. The image scaler 220 scales the camera image of the current frame output from the buffer 240 into the display image. The clock generator 400 selects the low speed clock or the high speed clock under the control of the controller 100 and supplies the clock signal to the operation clock of the image processor 120. The multiplexer 250 selects and outputs the output of the image scaler 220 in the preview mode to process the display image under the control of the controller 100, and outputs the image scaler 220 in one frame section in the capture mode to capture the image of the camera 110. Time division and selection of the display image and the divided image of the camera image output from the buffer 240 are sequentially performed. The preprocessor 210 preprocesses the display image and / or the segmented image which is time-divided and output by the multiplexer 250 under the control of the controller 100. The post processor 230 post-processes the preprocessed display image and / or the segmented image under the control of the controller 100. The demultiplexer 255 outputs the display image output from the post-processing unit 230 to the display unit 140 in the preview mode under the control of the control unit 100, and time-divisionally selects the display image and the split image output from the post-processing unit 230 in the capture mode. Output is sequentially performed to the display unit 140 and the adder 600. The adder 600 adds the split image to generate a camera image of one frame, and the still image codec 260 compresses and encodes the camera image output from the adder 600 in the storage 130.

In the preview mode for processing a display image, the buffer 240 buffers the camera image under the control of the controller 100, and the image scaler 220 scales and outputs the camera image of the current frame in the buffer 240 as the display image. In addition, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a display image processing path, and controls the clock generator 40 to supply a low speed clock or a high speed clock to the operation clock of the image processor 120. When the display image processing path is set, the multiplexer 250 selects the display image generated by the image scaler 220 and outputs it to the preprocessor 210. The demultiplexer 255 selects the output of the postprocessor 230 and displays the output on the display 140. Output In this case, when the low speed clock is supplied to the operation clock, the image processing unit 120 processes the display image as shown in 915 of FIG.

In addition, when a camera image capture request is generated, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a display image processing path, and controls the clock generator 400 to operate a high speed clock in the image processor 120. Supply to the clock. Then, the multiplexer 250 selects a display image of the current frame processed by the image scaler 240 and outputs the displayed image to the preprocessor 210. The preprocessor 210 and the postprocessor 230 process the display image at intervals 951 and 961 of FIG. The display image is displayed on the display unit 140 through the demultiplexer 255. Thereafter, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to set a capture image processing path, and displays a split image having a size set in a camera image of a frame (for example, an image having zero shutter rec) set in the buffer 240. Access and print Then, the multiplexer 250 outputs the split image output from the buffer 240 to the preprocessor 210, and the preprocessor 210 and the post processor 230 process the divided image in the sections 953 and 963 of FIG. 9, and the demultiplexer. 255 outputs the post-processed split images to the adder 600. The processing of the captured image is repeated as many times as the number of images divided from the camera image of one frame (that is, the number of divided images), and when the processing of the divided images is completed, the adder 600 processes the camera image of one frame. It generates and outputs to the still image codec 260. The still image codec 260 compresses and encodes the added capture camera image in the storage 130. In this case, the operation of dividing the camera image by adding the same after processing the image is the same as the method described with reference to FIGS. 6A and 7A to 7C.

Also, although not shown, the image processor 120 has an output of the buffer 240 connected to the preprocessor 210, an output of the preprocessor 240 and the buffer 240 connected to the multiplexer 250, and the output of the multiplexer 240 is an image scaler. It is connected to the 220, the output of the image scaler 240 may have a configuration that is connected to the post-processing unit 230.

The image processing unit 120 having the configuration as shown in FIGS. 8A to 8C divides and processes a camera image of a frame set at high speed when a capture request is made. This is to simplify the hardware configuration of the image processor 120 (ie, reduce the number of gates of the preprocessor 210 and / or the post processor 230), and to reduce power consumption of the image processor 120. In addition, as described above, when the camera image is divided and processed at a high speed (pre-processing and / or post-processing), the image processor 120 may time-division and process the display image and the camera image even in the case of a camera image having a large amount of high pixels. .

In the case of the camera device having the image processing unit 120 (FIGS. 2A to 2D, 4A to 4C, 6A to 6C, and 8A to 8C) having the above-described configuration, a video and a captured image may be processed together. Can be. That is, the display image is an image that can be stored as a video. Therefore, in the video storage mode, the image processing unit 120 having the configuration as described above generates a display image in every frame section, and outputs it to the display unit 140 and to the video codec 270 at the same time. Therefore, in the video storage mode, the image processor 120 displays the display image of each frame on the display unit 140 and simultaneously encodes the video codec 270 and stores the encoded image codec 270 in the storage unit 130. When the capture request command is generated in the above state, the image processing apparatus encodes the frame image of the zero shutter rack together with the display image. The image processing unit 120 displays the display image on the display unit 140 and encodes the image through the video codec 270 and stores the encoded image in the storage unit 130, and also stores the encoded captured image in the storage unit 130. Therefore, snapshot shooting can be performed simultaneously during video recording.

10A and 10B illustrate an example of a specific configuration of the image processor 120 according to an embodiment of the present invention.

10A is a diagram illustrating a specific configuration example of the image processing unit 120 having the configuration as illustrated in FIG. 8A. 10A illustrates a buffer 240 including a ring buffer capable of storing five camera images in the configuration of FIG. 8A, a ring buffer 910 capable of buffering a display image output from the preprocessor 210, and the adder 600. An example further includes a scaler 920 that scales the output of the display to a size of data to be stored, and a scaler 930 that scales the display image to fit the display size of the display unit 140. In addition, although FIG. 10A illustrates a configuration example of the image processor 120 including the clock generator 400 and the adder 600, the clock generator 400 and / or the adder 600 may not be included. That is, the image processing unit as shown in FIG. 10A may be configured as the image processing unit 120 as shown in FIGS. 2A, 4A, and 6A.

Referring to FIG. 10A, first, in the preview mode, the controller 100 controls the multiplexer 250 so that the output of the ring buffer 910 is transmitted to the post processor 230, and the demultiplexer 255 controls the output of the post processor 230. To be delivered to. In this state, the image obtained from the camera 110 is preprocessed by the preprocessor 210, scaled to the display size of the display unit 140 by the image scaler 220, and then applied to the ring buffer 910. The ring buffer 910 may buffer the display image and set a size to buffer the display image of 2 to 3 frames. The reason why the ring buffer 910 is used is to compensate for a frame delay that may occur when processing the captured image. That is, the captured image is an image of the camera 110, and a frame delay may occur when processing the captured image. In this case, the display image is buffered in the ring buffer 910, and the captured image is processed to process the display image buffered in the ring buffer 910. Therefore, it is possible to prevent the display image from being cut off, which may occur when processing the captured image. The display image output from the ring buffer 910 is applied to the post-processing unit 230 through the multiplexer 250 and post-processed, and is applied to the scaler 930 through the demultiplexer 255. Then, the scaler 930 finally scales and outputs the post-processed display image to fit the size of the display unit 140, and the display unit 140 displays the display image output from the scaler 930. In addition, in the preview mode, the image of the camera 110 preprocessed by the preprocessor 210 is applied to the buffer 240 and buffered. In this case, the buffer 240 in FIG. 10 illustrates an example of a ring buffer capable of buffering 5 frame camera images. The number of buffers of the ring buffer 240 may be set to a size sufficient to compensate for the shutter lag of the camera device. In the preview mode, the controller 100 buffers images captured by the camera 110 in the buffer 240 but does not access them.

Here, it is assumed that the image processor 120 uses a low speed clock in the preview mode, uses a high speed clock in the capture image processing mode, and processes a camera image of one frame into two divided images. In addition, it is assumed that the frame image compensating the shutter lag when capturing a request is raw data 4. When the capture request is generated in the above state, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to time-division into the display image and the capture image processing section in the frame section 921 and 923 of FIG. To form an image processing path. In addition, the controller 100 controls the selector 430 to select a high speed clock.

The image processing unit 120 selects a display image processing path under the control of the controller 100 to process the display image as described above in the section 951 of the frame section 921 of FIG. 9. Subsequently, the controller 100 accesses a split image having a half frame image size as shown in 710 of FIG. 7B from raw data 4 buffered by the buffer 240 in section 953 of the frame section 921 of FIG. 9, and the accessed divided image is The image is post-processed in the section 953 of FIG. 9 and then applied to the adder 600. Thereafter, in the section 961 of the frame 923 of FIG. 9, the image processor 120 selects the display image processing path again to process the display image of the next frame. Subsequently, the controller 100 accesses a segmented image having a half frame image size as shown in 720 of FIG. 7B from raw data 4 buffered by the buffer 240 in section 963 of the 923 frame section of FIG. 9, and the accessed segmented image is After the post-processing in the section 963 of FIG. 9, it is applied to the adder 600.

The adder 600 adds two split images to generate a camera image of one frame, the scaler 920 scales to the size of the captured image to store the camera image, and the still image codec 260 encodes the scaled captured image. Store in the storage 130. In this case, the display unit 140 may display the captured image when the captured image is processed. That is, the control unit 100 can display the captured image on the display unit 140. In this case, when the storage is completed for a predetermined time or until the user returns the camera device to the preview mode (for example, including a picture name). Up to) the captured image may be displayed, in which case the display image may not be displayed.

FIG. 10B is a diagram illustrating a specific configuration example of the image processing unit 120 having the configuration as illustrated in FIG. 8B. 10B illustrates a buffer 240 including a ring buffer capable of storing five camera images in the configuration of FIG. 8B, a buffer 910 capable of buffering a display image output from the image scaler 220, and an output of the adder 600. An example further includes a scaler 920 that scales the data to a size of data to be stored, and a scaler 930 that scales the display image to fit the display size of the display unit 140. In addition, although FIG. 10B illustrates a configuration example of the image processor 120 including the clock generator 400 and the adder 600, the clock generator 400 and / or the adder 600 may not be included.

Referring to FIG. 10B, in the preview mode, the controller 100 controls the multiplexer 250 to transmit the output of the ring buffer 910 to the preprocessor 210, and controls the demultiplexer 255 to output the scaler 930. To be delivered to. In this state, the image obtained from the camera 110 is scaled to the display size of the display unit 140 in the image scaler 220 and then applied to the ring buffer 910. Then, the ring buffer 910 may buffer the display image and set a size to buffer the display image of 2 to 3 frames. The reason why the ring buffer 910 is used is to compensate for a frame delay that may occur when processing the captured image. That is, the captured image is an image of the camera 110, and a frame delay may occur when processing the captured image. In this case, the display image is buffered in the ring buffer 910, and the captured image is processed to process the display image buffered in the ring buffer 910. Therefore, it is possible to prevent the display image from being cut off, which may occur when processing the captured image. The display image output from the ring buffer 910 is preprocessed by being applied by the preprocessing unit 210 through the multiplexer 250. do. Then, the scaler 930 finally scales and outputs the post-processed display image to fit the size of the display unit 140, and the display unit 140 displays the display image output from the scaler 930. In addition, the camera image output from the camera 110 in the preview mode is applied to the buffer 240 is buffered. In FIG. 10B, the buffer 240 is shown as a ring buffer capable of buffering five frame images. The number of buffers of the buffer 240 may be set to a size sufficient to compensate for the shutter lag of the camera device.

When the capture request is generated in the above state, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to process the display image and the captured image in one frame section (for example, the frame section 921 and 923 in FIG. 9). Time division is performed by the interval to form an image processing path. In addition, the controller 100 controls the selector 430 to select a high speed clock. The image processor 120 selects a display image image path under the control of the controller 100 in a section 951 of the frame section 921 of FIG. 9 to process the display image as described above. Thereafter, the image processor 120 accesses a segmented image of the camera image (eg, raw data 4) of the set frame buffered in the buffer 240 in section 953 of the frame section 921 of FIG. 9, and the segmented image is preprocessed and post-processed. It is applied to the adder 600. Thereafter, the image processor 120 selects the display image path again under the control of the controller 100 in the 961 section of the 923 frame section of FIG. 9 to process the display image of the next frame. Thereafter, the controller 100 accesses the next divided image of the camera image in the buffer 240 in the section 963 of the 923 frame section of FIG. 9, and the accessed divided image is pre-processed and post-processed and applied to the adder 600. Then, the adder 600 adds two divided images to generate a camera image of one frame, the scaler 920 scales to the size of an image to store the frame image, and the still image codec 260 encodes the scaled captured image. Stored in the storage unit 130.

11 is a flowchart illustrating an image processing method of a camera apparatus or a terminal apparatus including a camera according to an exemplary embodiment of the present invention.

Referring to FIG. 11, when a camera driving command is generated through the input unit 150, the controller 100 drives the camera 110 in step 1111, and executes a preview mode for processing a display image in step 1113. In the preview mode, the controller 100 processes the display image in the same manner as in FIGS. 12A-12D. In addition, if a capture request is generated in the process of processing the display image, the controller 100 detects it in step 1115 and processes the captured image in step 1117. At this time, the control unit 100 performs time division control on the display image and the captured image by the procedure as shown in FIGS. 14 to 17. The preview mode or the capture mode is performed until the camera stops driving. When the user generates a termination request through the input unit 150, the control unit 100 detects this in step 1119 and stops driving the camera 110 in step 1121. Quit.

12A to 12D are flowcharts illustrating a procedure of processing a display image in a camera device or a portable terminal having a camera according to an embodiment of the present invention.

12A, in the preview mode, the controller 100 controls the multiplexer 250 and the demultiplexer 255 of the image processor 120 to set a display image processing path in step 1211. In step 1213, the controller 100 preprocesses the camera image output from the camera 110. In the preprocessing process, the image processor 120 extracts 3A (auto white balance), auto exposure (AE), and auto focusing (AF) from the camera image, and performs lens shading correction of the camera image. Perform dead pixel correction, knee correction, etc.

In operation 1215, the controller 100 buffers the preprocessed camera image in the buffer 240 and scales the camera image to a size that can be displayed on the display 140 to generate a display image. Thereafter, the control unit 100 post-processes the display image multiplexed in step 1217 and then displays it on the display unit 140 in step 1219.

FIG. 13 is a flowchart illustrating a scaling process of generating a display image and a process of post-processing and displaying the display image in FIGS. 12A to 12D. Referring to FIG. 13, image scaling may be performed through a resizing or averaging operation in an embodiment of the present invention. In this case, when the image scaler 220 is implemented as a resizer, the control unit 100 detects this in step 1311 and performs a resizing operation in step 1313 to scale the camera image to the display image. At this time, in step 1313, the controller 100 generates a display image by performing operations such as decimation, interpolation, and cropping. In addition, when the image scaler 220 is implemented as an averaging unit, the control unit 100 detects this in step 1311 and in step 1315 adds and averages adjacent pixels in the camera image to reduce the number of pixels. In this case, the averaging operation may be performed in the same manner as in FIGS. 3A and 3B.

After performing image scaling as described above, the controller 100 pre-processes and / or post-processes the display image. In this case, the post-processing operation of the display image may be performed as in steps 1317 to 1321. Here, the post-processing operation of the image performs color interpolation, IPC processing, and image conversion operation as described above, and when the post-processing process is performed, a YUV display image is generated. In step 1323, the controller 100 outputs the YUV display image to the display unit 140 to display the YUV display image.

Therefore, when the preview mode is performed in step 1613, the controller 100 scales the image of the camera 110 to the display image size in the same procedure as in FIG. 13 in every frame period, and post-processes the scaled image on the display unit 140. The buffering operation of the full resolution image is performed.

13 illustrates an example of performing post-processing after image scaling. However, when the display image is processed, the controller 100 may preprocess the display image after performing step 1313, and may postprocess the preprocessed display image through steps 1317-1321.

12B, in the preview mode, the controller 100 controls the multiplexer 250 and the demultiplexer 255 of the image processor 120 to set a display image processing path in step 1231. In step 1233, the controller 100 scales the camera image output from the camera 110 into the display image, and buffers the camera image in the buffer 240. Thereafter, the control unit 100 preprocesses the display image in step 1235, postprocesses the preprocessed display image in step 1237, and displays the postprocessed display image through the display unit 140 in step 1239.

12C, in the preview mode, the controller 100 controls the multiplexer 250 and the demultiplexer 255 of the image processor 120 to set a display image processing path in step 1251. The controller 100 buffers the camera image output from the camera 110 in step 1253, and scales the camera image of the current frame into the display image in the buffered camera image in step 1255. Thereafter, the control unit 100 preprocesses the display image in step 1257, postprocesses the preprocessed display image in step 1259, and displays the post-processed display image through the display unit 140 in step 1261.

12D, in the preview mode, the controller 100 controls the multiplexer 250 and the demultiplexer 255 of the image processor 120 to set a display image processing path in step 1271. The controller 100 buffers the camera image output from the camera 110 in operation 1273, and preprocesses the camera image of the current frame from the buffered camera image in operation 1275. Thereafter, the controller 100 scales the preprocessed camera image into a display image, postprocesses the display image in step 1279, and displays the postprocessed display image through the display unit 140 in step 1281.

In the process of processing the display image as described above, the user may request an image capture. In this case, when a capture request is generated, the controller 100 time-divisions the display image and the captured image path in one frame section, and also sets a camera image (eg, a camera image of zero shutter rack) set among the buffered camera images. ) To process the captured image. 14 to 17 are flowcharts illustrating a procedure of processing a captured image in a camera device or a portable terminal including a camera according to the first to fourth embodiments of the present invention, respectively.

Referring to FIG. 14, when a capture request is made, the controller 100 controls the multiplexer 250 and the demultiplexer 255 to form a display image processing path in step 1411. In step 1413, the control unit 100 scales the image captured by the camera 110 to the display image through the display image processing path, and displays the image through the display unit 140 by preprocessing and postprocessing the image. The display image processing method of step 1413 may be performed by any one of FIGS. 12A to 12D. In step 1415, the control unit 100 forms a capture image processing path, and in step 1417, accesses a camera image (eg, a frame image of zero shutter rack) set among the images buffered in the buffer 240, and in step 1419. Preprocesses and postprocesses the accessed camera image. Thereafter, the controller 100 codes the post-processed image in step 1421 and stores it in the storage 130 in step 1423. In this case, the captured image processing method may be performed in the same manner as the processing of the display image.

Referring to FIG. 15, the controller 100 may supply a high speed clock as an operation clock of the image processor 120 when processing a captured image. In this case, the image processor 120 may time-division the display image processing path and the captured image path in one frame section under the control of the controller 100, and may process the display image and the captured image at high speed by the high speed clock. In this case, when the capture request is made, the control unit 100 supplies a high-speed clock to the operation clock of the image processor 120 in step 1511, and controls the multiplexer 250 and the demultiplexer 255 of the image processor 120 to control the display image processing path in step 1513. Form. In step 1515, the control unit 100 scales the image captured by the camera 110 to the display image through the display image processing path, and also displays the image on the display unit 140 by pre-processing and post-processing the image by a fast clock. The display image processing method of step 1515 may be performed by one of FIGS. 12A to 12D. In step 1517, the control unit 100 forms a capture image processing path, and in step 1519, accesses a camera image (eg, a frame image of zero shutter rack) set among the images buffered in the buffer 240, and in step 1521. Preprocesses and postprocesses the accessed camera image by a high speed clock. Thereafter, the controller 100 codes the post-processed camera image in operation 1523 and stores the stored image in the storage unit 130 in operation 1525. In this case, the captured image processing method may be performed in the same manner as the processing of the display image.

In addition, the captured image may be divided and processed. The captured image may be an image of the camera 110. If the camera image is a high pixel image, the configuration of the preprocessor 210 and / or the post processor 230 is complicated, and the current consumption of the camera device is increased according to the camera image processing. Can be large. Therefore, the camera image is divided into a predetermined size when the captured image is processed, and the pre-processing unit 210 and / or the post-processing unit 230 are configured to process the divided image, and the post-processed image is added and processed. This can be efficient. 16 is a flowchart illustrating a procedure of regenerating a carrera image of one frame by dividing a camera image, processing and adding the divided images when the captured image is processed by the image processor 120.

Referring to FIG. 16, when a capture request is made, the controller 100 sets a display image path for processing an image acquired by the camera 110 as a display image in step 1611. In step 1613, the controller 100 scales the camera image to convert the camera image into a display image, and displays the display image by pre-processing and / or post-processing. The display image processing method of step 1613 may use one of the methods of FIGS. 12A-12D.

In step 1615, the controller 100 sets a capture image processing path for storing an image acquired by the camera in the remaining frame sections of the corresponding frame section. In operation 1617, the controller 100 selects a frame (eg, a frame image (zero shutter image) at the time of shutter-on) from among the buffered camera images, and the image having the size set in the camera image of the selected frame. Split access The controller 100 may access the divided image from the camera image in the same manner as in FIGS. 7A to 7C. In step 1619, the controller 100 preprocesses and / or post-processes the split image. In this case, the method of preprocessing and / or postprocessing the split image may be performed in the same manner as the processing method of the display image. After processing the divided image, the controller 100 checks whether all the camera images of one frame are processed in step 1621. In this case, if the processing of the 1-frame camera image is not completed (that is, there are more split images to be processed), the controller 100 detects this in step 1621 and returns the post-processed split image to a previous state in step 1623 (eg, For example, the divided image is added to the processed image in the previous frame section, and in step 1625, the divided image to be accessed next is set. When the processing of the split images is completed while repeating the above operation, a single frame image is generated. In this case, the controller 100 detects this in step 1621, adds the post-processed split image in step 1627, and step 1629. In step 1637, the camera image of the frame is encoded and stored in the storage 130.

FIG. 16 illustrates a method of accessing and processing a camera image as a divided image in a capture mode, and illustrates an example of processing one divided image in one frame section as shown in FIG. 9. However, when the divided images may be processed in one frame section, the controller 100 may proceed to step 1617 in step 1625 to access and process the next divided image. In this case, the image processing unit 120 is time-division controlled in one frame section to form a display image and a capture image processing path, and after processing the display image, a plurality of divided images (that is, one frame of the frame) are formed. Split images of camera images) can be processed continuously.

In addition, during the capture image processing, the operation clock of the image processor 120 may be set to a high speed clock, and the camera image may be divided into a predetermined size and processed. 17 is a flowchart illustrating a procedure of selecting an operation clock of the image processing unit 120 as a high speed clock and dividing a camera image when processing the captured image by the image processing unit 120.

Referring to FIG. 17, when a capture request is made, the controller 100 sets the operation clock of the image processor 120 to a high speed clock in step 1711. This is to allow the image processor 120 to process the display image and the captured image at high speed in the capture mode. In operation 1713, the controller 100 sets a display image path for processing the image acquired by the camera 110 as a display image. In step 1715, the control unit 100 scales the camera image to convert the camera image into a display image, and displays the display image by pre-processing and / or post-processing at high speed. The display image processing method of step 1715 may use one of the methods of FIGS. 12A to 12D.

In step 1717, the control unit 100 sets a capture image processing path for storing an image acquired by the camera in the remaining frame sections of the corresponding frame section. In operation 1719, the control unit 100 selects a set frame (eg, a frame image of a shutter-on time (zero shutter rec image)) from among the buffered camera images, and an image having a size set in the camera image of the selected frame. Split access The controller 100 may access the divided image from the camera image in the same manner as in FIGS. 7A to 7C. In step 1721, the controller 100 preprocesses and / or post-processes the divided image at high speed. In this case, the method of preprocessing and / or postprocessing the split image may be performed in the same manner as the processing method of the display image. After the split image processing, if the processing of the 1-frame camera image is not completed (that is, there are more split images to be processed), the controller 100 detects this in step 1723 and detects the post-processed split image in step 1725. The divided image is added to the previous state (for example, the previous frame section) and the divided image to be accessed next is set in step 1727, and then the process returns to step 1711. If the processing of the split images is completed while repeating the above operation, a single frame image is generated. In this case, the control unit 100 detects this in step 1723, adds the post-processed split image in step 1729, and step 1731. In step 1733, the camera image of the frame is encoded and stored in the storage unit 130.

FIG. 17 illustrates a method of accessing and processing a camera image as a split image in a capture mode, and illustrates an example of processing one split image in one frame section as shown in FIG. 9. However, when the divided images can be processed in one frame section, the controller 100 proceeds from step 1727 to step 1719 to access and process the next divided image. In this case, the image processing unit 120 is time-division controlled in one frame section to form a display image and a capture image processing path, and after processing the display image, a plurality of divided images (that is, one frame of the frame) are formed. Split images of camera images) can be processed continuously.

When the captured image is processed by one of the methods shown in FIGS. 14 to 17, the control unit 100 performs time division control on the image processor 120 in the frame section, thereby allowing the image processor 120 to display the display image in one frame section. Process the captured images sequentially. In this case, the display unit 140 displays a display image, and the storage unit 130 stores the captured image. In this case, the controller 100 may control the display unit 130 to display the captured image when the captured image is processed. In this case, the captured image can be displayed for a set time or until the user returns the camera device to the preview mode (e.g., until the recording is completed including the picture name, etc.), in which case the display image is displayed. It may not be.

As described above, the camera apparatus or the portable terminal including the camera according to the embodiment of the present invention can configure the image processing apparatus in a compact manner by time-division processing the preview / video image processing and the captured image processing.

18 is a diagram showing the configuration of a portable terminal device including a camera device according to an embodiment of the present invention. The portable terminal device may include a mobile phone having a camera, a smart phone, a tablet, a portable computer, a music player (MP), and the like.

Referring to FIG. 18, the camera 110 includes an image sensor and acquires a camera image through the sensor when the camera is driven.

The image processor 120 generates a display image for displaying a camera image obtained from the camera 110 on the display unit 140 and a capture image for storing when a capture request is made. The display image may be a YUV image, and the captured image may be a JPEG image.

The application processor 1810 performs various application functions of the portable terminal device. In addition, the application processor 1810 processes a display image and / or a captured image generated by the image processor 120 when the camera is driven according to an exemplary embodiment of the present invention. In this case, the application processor 1810 controls the display image to be displayed and displayed on the display unit 140 in the preview mode. When the capture request command is generated, the application processor 1810 selects and processes a camera image of a set frame from among the buffered camera images. Control 120. The application processor 1810 includes a controller function of the camera apparatus.

The communication unit 1830 performs a communication function with an external device or a base station. The communication unit 1830 includes a transmitter including a frequency up converter for raising a transmission signal to an RF band, a power amplifier, an amplifier for low noise amplifying the RF reception signal, and a converter for down converting an RF signal to a base band (frequency down). converter) and the like. The communication unit 1830 may include a modulator for modulating the transmission signal and transmitting the modulated signal to a transmitter, and a demodulator for demodulating a signal output from the receiver. The modulation and demodulation unit may be a modulation / demodulation unit of WCDMA, GSM, LTE, or a modulation / demodulation unit of WIFI, WIBRO, etc. according to a communication method.

The terminal controller 1820 controls overall operations of the portable terminal device and performs functions of voice call and data communication. Here, the terminal controller 1820 may include a modulation and demodulation unit of the communication unit 1830, and in this case, the communication unit 1830 may include only an RF communication unit. In this case, the terminal controller 1820 may perform a function of modulating and demodulating and encoding / decoding the communication signal. The audio processor 1840 is connected to a handset (microphone and a speaker) of the portable terminal and processes a voice call during voice communication under the control of the terminal controller 1820.

The input unit 150 generates various operation commands and data input signals of the portable terminal device to the terminal controller 1820 and the application processor 1810. In particular, the application processor 1810 may generate a camera driving command and a capture command. The display unit 140 displays the communication and application execution states of the portable terminal device under the control of the terminal controller 1820 and the application processor 1810. In addition, the display unit 140 displays a display image processed by the image processor 120 in the preview mode according to an embodiment of the present invention. The input unit 150 may be a touch panel that senses a user's touch input, and the display unit 140 may be an LCD or an OLED panel displaying data and images generated during program execution. The input unit 150 and the display unit 140 may be an integrated touch screen. In addition, the input unit 150 may include buttons located outside the portable terminal.

The storage unit 130 may include a program memory for storing an operation program of the portable terminal device and programs according to an embodiment of the present invention, and a data memory for storing data for program execution and data generated during program execution. . The storage unit 130 may include a nonvolatile memory such as a flash memory and a volatile memory such as a RAM. In particular, the storage unit 130 stores a compressed image of a camera image or a scaled image in the image processing unit 120 under the control of the application processor 1810 as a capture image.

In the case of the portable terminal as shown in FIG. 18, the application processor 1810 performs a control function of the image processor 120. The image processor 120 may be any one of FIGS. 2A to 2D, 4A to 4C, 6A to 6C, and 8A to 8C, and the display image and the still image codec output from the demultiplexer 255. The apparatus may further include a second multiplexer configured to multiplex the output of 260. The application processor 1810 includes a second demultiplexer which demultiplexes the display image and the captured image multiplexed by the second multiplexer of the image processor 120, and outputs the demultiplexed display image to the display unit 140. Store in the storage 130. The video codec 270 of the image processor 120 may be provided in the application processor 1810.

Referring to FIG. 18, the portable terminal device can perform an incoming / outgoing call and communication service, and processes an image captured by the camera unit 110 according to an embodiment of the present invention in the camera photographing mode.

The application processor 1810 processes an image processing application of the camera 110 and various applications of the portable terminal device according to an embodiment of the present invention.

During an outgoing call, the user may generate an outgoing call number and a call request signal through the input unit 150. The terminal controller 1820, which detects this, forms a call path by controlling the communication unit 1830, and performs an outgoing call function through a base station. In addition, when an incoming call is generated, the terminal controller 1820 detects this through the communication unit 1830, alerts the incoming call through the audio processor 1840, and displays the incoming information through the display unit 140. At this time, when the user responds through the input unit 150, the terminal controller 1820 forms the incoming call path through the communication unit 1830 to perform a call service. In the case of data communication, a communication path is formed by an operation similar to the above-mentioned communication.

In addition, the portable terminal apparatus can execute various applications. At this time, if the user requests the execution of a specific application through the input unit 150, the application processing unit 1810 executes the application, and displays the result on the display unit 140. At this time, when the user generates a camera driving request command, the application processor 1810 detects this and drives the camera unit 110 through the image processor 120. Then, the image processing unit 120 forms a display image processing path in the preview mode under the control of the application processing unit 1810, and in the capture mode, time division control is performed to sequentially form the display image processing path and the capture image processing path in the frame section.

In this case, the image processing unit 120 for processing the display image buffers the camera image in the buffer 240, and converts the camera image into a display image and processes the converted display image and outputs it to the application processing unit 1810. Then, the application processing unit 1810 displays the display image on the display unit 140. In this case, the display image processing may include preprocessing the camera image as described above, scaling the preprocessed camera image to convert the display image, and postprocessing the converted display image. In addition, the image processor 120 may scale the camera image to convert the image into a display image, and preprocess and post-process the converted display image. In addition, the image processor 120 may buffer the camera image, scale the buffered camera image to the display image, and preprocess and postprocess the display image. In addition, the image processor 120 may buffer an image of the camera, preprocess the buffered image, scale the preprocessed image into a display image, and then postprocess the display image. After processing the display image, the image processing unit 120 outputs the processed display image to the application processing unit 1810, and the application processing unit 1810 displays it on the display unit 140.

In addition, in the capture mode, the image processor 120 is time-divided and controlled in the frame section as described above under the control of the application processor 1810 to sequentially form the display image processing path and the capture image processing path. The display image processing method is performed as described above. When the captured image is processed, the image processor 120 selects a camera image of a set frame from among the camera images buffered under the control of the application processor 1810, and pre-processes and post-processes the selected camera image, and converts the captured image into a captured image. In this case, the method for processing the captured image may use a fast clock as described above, may be processed as a split image, and may also be processed as a split image using a fast clock. As described above, the display image and the captured image which are time-divided and sequentially output from the image processing unit 120 are multiplexed and output to the application processing unit 1810. The application processing unit 1810 demultiplexes the multiplexed display image and the captured image, and then displays the display image. The image is displayed through the display unit 140 and the captured image is stored in the storage unit 130.

In addition, the image processor 120 of the portable terminal has a pre-processing unit 210, an image scaler 220, and a post-processing unit 230 in any one of FIGS. 2A to 2D, 4A to 4C, 6A to 6C, and 8A to 8C. And a buffer 240, and the demultiplexer 255 and the still image codec 260 (including the adder 600 when processing the captured image as a split image) may be configured in the application processor 1810. When the image processing unit 120 has the above configuration, the image processing unit 120 outputs an image (display image or display image / capture image) output from the post processing unit 230 to the application processing unit 1810, and the application processing unit 1810 performs the post-processed image. The display image is displayed on the display unit 140 by demultiplexing (display image or display image / capture image), and the captured image is compressed and encoded through the still image codec and stored in the storage unit 130. In this case, when the image processing unit 120 processes the camera image as a divided image, the application processing unit 1810 includes an adder to add the divided images output from the image processing unit 120 and compresses the image when the image processing unit 1 processes the camera image. Encode and save.

The embodiments of the present invention disclosed in the present specification and drawings are merely illustrative of specific embodiments of the present invention and are not intended to limit the scope of the present invention in order to facilitate understanding of the present invention. It will be apparent to those skilled in the art that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein.

Claims (36)

An image processing apparatus comprising:
An image scaler that scales the camera image to a display image,
A buffer for buffering the camera image,
A multiplexer which multiplexes the output of the image scaler and the buffer;
An image processor which processes an output image of the multiplexer;
A demultiplexer for outputting a display image to demultiplex the output of the image processor;
A codec for encoding a camera image output from the demultiplexer and outputting the captured image as a captured image,
And a controller configured to control the multiplexer and the demultiplexer to time-divisionally control the display image and the camera image in a frame section in a capture mode. The method of claim 1, wherein the image processing unit
A preprocessor for preprocessing the image;
And a post processor configured to post-process the preprocessed image. 3. The method of claim 2,
The control unit sets the display image processing path by controlling the multiplexer and the demultiplexer in the preview mode.
And a display image generated by the image scaler when the display image processing path is set is output through the pre-processing unit and the post-processing unit. 3. The method of claim 2,
The controller sequentially sets the display image processing path and the captured image processing path by time division controlling the multiplexer and the demultiplexer in one frame section in the capture mode.
When the display image processing path is set, the display image generated by the image scaler is processed through the preprocessor and the post processor and then output to the display unit.
When setting the capture image processing path, a camera image of a set frame is accessed among the camera images buffered in the buffer, and the accessed camera image is processed through the preprocessor and the post processor and then output to a codec to be encoded as a capture image. Apparatus characterized in that the. The apparatus of claim 4, wherein the camera image of the frame set in the capture mode is a camera image of a frame whose shutter delay is compensated for. The method of claim 5,
Further comprising a clock generator for generating a high speed clock or a low speed clock under the control of the controller,
And the output of the clock generator is supplied to the operation clocks of the buffer, the preprocessor and the post processor. The method of claim 6, wherein
And the controller controls the clock generator to select the low speed clock as the operation clock in the preview mode and the high speed clock as the operation clock in the capture mode. The method of claim 5,
And an adder configured to add the divided images processed by the post processor to one frame image.
The control unit accesses the camera image of the frame set in the buffer as the divided image of the set size in the capture mode,
The adder adds the post-processed split image and outputs the added image to the codec when added to a camera image of one frame. 9. The method of claim 8,
The control unit controls the buffer to output one divided image in one frame section in the capture mode,
The image processing unit is characterized in that for processing one divided image in one frame interval. The method of claim 5,
A clock generator for generating a high speed clock or a low speed clock under the control of the controller;
And an adder configured to add the divided images processed by the post processor to one frame image.
The control unit selects the fast clock as the operation clock in the capture mode, accesses the camera image of the frame set in the buffer as a divided image of a set size,
The adder adds the post-processed split image and outputs the added image to the codec when added to a camera image of one frame. 11. The apparatus of claim 10, wherein the control unit
In the preview mode, the clock generator is controlled to select a low speed clock as an operation clock.
In the capture mode, the controller controls the buffer to output one divided image in one frame section in the capture mode, the image processing unit is characterized in that for processing one divided image in one frame section. The method of claim 5,
The preprocessor compensates for dead pixel correction and lens shading of the input image,
And the post processor converts the preprocessed image into a YUV image by color interpolation. The method of claim 5,
And the image scaler is a resizer. 6. The apparatus of claim 5, wherein the image scaler is an averaging unit that adds and averages adjacent pixels of the image to reduce the number of pixels to the size of the display image of the display unit. In the camera image processing method,
A display image processing process of buffering a camera image by controlling the image processing unit as a display image processing path in the preview mode, converting the image into a display image, and processing and displaying the image;
In the capture mode, time-division control of the image processing unit in the frame section sets the display image processing path and the capture image processing path sequentially, and performs the display image processing process in the display image processing path, and when the capture image processing path is set And a capture image processing process of selecting and processing a camera image of a set frame from the buffered camera image. The method of claim 15, wherein the image processing of the capture mode is performed.
Setting a display image processing path of the image processing unit;
A display image processing process of buffering a camera image and converting and processing and displaying the camera image into a display image through the set display image path;
Setting a capture image processing path of the image processor;
Selecting a camera image of a set frame from among the buffered camera images, and processing the selected camera image through a set display image path and then compressing and storing the captured image process. The method of claim 16, wherein the processing of the display image comprises:
Scaling the camera image into buffering and display images,
Preprocessing the display image;
And post-processing the preprocessed display image. The method of claim 16, wherein the processing of the display image comprises:
Preprocessing the camera image,
Scaling the preprocessed camera image into a buffering and display image;
And post-processing the display image. The method of claim 16, wherein the processing of the display image comprises:
Buffering camera images,
Scaling a camera image of the buffered current frame into a display image;
Preprocessing the display image;
And post-processing the preprocessed display image. The method of claim 16, wherein the processing of the display image comprises:
The process of buffering camera images
Preprocessing the camera image of the buffered current frame;
Scaling the preprocessed camera image into a buffering and display image;
And post-processing the display image. 17. The method of claim 16, wherein the camera image of the frame set during the capture image processing is a camera image of a frame compensated for by a shutter delay of the camera. 17. The method of claim 16,
Equipped with high speed clock or low speed clock,
And supplying the high speed clock to the operation clock of the image processor in the capture mode. The method of claim 22,
And supplying a low speed clock to the operation clock of the image processor in the preview mode. The method of claim 16, wherein the captured image processing process,
Accessing a camera image of a frame set in the buffer as a split image having a predetermined size;
And adding the post-processed split image and outputting the split image to the codec when added to a camera image of one frame. 25. The method of claim 24,
The access of the divided image is accessed once in one frame period. 17. The method of claim 16,
And a high speed clock or a low speed clock, and supplying the high speed clock to the operation clock of the image processor in the capture mode.
The capture image processing process,
Accessing a camera image of a frame set in the buffer as a split image having a predetermined size;
And adding the post-processed split image and outputting the split image to the codec when added to a camera image of one frame. In the camera device,
A camera which is operated when the camera is driven to generate a full resolution image;
A display unit for displaying a display image,
An input unit for generating a preview mode and a capture mode signal,
A storage unit for storing the captured image,
A control unit configured to control the display image processing path setting when the preview mode signal is detected, and to sequentially set the display image processing path and the capture image processing path by time-dividing the frame section when detecting the capture mode signal;
Under the control of the controller, a display image processing path is set in the preview mode, and the camera image is converted into a buffering and a display image and processed. The display image is output to the display unit. A camera image is buffered and converted into a display image and output to the display unit, and then the captured image path is set to access and process the camera of the set frame in the buffered camera image to compress-encode the captured image into a captured image, Apparatus comprising an image processing unit for outputting to. The method of claim 27, wherein the image processing unit,
A buffer for buffering the camera image;
An image scaler for scaling the camera image into a display image;
A multiplexer which multiplexes the output of the buffer and the image scaler;
A preprocessor for preprocessing the output of the multiplexer;
A post-processing unit for post-processing the output of the pre-processing unit;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. An image processing path is set to process a display image, a capture image processing path is set to select a multiplexer output of the buffer, and a demultiplexer sets a capture image path for outputting the output of the post-processor to a still image codec Device characterized in that. The method of claim 27, wherein the image processing unit,
A preprocessor for preprocessing the camera image;
A buffer that buffers the preprocessed camera image,
An image scaler that scales the preprocessed camera image to the display image,
A multiplexer which multiplexes the output of the buffer and the image scaler;
A post processor configured to post-process the output of the multiplexer;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. After setting the image processing path to process the display image and setting the capture image processing path, the multiplexer selects the output of the buffer, and the demultiplexer sets the capture image path for outputting the output of the post-processing part to the still image codec. Device characterized in that. The method of claim 27, wherein the image processing unit,
A buffer for buffering the camera image;
An image scaler for scaling a camera image of the buffered current frame into a display image;
A multiplexer which multiplexes the output of the buffer and the image scaler;
A preprocessor for preprocessing the output of the multiplexer;
A post-processing unit for post-processing the output of the pre-processing unit;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. After setting the image processing path to process the display image and setting the capture image processing path, the multiplexer selects the output of the buffer, and the demultiplexer sets the capture image path for outputting the output of the post-processing part to the still image codec. Device characterized in that. The method of claim 27, wherein the image processing unit,
A buffer for buffering the camera image;
A preprocessor for preprocessing the camera image output from the buffer;
An image scaler that scales the preprocessed camera image to the display image,
A multiplexer which multiplexes the output of the preprocessor and the image scaler;
A post processor configured to post-process the output of the multiplexer;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. After setting the image processing path to process the display image and setting the capture image processing path, the multiplexer selects the output of the preprocessor and the demultiplexer sets the capture image path to output the output of the postprocessor to the still image codec. Device characterized in that. In a mobile terminal device,
A camera which is operated when the camera is driven to generate a full resolution image;
A display unit for displaying a display image,
An input unit for generating a preview mode and a capture mode signal,
A storage unit for storing the captured image,
And controls the display image processing path setting when the preview mode signal is sensed by the camera, and controls the display image processing path and the captured image processing path by time-division controlling the frame period when the capture mode signal is sensed. An application processing unit which is set sequentially;
A terminal control unit controlling a communication unit to perform a communication function of the device, and communicating with the application processing unit;
By setting the display image processing path in the preview mode under the control of the application processing unit, converts and processes the camera image into a buffering and display image and outputs it to the display unit, and sets the display image processing path in the frame section in the capture mode. The camera image is buffered and converted into a display image, processed and outputted to the display unit, and the captured image path is set to access and process a camera of a frame set in the buffered camera image to compress and encode the captured image into the captured image. Apparatus comprising an image processing unit for outputting to the unit. The method of claim 32, wherein the image processing unit,
A buffer for buffering the camera image;
An image scaler for scaling the camera image into a display image;
A multiplexer which multiplexes the output of the buffer and the image scaler;
A preprocessor for preprocessing the output of the multiplexer;
A post-processing unit for post-processing the output of the pre-processing unit;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. After setting the image processing path to process the display image and setting the capture image processing path, the multiplexer selects the output of the buffer, and the demultiplexer sets the capture image path for outputting the output of the post-processing part to the still image codec. Device characterized in that. The method of claim 32, wherein the image processing unit,
A preprocessor for preprocessing the camera image;
A buffer that buffers the preprocessed camera image,
An image scaler that scales the preprocessed camera image to the display image,
A multiplexer which multiplexes the output of the buffer and the image scaler;
A post processor configured to post-process the output of the multiplexer;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. After setting the image processing path to process the display image and setting the capture image processing path, the multiplexer selects the output of the buffer, and the demultiplexer sets the capture image path for outputting the output of the post-processing part to the still image codec. Device characterized in that. The method of claim 32, wherein the image processing unit,
A buffer for buffering the camera image;
An image scaler for scaling a camera image of the buffered current frame into a display image;
A multiplexer which multiplexes the output of the buffer and the image scaler;
A preprocessor for preprocessing the output of the multiplexer;
A post-processing unit for post-processing the output of the pre-processing unit;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. After setting the image processing path to process the display image and setting the capture image processing path, the multiplexer selects the output of the buffer, and the demultiplexer sets the capture image path for outputting the output of the post-processing part to the still image codec. Device characterized in that. The method of claim 32, wherein the image processing unit,
A buffer for buffering the camera image;
A preprocessor for preprocessing the camera image output from the buffer;
An image scaler that scales the preprocessed camera image to the display image,
A multiplexer which multiplexes the output of the preprocessor and the image scaler;
A post processor configured to post-process the output of the multiplexer;
A still image codec that compresses and encodes a post-processed camera image as a captured image,
And a demultiplexer for demultiplexing the output of the post-processing unit and outputting the demultiplexer to the display unit and the still image codec.
Under the control of the control unit, the multiplexer selects the output of the image scaler in the preview mode, and the demultiplexer sets the display image processing path for outputting the output of the post-processing unit to the display unit. After setting the image processing path to process the display image and setting the capture image processing path, the multiplexer selects the output of the preprocessor and the demultiplexer sets the capture image path to output the output of the postprocessor to the still image codec. Device characterized in that.

Images