JP2005323293A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To attain improvement of image processing efficiency by attaining reduction of processing time, power consumption and memory capacity in image superimposing processing. <P>SOLUTION: A format converting unit 10 includes a superimpose image region selecting section 12 and an arithmetic processing section 13. The superimpose image region selecting section sets a threshold value for determining whether or not the region is required for a superimpose image, when superimposing a plurality of image data, and selects an image region based on the threshold value. The arithmetic processing section 13 operates the selected image region using a conversion coefficient and converts the YUV format and RGB format of the image data. An image superimposing unit 20 writes a plurality of format-converted image data in a memory and generates a superimposed image and outputs it. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that processes a plurality of image data.

  In recent years, image processing systems have become very complex, and various types of macros (applications) are mounted on one system and various image formats such as RGB format and YUV format are mixed.

  RGB is an abbreviation with the initial letters of the three primary colors of red (red), green (green), and blue (blue). Expresses light and shade, strength, etc.).

RGB 888 and RGB 555 are frequently used in the RGB format. RGB888 is a format that enables color tone expression of 16,777,210 (= 2 ⁸ × 2 ⁸ × 2 ⁸ ) colors by dividing RGB into 256 gradations (8 bits). RGB555 is a format in which RGB can be divided into 32 gradations (5 bits) and color tone expression of 32,768 (= 2 ⁵ × 2 ⁵ × 2 ⁵ ) colors can be expressed.

  YUV is a format in which color is represented by three pieces of information: a luminance signal (Y), a difference between the luminance signal and the red component (U), and a difference between the luminance signal and the blue component (V). By utilizing the property that the human eye is more sensitive to changes in brightness than changes in color and allocating a larger amount of data to the luminance signal, a high data compression rate can be obtained with little degradation in image quality. It is used in compression techniques such as MPEG. The RY color difference signal U may be referred to as Cr, and the BY color difference signal V may be referred to as Cb.

  YUV format is sub-sampling (degradation of image quality of luminance can be clearly identified, but it is difficult to judge deterioration of image quality of color, so color information is reduced and averaged. This process is called sub-sampling) Are classified into several types according to the method of thinning out color difference components. For example, there are YUV422 which thins out by two pixels in the horizontal direction and shares one U and one V by two pixels. A main YUV format classification table T3 is shown in FIG.

  On the other hand, in a system in which a plurality of image formats are mixed, data is exchanged between macros at high speed and smoothly, and it is different from the format (color space) in which the current color is expressed, such as YUV → RGB. Format conversion (color space conversion) to the above format is performed. For example, when digital video in which colors are expressed in YUV is to be output to a personal computer display, conversion to RGB is required.

  FIG. 7 illustrates an image processing system that performs format conversion. The image processing system 100 includes a CPU 101, a format converter 102, a graphic controller 103, macros 104-1 to 104-n, a memory 105, and a memory controller 106.

  The CPU 101 performs overall image processing control. The format converter 102 is a device that performs YUV / RGB format conversion, and converts image formats such as RGB888 and YUV420 to each other by hardware. The graphic controller 103 performs drawing processing such as animation, and generally handles data in the RGB format (RGB888 or RGB555).

  The macros 104-1 to 104-n include functions such as JPEG and MPEG, and perform compression / decompression processing set for each. The memory 105 stores image data, and the memory controller 106 performs R / W control of the memory 105.

  In such an image processing system 100, a process of superimposing images having different formats output by the plurality of macros 104-1 to 104-n and generating a final output image is frequently performed. The superimposing process is, for example, a process of superimposing character / graphic data on the background image data.

  The following (a) to (d) show the flow of conventional image superposition processing. 2 shows a process of superimposing two pieces of YUV format image data on the image processing system 100 to generate one piece of RGB image data.

  (A) The format converter 102 reads the first YUV format image data from the area A of the memory 105, and converts the data of one frame into the RGB image data format. The format converter 102 has a function of performing format conversion on the entire rectangular area (for example, the entire screen) surrounded by the start address and end address of the image. Then, the format converter 102 writes the format-converted RGB image data back to the area A of the memory 105.

  (B) The format converter 102 reads the second YUV format image data from the area B of the memory 105 and converts the data of the entire frame into the RGB image data format. Then, the format converter 102 writes the format-converted RGB image data back to the area B of the memory 105.

  (C) The CPU 101 (or graphic controller 103) reads out the data processed in (a) from the memory 105, performs superimposition determination as to whether or not to superimpose pixels for each pixel, and then stores the data to be superimposed on the memory. 105 is written back into area C (output image area).

  (D) The CPU 101 (or the graphic controller 103) reads out the data processed in (b) from the memory 105, performs the same superimposition determination for each pixel, and then transmits the data to be superimposed on the area C (output) of the memory 105. Write back to the image area. By such processing, one piece of RGB image data in which two pieces of YUV format image data are superimposed is generated.

On the other hand, as a conventional image superimposition processing technique, a technique has been proposed in which a portion where character / graphic data is superimposed on image data is replaced with a simple image before synthesis to reduce the burden of preprocessing of image data ( For example, Patent Document 1).
Japanese Patent Laid-Open No. 07-131632 (paragraph numbers [0012] to [0016], FIG. 1)

  When the image processing system 100 performs the operation of superimposing images of different formats, the entire image data (for one screen) is first converted in format, and the data is converted into software (CPU 101) or hardware (graphics). After the controller 103) reads out and determines whether or not the data is to be superimposed, it is necessary to repeat a series of operations for writing the necessary data back to the output image area by the number of superimposed images.

  As described above, conventionally, the format conversion process and the superimposition determination process are executed by separate components, and therefore, data exchange between the components is necessary. In the conventional format converter 102, In addition, the entire rectangular area surrounded by the start address, end address, and the like of the image did not have a function of performing format conversion only on the designated superimposition area in the rectangular area.

  As a result, when image superimposition processing is conventionally performed, format conversion must be performed up to an unnecessary image region on the finally generated superimposition screen, which significantly increases processing time and power consumption. There was a problem. Further, at the present time when the image data to be handled has become enormous, the memory capacity to be used will also increase, resulting in a large cost burden.

  Further, in the above-described conventional technique (Japanese Patent Laid-Open No. 07-131632), the burden of compression / decompression processing such as JPEG is reduced by painting an image area where characters and figures overlap with a certain color. However, it is not necessarily an optimal process for improving the processing efficiency of the entire image processing because it performs a complicated process of extracting the outline of the superimposed region and filling it with a certain color.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an image processing apparatus that performs format conversion only on a region necessary for a superimposed image to improve image processing efficiency.

  In the present invention, in order to solve the above-described problem, in the image processing apparatus 1 that processes a plurality of image data as shown in FIG. 1, it is a necessary area for a superimposed image when a plurality of image data is superimposed. A threshold value for determining whether or not the image area is selected, and based on the threshold value, an overlapped image area selecting unit 12 that selects an image area, and the selected image area is calculated by a conversion coefficient to obtain image data. A format conversion unit 10 configured to perform the format conversion, and an image superimposition unit 20 that writes a plurality of image data after format conversion to a memory, generates a superimposed image, and outputs the superimposed image. An image processing apparatus 1 is provided.

  Here, the superimposed image region selection unit 12 sets a threshold value for determining whether the region is necessary for the superimposed image when a plurality of image data is superimposed, and based on the threshold value, Select the image area. The arithmetic processing unit 13 performs the format conversion of the image data by calculating the selected image area using the conversion coefficient. The image superimposing unit 20 writes the plurality of image data after the format conversion to the memory, generates a superimposed image, and outputs it.

  The image processing apparatus of the present invention sets a threshold value for determining whether or not a region is necessary for a superimposed image when a plurality of image data are superimposed, and the image region is determined based on the threshold value. The selected image area is calculated using a conversion coefficient, and the format conversion of the image data is performed. Then, a plurality of image data after format conversion is written in a memory, and a superimposed image is generated and output. As a result, format conversion can be performed only in a region necessary for the superimposed image, so that it is possible to reduce processing time, power consumption, and memory capacity in the image superimposing process, and to improve image processing efficiency.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a principle diagram of an image processing apparatus according to the present invention. The image processing apparatus 1 includes a format conversion unit 10 and an image superimposing unit 20, and is an apparatus that superimposes image data at high speed.

  The format conversion unit 10 includes a superimposed image region selection unit 12 and an arithmetic processing unit 13. The superimposed image area selection unit 12 sets a threshold value for determining whether or not the area is necessary for the superimposed image when superimposing a plurality of image data, and determines the image area based on the threshold value. Sort out. The arithmetic processing unit 13 calculates the selected image area using a conversion coefficient, and performs format conversion of the image data (YUV format → RGB format, etc.).

  The image superimposing unit 20 writes the plurality of image data after the format conversion to the memory, generates a superimposed image, and outputs it. In the example shown in the figure, a single superimposed image D3 is generated by superimposing (superimposing) a character / graphic image D2 on the background image D1.

  FIG. 2 is a diagram illustrating the overall configuration of the image processing apparatus 1. The image processing apparatus 1 includes a format conversion unit 10 and an image superimposition unit 20, and the format conversion unit 10 includes an input buffer 11, a superimposed image region selection unit 12, an arithmetic processing unit 13, and an output buffer 14.

  The input buffer 11 captures and holds input image data to be subjected to format conversion placed on a memory in the apparatus. The superimposed image area selection unit 12 includes a threshold value table T1, and the threshold value table T1 holds threshold information for selecting image data to be input to the arithmetic processing unit 13.

  The arithmetic processing unit 13 includes a conversion coefficient table T2, and the conversion coefficient table T2 holds conversion coefficient information for format conversion. The output buffer 14 holds data output from the arithmetic processing unit 13. The image superimposing unit 20 has a memory 21, takes in image data from the output buffer 14 into the memory 21, and stores the superimposed image in the memory 21.

  Next, the configuration and operation of the image processing apparatus 1 of the present invention will be described in detail below. As the format conversion, conversion from YUV format to RGB format is performed.

  FIG. 3 is a diagram showing the configuration of the input buffer 11 and the superimposed image region selection unit 12. The image data is composed of three color components, a first color component, a second color component, and a third color component. In the present invention, each of the three color components is processed as Y, Cr, and Cb.

  The input buffer 11 includes a Y storage area 11a, a Cr storage area 11b, and a Cb storage area 11c. The superimposed image region selection unit 12 includes a threshold table T1, a Y component comparator 12a-1, a Cr component comparator 12a-2, a Cb component comparator 12a-3, and a determination circuit 12b.

For the input buffer 11, the Y storage area 11a, the Cr storage area 11b, and the Cb storage area 11c store Y data, Cr data, and Cb data, respectively.
The threshold value table T1 is realized as a part of a register group. Regarding Y data, a Y lower limit register r1a for setting a lower limit threshold value for Y data and a Y upper limit register r1b for setting an upper limit threshold value for Y data. Consists of

  Similarly, the Cr data includes a Cr lower limit register r2a for setting a lower limit threshold for Cr data and a Cr upper limit register r2b for setting an upper limit threshold for Cr data. For Cb data, the lower limit for the Cb data is set. It consists of a Cb lower limit register r3a for setting a threshold and a Cb upper limit register r3b for setting an upper limit threshold for Cb data.

  The Y component comparator 12a-1 compares and determines whether or not the Y data output from the Y storage area portion 11a falls within the range of the Y upper limit threshold and the Y lower limit threshold. Output data. The Cr component comparator 12a-2 compares and determines whether or not the Cr data output from the Cr storage area unit 11b falls within the range of the Cr upper limit threshold and the Cr lower limit threshold. Output data. The Cb component comparator 12a-3 compares and determines whether or not the Cb data output from the Cb storage area unit 11c falls within the range of the Cb upper limit threshold and the Cb lower limit threshold. Output data.

  The determination circuit 12b receives Y data, Cr data, and Cb data and the respective determination results, and all the three determination results indicate that the Y data, Cr data, and Cb data are within the threshold range. If it is, enable is asserted, and Y data, Cr data, and Cb data are transmitted to the arithmetic processing unit 13 at the next stage.

  Next, the operation until the input image is determined to be superimposed will be described with reference to FIG. 3 (the operation after the arithmetic processing unit 13 that performs format conversion will be described with reference to FIG. 4). Specifically, when an image in YUV format (YUV420 or YUV422) is converted to RGB888 data and superimposed, only a black character portion written on a blank sheet is superimposed on a certain photographic image as a background. An operation example will be described below.

  Since the image processing is performed for each of the background image and the character data, first, the superimposition determination processing for the photographic image as the background will be described. When it is desired to convert the format of the entire image like a background image, the six upper limit / lower limit registers for each color component are set as follows so that all input data is to be converted.

  Y lower limit register r1a = 0x00, Y upper limit register r1b = 0xff, Cr lower limit register r2a = 0x00, Cr upper limit register r2b = 0xff, Cb lower limit register r3a = 0x00, and Cb upper limit register r3b = 0xff.

  Each data of Y, Cr, Cb stored in the input buffer 11 can take various values from 0x00 to 0xff, but if the lower and upper thresholds are set as described above, all Will fall within the threshold.

  Then, the Y component comparator 12a-1, the Cr component comparator 12a-2, and the Cb component comparator 12a-3 have OK determination results (data falls within the threshold value) for all data. When the determination circuit 12b receives the three determination results and the data of Y, Cr, and Cb, since all the determination results are OK, the asserted enable signal and the data of Y, Cr, and Cb are next processed. To the stage arithmetic processing unit 13.

  Next, the superimposition determination of the character portion written on the blank paper will be described. The white and black color components have the following characteristics as Y, Cr, and Cb formats. The characteristics of the white color Y, Cr, and Cb data include values close to Y: 0xff, values close to Cr: 0x80, and values close to Cb: 0x80.

  Further, the characteristics of black Y, Cr, and Cb data have values close to Y: 0x00, values close to Cr: 0x80, and values close to Cb: 0x80. Therefore, in order for only the black color (character portion) to be subject to format conversion, the six upper / lower limit registers for each color component can be set as follows, for example.

  Y lower limit register r1a = 0x00, Y upper limit register r1b = 0x10, Cr lower limit register r2a = 0x78, Cr upper limit register r2b = 0x88, Cb lower limit register r3a = 0x78, Cb upper limit register r3b = 0x88.

  In such a threshold setting, for example, when white data (Y = 0xff / Cr = 0x80 / Cb = 0x80) is taken into the input buffer 11, Cr and Cb are within the threshold. An OK judgment is issued, but Y does not fall within the threshold value. Therefore, in this case, enable is not asserted from the determination circuit 12b.

  On the other hand, when black data (Y = 0x00 / Cr = 0x80 / Cb = 0x80) is taken into the input buffer 11, since all the color components are within the threshold value, the determination circuit 12b is enabled. The black data is transmitted to the arithmetic processing unit 13 at the next stage.

  Next, operations after the arithmetic processing unit 13 will be described. FIG. 4 is a diagram for explaining operations of the arithmetic processing unit 13, the output buffer 14, and the image superimposing unit 20. When the enable signal from the determination circuit 12b is asserted, the product-sum operation unit 13a in the operation processing unit 13 is activated, and the format conversion operation is performed using the conversion coefficient table T2.

  When conversion from the YUV format to the RGB format is performed, conversion coefficients defined in ITU-R BT.601 are set in the conversion coefficient table T2 (note that conversion coefficients stored in the conversion coefficient table T2 are stored). May be rewritable from the outside). Then, the product-sum operation unit 13a performs the product-sum operation using the following formulas (1a) to (1c) including the conversion coefficient, so that the conversion from the YUV format to the RGB format (RGB888 data) is executed. The

  The arithmetic processing unit 13 performs the above-described calculation on the background image and the character image, outputs the background image RGB888 data, and outputs the character image RGB888 data. The output buffer 14 receives these image data and stores them in the FIFO format for each of R, G, and B pixels.

  The image superimposing unit 20 writes the image data output from the output buffer 14 in a designated area in the memory 21. Here, a background image and a character image are written in the output image area 21a, thereby generating a superimposed image in which characters are superimposed on the background image. Note that the memory 21 has a plurality of configurations, and a plurality of superimposed images can be stored corresponding to each memory surface.

  As described above, according to the present invention, only necessary data is format-converted (all background images are format-converted and only character data is format-converted as an image superimposed on the background) and written to the memory. Can do. Thereby, it is possible to improve the efficiency of the image superimposition processing.

  In addition, since only the superimposition data is written in the memory 21 (unnecessary data not related to the superimposition is not written), the output image area 21a can be directly designated as the data output area, and the storage address can be separately specified by an output address register or the like. It becomes possible to specify and output.

  In the above description, an example in which a color component threshold value register is prepared in the threshold value table T1 for one superimposed image and the superimposed image processing is performed has been described. However, the threshold value table T1 is set. The threshold value to be set can be arbitrarily set from the outside. For this reason, in order to generate a plurality of different superimposed images, it is possible to generate a plurality of superimposed images by setting threshold values corresponding to the images one by one.

  Next, a mobile phone to which the image processing apparatus 1 of the present invention is applied will be described. FIG. 5 is a diagram showing the configuration of the mobile phone of the present invention. The mobile phone 3 includes a camera 3 a and includes a menu display unit 31, an image storage unit 32, and an image processing unit 33.

  The menu display unit 31 has a user interface function, and displays a mode selection menu including a superposition mode for generating a superposition image to the user. The image storage unit 32 includes a background image storage unit 32a and a character / graphic image storage unit 32b. The background image storage unit 32a stores a background image captured by the camera 3a. The character / graphic image storage unit 32b stores a character / graphic image captured by the camera 3a. The background image storage unit 32a and the character / graphic image storage unit 32b actually correspond to different storage areas on the same memory. The image processing unit 33 has a function corresponding to the image processing apparatus 1 of the present invention.

Here, a series of flow of creating a superimposed image using the mobile phone 3 is shown in the following (a) to (f).
(A) The user draws a character or a picture (character / graphic image D2) with a black pen on white paper.

(B) The menu display unit 31 of the mobile phone 3 is set to a mode for superimposing an image.
(C) The user takes an image (background image D1) to be used as a background with the camera 3a of the mobile phone 3, and the background image storage unit 32a stores the background image D1.

(D) A user photographs paper (character / graphic image D2) on which characters and pictures are drawn with the camera 3a of the mobile phone 3, and the character / graphic image storage unit 32b stores the character / graphic image D2.
(E) The superimposing mode of the mobile phone 3 operates (the image processing unit 33 operates), and one image (superimposed image D3) in which characters and pictures are superimposed on the background image is completed.

(F) The user uses the completed image as wallpaper, or attaches it to an email and sends it to the other party.
As described above, according to the present invention, the format conversion unit 10 is provided with a function for performing superimposition determination based on a set threshold value, and is configured to perform format conversion only on a region necessary for a superimposed image. As a result, superposition determination and format conversion are performed within one component, and the minimum necessary format conversion is performed, so that it is possible to reduce the image processing time.

  In addition, as an operation flow when the function of the present invention is used in the image processing system 100 of FIG. 7, the first YUV format image data is read out, and superimposition determination based on a threshold is performed, and only the superimposition data is formatted. Conversion is performed and the output image area on the memory 105 is written. Similarly, the second YUV format image data is read out, superposed by a threshold value, format conversion is performed only on the superposed data, written in the output image area on the memory 105, and the image is superimposed. From such an operation, it can be seen that the image processing time is greatly reduced in the present invention as compared with the prior art.

On the other hand, in the present invention, since the image format conversion and the writing to the memory are performed only for the superimposed data, the memory capacity and the power consumption can be greatly reduced.
In the above description, YUV → RGB is used as the format conversion, but it is also possible to perform the RGB → YUV format conversion by performing the inverse operation of the matrix calculation of the arithmetic processing unit 13. Also, the case where a single superimposed image is generated by superimposing two images has been shown, but if a register for setting a threshold value corresponding to a plurality of images is provided, the basic control remains unchanged. In addition, a single superimposed image can be generated by superimposing a plurality of images.

  Furthermore, although the example which applied this invention to the mobile telephone was shown above, it is applicable not only to a mobile telephone but to the apparatus which has various image processing functions. For example, for a system having a graphic controller, a superimposed image can be generated by superimposing a plurality of drawing areas as animation drawn by the graphic controller. Also, the present invention can easily perform processing such as taking out only the portion drawn in red from the animation drawn by the graphic controller and overlaying it with the natural image.

1 is a principle diagram of an image processing apparatus according to the present invention. It is a figure which shows the whole structure of an image processing apparatus. It is a figure which shows the structure of an input buffer and a superimposition image area | region selection part. It is a figure for demonstrating operation | movement of an arithmetic processing part, an output buffer, and an image superimposition part. It is a figure which shows the structure of the mobile telephone of this invention. It is a figure which shows the classification table of a YUV format. It is a figure which shows the image processing system which performs format conversion.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Format conversion part 12 Superimposed image area | region selection part 13 Operation processing part 20 Image superimposition part D1 Background image D2 Character and figure image D3 Superimposed image

Claims (4)

In an image processing apparatus that processes a plurality of image data,
A superimposed image region selecting unit that sets a threshold value for determining whether or not a region is necessary for a superimposed image when a plurality of image data are superimposed, and selects an image region based on the threshold value; A format conversion unit comprising: an arithmetic processing unit that calculates the selected image region by a conversion coefficient and performs format conversion of the image data;
An image superimposing unit that writes a plurality of image data after format conversion to a memory, generates a superimposed image, and outputs;
An image processing apparatus comprising:   The superimposed image region selection unit sets a lower limit threshold and an upper limit threshold for each of the first color component, the second color component, and the third color component constituting the image data, and the three color components are The image processing apparatus according to claim 1, wherein only an image area that falls within a threshold range is determined as an image area to be superimposed. In a format conversion device that performs format conversion of image data,
When superimposing multiple image data, set a threshold value to determine whether the area is necessary for the superimposed image, and configure the image data when selecting the image area based on the threshold value A lower limit threshold and an upper limit threshold are set for each of the first color component, the second color component, and the third color component, and an image area in which the three color components fall within the threshold range, A superimposed image region selection unit that determines an image region to be superimposed;
An arithmetic processing unit that performs a format conversion of the image data by calculating the selected image area using a conversion coefficient;
A format conversion apparatus comprising: In a mobile phone having a camera function,
A menu display unit for displaying a mode selection menu including a superposition mode for generating a superposition image to the user;
An image storage unit including a background image storage unit that stores a background image captured by the camera, and a character / graphic image storage unit that stores a character / graphic image captured by the camera;
When superimposing a background image and a character / graphic image, a threshold is set for determining whether the area is necessary for the superimposed image, and the image area is selected when selecting the image area based on the threshold. An image in which the lower limit threshold and the upper limit threshold are set for each of the first color component, the second color component, and the third color component constituting the data, and the three color components fall within the threshold range. Superimposed image area selection unit that determines an area as an image area to be superimposed, an arithmetic processing unit that calculates the selected image area using a conversion coefficient, and converts the YUV format and RGB format of the image data, after format conversion An image processing unit including an image superimposing unit that writes a background image and a character / graphic image into a memory and generates and outputs a single superimposed image in which the character / graphic image is superimposed on the background image;
A mobile phone characterized by comprising:

Images