JP2005182412A - Image processor and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically set an appropriate parameter value corresponding to moving image data and to generate and output images whose resolution is increased appropriately without putting burdens on a user. <P>SOLUTION: An image processor is provided with a means for selecting a frame (called "support frame" hereafter) to be composited with a base frame on the basis of the degree of image change from the base frame, a means for executing a compositing processing on the basis of the image data (called "base image data" hereafter) of the base frame and the image data of the support frame and generating the image data (called "composite image data" hereafter) in which the number of pixels is larger than the base image data, and a means for executing the output processing of the base frame on the basis of the composite image data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing technique for increasing the resolution of image data, and more particularly to an image processing technique suitable for outputting a low-resolution moving image.

  In recent years, devices that can easily shoot moving images, such as mobile phones and digital cameras, are becoming widespread, in addition to devices that mainly use moving images such as video cameras.

  The resolution (size) of moving images that can be shot with such a device is, for example, QCIF size (176 × 144 pixels), QVGA size (320 × 240 pixels), etc. There is a current situation that it cannot be said that the resolution (large size) is sufficiently higher than the resolution.

  In addition, services that distribute moving images via a communication network are becoming common, but there are situations where it is actually difficult to distribute high-resolution moving image data due to problems such as communication speed.

For this reason, a technique has been proposed and developed for generating high-resolution data by combining low-resolution moving image data when displaying or printing based on captured or distributed moving image data. (See Patent Document 1).
JP 2000-194845 A

  Many techniques for synthesizing moving image data to achieve high resolution involve many parameters (for example, output resolution, number of synthesized frames, etc.) in the processing process, and appropriately increase the resolution. Therefore, it is necessary to set appropriate values according to the moving image data for these parameters.

  However, since it is very difficult for general users to set parameter values appropriately according to individual moving image data, some users have to use the default parameter values fixedly. There are many problems that it cannot be said that high resolution technology is fully utilized.

  Accordingly, an object of the present invention is to automatically set an appropriate parameter value according to moving image data, and to generate and output an image with high resolution appropriately without imposing a burden on the user.

  The image processing apparatus according to the present invention includes means for selecting a frame to be combined with a base frame (hereinafter referred to as “support frame”) based on the degree of image change from the base frame, image data of the base frame, and support frame Image data having a larger number of pixels than the base image data (hereinafter referred to as “composite image data”) is executed based on the image data (hereinafter referred to as “base image data” and “support image data”). And a means for executing a base frame output process based on the composite image data.

  According to such a configuration, it is possible to automatically determine the number of frames to be used for synthesis while automatically selecting an appropriate support frame based on the image change, and the user can select the number of frames to be used for selection and synthesis of the support frame by himself / herself. Therefore, it is possible to execute printing or the like by increasing the resolution easily and appropriately without having to make the determination.

  Preferably, the selecting means selects the support frame so that the number of selections is equal to or less than a predetermined upper limit value. Further preferably, the selecting means selects the support frame so that a frame interval between the base frame and the support frame is equal to or less than a predetermined upper limit value. The upper limit value is preferably determined based on the number of pixels of the composite image data.

  According to such a configuration, the support frame can be appropriately selected within a range in which the image quality is improved by the synthesis without setting or operating by the user himself / herself.

Preferably, the image processing apparatus includes means for determining whether or not to perform the combining process on the base combined data based on the degree of blur of the base image data. According to such a configuration, the combining can be performed without setting or operating the user himself / herself. If there is a possibility that it cannot be properly performed, the synthesis process can be stopped appropriately.

  An image forming apparatus according to the present invention includes the image processing apparatus according to the present invention and means for forming an image on a medium based on image data.

  The image processing method of the present invention includes a step of selecting a frame to be combined with a base frame (hereinafter referred to as a “support frame”) based on the degree of image change with the base frame, and image data of the base frame and the support frame. Image data having a larger number of pixels than the base image data (hereinafter referred to as “composite image data”) is executed based on the image data (hereinafter referred to as “base image data” and “support image data”). And a step of executing a base frame output process based on the composite image data.

  The image processing method of the present invention can be implemented by a computer, and a computer program therefor can be installed or loaded on the computer through various media such as a CD-ROM, a magnetic disk, a semiconductor memory, and a communication network. . It also includes the case where a computer program is recorded and distributed on a printer card or printer option board.

  In the present specification, the function of one “means” may be realized by two or more physical means, or the functions of two or more “means” may be realized by one physical means. .

  According to the present invention, it is possible to automatically set an appropriate parameter value according to moving image data, and to generate and output an image with an appropriately high resolution without imposing a burden on the user.

  Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a hardware configuration of the printer 1 according to the present embodiment.

  The printer 1 includes a power supply mechanism 2 that includes a paper feed mechanism 10 that supplies paper into the printer, a print engine 11 that performs printing, and a paper discharge mechanism 12 that discharges paper to the outside of the printer. The print engine 11 includes a paper feed mechanism, a carriage mechanism, a print head, and the like.

  The power mechanism unit 2 is controlled to perform a printing operation by a CPU (processor) 13, a ROM 14, a RAM 15, a display panel and panel controller 16, a PC card slot and a PC card controller 17, an interface 18 such as a USB, and the like. It is the control unit 3. CPU13 is comprised so that each means 14-18 can be accessed via a bus | bath. The power mechanism unit 2 may be provided with a CPU independently. In this case, the CPU of the power mechanism unit 2 communicates with the CPU 13 via a parallel interface or the like to control the print engine 11 to perform printing. It will cause the action to be performed.

  The configuration and operation of the power mechanism unit 2 and the control unit 3 are basically the same as the configuration and operation of the conventional printer. For example, the printer 1 receives a print job from the host device via the interface 18, or receives image data from an external memory via the PC card slot 17 or the like / from an information device such as a digital camera via the interface 18. A function of directly reading and controlling the power mechanism unit 2 based on the print job or image data to perform a printing operation is provided.

  However, the printer 1 has a function of decompressing moving image compressed data, and further selects a frame to be combined with the base frame (hereinafter referred to as “support frame”) based on the degree of image change from the base frame. Then, based on the image data of the base frame and the image data of the support frame (hereinafter referred to as “base image data” and “support image data”, respectively), the synthesis process is executed (the pixel value interpolation process is executed), and the base This is different from the conventional configuration in that it has a function of generating image data having a larger number of pixels than image data (hereinafter referred to as “composite image data”).

  In FIG. 2, the main function block diagram in the control part 3 is shown. As shown in the figure, the control unit 3 includes a panel IF unit 20, an expansion unit 21, a synthesis unit 22, a print control unit 23, and the like. Each means described above is functionally realized by the CPU 13 executing an application program stored in the ROM 14 or the RAM 15. The control unit 3 can include image processing (size conversion processing, noise removal, sharpness, etc.) included in a general printer in addition to the above-described units.

  The panel IF unit 20 receives, for example, a print control command for the printer 1 from the user via the display panel including the LCD and operation keys and the panel controller 16 and outputs a print status and the like to the user.

  The decompression means 21 performs decompression processing or the like on the compressed image data (JPEG data, motion JPEG data, MPEG data, etc.), restores the image data for each frame, and stores it in a predetermined area of the RAM 15. For the decompression process, a known process corresponding to the compression method can be used.

  The synthesizing unit 22 selects the support frame to be synthesized with the base frame based on the degree of image change from the base frame for the moving image data (temporally continuous image data), and the base image data and the support image Composition processing is executed based on the data, and image data having a larger number of pixels than the base image data (hereinafter referred to as “composite image data”) is generated. Note that a large number of pixels means that the resolution (number of pixels per unit area) is high when outputting to the same size, and the size is large (expanded) when outputting at the same resolution. It will be said.

  The print control means 23 executes the analysis of the print job, the generation of the print image, the control of the print engine 11 and the like as in the conventional printer apparatus.

  For example, based on a print request sent from an information device such as a host device or a digital camera, print data is sequentially read from the reception buffer and analyzed to generate raster format image data. When the print data includes compressed image data, the decompression unit 21 is controlled to restore the image data. Besides the external print request, for example, based on a print instruction command received by the panel IF unit 20, the compressed image data is read from the external memory or the like via the PC card slot 17, and the decompression unit 21 is controlled. To restore the image data.

  At this time, for example, when the print request includes designation of moving image compression data and automatic composite printing, or when the print instruction command is received in the state where the moving image compression data and automatic synthetic printing are designated in the panel IF unit 20 The print control unit 23 controls the decompressing unit 21 and the synthesizing unit 22 to generate synthesized image data based on the compressed moving image data.

  Then, a printing process (output process) is executed based on the restored image data or the generated composite image data. Specifically, predetermined image processing (for example, noise removal) that is normally performed in the printing operation is performed on the image data or the composite image data to generate a print image. Further, based on a control command read from the reception buffer or a command input via the panel IF means 20, for example, the paper feed mechanism of the print engine 11 is controlled to prepare a state necessary for starting printing, and a predetermined unit. The print image for one minute (for example, one pass) is transferred to the print engine 11, and printing is executed while controlling the print engine 11.

(Automatic synthesis processing)
Hereinafter, the operation of the synthesizing means 22 will be described in detail with reference to the flowcharts shown in FIGS. Each step can be executed in any order or in parallel within a range that does not cause a contradiction in processing contents.

  In the present embodiment, the processing will be described on the assumption that the compressed image data included in the print data or the compressed image data read from the external memory or the like is the compressed motion image data of the motion JPEG method. The motion JPEG method is a method for realizing moving image display by continuously reproducing a color still image compressed based on the JPEG method, which is an international standard for color still image coding. Each JPEG data is included.

  First, the synthesizing unit 22 selects a base frame that is a reference for synthesis (step S100).

  The base frame is selected based on frame designation data in a print request sent from a host device, a digital camera, or the like, or based on frame designation data received by the panel IF unit 20.

  In this case, for example, a moving image is displayed on the host device or the digital camera, or if the printer 1 has a moving image display function, the moving image is displayed on the printer 1 and the user stops the moving image at a desired scene (or It is conceivable to specify a frame by selectively displaying a still image (by specifying a frame number).

  Next, the synthesizing unit 22 performs an expansion process on the base frame to generate base image data (step S101). Specifically, for JPEG data corresponding to the base frame read from the print data in the reception buffer or from an external memory, the Huffman code is decoded for the DC component and the Huffman code is decoded for the AC component every 8 × 8 blocks. The base image data is generated by performing run-length compression decoding, performing inverse quantization on each decoded component to calculate frequency components, and performing inverse DCT on the frequency components.

  Next, the synthesizing means 22 is based on the number of pixels (resolution) of the base image data and the number of pixels of the synthesized image data in the print request sent from, for example, a host device or a digital camera, or the panel IF means. An enlargement ratio is obtained based on the number of pixels of the composite image data received at 20 (step S102).

  The enlargement ratio is obtained based on, for example, (number of vertical pixels of composite image data) / (number of vertical pixels of base image data) × (number of horizontal pixels of composite image data) / (number of horizontal pixels of base image data). Can do. When the number of pixels of the composite image data is not specified, for example, a default value such as a VGA size (640 pixels × 480 pixels) is used as the number of pixels of the composite image data, or the enlargement ratio itself is, for example, (2 × 2) A default value such as double may be used.

  Next, the synthesizing unit 22 determines whether or not the enlargement ratio is greater than 1 (step S103). This is because the composition process is intended to increase the number of pixels (increase the resolution) in the first place, and is not targeted when the enlargement ratio at which the number of pixels does not change or decreases is 1 or less.

On the other hand, when the enlargement ratio is greater than 1, the synthesizing unit 22 determines, based on the enlargement ratio, the parameter N _max indicating the upper limit value of the number of support frames and the parameter D indicating the upper limit value of the frame interval between the base frame and the support frame. _max is set (step S104).

Specifically, for example, a table as shown in FIG. 5 may be stored in the RAM 15 and set by referring to this table. As can be seen from the figure, in this table, the correspondence relationship is determined such that N _max and D _max take larger values as the enlargement ratio increases. The number of support frames that can contribute to improving the image quality of the composite image data and the frame interval between the base frame and the support frame are considered to have an upper limit, while the upper limit value is the size of the composite image data (more specifically, the base This is because it is considered to depend on the enlargement ratio from the image data and the support image data.

  Next, the synthesizing unit 22 sets an initial value 0 to a parameter N indicating the number of selected support frames and a parameter M indicating the number of selected candidate frames (step S105).

  Next, the synthesizing unit 22 measures the degree of blur of the base image data (step S106). The degree of blur can be measured by, for example, the average value of the edge strength, and the edge strength can be calculated using the output value of a conventional edge detection filter such as Sobel.

  Next, the synthesizing unit 22 determines whether or not to perform the synthesizing process on the base synthesized data based on the degree of blur of the base image data (step S107). Specifically, when the degree of blur is equal to or greater than a certain limit (for example, the average value of edge strength is equal to or greater than a predetermined threshold (for example, 32)), the composition process is set to “No” and the process is stopped. This is because if the base image data is blurred, the degree of image change with respect to the candidate frame cannot be accurately measured in the subsequent process, and the support frame may not be selected appropriately. Further, the image change with respect to the support frame cannot be accurately measured, and there is a possibility that it cannot be appropriately combined. In this case, it is conceivable that the print control unit 23 performs the sharpening process on the base image data as necessary to execute the printing process.

  On the other hand, when the degree of blur is not more than the limit, the synthesizing unit 22 sets “OK” for the synthesizing process and selects a candidate frame (step S108).

  As a method for selecting candidate frames, for example, the frame interval D between the base frame and the support frame is set to D = (INT (M / 2) +1) so that the preceding and succeeding frames are alternately selected based on the base frame. When M is 0 or an even number, a D frame is left in the forward direction (time axis + direction), and when M is an odd number, a D frame is left in the reverse direction (time axis-direction) to select a candidate frame. To. At this time, if there is no such candidate frame, the process proceeds to step S115, and the process may be shifted to the synthesis process. Note that INT (X) is a function that gives X a value obtained by rounding down the decimal point to an integer.

  Next, the synthesizing unit 22 performs a decompression process on the selected candidate frame to generate candidate image data (step S109).

  Next, the synthesizing unit 22 measures the degree of image change between the base frame and the candidate frame (step S110). Here, various methods are conceivable as image change measurement methods. In the present embodiment, a change amount d representing the degree of image change is obtained as follows.

Change amount d = (root mean square of pixel value difference of pixels at the same coordinates in each frame) ^1/2
Next, the synthesizing unit 22 determines whether the degree of image change (change amount d) is smaller than a predetermined threshold value α (step S111), and if it is equal to or larger than the threshold value α, selects a candidate frame as a support frame. Without proceeding to step S113.

  On the other hand, when the degree of image change is smaller than the threshold value α, the synthesizing unit 22 selects the candidate frame as a support frame and increments N by +1 (step S112).

  As will be described later, in the synthesis, pseudo base image data is obtained by reversely changing the support image data based on the image change between the base frame and the support frame. If the image change is large at this time, the base image The deviation between the data and the pseudo base image data also becomes large, and there is a possibility that adverse effects will be caused when they are combined. Therefore, in the present embodiment, the degree of image change is measured as described above, and is selected as a support frame only when it is smaller than the threshold value α. The threshold value α can be appropriately determined according to the image change measurement method. For example, when the image change is measured based on the change amount d as described above, α = 1 to about 10 (the pixel value is 0 to 255). The threshold value α may be configured to be adjustable based on a user instruction or the like.

  Next, the synthesizing unit 22 increments M by +1 (step S113).

Next, the synthesizing unit 22 determines whether (N ≦ N _max ) and (D ≦ D _max ) are satisfied (step S < _b > 114), and if satisfied, reverts to step S < _b > 107.

  On the other hand, if it does not hold, the synthesizing means 22 stops the selection of the support frame, executes the synthesizing process based on the support image data of the N support frames selected so far and the base image data. Composite image data corresponding to the frame is generated (step S115).

  For example, the synthesis process includes 1) obtaining pseudo base image data by reversely changing the support image data based on an image change from the base frame to the support frame, and 2) interpolation processing based on the base image data and each pseudo base image data. To obtain the pixel value of the composite image data with high resolution.

  The pseudo base image data can be obtained by the following method. First, as an image change from the base frame to the support frame, for example, an affine transformation parameter matrix is obtained so that an overlap between the support frame and an affine transformation of the base frame becomes large. Next, an inverse matrix of the obtained affine transformation parameter matrix is obtained as an image change (inverse change) from the support frame to the base frame. Next, based on the obtained inverse matrix, the support frame is subjected to affine transformation to obtain pseudo base image data (in this case, the coordinates of each pixel of the pseudo base image data are not necessarily integer values). Absent). The affine transformation parameter matrix and the inverse matrix may be obtained for each MCU (Minimum Coded Unit) of the base frame, for example.

As the interpolation process, a known method such as a nearest neighbor interpolation method or an Nth-order function interpolation method can be used. When using the nearest neighbor interpolation method, the coordinates (x ′, y ′) are obtained by multiplying the coordinates (x, y) of the base image data and each pseudo base image data by the vertical and horizontal enlargement rates, respectively, and the coordinates (x ′, y ′) of the composite image data ( x _high, as the pixel value of y _high), the coordinates (x _high, coordinates closest to _{y high) (x ', y} ' may be adopted pixel value corresponding to). N-order function interpolation methods include N = 1 bilinear (linear) interpolation method and N = 3 bicubic interpolation method. If the former is used, the coordinates (x _high , y _high ) may be interpolated using a linear function based on the pixel values of four coordinates (x ′, y ′) in the vicinity of the coordinates (x _high , y _high ), for example. If it is used, similarly, for example, interpolation may be performed using a cubic function based on pixel values of neighboring 16 coordinates.

As described above, in the present embodiment, the parameter N _max indicating the upper limit value of the number of support frames and the parameter D _max indicating the upper limit value of the frame interval between the support frame and the base frame are automatically determined according to the enlargement ratio, and the base image It is configured to automatically determine whether or not to perform combining processing based on the degree of data blur, and to automatically determine the number of frames used for combining while automatically selecting an appropriate support frame based on image changes. Therefore, the user can execute the printing or the like by increasing the resolution easily and appropriately without having to perform the above settings and operations.

(Modification)
The present invention is not limited to the above-described embodiment, and can be variously modified and applied.

  For example, in the above embodiment, the configuration in which the printer 1 includes the decompression function and the composition function is described. However, for example, an information device such as a host device or a digital camera may have the decompression function or the composition function. In this case, the configuration of the printer 1 can be a configuration in which the decompressed image data is received (read out) and combined and printed, or a configuration in which the combined image data is received (read out) and printed.

  Further, for example, when moving image compressed data is transmitted from a host device or the like, it is not necessary to transmit image compressed data for all frames, and only the image compressed data near the base frame may be transmitted. Or it is good also as a structure which transmits the image compression data of a corresponding frame according to the request | requirement from the printer 1 (side which has a synthetic | combination function).

  Further, for example, when the printer 1 has not only a printing function but also a function of displaying an image on a panel or the like as an image output function, high-resolution still image display may be performed based on the composite image data. High-resolution moving image display may be performed based on a plurality of temporally continuous composite image data.

  Further, for example, in the above embodiment, the previous and subsequent frames are alternately selected as candidate frames based on the base frame. For example, a unidirectional frame with a small degree of image change is selected as a candidate frame. You may comprise.

  Further, for example, in the above embodiment, the composition process is stopped when the degree of blur of the base image data is equal to or greater than the limit, but instead of stopping the composition process, for example, base image data, candidate image data, support image It may be configured that the sharpness process is performed on the data under the same conditions, and then the degree of image change or the image change is obtained.

  For example, in addition to / in place of the condition that the degree of image change (amount of change) is smaller than the threshold value α, a condition that the degree of blurring of the candidate image data is not greater than the limit may be used as the support frame selection condition.

  Further, for example, in the above embodiment, even when the image change amount between the base frame and the candidate frame is equal to or greater than the threshold value α, the support frame can be further selected after that. If the amount is greater than or equal to the threshold value α, it is considered that other candidate frames that are further away from the base frame than the candidate frame are more likely to have an image change amount that is greater than or equal to the threshold value α. Alternatively, the support frame selection process may be stopped at the stage where the frame is found, and the process may proceed to the synthesis process in step S115.

  Also, for example, in the above embodiment, the processing has been described for the motion JPEG format moving image compressed data, but the present invention is a moving image compressed data of a scheme that can obtain image data in units of frames, such as the MPEG scheme. If applicable, it is applicable. In the case of MPEG moving image compressed data as a target, since the motion information between frames is included in the compressed data, instead of obtaining the degree of image change (change amount d) as in the above embodiment, The degree of image change (change amount d) may be obtained using motion information. For example, it is conceivable to obtain the maximum value or average value of motion information in units of MCU (Minimum Coded Unit) in two frames as the change amount d.

  Finally, the present invention is not limited to application to a printer device, but a device having an image output function or a device that can be connected / communicated with such a device (for example, a digital camera, a mobile phone, a copier, a general-purpose computer, etc.) ).

FIG. 2 is a block diagram illustrating a hardware configuration of the printer 1 according to the embodiment of the present invention. 3 is a block diagram illustrating a functional configuration diagram of a control unit 3. FIG. 3 is a flowchart showing the operation of the combining means 22. 3 is a flowchart showing the operation of the combining means 22. It is a figure which shows the relationship between an enlarging rate and each upper limit.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Printer, 2 Power mechanism part, 3 Control part, 10 Paper feed mechanism, 11 Print engine, 12 Paper discharge mechanism, 13 CPU, 14 ROM, 15 RAM, 16 LCD panel and LCD controller, 17 PC card slot and PC card controller , 18 interface, 20 panel IF means, 21 expansion means, 22 composition means, 23 print control means

Claims (8)

Means for selecting a frame to be combined with the base frame (hereinafter referred to as “support frame”) based on the degree of image change with the base frame;
Based on the image data of the base frame and the image data of the support frame (hereinafter referred to as “base image data” and “support image data”), image data (hereinafter referred to as “base image data” and “support image data”) having more pixels than the base image data. , Referred to as “composite image data”),
Means for executing a base frame output process based on the composite image data.   The image processing apparatus according to claim 1, wherein the selecting unit selects a support frame so that a selection number is equal to or less than a predetermined upper limit value.   The image processing apparatus according to claim 1, wherein the selecting unit selects the support frame so that a frame interval between the base frame and the support frame is equal to or less than a predetermined upper limit value.   4. The image processing apparatus according to claim 2, wherein the upper limit value is determined based on the number of pixels of the composite image data.   5. The image processing apparatus according to claim 1, further comprising: a unit that determines whether or not to perform synthesis processing on the base composite data based on a degree of blur of the base image data. The image processing apparatus according to any one of claims 1 to 5,
Means for forming an image on a medium based on image data. Selecting a frame to be combined with the base frame (hereinafter referred to as “support frame”) based on the degree of image change with the base frame;
Based on the image data of the base frame and the image data of the support frame (hereinafter referred to as “base image data” and “support image data”), image data (hereinafter referred to as “base image data” and “support image data”) having more pixels than the base image data. , Referred to as “composite image data”),
And a step of executing a base frame output process based on the synthesized image data. A program for causing a computer to execute the image processing method according to claim 7.

Images