JP4097130B2 - Image processing apparatus, image processing method, electronic camera apparatus, program, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates generally to the field of image processing, and more particularly to a compression encoding technique for wide-angle images having different scaling factors depending on directions or areas.
[0002]
[Prior art]
As a method of recording or transmitting a wide-angle image, an electronic camera device such as a digital camera with a limited angle of view and a curved mirror or fisheye lens are used. There is a method of taking an image (with distortion), recording or transmitting the image, and removing the image distortion by image processing during image reproduction. As a specific example, a curved mirror having zero curvature in the vertical direction (horizontal direction) and a large curvature in the horizontal direction (vertical direction) is used to photograph an image with a wide angle of view in the horizontal direction (vertical direction). Or a 360-degree panoramic image using a hyperbolic convex mirror or a fish-eye lens as described in Japanese Patent No. 2939087.
[0003]
Another method is to shoot a scene by dividing it into a plurality of images using an electronic camera device such as a digital still camera, and from the plurality of images by image processing (such as planar projection) in the horizontal direction or the vertical direction. This is a method of synthesizing a wide-angle image with a wide angle of view or a 360 ° panoramic image. Also in this method, it is necessary to remove image distortion by image processing during image reproduction.
[0004]
In general, such wide-angle image data is compressed and encoded in advance and then recorded or transmitted.
[0005]
[Problems to be solved by the invention]
There is a problem that a wide-angle image having a variable magnification depending on the direction and area as described above is an image with a poor balance of resolution (image quality) when reproduced.
[0006]
For example, in Japanese Patent Application No. 2001-13802, the present applicant has proposed an imaging apparatus that captures a wide-angle image compressed (reduced) in the horizontal direction or the vertical direction. The principle will be described with reference to FIG. In the figure, M is a mirror, I is an imaging surface, and O is the center of the optical system. The mirror M is a mirror that has a straight cross section in the y-axis (perpendicular to the paper surface) direction and is bent only in the xz direction. The imaging surface I is orthogonal to the z-axis at a distance f from the optical center O. According to such an imaging system, a wide-angle image reduced (compressed) in the x direction (horizontal direction or vertical direction) can be captured.
[0007]
Here, on the imaging surface I, the light ray reaching the point p at the distance u from the z axis is traced in reverse. That is, if a point that extends a straight line connecting the point p and the optical center O and intersects the mirror M is q, the light ray is reflected to the object with respect to the normal line qq ′ of the mirror M at the point q and reaches the point r. The angle between the ray qr and the z-axis is θ. And
θ = αu
Thus, a natural wide-angle image can be reproduced only by the process of uniformly enlarging the captured image in the reduction direction, that is, the x direction.
[0008]
However, the captured image is simply compressed and encoded and transmitted or stored, and the decoded image is reproduced by performing the enlargement process, compared with the direction in which the resolution in the direction reduced at the time of imaging is not reduced. There was a problem that the resolution was low and the resolution balance was poor overall.
[0009]
Further, for example, as described in Japanese Patent No. 305416, a 360 ° panoramic image photographed using a hyperbolic convex mirror is an image of the camera itself in a central area A as shown in FIG. Is captured, and an image in which a useful subject is captured in an annular region B around the image is obtained. A planar 360 ° panoramic image C as shown in FIG. 10B can be reproduced from the image of the annular region B by image processing. However, in the annular region B of FIG. 10A, the inner peripheral side is greatly reduced (compressed) in the circumferential direction compared to the outer peripheral side, so that a 360 ° panoramic image as shown in FIG. When the 360 ° panoramic image C as shown in FIG. 10B is reproduced by simply compressing and encoding and transmitting or storing it, the resolution on the upper side (inner side) becomes lower (outer side). ), And the overall resolution is poorly balanced.
[0010]
Accordingly, an object of the present invention is to provide means for enabling reproduction of a wide-angle image with a good resolution balance as a whole.
[0011]
[Means for Solving the Problems]
In order to achieve the above object, the invention according to claim 1 is characterized in that the vertical direction is reduced / magnified at a larger magnification than the horizontal direction, or the horizontal direction is reduced / magnified at a larger magnification than the vertical direction. Input means for inputting the doubled wide-angle image data, and JPEG2000-compliant compression encoding processing for the wide-angle image data input by the input means. In the compression encoding processing, Compression coding means for preferentially assigning a code to a high-frequency component in a direction with a large scaling ratio of image data as compared with a high-frequency component in a direction with a small scaling ratio of the wide-angle image data. An image processing apparatus.
[0012]
According to the second aspect of the present invention, a wide-angle image in which the vertical direction is reduced / magnified with a larger magnification than the horizontal direction or the horizontal direction is reduced / magnified with a larger magnification than the vertical direction. Input means for inputting data, and JPEG2000-compliant compression encoding processing on the wide-angle image data input by the input means. In the compression-encoding processing, the wide-angle image data has a large scaling factor. And a compression encoding means for performing post-quantization for increasing the code amount of the high-frequency component in the direction as compared with the code amount of the high-frequency component in the direction with a small scaling ratio of the wide-angle image data. It is a processing device.
[0013]
According to a third aspect of the present invention, a wide-angle image in which the vertical direction is reduced / magnified at a larger magnification than the horizontal direction or the horizontal direction is reduced / magnified at a larger magnification than the vertical direction. An image pickup means for inputting data, and a JPEG2000-compliant compression encoding process on the wide-angle image data input by the image pickup means. In the compression-encoding process, the wide-angle image data has a large scaling factor. An image camera device comprising compression encoding means for preferentially assigning a code to a high frequency component in a direction as compared with a high frequency component in a direction with a small scaling ratio of the wide-angle image data.
[0014]
According to a fourth aspect of the present invention, a wide-angle image in which the vertical direction is reduced / magnified at a larger magnification than the horizontal direction or the horizontal direction is reduced / magnified at a larger magnification than the vertical direction. An image pickup means for inputting data, and a JPEG2000-compliant compression encoding process on the wide-angle image data input by the image pickup means. In the compression-encoding process, the wide-angle image data has a large scaling factor. And compression encoding means for performing post-quantization for increasing the code amount of the high-frequency component in the direction as compared with the code amount of the high-frequency component in the direction with a small scaling ratio of the wide-angle image data. It is a camera device.
[0015]
According to the fifth aspect of the present invention, a wide-angle image in which the vertical direction is reduced / magnified with a larger magnification than the horizontal direction or the horizontal direction is reduced / magnified with a larger magnification than the vertical direction. An input step for inputting data, and JPEG2000-compliant compression encoding processing on the wide-angle image data input in the input step, and in the compression-encoding processing, the wide-angle image data has a large scaling factor. And a compression encoding step for preferentially assigning a code to a high-frequency component in a direction as compared with a high-frequency component in a direction with a small scaling ratio of the wide-angle image data.
[0016]
The invention according to claim 6 is a wide-angle image in which the vertical direction is reduced / magnified at a larger magnification than the horizontal direction, or the horizontal direction is reduced / magnified at a larger magnification than the vertical direction. An input step for inputting data, and JPEG2000-compliant compression encoding processing on the wide-angle image data input in the input step, and in the compression-encoding processing, the wide-angle image data has a large scaling factor. And a compression coding step for performing post-quantization for increasing the code amount of the high-frequency component in the direction as compared with the code amount of the high-frequency component in the direction with a small scaling ratio of the wide-angle image data. It is a processing method.
[0018]
Since the resulting compressed code data by the present invention lever, the information lost prone particular direction of the high frequency components by scaling and stored preferentially as described above, overall resolution by extending it It is possible to reproduce an image with a good balance of (image quality).
[0019]
A seventh aspect of the invention is a program for causing a computer to execute each step of the image processing method according to the fifth or sixth aspect .
[0020]
The invention according to claim 8 is a computer-readable recording medium on which the program according to claim 7 is recorded.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
[0022]
FIG. 1 is a block diagram for explaining a first embodiment of the present invention. Further, FIG. 2 shows a processing flow in this embodiment.
[0023]
An image processing apparatus shown in FIG. 1 has an input device 100 (step in FIG. 2) for inputting wide-field-angle image data having a variable magnification depending on a direction or a region from a recording medium on which the image data is recorded or from an external device. S101), the compression encoding processing apparatus 101 (corresponding to step S102 in FIG. 2) for encoding and compressing the input image data, and the compression encoded data obtained thereby to an external apparatus via a communication line A transmission apparatus 102 (corresponding to step S103 in FIG. 2) for transmission and a recording apparatus 103 (corresponding to step S103 in FIG. 2) for recording the compressed encoded data on a recording medium are configured.
[0024]
In this embodiment, the compression encoding processing apparatus 101 is configured to perform compression encoding processing of image data using an algorithm compliant with JPEG2000 (ISO / IEC FCD 15444-1). The JPEG2000 compression encoding algorithm is described in detail in, for example, the book “Next Generation Image Encoding System JPEG2000” (by Yasuyuki Nomizu, Triqueps Co., Ltd.), and the outline thereof will be described below.
[0025]
FIG. 3 is a simplified processing flow for explaining the basic algorithm of JPEG2000. The image data to be processed is processed for each non-overlapping rectangular area called a tile. First, the image data of each tile is subjected to a DC level / color space conversion process 200. The DC level / color space conversion process 200 is intended to improve the compression ratio, and includes a level shift operation of input image data and a color space conversion process. Level shift is an operation of subtracting half of the dynamic range from an unsigned integer value such as an RGB value. When the input image data is YCrCb data, for example, level shift is not performed on signed integer values such as Cr and Cb components. The color space conversion process is a process of converting into YCrCb data when the input image data is RGB data or CMY data.
[0026]
After the DC level / color space conversion process 200, the image data of each tile of each component is subjected to a two-dimensional wavelet conversion process (discrete wavelet conversion: DWT) 202.
[0027]
FIG. 4 is an explanatory diagram of wavelet transformation when the number of decomposition levels is 3. FIG. By the two-dimensional wavelet transform on the tile image shown in FIG. 4A, the wavelet coefficients of the 1LL, 1HL, 1LH, and 1HH subbands shown in FIG. 4B are obtained. By applying the two-dimensional wavelet transform to the 1LL subband coefficient, the wavelet coefficients of the subbands 2LL, 2HL, 2LH, and 2HH are obtained as shown in FIG. By applying the two-dimensional wavelet transform to the coefficients of the 2LL subband, 3LL, 3HL, 3LH, and 3HH subband wavelet coefficients are obtained as shown in FIG.
[0028]
The coefficients of each subband obtained by recursive division (octave division) of such low-frequency components (LL subband coefficients) are subjected to entropy coding processing 206 after being subjected to quantization processing 204. . The deepest LL subband coefficient (3LL subband coefficient in the example of FIG. 4) is not encoded. In JPEG2000, both lossless compression (lossless compression) and lossy compression (lossy compression) are possible. In the case of lossless compression, the quantization step width is always 1, and at this stage, quantization is not substantially performed.
[0029]
The entropy encoding process 206 performs entropy encoding of wavelet coefficients. For this entropy coding, a block-based bit-plane coding method called EBCOT (Embedded Block Coding with Optimized Truncation) consisting of block division, coefficient modeling, and binary arithmetic coding is used. The wavelet coefficient to be encoded is an integer with a positive or negative sign (or an integer expressed as a real number), and scans them in a predetermined order, while the coefficients are expressed in absolute values, from the upper bits to the lower bits Encoding processing is executed in bit plane units.
[0030]
The codes generated by the entropy encoding unit 206 are combined into one code stream by the tag processing 208 and tag information is added. FIG. 5 shows a schematic structure of the code stream. As shown in FIG. 5, the main header describing the encoding parameters, the quantization parameters, etc. follows the SOC marker indicating the beginning of the code data, followed by the code for each tile. The code for each tile starts with an SOT marker, and includes a tile header, an SOD marker, and tile data (a tile image code string). An EOC marker indicating the end is placed after the last tile data.
[0031]
This JPEG2000 algorithm has good image quality at a high compression rate (low bit rate) and has many advantages.
[0032]
One of them is that the overall code amount (compression rate) can be adjusted without re-compression by quantization (post-quantization) by discarding (truncating: truncation) the code of the lower bit plane after encoding. Also, the compression ratios in the horizontal direction and the vertical direction can be individually adjusted. In JPEG2000, a coefficient bit plane is ordered for each area called a precinct, and a layer composed of one or two bit planes can be configured. When a multi-layer configuration is adopted, by performing code truncation in units of layers from a lower layer, a finer code amount adjustment is possible.
[0033]
The other is a processing function called selective area image quality improvement (ROI: Region of Interest) that raises the image quality of a selected area of an image over other areas. In the basic method of JPEG2000 (JPEG2000 Part 1), an ROI method is used in which the wavelet coefficient value of a selected area is shifted to the upper bit side before the wavelet coefficient is encoded, and the wavelet coefficient value outside the area is shifted to the lower bit side. Although adopted, ROI by quantizing a selected region with a smaller quantization step than other portions is also possible. With this ROI function, it is possible to reduce the overall code amount without degrading the image quality of the portion of interest in the image.
[0034]
Now, with reference to FIG. 1 and FIG. 2 again, the contents of processing by the compression coding processing apparatus 101 or step S101 will be described.
[0035]
The wide-angle image data input by the input device 100 (step S101) is scaled or scaled in the vertical direction (or horizontal direction) of the image at a normal ratio or substantially equal magnification, and the horizontal direction (or vertical direction) is reduced at a large rate. Assume that the image data is doubled. In this case, the compression coding processing apparatus 101 (step S102) generates a code stream in which codes are preferentially assigned to high-frequency components in the direction with a large scaling ratio. Since this code stream is preferentially stored with high-frequency components in the direction with a large variable magnification, it is expanded to reproduce a wide-angle image with a good overall balance of resolution (image quality). be able to.
[0036]
An example of specific processing of the compression coding processing apparatus 101 (step S102) will be described next.
[0037]
For example, lossless compression coding or compression coding with a low compression rate is performed, and post-quantization is performed so that the code amount of a high-frequency component in a large direction with a variable magnification is increased. In other words, when the lateral scaling factor is large, the signs of the LH subband coefficient and the HH subband coefficient including the high frequency component in the vertical direction are changed from those having a low composition level until the target compression rate is reached. Perform quantization. For example, when the wavelet transform with the number of decomposition levels of 3 is performed as in the example shown in FIG. 4, the codes of 1LH subband coefficient and 1HH subband coefficient, 2LH subband coefficient and 2HH subband coefficient Post-quantization is performed in the order of the code of the code, 3LH subband coefficient, and 3HH subband coefficient. By such post-quantization, it is possible to generate a code stream in which codes are preferentially assigned to HL subband coefficients which are high-frequency components in the horizontal direction. When the vertical scaling factor is large, the post-quantization from the low decomposition level of the HL subband coefficient including the high frequency component in the horizontal direction and the sign of the HH subband coefficient until the target compression rate is achieved. I do. For example, when the wavelet transform with the number of decomposition levels of 3 is performed as in the example shown in FIG. 4, the codes of 1HL subband coefficients and 1HH subband coefficients, 2HL subband coefficients and 2HH subband coefficients Post quantization is performed in the order of the code of the code, the 3HL subband coefficient, and the code of the 3HH subband coefficient. By such post-quantization, it is possible to generate a code stream in which codes are preferentially assigned to LH subband coefficients that are high-frequency components in the vertical direction.
[0038]
Alternatively, at the stage of quantization processing, if the horizontal scaling factor is large, the quantization step for the HL subband coefficient is reduced, and if the vertical scaling factor is large, the quantum for the LH subband coefficient is reduced. A similar code stream can also be generated by a method of reducing the conversion step. Alternatively, in the entropy encoding stage, when the horizontal scaling factor is large, the HL subband coefficient is shifted to the upper bitplane side, and when the vertical scaling factor is large, the LH subband coefficient is shifted to the upper bitplane. A method of shifting to the side is also possible. It is also possible to combine such a method with the post-quantization described above.
[0039]
On the other hand, if the input wide-angle image data is a 360 ° panoramic image in which the magnification (circumference reduction ratio) of the image center region is larger than that of the outer region as shown in FIG. In the compression coding processing apparatus 101 (step S102), as schematically shown in FIG. 6, the center region 302 of the image 301 is designated as a region of interest (ROI), and selective area image quality improvement processing is performed to obtain the center of the image. A code stream in which codes are preferentially assigned to areas is generated. In this code stream, information of a region with a large scaling ratio is preferentially stored. Therefore, the code stream is expanded and a flat 360 ° panoramic image as shown in FIG. When reproduced, an image with a good balance of resolution (image quality) can be obtained as a whole.
[0040]
FIG. 7 is a block diagram for explaining a second embodiment of the present invention. In this embodiment, the same processing or function as in the first embodiment is realized on a computer system. Therefore, since the processing flow is the same as that of FIG. 2, FIG. 2 is used in the following description.
[0041]
In FIG. 7, 401 is a CPU, and 402 is a main memory used as a storage area for programs executed by the CPU 401 and a working storage area for processing. An image processing program 403 for realizing each step of the image processing method of the present invention is also loaded on the main program 402 at the time of execution. When processing wide-angle image data generated by the image processing application program 404 operating on this computer system, the application program 404 is also loaded into the main memory 402 and executed. Reference numeral 405 denotes an auxiliary storage device such as a hard disk device for storing programs and data. 406 is an external for inputting wide-angle image data to be processed from a recording medium 407 such as various memory cards and an external device 408 such as an electronic camera device, and for outputting compression-coded code data to the recording medium 407. It is an interface part. Reference numeral 409 denotes a communication control unit for transmitting compression-encoded data of wide-angle image data. The image processing program 403 and various recording media on which the image processing program 403 can be read by the computer system are also included in the present invention.
[0042]
In this embodiment, each step shown in FIG. 2 is executed in order according to the image processing program 403 on the computer system.
[0043]
First, in step S <b> 101, wide-angle image data to be processed is input and written in a storage area in the main memory 402. More specifically, wide-angle image data is input from the recording medium 407 or the external device 408 via the external interface unit 406. Alternatively, wide-angle image data generated by the application program 404 and stored in the auxiliary storage device 405 is read and written in a storage area of the main memory 402.
[0044]
In step S102, as in the first embodiment, a wide-angle image data compression / encoding process is executed using an algorithm compliant with JPEG2000, and a code stream capable of reproducing an image with a balanced resolution is stored in the main memory 402. Generated in a specific storage area.
[0045]
In step S103, the code stream is written to the auxiliary storage device 405, or transmitted to a device on the network via the communication control unit 409, or recorded on the recording medium 407 via the external interface unit 406. The
[0046]
FIG. 8 is a block diagram for explaining a third embodiment of the present invention. This embodiment relates to an electronic camera device such as a digital still camera, and has a configuration in which the input device 100 in FIG. The compression encoding processing device 501 has the same configuration as the compression encoding device 101 in FIG. 1, and the recording device 503 records the code stream generated by the compression encoding processing device 502 as a file on a recording medium such as various memory cards. It is means to do. The imaging device 501 may have the same configuration as an imaging unit of a general digital still camera. Since the processing flow in this electronic camera apparatus is the same as that of FIG. 2, FIG. 2 is used in the following description. However, step S101 is a step of imaging processing.
[0047]
In step S101, the imaging device 501 captures wide-angle images with different magnification ratios depending on the direction or region via a curved mirror or fisheye lens. Data of the captured image is input to the compression encoding processing device 502.
[0048]
In step S102, the compression encoding processing device 502 performs the same compression encoding processing as in the first embodiment on the wide-angle image data.
[0049]
In step S103, the code stream generated by the compression encoding processing device 502 is recorded on the recording medium as a file by the recording device 503. If this code stream is expanded, a wide-angle image with a good balance of resolution can be reproduced.
[0050]
【The invention's effect】
As is apparent from the above description, wide-angle images with different scaling ratios depending on directions can be processed by the apparatus or method of the present invention, and wide-angle images with a good overall balance of resolution can be reproduced. It becomes.
[Brief description of the drawings]
FIG. 1 is a block diagram for explaining a first embodiment of the present invention;
FIG. 2 is a flowchart showing a processing flow in the image processing apparatus or method according to the present invention.
FIG. 3 is a diagram for explaining a basic algorithm of JPEG2000.
FIG. 4 is a diagram for explaining a two-dimensional wavelet transform.
FIG. 5 is a diagram for explaining the structure of a code stream generated by a JPEG2000 algorithm.
FIG. 6 is a diagram for explaining selective area image quality improvement processing in 360 ° panoramic image data;
FIG. 7 is a block diagram for explaining a second embodiment of the present invention.
FIG. 8 is a block diagram for explaining a third embodiment of the present invention;
FIG. 9 is a diagram for explaining an example of an imaging system for capturing an image with a wide horizontal or vertical angle of view using a curved mirror.
FIG. 10 is a diagram for explaining problems relating to a 360 ° panoramic image photographed using a hyperbolic convex mirror.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 100 Input device 101 Compression encoding processing apparatus 102 Transmission apparatus 103 Recording apparatus 403 Image processing program 501 Imaging apparatus 502 Compression encoding processing apparatus 503 Recording apparatus

Claims (8)

An input means for inputting wide-angle image data in which the vertical direction is reduced / magnified at a larger magnification than in the horizontal direction, or the horizontal direction is reduced / magnified at a larger magnification than in the vertical direction; ,
  JPEG2000-compliant compression encoding processing is performed on the wide-angle image data input by the input means, and the wide-angle image data in the direction with a large scaling factor is added to the wide-angle image data in the compression encoding processing. An image processing apparatus comprising compression encoding means for preferentially assigning a code as compared with a high frequency component in a direction in which the magnification of the angle of view image data is small. An input means for inputting wide-angle image data in which the vertical direction is reduced / magnified at a larger magnification than in the horizontal direction, or the horizontal direction is reduced / magnified at a larger magnification than in the vertical direction; ,
  JPEG2000-compliant compression encoding processing is performed on the wide-angle image data input by the input means. In the compression-encoding processing, the code amount of the high-frequency component in the direction in which the magnification of the wide-angle image data is large is determined. An image processing apparatus, comprising: compression encoding means for performing post-quantization that is larger than a code amount of a high-frequency component in a direction in which the scaling ratio of the wide-angle image data is small. An image pickup means for inputting wide-angle image data in which the vertical direction is reduced / magnified at a larger magnification than the horizontal direction, or the horizontal direction is reduced / magnified at a larger magnification than the vertical direction; ,
  JPEG2000-compliant compression encoding processing is performed on the wide-angle image data input by the imaging unit, and the wide-angle image data in the direction with a large scaling factor is added to the wide-angle image data in the compression encoding processing. An image camera device comprising compression encoding means for preferentially assigning a code as compared with a high frequency component in a direction with a small scaling factor of field angle image data. An image pickup means for inputting wide-angle image data in which the vertical direction is reduced / magnified at a larger magnification than the horizontal direction, or the horizontal direction is reduced / magnified at a larger magnification than the vertical direction; ,
  JPEG2000-compliant compression encoding processing is performed on the wide-angle image data input by the imaging unit, and in the compression-encoding processing, the code amount of the high-frequency component in the direction in which the magnification of the wide-angle image data is large is determined. An electronic camera apparatus comprising compression encoding means for performing post-quantization that is larger than a code amount of a high-frequency component in a direction in which the magnification of the wide-angle image data is small. An input step for inputting wide-angle image data in which the vertical direction is reduced / scaled with a larger scaling ratio than the horizontal direction, or the horizontal direction is reduced / scaled with a larger scaling ratio than the vertical direction; ,
  The wide-angle image data input in the input step is subjected to JPEG2000-compliant compression encoding processing. In the compression-encoding processing, the wide-angle image data is subjected to the high-frequency component in the direction of a large scaling factor. And a compression encoding step for preferentially assigning a code as compared with a high-frequency component in a direction in which the magnification of the angle-of-view image data is small. An input step for inputting wide-angle image data in which the vertical direction is reduced / scaled with a larger scaling ratio than the horizontal direction, or the horizontal direction is reduced / scaled with a larger scaling ratio than the vertical direction; ,
  JPEG2000-compliant compression encoding processing is performed on the wide-angle image data input in the input step. In the compression-encoding processing, the code amount of the high-frequency component in the direction in which the magnification ratio of the wide-angle image data is large is determined. An image processing method comprising: a compression encoding step for performing post-quantization that is larger than a code amount of a high-frequency component in a direction in which the magnification of the wide-angle image data is small. A program for causing a computer to execute each step of the image processing method according to claim 5 or 6. A computer-readable recording medium on which the program according to claim 7 is recorded.

Images