JP2008061172A - Image processing apparatus, image processing method and program therefor and recording medium with the program stored - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus, image processing method, program therefor and recording medium with the program recorded thereon where the computational complexity in distortion correction can be reduced. <P>SOLUTION: When performing distortion correction upon an image in a selected region selected in an image as a whole over a wide field of view, coordinates (x, y) of a point (p) coordinate-calculated by the distortion correction processing are frequently real numbers that include decimal points. When pixel data of four points (luminance data (Y data) of the four points and color difference data (CrCb data)) are read out, the luminance data of the point (p) are subjected to linear interpolation, and the color difference data are used, as they are. As a result, computational complexity in distortion correction can be reduced. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, a program thereof, and a recording medium for correcting distortion of an image captured using a wide-angle lens.

  Conventionally, there are the following techniques for correcting geometric distortion (distortion) of an image captured by a camera. For example, there is a technique in which image data with distortion is converted into YUV data by calculation, approximate polynomial calculation is performed, and different interpolation calculations are performed for the luminance signal and the color difference signal. Specifically, among YUV data, for Y data, interpolation calculation is performed with a second-order polynomial by applying Lagrange's interpolation formula from 9-point pixel data. On the other hand, among the YUV data, for the UV data, a polynomial interpolation calculation is performed from four points of pixel data using a linear interpolation formula. This is because for luminance information, human visual sensitivity is high, so even if the amount of calculation is large, an interpolation method with little deterioration in image quality must be selected. This is because even if a small number of interpolation methods are selected, there is no significant deterioration in visual image quality (see, for example, Patent Document 1).

Japanese Patent Laid-Open No. 11-250239 (paragraph [0017], FIG. 7)

  However, recently, since the number of pixels of the image sensor has increased, the amount of calculation for distortion correction has also increased, and processing takes time. In particular, when the captured image is a moving image, the amount of calculation for distortion correction becomes enormous, or when high-speed distortion processing is required, high-speed arithmetic processing is required.

  In view of the circumstances as described above, an object of the present invention is to provide an image processing apparatus, an image processing method, a program thereof, and a recording medium on which the program is recorded, which can reduce the amount of calculation in distortion correction. .

  In order to achieve the above object, an image processing apparatus according to the present invention corrects the distortion of an image of a selected region selected from a wide-field image including distortion caused by the wide-angle lens imaged using a wide-angle lens. Furthermore, coordinate calculating means for calculating the correction coordinates of the selection area image and interpolation means for interpolating each predetermined pixel data of the selection area image corresponding to the calculated correction coordinates are provided.

  In the present invention, for example, when distortion correction is performed on a selected region image selected from a wide-angle visual field image, pixel data interpolation processing is performed. In particular, for example, since distortion correction is performed on a selected region image selected from the wide-angle field image, the amount of calculation can be reduced as compared with the case where the entire wide-field image is corrected.

  The “pixel data” may be RGB data, or may be luminance / color difference data converted from RGB data, as will be described later. “Luminance / color difference data” is YUV data or YCrCb data including luminance data and color difference data.

  “Corresponding predetermined pixel data” is pixel data around a certain point that is an object of interpolation processing by an interpolation means, for example.

  “Image” mainly means moving images, but of course includes still images.

  According to an aspect of the present invention, the image processing apparatus further includes conversion means for converting the RGB data of the wide-field image into luminance / color difference data including luminance data and color difference data, and the coordinate calculation means. Calculates the correction coordinates of the selected area luminance / color difference data corresponding to the selected area image among the luminance / color difference data.

  According to one aspect of the present invention, the interpolation means interpolates selection area RGB data corresponding to the selection area image among the RGB data of the wide field image for each pixel of the selection area image. Means, and calculating means for calculating an interpolation value based on the interpolated selection area RGB data and the selection area luminance / color difference data. In the present invention, since the selected area RGB data interpolated by the RGB interpolation means is used to calculate the interpolation value, it is possible to reduce the amount of calculation while suppressing deterioration in image quality.

  According to an embodiment of the present invention, the coordinate calculation unit calculates a real number including a decimal point as the correction coordinate, and the interpolation unit calculates the luminance of four pixels obtained by rounding up or down the decimal point of the real number. Luminance data interpolation means for linearly interpolating data is provided. In many cases, the correction coordinates calculated for distortion correction do not correspond 1: 1 to each pixel constituting a wide-field image including distortion (or a selected area image including distortion). Therefore, in the present invention, the luminance data of four pixels obtained by rounding up or down the decimal point is interpolated.

  According to one aspect of the present invention, the interpolation means includes color difference data interpolation means for linearly interpolating two of the four pixels. Alternatively, the interpolation means includes color difference data interpolation means for selecting color difference data of a predetermined pixel among the four pixels. In the present invention, since the color difference data only selects a predetermined pixel, the calculation amount is further reduced.

  According to an aspect of the present invention, the image processing apparatus further includes update means for updating the selection area according to information indicating the selection area. In the present invention, every time the selected area image is updated, an interpolation process is executed and distortion correction is performed. Even in such a case, since the calculation amount is reduced as much as possible in the present invention, smooth processing can be executed.

  According to an aspect of the present invention, the image processing apparatus includes: a first output unit that outputs an image including an enhanced region in which the selected region is enhanced in the wide-field image as an entire image; and the selection A second output unit that outputs a distortion-corrected image in which the distortion of the region image is corrected; and a composite image output unit that outputs a composite image in which the entire image and the selected region image are combined as one-screen image data. In addition. In the present invention, the entire image including the enhancement region and the distortion-corrected image in which the distortion is corrected can be output on one screen, so that the user recognizes which part of the entire image is selected, for example. Can be convenient. Also in the present invention, as described above, the amount of calculation is reduced as much as possible, so that a smooth composite image output process can be executed.

  The “entire image” is not limited to the wide-field image itself, as long as it is an image including an enhancement region in the wide-field image.

  As described above, according to the present invention, the amount of calculation in distortion correction can be reduced.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an image processing system according to an embodiment of the present invention. The image processing system 100 includes a wide-angle lens 1, an image processing device 10, and a display unit 12, and further includes an external device 13, an input device 14, and the like. The image processing apparatus 10 includes an image sensor 2, a color processing circuit 3, a distortion correction processing circuit 4, a controller 5, a memory 6, an image compression circuit 7, a display interface 8, a USB (Universal Serial Bus) interface 9, and an input device interface 11. Prepare.

  The image sensor 2 is a sensor that converts light taken in through the wide-angle lens 1 into an electrical signal. For example, a complementary metal-oxide semiconductor (CMOS) sensor, a charge coupled device (CCD), or the like is used. As the wide-angle lens 1, a lens having a wide viewing angle or a wide field angle is used. For example, a fish-eye lens or a PAL (Panoramic Annular Lens) (annular lens) from which an annular image can be obtained may be mentioned. Alternatively, by using two fisheye lenses having an angle of view of about 180 °, it is possible to capture an image in a field of view in all directions (spherical space (360 °)).

  As shown in FIG. 2, the color processing circuit 3 includes an RGB interpolation circuit 17 that interpolates RGB data of Bayer data of the image sensor 2, for example. The color processing circuit 3 includes a YCrCb conversion circuit 18 that converts the interpolated RGB data into YCrCb data. In addition, although not shown, the color processing circuit 3 includes an AE (Automatic Exposure) arithmetic circuit that controls the amount of light, an AWB (Automatic White Balance) arithmetic circuit that controls the white balance, and a gain that performs gain adjustment using a non-linear table for each RGrGbB pixel filter. An adjustment circuit is included. Further, the color processing circuit, as will be described with reference to FIG. 4C, is a screen addition processing circuit that adds the wide-field entire image 25 with thinned data to a part of the image displayed on the display unit 12, In addition, necessary or additional circuits are provided.

  The distortion correction processing circuit 4 mainly performs distortion correction processing on an image including distortion processed by the color processing circuit 3. Further, the distortion correction processing circuit 4 performs interpolation processing of pixel data (the YCrCb data) including luminance data and color difference data during the distortion correction processing. The memory 6 functions as a buffer at the time of processing of the distortion correction processing circuit 4 or has a function as a frame memory for image output. In particular, the memory 6 has a function of storing pixel data (RGB data or YCrCb data) of a selection area image described later. The memory 6 may be connected to a DMA (Direct Memory Access) circuit (not shown). In this case, data is input / output to / from the memory 6 without being controlled by the controller 5.

  The controller 5 controls the entire image processing apparatus 10 in an integrated manner, and is configured by, for example, a CPU (Central Processing Unit).

  The image compression circuit 7 compresses the image processed by the distortion correction processing circuit 4 in a predetermined format. For example, JPEG (Joint Photographic Expert Group) is used as the compression format. However, other known formats may be used as appropriate.

  The display interface 8 is an interface that outputs the image processed by the distortion correction processing circuit 4 to the display unit 12. The display interface 8 generates output data such as an NTSC (National Television System Committee) system or a PAL (Phase Alternating Line) system.

  The USB interface 9 controls input / output of an image processed by the image compression circuit 7 with respect to an external device 13 such as an external computer. Note that the display interface 8 may also output an image compressed by the image compression circuit 7.

  For example, the input device interface 11 receives a signal of an operation input by a user. Examples of the input device 14 include a mouse, a keyboard, a switch device, a touch sensor, a game device controller, and a stick-like operation device that can be held by a user.

  Note that the “image” in the present embodiment includes a still image, but in reality, it is often processed as a moving image.

  The color processing circuit 3, the distortion correction processing circuit 4, the controller 5, and the like can realize processing by at least one of, for example, an FPGA (Field Programmable Gate Array) and a DSP (Digital Signal Processor).

  FIG. 3 is a block diagram showing a configuration of the distortion correction processing circuit 4. The distortion correction processing circuit 4 includes an update processing circuit 24, a pixel data interpolation circuit 19, an output pixel synthesis circuit 20, and a filter circuit 21.

  The update processing circuit 24 includes a coordinate calculation unit 22 and a display direction designation register 23. The display direction designation register 23 stores information indicating the selected area analyzed by the controller 5 in accordance with input information from the input device interface 11. The coordinate calculation unit 22 calculates a selection region in the wide-field image 25 (see FIG. 4A) captured by the image sensor 2 via the wide-angle lens 1 according to information indicating the selection region. As will be described later, the distortion correction processing circuit 4 mainly corrects the distortion of the image in the selected region, and outputs the distortion-corrected image subjected to the distortion correction from the various interfaces 12, 13, or 14. “Information indicating a selected area” is position information of an area selected by the user in the wide-field image 25, for example. In other words, the update processing circuit 24 updates the distortion-corrected image in the selected area according to the position information (changes) in the selected area.

  The pixel data interpolation circuit performs processing such as linear interpolation on pixel data, that is, luminance data and color difference data for each pixel by calculating coordinates of an image including distortion stored in the memory.

  The output pixel synthesizing circuit 20 converts the pixel data once converted into the YCrCb 4: 4: 4 format into the 4: 2: 2 format or the 4: 2: 0 format by averaging. The filter circuit 21 performs processing for applying an averaging or sharpening filter to the output image. The output pixel synthesis circuit 20 and the filter circuit 21 are not particularly important elements in the present embodiment.

  FIG. 4 shows examples of various images that are image-processed by the image processing apparatus 10 and displayed on the display unit 12. FIG. 4A shows an entire image including the enhanced region 16 in which the selected region is enhanced in the wide-field image 25. The highlight area 16 is, for example, a frame-shaped cursor. However, the highlight area 16 is not limited to the cursor 16 as described above, and may be displayed in a different color from the area other than the highlight area 16 in the entire image 25. That is, the emphasis region 16 may be in any form that allows the user to distinguish the emphasized region from other regions by viewing the wide-field image 25. The user uses the input device to select a desired region (range surrounded by the cursor 16) in the entire image 25 with the cursor 16, and the controller 5 performs various processes using the range as the selection region.

  Hereinafter, the image of FIG. 4A that does not include the enhancement region 16 is referred to as a “wide-field image”. Further, as shown in FIG. 4A, an image including the enhancement region 16 in the wide-field image is defined as “entire image 25”, and “wide-field image” and “entire image 25” are distinguished. Hereinafter, an image in a range surrounded by the emphasis area 16 is referred to as a “selected area image”.

  The selection area in the whole image 25 is not limited to the form selected by the user. For example, a mode in which a preset area in the entire image 25 is selected is also conceivable. Information (position information or the like) indicating the area in that case may be stored in advance in a ROM (not shown), an external storage device, or the like. Thus, not only when the selection area is set in advance, but when a specific area in the entire image 25 is automatically recognized, the automatically recognized “specific area” is processed as the “selection area”. The form to be considered is also conceivable. Alternatively, there is also a form in which a part of the image area detected in the entire image 25 is processed as a “selected area” by various sensors (not shown). In this case, examples of the sensor include an optical sensor, a temperature sensor, a pressure sensor, a sound sensor, a humidity sensor, a vibration sensor, a gas sensor, and various other sensors.

  FIG. 4B is a diagram illustrating a distortion corrected image in which the distortion of the selected region is corrected. FIG. 4C shows a composite image 36 in which the whole image thinning screen 45 in which the whole image 25 is thinned out and the distortion corrected image 35 in which distortion is corrected are combined as image data of one screen. The distortion correction processing circuit 4 also generates such a composite image and outputs it to the display unit 12 or the like. For example, a technique such as OSD (On Screen Display) may be used to generate the composite image 36. The controller 5 can switch the output of the whole image 25, the distortion corrected image 35, and the composite image 36 shown in FIGS. 4A, 4 </ b> B, and 4 </ b> C in accordance with, for example, input information from the input device interface 11. it can. That is, the controller 5 mainly has three display modes, and switches between them appropriately. Note that the mode in which the entire image 25 in FIG. 4A is displayed may be a mode in which a wide-field image in which the highlight region 16 is not displayed is displayed. FIG. 14 shows a display image displayed on the display unit when the display modes are sequentially switched. FIG. 14A shows only a wide-field image (the whole image 25 may be used). FIG. 14B is an image of only the distortion corrected image 35 in which the distortion of the selected region is corrected. FIG. 14C shows a composite image 36 including the both.

  Next, the aspect of the installation state of the camera equipped with the wide-angle lens 1 will be described.

  Referring to FIG. 5, when the viewpoint is at the center O of the sphere 31, the visual field that can be captured by the viewpoint O is a three-dimensional space represented by the sphere 31. For example, when a fisheye lens having a visual field of about 180 ° is used as the wide-angle lens 1, the visual field captured by the fisheye lens is a hemispherical portion of the sphere 31. For example, FIG. 6A shows a field of view of the upper half (upward in an actual three-dimensional space) of the sphere 31 (hereinafter referred to as the upper hemisphere field of view). FIG. 6B shows a field (hereinafter referred to as a lower hemisphere field) of the lower half (downward in the actual three-dimensional space) of the sphere 31. FIG. 6C shows a horizontal half field of view of the sphere 31 (hereinafter referred to as a horizontal hemisphere field). In particular, in the example of FIG. 6A, it is assumed that the user is looking upward from the ground, the floor, or the desk. In the example of FIG. 6B, it is assumed that the user looks down from the ceiling or the sky. In the example of FIG. 6C, it is assumed that the horizontal direction or the horizontal direction is viewed from a wall or the like perpendicular to the ground.

  In addition to these, a field of view of the upper half and the lower half can be considered. As described above, various hemispherical fields of view can be obtained by setting various orientations or inclinations of the fisheye lens when the fisheye lens is installed.

  In addition, FIG. 6 demonstrated the case where a fisheye lens was used. The present invention is not limited to this, and when various wide-angle lenses 1 are used, of course, the field of view changes depending on the direction and inclination of the lens.

  Next, the operation of the image processing apparatus 10 configured as described above will be described. FIG. 7 is a flowchart showing the operation.

  The controller 5 causes the image sensor 2 to convert the light taken in through the wide angle lens 1 into an electrical signal (step 701). The controller 5 performs color processing by the color processing circuit 3 (step 702). In color processing, in particular, as described above, RGB data is interpolated (step 702-1), and the interpolated RGB data is converted into YCrCb data (step 702-2). This conversion process may be performed by the 4: 4: 4 method. Alternatively, thinning may be performed by a 4: 2: 2 method, a 4: 1: 1 method, or a 4: 2: 0 method.

The conversion formula from RGB to YCrCb can be expressed by the following formulas (1) to (3). However, the numerical value of each term in this formula may be slightly different.
Y = 0.299R + 0.587G + 0.114B (1)
Cr = 0.500R-0.419G-0.081B (2)
Cb = -0.169R-0.331G + 0.500B (3)

FIG. 8 is a diagram illustrating a Bayer array of the image sensor 2. Each red pixel is denoted by r mn, each green pixel is denoted by g mn, and each blue pixel is denoted by b mn. The interpolation of RGB data in step 702-1 is performed as follows. For example, the RGB values of four pixels g11 (= p11), r12 (= p12), b21 (= p21), and g22 (= p22) are roughly combined using the RGB values of the surrounding pixels. Then, the RGB values of the four pixels p11, p12, p21, and p22 are as follows.
p11 = (r11 = (r10 + r12) / 2, g11, b11 = (b01 + b21) / 2) (4)
p12 = (r12, g12 = (g02 + g11 + g13 + g22) / 4, b12 = (b01 + b03 + b21 + b23) / 4) (5)
p21 = (r21 = (r10 + r12 + r30 + r32) / 4, g21 = (g11 + g20 + g22 + g31) / 4, b21) (6)
p22 = (r22 = (r12 + r32) / 2, g22, b22 = (b21 + b23) / 2) (7)
The controller 5 may perform the above-described RGB interpolation processing not on all the RGB data of the entire image 25 but on the RGB data (selected region RGB data) in the selected region image.

  Here, the controller 5 adjusts the interpolated RGB values so as to suppress color misregistration by multiplying the interpolated RGB values by the matrix A shown in Equation 1 below, and obtains final RGB values.

In that case, the pixel values P11, P12, P21, P22 after color misregistration adjustment are
P11 = (R11, G11, B11) = Ap11 (8)
P12 = (R12, G12, B12) = Ap12 (9)
P21 = (R21, G21, B21) = Ap21 (10)
P22 = (R22, G22, B22) = Ap22 (11)
It can be expressed. In the present embodiment, this color misregistration adjustment process is not necessarily performed. This color misregistration adjustment process may also be performed on the selected region RGB data.

  The controller 5 sets the RGB value obtained by the equations (4) to (7) (or (8) to (11)) as the final RGB value, and converts the RGB data into YCrCb data as described above. To do. The YCrCb data is stored in the memory 6 (step 703).

  Returning to the description of FIG. The controller 5 uses the distortion correction processing circuit 4 to perform distortion correction processing on the selected area image in the entire image 25 in accordance with the input information from the input device interface 11, and generates a distortion corrected image (step 704). As will be described in detail later, the distortion correction processing circuit 4 obtains the address of the memory 6 corresponding to the coordinates of the pixel on the display plane displayed by the display unit 12 by the calculation of the coordinate calculation unit 22 by calculation. That is, a coordinate conversion process is performed. The address of the memory 6 corresponds to the correction coordinates of the selected area image. The correction coordinates obtained by calculation, that is, the memory address often includes a decimal point. Therefore, the pixel data interpolation circuit 19 generates pixel data by coordinate interpolation, and generates a distortion corrected image (step 704).

  The controller 5 converts the generated distortion corrected image into a pixel format corresponding to the display unit 12, the external device 13, and the like. After saving in the output memory, the controller 5 outputs the distortion corrected image via the various interfaces 8, 9, and 11 (step 705).

  Next, the operation of the update processing circuit 24 will be described. FIG. 9 is a flowchart showing the operation. If there is a change in information from the input device 14 from the standby state of the image processing apparatus 10 (YES in step 901), the controller 5 acquires the input information via the input device interface 11 (step 902). The controller 5 calculates position information of the emphasis region 16 in the entire image 25, a calculation process for distortion correction processing, a calculation process for switching the display mode, or two display direction modes (direct mode) described later. Further, a calculation process for switching the rotation mode) is performed (step 903). In particular, the position information and display direction mode switching information are stored in the display direction designation register 23.

  Next, the operation of the distortion correction processing circuit 4 will be described in more detail.

  In FIG. 10A, the above-mentioned upper hemisphere with the viewpoint o (0, 0, 0) as the center is viewed from the side. In the figure, a point P (u, v, w) is a surrounding subject point, a point Q is an imaging point of the whole image 25 that is a fisheye image, an angle θ is an incident angle from the point P to the viewpoint, and a distance h is Indicates the image height. The point P and the point Q are associated with each other by the image height characteristic. FIG. 10B is a graph showing the image height characteristics. The horizontal axis indicates the angle (incident angle) θ, and the vertical axis indicates the image height h. The data may be stored in advance in the memory 6 or another memory (not shown) as a conversion table.

  Here, FIG. 11A shows a display plane which is a plane of a range displayed on the display unit 12. FIG. 11B is a three-dimensional view of the upper hemisphere shown in FIG. An image obtained by projecting the spherical surface 33 shown in FIG. 11B on the xy plane is an entire image 25 shown in FIG. This whole image 25 is an image before distortion correction obtained by the image sensor 2. The center O of the display plane 32 is in contact with the spherical surface 33 formed by the upper hemisphere field of view, and the display plane 32 is a screen on which the selected area in the entire image 25 is displayed.

  In describing the coordinate conversion processing, zoom processing and pan processing (vertical pan processing and horizontal pan processing) will be described first for easy understanding. The zoom process is a process of changing the size of the distortion-corrected image by the controller 5 changing the size of the selected region in accordance with input information from the input device interface 11. The change in the size of the selection area is also included in the concept of the change in the position information. The pan process is a process in which the controller 5 changes the position of the selected area in accordance with input information from the input device interface 11.

Referring to FIG. 12, first, the initial position P ₀ (x ₀ , y ₀ , z ₀ ) = P of the spherical surface 33 with radius R (= 100% image height) is set at the center O of the display plane 32 (selected area). Contact with ₀ (R, 0, 0). In zoom processing, when a certain point P (X, Y) is converted into P ₀ (x ₀ , y ₀ , z ₀ ) by zoom processing, the conversion formula is as follows. In this zoom process, it is assumed that the center O of the display plane 32 coincides with the point P for easy understanding.
x ₀ = R (12)
y ₀ = X × Zoom (13)
z ₀ = Y × Zoom (14)
“Zoom” is a zoom magnification. In the zoom process, the controller 5 changes the size of the cursor 16 in accordance with the input position information of the selected area in FIGS. 4 (A) and 4 (C). In the zoom process, since only the enlargement or reduction of the display plane 32 is performed, the position of x ₀ = R does not change.

The vertical pan process will be described. P ₁ (x ₁ , y ₁ , z ₁ ) is a position where the display plane 32 is moved from the initial position P ₀ by the vertical pan angle V around the y axis while keeping the center O in contact with the spherical surface 33. The movement from P ₀ to P _{1 corresponds} to the movement of the cursor 16 in the vertical direction (vertical direction in the figure), as described with reference to FIG. In this case, the distortion correction image displayed on the display unit 12 is displayed so as to move in the direction perpendicular to the horizon. However, it should be noted that FIG. 4A is a horizontal hemisphere field of view, and FIG. 12 is an upper hemisphere field of view. For convenience of explanation, the field of FIG. 4A is now an upper hemisphere field of view. It is assumed and explained. The coordinate conversion formula for the vertical pan angle is as follows.
x ₁ = x ₀ cosV + z ₀ sinV (15)
y ₁ = y ₀ (16)
z ₁ = −x ₀ sinV + z ₀ cosV (17)
That is, these formulas are rotation matrix conversion formulas around the y-axis.

The horizontal panning process will be described. P ₂ (x ₂ , y ₂ , z ₂ ) is a position moved from the P ₁ by the horizontal pan angle H around the z axis while keeping the center O in contact with the spherical surface 33. The movement from P ₁ to P _{2 corresponds} to the movement of the cursor 16 at the rotation angle H about the center of the entire image 25 as described with reference to FIG. In this case, the distortion correction image displayed on the display unit is displayed so as to move in the horizontal direction parallel to the horizon. The coordinate conversion formula for the horizontal pan angle is as follows.
_{x 2 = x 1 cosH-y} 1 sinH ···· (18)
y ₂ = x ₁ sinH + y ₁ cosH (19)
z ₂ = z ₁ (20)
That is, these equations are rotation matrix conversion equations centered on the z-axis.

FIG. 16 shows each rotation matrix described above. As shown in FIG. 16, [X] is a rotation matrix centered on the x axis, [Y] is a rotation matrix centered on the y axis, [Z] is a rotation matrix centered on the z axis, and [M] is Indicates the zoom magnification. Then, [P ₂ ] can be expressed by the following equation (21) in the direct mode and (22) in the circulation mode.
[P ₂ ] = [Z] [Y] [X] [M] [P ₀ ] (21)
[P ₂ ] = [Z] [Y] [M] [P ₀ ] (22)
The “direct mode” and “circulation mode” are modes for defining position information of a selected region on the entire image 25 using the above formulas (12) to (20).

In the “direct mode”, it is possible to move the point P around all three axes by the above equation (21) with the above P ₀ as an initial position. In this mode, the position of P ₂ ) is defined. As shown in FIG. 6C, this is a mode mainly used when a camera is installed so that a horizontal hemisphere field of view can be obtained. In the horizontal hemisphere field of view, the z-axis direction shown in FIGS. 11 to 13 coincides with the direction of the horizontal line.

In the “circulation mode”, the above-described P ₀ is set as an initial position, and the point P can be moved around two axes of the z-axis and the y-axis by the above equation (22). In this mode, the position of the point P is defined. This is a mode mainly used when the camera is installed so that the upper and lower hemispherical fields of view can be obtained as shown in FIGS. 6 (A) and 6 (B).

Next, coordinate conversion processing for distortion correction will be described. Referring to FIG. 13, first, an incident angle (elevation angle) θ of P ₂ (x ₂ , y ₂ , z ₂ ) is obtained. In FIG. 13, if the coordinate center is o (0, 0, 0),
OP ₂ = (x ₂ ² + y ₂ ² + z ₂ ² ) ^1/2 (23)
Therefore, θ can be expressed by the following equation.
θ = arccos (z ₂ / OP ₂ ) (24)
Note that P ₂ (x ₂ , y ₂ , z ₂ ) shown in FIGS. 12 and 13 may be considered as coordinates corresponding to the point P (u, v, w) shown in FIGS. 10 and 11.

  An image height h (see FIG. 10A) is obtained from θ obtained by the equation (24). As described above, the image height h is obtained from the image height characteristics shown in FIG.

Referring to FIG. 12, when the image height h is obtained, P ₂ that from the point P ₂ downlink vertically _{'(x 2, y 2,} 0) point is the position of the image height h to Q (x, y ) (See FIG. 10A). Specifically, the coordinates of the point Q are obtained from the following equation.
OP ₂ '= OP ₂ sin θ (25)
x = x ₂ h / OP ₂ '(26)
y = y ₂ h / OP ₂ '(27)
As described above, when the coordinates of the point Q (corrected coordinates) are calculated, the coordinates of the point Q shown in FIG. 11C are changed to the points P (u, v, w) shown in FIG. Match with coordinates. That is, when the coordinates of the point Q are calculated, the pixel data on the memory address corresponding to the point Q (the pixel data of the coordinates corresponding to the point Q among the pixel data stored in step 703) is Read from memory. Such coordinate calculation of the point Q is performed for all the points on the display plane 32, whereby a distortion correction image 35 (see FIG. 11D) of the selected region corresponding to the display plane 32 is generated.

  Here, the coordinates of the point Q obtained by calculation for distortion correction (memory address obtained by calculation) are 1: 1 for each pixel constituting the distortion correction image 35 of the display plane 32 as described above. In many cases, it does not correspond. Therefore, the pixel data interpolation circuit 19 performs an interpolation process on the pixel data as described below using the coordinates obtained by the calculation.

FIG. 14 is a diagram for explaining the operation of the interpolation processing. For example, it is assumed that the pixel data interpolation circuit 19 obtains pixel data of coordinates of a certain point p (corresponding to the point Q in the above description) obtained by calculation with reference to FIG. As shown in FIG. 14, for example, it is assumed that the coordinates (x, y) of the point p are real numbers (256.7, 678.9) including a decimal point. Four pieces of peripheral pixel data including this point p are read from the memory 6. That is, pixel data of four pixels obtained by rounding up or down the decimal point is read out. The coordinates of the four pixels are as follows.
Upper left (256,678)
Upper right (256,679)
Lower left (257, 678)
Lower right (257,679)
These four points correspond to g11, r12, b21, and g22 in FIG. When these four points of pixel data (the luminance data (Y data) and the color difference data (CrCb data) of each of the four points) are read out, the luminance data at the point p is linearly interpolated, and the color difference data is The color difference data itself is used. This is due to the fact that human beings have only half the sensitivity to luminance and the sensitivity to luminance. Alternatively, linear interpolation may be performed on the color difference data.

  There are various methods for linear interpolation. For example, the pixel data of the point p is obtained by calculating the average value from the pixel data of four points or calculating the median value. Alternatively, interpolation processing may be performed as follows.

P, which is an RGB value interpolated with respect to the point p as follows from the RGB values of the four points shown in the above formulas (8) to (11) (also may be the formulas (4) to (7)) (1.m, 1.n) is required.
P (1.m, 1.n) = (1-m) (1-n) P11 + (1-m) nP12 + m (1-n) P21 + mnP22 (28)
More precisely,
P (1+ (0.m), 1 + (0.n)) = (1- (0.m)) (1- (0.n)) P11 + (1- (0.m)) (0. n) P12
+ (0.m) (1- (0.n)) P21 + (0.m) (0.n) P22 (29)
m and n are integers, and the center point of pixels g11 (= p11), r12 (= p12), b21 (= p21), and g22 (= p22) in FIG. Further, the coordinates of a point separated by 1 pixel in the vertical direction from the point C are m = 1, and the coordinates of the point separated by 1 pixel in the horizontal direction from the point C are n = 1. For m and n, positive / negative is determined in the vertical direction from the point C, and positive / negative is determined in the horizontal direction.

As described above, since the human sensitivity to the color difference is only about half of the sensitivity to the luminance, the approximate expression of the following expression (30) is established as a method of the interpolation process for the point p. In the example of Expression (30), YCrCb data converted by the 4: 2: 0 method in Step 702-2 is handled.
YCrCbP (1+ (0.m), 1 + (0.n))
= YCrCb {(1- (0.m)) (1- (0.n)) P11 + (1- (0.m)) (0.n) P12 + (0.m) (1- (0.n )) P21 + (0.m) (0.n) P22}
= (1- (0.m)) (1- (0.n)) YCrCbP11 + (1- (0.m)) (0.n) YCrCbP12 + (0.m) (1- (0.n)) YCrCbP21
+ (0.m) (0.n) YCrCbP22
≒ ((1- (0.m)) (1- (0.n)) YP11 + (1- (0.m)) (0.n) YP12 + (0.m) (1-n) YP21 + (0. m) (0.n) YP22,
CrP12, CbP21) (30)
In this equation (30), only the luminance needs to be calculated, so the calculation amount is about 1/3. In addition, since the selection area RGB data interpolated by the RGB interpolation circuit and the YCrCb data are multiplied in order to calculate the interpolation value, the amount of calculation can be reduced while suppressing the deterioration of the image quality. This equation (30) handles YCrCb data converted by the 4: 2: 0 method as described above. However, of course, the selection area RGB interpolated for YCrCb data converted by another method may be used.

  As shown in FIG. 14, the output pixel composition circuit temporarily changes the pixel data obtained by the 4: 2: 0 method to the 4: 4: 4 method. Then, it is converted into 4: 2: 2 or the like as appropriate according to the output format. Of course, in addition to the 4: 2: 2 scheme, the 4: 2: 0 scheme or other schemes may be used. When the processing in the output pixel synthesis circuit 20 is completed, the controller 5 writes the pixel data in the output frame memory.

  As described above, in the present embodiment, pixel data interpolation processing is performed when distortion correction is performed on a selected region image selected from the entire image 25. In particular, for example, since distortion correction is performed on a selected region image selected from the entire image 25, the amount of calculation can be reduced as compared with the case where the entire entire image 25 is corrected.

  In the present embodiment, as described above, every time the selected area image is updated in the update processing circuit 24, the pixel data interpolation circuit 19 performs an interpolation process. Even in such a case, since the calculation amount is reduced as described above, smooth processing can be executed.

  In the present embodiment, the entire image 25 including the cursor 16 and the distortion-corrected image 35 in which the distortion is corrected can be output on one screen. For example, which part of the entire image 25 is selected. Can be recognized by the user, which is convenient. Also in this case, since the amount of calculation is reduced as much as described above, it is possible to execute a smooth composite image output process.

  Embodiments according to the present invention are not limited to the embodiments described above, and other various embodiments are conceivable.

1 is a block diagram illustrating a configuration of an image processing system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of a color processing circuit illustrated in FIG. 1. It is a block diagram which shows the structure of the distortion correction processing circuit shown in FIG. The example of the various images which are image-processed by an image processing apparatus and are displayed on a display part is shown. It is a figure which shows the visual field represented by a spherical body. It is a figure which each shows a visual field (imaging range) when imaged with the camera using a wide angle lens. It is a flowchart which shows operation | movement of an image processing apparatus. It is a figure which shows the Bayer arrangement | sequence of an image sensor. It is a flowchart which shows operation | movement of an update process circuit. It is a figure for demonstrating an example of the principle of coordinate transformation. It is a figure for demonstrating an example of the principle of coordinate transformation. It is a figure for demonstrating the coordinate conversion of a pan process. It is a figure for demonstrating an example of the principle of coordinate transformation. It is a figure for demonstrating the operation | movement of an interpolation process. It is a figure which shows the display image displayed on a display part when a display mode is switched in order. It is a diagram for explaining the position of each rotation matrix and P _2.

Explanation of symbols

4 ... distortion correction processing circuit 5 ... controller 6 ... memory 10 ... image processing device 16 ... emphasis region (cursor)
DESCRIPTION OF SYMBOLS 17 ... RGB interpolation circuit 18 ... YCrCb conversion circuit 19 ... Pixel data interpolation circuit 22 ... Coordinate calculation part 23 ... Display direction designation register 24 ... Update processing circuit 25 ... Wide field image, whole image 35 ... Distortion correction image 36 ... Composite image

Claims (10)

Coordinate calculation means for calculating correction coordinates of the selected area image in order to correct the distortion of the image of the selected area selected from the wide-field image including distortion caused by the wide-angle lens imaged using the wide-angle lens; ,
Interpolating means for interpolating each predetermined pixel data of the selected area image corresponding to the calculated correction coordinates. An image processing apparatus comprising: The image processing apparatus according to claim 1,
Further comprising conversion means for converting the RGB data of the wide-field image into luminance / color difference data including luminance data and color difference data;
The coordinate calculation means calculates the correction coordinates of selected area luminance / color difference data corresponding to the selected area image out of the luminance / color difference data. The image processing apparatus according to claim 2,
The interpolation means includes
RGB interpolation means for interpolating selection region RGB data corresponding to the selection region image among the RGB data of the wide field image for each pixel of the selection region image;
An image processing apparatus comprising: calculation means for calculating an interpolation value based on the interpolated selection area RGB data and the selection area luminance / color difference data. The image processing apparatus according to claim 2,
The coordinate calculation means calculates a real number including a decimal point as the correction coordinates,
The image processing apparatus according to claim 1, wherein the interpolation means includes luminance data interpolation means for linearly interpolating luminance data of four pixels obtained by rounding up or down the decimal point of the real number. The image processing apparatus according to claim 3,
The image processing apparatus, wherein the interpolation means includes color difference data interpolation means for linearly interpolating two of the four pixels. The image processing apparatus according to claim 3,
The image processing apparatus, wherein the interpolation means includes color difference data interpolation means for selecting color difference data of a predetermined pixel among the four pixels. The image processing apparatus according to claim 1,
An image processing apparatus, further comprising updating means for updating the selection area according to information indicating the selection area. The image processing apparatus according to claim 1,
A first output means for outputting an image including an enhanced region in which the selected region is enhanced in the wide-field image as a whole image;
Second output means for outputting a distortion corrected image in which the distortion of the selected area image is corrected;
An image processing apparatus, further comprising: a synthesized image output unit that outputs a synthesized image obtained by synthesizing the entire image and the selected region image as image data of one screen. In order to correct the distortion of the image of the selected area selected from the wide-field image including distortion caused by the wide-angle lens imaged using the wide-angle lens, the correction coordinates of the selected area image are calculated,
An image processing method comprising: interpolating predetermined pixel data of the selected area image corresponding to the calculated correction coordinates. In order to correct the distortion of the image of the selected area selected from the wide-field image including distortion caused by the wide-angle lens imaged using the wide-angle lens, the correction coordinates of the selected area image are calculated,
A program that causes a computer to execute interpolation of each predetermined pixel data of the selected area image corresponding to the calculated correction coordinates.

Images