JP4907470B2 - Image signal processing apparatus, image encoding apparatus and image decoding apparatus, method thereof, processor thereof and image processor for video conference system - Google Patents

Description

  The present invention relates to an image signal processing method for correcting distortion generated in an image photographed based on the installation position and direction of a camera, mainly in a TV conference / TV phone system, and transforming the image into a less uncomfortable image.

  As digital broadcasting and next-generation optical discs are compatible with HDTV, large-screen display devices such as HDTV compatible PDP displays and liquid crystal displays are rapidly spreading. Furthermore, high-speed network environments using optical fibers have become widespread for networks, and transmission and reception are possible at bit rates exceeding several Mbit / s even in ordinary homes. In the next few years, it is expected that transmission and reception of several tens of Mbit / s will be possible. By using the above-mentioned image encoding technology, high-quality videophones can be used not only for companies using dedicated lines but also for general households. The introduction of the TV conference system is expected to progress.

  In a TV conference / TV phone, it is preferable to take a front image of a speaker and send it to the other party. This can be easily inferred from the fact that a good impression is given by speaking with a gaze at the other party in the conversation, and the conversation can be performed smoothly. However, in an actual TV conference / TV phone, it is not easy to take an image from the front of a speaker who often faces the display surface of the display device.

  FIG. 23 is a diagram showing an installation example of a display device and a camera of the TV conference system. The camera 101 is installed on the display device 102. Considering that a front image is taken, it is desirable to install the camera 101 in the center of the display device 102. However, when the camera 101 is installed in the center of the display device 102, the dialogue that the camera 101 is displayed on the display device 102 is obstructive. The person's image becomes difficult to see. Therefore, it is common to install as shown in FIG.

  In recent years, the size of the display device 102 has increased, and a large display exceeding 40 inches has been often installed even at home. The use of the large display device 102 makes it easier to see the image of the conversation person, but on the other hand, the position of the camera 101 installed on the upper part of the display device 102 is greatly deviated from the front of the speaker.

  FIG. 24 is a diagram showing an example of an image that is taken with a vertical one distorted. The speaker stands upright in front of the wall 104 perpendicular to the floor and the pillar 103 that supports it. The camera 101 is installed on the upper part of the display device 102, and the camera 101 faces slightly downward to photograph a speaker. 24A is a view of the installed state as viewed from the side of the speaker, FIG. 24B is a view of the speaker as viewed from the front, and FIG. 24C is a view of the speaker as viewed from above. It is.

  Now, since the camera 101 is installed in the upper part of the display apparatus 102, the distance from the camera 101 to the speaker, the wall 104, and the pillar 103 varies greatly depending on the position. That is, the distance Lb from the camera 101 to the lower part of the pillar 103 is considerably longer than the distance Lt from the camera 101 to the upper part of the pillar 103. It is well known that an image taken with a camera is taken smaller as the distance to the subject increases. Accordingly, an image taken by the camera 101 installed in this way is as shown in FIG. Comparing FIG. 24B and FIG. 24D, in FIG. 24D, the lower part of the image (that is, the part far from the camera 101) is taken small, so it should be vertical in nature. It can be seen that the pillar 103 is photographed obliquely, and the discomfort is great.

The distortion caused by the installation position of the camera 101 not being in front of the speaker can be corrected by calculation if the distance between the camera 101 and the subject (in this case, the speaker, the wall 104, and the pillar 103) is known. Is possible. For example, if the position and orientation of the camera 101 and the subject are fixed, the distance can be measured in advance and a correction method can be calculated (see Patent Document 1). The camera disclosed in Patent Document 1 is a document camera, and the distance between the camera 101 and a subject (document table) and the orientation of the camera 101 are measured by a sensor, and an image captured based on the measured distance and orientation is an image signal. It is corrected by processing.
Japanese Patent No. 3109580

  However, in a video conference / telephone call, the installation location may change before the conference starts, the number of participants in the conference and the shooting range may change frequently. The distance between the camera 101 and the subject and the orientation of the camera 101 Is difficult to measure. Therefore, there is a problem that the image cannot be easily corrected using the conventional method.

  Further, the distance between the camera 101 and the subject and the orientation of the camera 101 are measured, and the image is not corrected based on the measured distance or orientation, but the captured image is analyzed to detect image distortion. A method for correcting the detected distortion is also considered. According to this method, there is a merit that the image can be corrected without being influenced even when the installation location of the camera 101, the shooting angle, and the like are frequently changed.

  However, even in this case, there is a delay of at least one picture in order to analyze the captured image and detect image distortion. Therefore, there is a problem that a memory for holding image data for at least one picture is required.

  In order to solve the above problems, the present invention provides an image signal processing apparatus capable of saving a memory for holding image data even when distortion is detected and corrected from a captured image. It is another object of the present invention to provide an image signal processing method.

In order to solve the above problems, an image signal processing apparatus of the present invention is an image signal processing apparatus that corrects image distortion of a moving image composed of a plurality of pictures that are sequentially input. A distortion detecting means for detecting a distortion of a vertical line of the image in the distortion detection picture, and a horizontal displacement of the image corresponding to the distortion of the vertical line detected by the distortion detecting means. , and a correcting means for correcting the image of the correction target picture to be inputted later than the strain detecting picture, the distortion detection unit is a pixel each other located in at least two horizontal on line of the distortion detection picture And a vertical correlation calculation unit that calculates a correlation between pixel values of pixels in a relative positional relationship having the same shift amount in the horizontal direction with respect to a plurality of shift amounts, and the correction unit includes Wherein the degree of correlation represented by the calculated the correlation in the distortion detection picture correcting the correction target picture to shift only in the opposite direction the shift amount in the horizontal direction becomes maximum.

  In the natural world, especially artificial buildings, there are many vertical and horizontal ones, and the vertical and horizontal correlations are large, but the diagonal correlations are not so many. Also, the distortion most noticeable in the captured video is the distortion in the vertical direction and the horizontal direction. Therefore, if the strength of the correlation is calculated from the captured image and the correlation is large in the vertical direction, it is assumed that the correlation should be the largest in the complete vertical direction, and the correlation in the vertical direction is assumed. The image is enlarged / reduced in the horizontal direction so that becomes larger. In this way, it is possible to reconstruct an image without using the actual distance between the camera 101 and the subject and the orientation of the camera 101, and to reconstruct an image that has a strong correlation in the vertical direction with little visual discomfort.

  Furthermore, in a TV conference / TV phone, there is almost no change in the installation position and orientation of the camera immediately after the start of the conference. Therefore, it is possible to detect almost the same distortion as the correction target picture even if the distortion is detected in advance in a picture input before the correction target picture, instead of detecting the distortion in the correction target picture for correcting the distortion. . Therefore, distortion detection, which requires a large amount of calculation and is difficult to calculate in real time, and a correction method for correcting the distortion are calculated in advance, and only the image deformation processing with a small amount of calculation is performed in real time on the correction target picture. Thus, it is possible to perform image signal processing with a short delay time in a short time.

  By transforming the image signal so that the part that is supposed to be vertical is vertical, the distortion of the captured image is greatly reduced even when shooting from the front of the subject is not possible, and uncomfortable feeling in video conferences and videophones Can be reduced.

  In the present invention, since the image signal is deformed using only the information of the captured image, the distance between the camera and the subject and the orientation of the camera need not be measured in advance or acquired by a sensor, and can be realized inexpensively and easily. can do.

  Hereinafter, embodiments of the present invention will be described with reference to FIGS.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an image signal processing apparatus 3000 according to the first embodiment. The image signal processing device according to the present embodiment calculates the correlation in the vertical direction of the picture represented by the input image signal Vin, and generates an output image by modifying the picture so that the calculated correlation becomes the highest. It is a processing device. However, the first picture for which the correlation in the vertical direction has been calculated is output as it is without being deformed, and the subsequent picture is deformed to generate an output image.

The image signal processing device 3000 includes a distortion detection unit 306 and a correction processing unit 300.
The distortion detection unit 306 inputs a predetermined image signal (distortion detection picture) from among the moving images composed of a plurality of pictures that are sequentially input, and the vertical direction in the picture ([i] -th picture) A line distortion signal D [i] (or a signal corresponding to distortion in the vertical direction) is output.

  The correction processing unit 300 receives a vertical distortion signal of the i-th picture, and cancels a horizontal shift of an image (correction target picture: [i + 1] -th and subsequent pictures) input after the picture. To correct.

  FIG. 2 is a block diagram illustrating a configuration of the distortion detection unit 306 illustrated in FIG. As shown in FIG. 2, the distortion detection unit 306 in the first embodiment includes a division unit 401 and a correlation calculation unit 402. The dividing unit 401 divides a picture of one screen (distortion detection picture) represented by the input image signal Vin, and outputs information indicating how the screen is divided. The correlation calculation unit 402 calculates the correlation between the pixel values in the vertical direction for each divided screen for each predetermined unit (for each line or for each division unit), and detects and outputs the distortion of the vertical line.

  FIG. 3 is a block unit illustrating a configuration of the correction processing unit 300 illustrated in FIG. As shown in FIG. 3, the correction processing unit 300 according to the first embodiment includes a parameter detection unit 5303, a parameter memory 5304, and an image correction unit 5305.

  The parameter detection unit 5303 receives the output of the distortion detection unit 306 derived for the distortion detection picture (i-th picture) and derives correction parameters for each image (or division unit) described later. The parameter memory 5304 holds the derived correction parameters and parameters as initial values (hereinafter referred to as correction parameters) in units of pictures. The image correction unit 5305 corrects an image (or division unit) for the correction target picture ([i + 1] th and subsequent pictures) using the correction parameters described above.

  FIG. 4 is a block diagram showing a detailed configuration of the correction processing unit 300 shown in FIG. As shown in the figure, the correction processing unit 300 includes a parameter detection unit 5303, a parameter memory 5304, an image correction unit 5305, a switch 310, and a switch 311. The parameter detection unit 5303 detects a parameter for correcting the distortion calculated by the correlation calculation unit 402 shown in FIG. The parameter detection unit 5303 stores the detected parameter in the parameter memory 5304 connected by the switch 311. The parameter memory 5304 holds stored parameters. When the parameter memory 5304 is connected to the image correction unit 5305 by the switch 310, the parameter memory 5304 outputs the stored parameters to the image correction unit 5305. The image correction unit 5305 applies the parameters input from the parameter memory 5304 to the input image to correct vertical line distortion.

  The parameter memory 5304 is composed of two parameter memories including the parameter memory 1 and the parameter memory 2. The parameter memory 1 and the parameter memory 2 are switched via the switch 311 and the switch 310, respectively. It is done. The switch 310 and the switch 311 are switched according to a control signal from the parameter detection unit 5303. For example, when a parameter is detected by the parameter detection unit 5303 for the first input image (picture), the switch 311 connects the parameter detection unit 5303 and the parameter memory 1. As a result, the parameters detected by the parameter detector 5303 are accumulated in the parameter memory 1. During this time, the switch 310 that connects the parameter memory 5304 and the image correction unit 5305 is controlled to connect the parameter memory 2 and the image correction unit 5305 by a control signal. The parameter memory 2 stores parameters that do not deform the image. As a result, the first input image for which the parameter is detected is output as an output image as it is from the image correction unit 5305 without applying the parameters stored in the parameter memory 1.

  When the next parameter detection process is started, the switch 310 is switched to connect the parameter memory 1 and the image correction unit 5305. As a result, the parameters stored in the parameter memory 1 are read out to the image correction unit 5305 and applied to the current input image input to the image correction unit 5305. As a result, the image correcting unit 5305 corrects and outputs the distortion of the vertical line included in the input image. On the other hand, the switch 311 is switched so as to connect the parameter detection unit 5303 and the parameter memory 2. As a result, parameters detected from the distortion signal currently input to the parameter detection unit 5303 are accumulated in the parameter memory 2.

  Further, when the next parameter detection process is performed, the switch 310 is switched to connect the parameter memory 2 and the image correction unit 5305 and apply the parameters stored in the parameter memory 2 to correct the distortion. Do.

  In the case where the parameter can be detected in real time, it is preferable to apply the detected parameter to the next picture, and sequentially perform correction using the parameter obtained in the immediately preceding picture for each picture. However, the present invention is not limited to this, and when the calculation amount of parameter detection is large, the parameters detected from the input picture may be applied to correct all the input images (pictures) thereafter. In addition, it is not necessary to continue to use the parameters detected from the first input picture until the end, the parameters are detected for each predetermined number of pictures, and the parameters are detected from the picture where the parameters are detected to the subsequent pictures. The parameters may be applied to correct the distortion of the vertical line.

  FIG. 5 is a diagram for explaining the relationship between a screen that is a target of parameter detection in this embodiment and a screen that is a target of deformation using the detected parameter. The moving image signal is composed of a plurality of temporally continuous images, and the correlation between the continuous images is very large. In a video conference / telephone, the position and orientation of the camera 101 and the position of the subject hardly change in a continuous image. Therefore, an image deformation parameter is detected by using an image at an earlier time as a “parameter detection target image” instead of the “deformation target image” that is an image currently captured by the camera 101, and the detected image deformation parameter is “ The concept of the first embodiment is applied to the “deformation target image”.

  In the distortion detection unit 306 illustrated in FIG. 2, the division unit 401 divides an input image (picture) into division units, and the correlation calculation unit 402 calculates a correlation value. The parameter detection unit 5303 detects an image deformation parameter that increases the vertical correlation from the correlation value calculated by the correlation calculation unit 402 and temporarily stores it in the parameter memory 5304. The image correction unit 5305 performs image deformation on the input image signal Vin based on the image deformation parameter read from the parameter memory 5304, and outputs the image as an output image signal Vout having a large vertical correlation.

  In FIG. 4, the parameter detection unit 5303 has been described as receiving an output signal from the correlation calculation unit 402 of the distortion detection unit 306. However, the parameter detection unit directly inputs the input image signal Vin and is similar to the correlation calculation unit. It is good also as processing.

  In this case, the parameter detection unit 5303 includes “a distortion detection unit that detects a distortion of a line in the vertical direction of an image in a distortion detection picture to be a distortion detection target” and “the distortion detection unit that detects a distortion in a specific picture”. The image correction unit 5305 is a correction target that is input after the picture so as to cancel the horizontal shift of the image corresponding to the distortion of the vertical line detected by the distortion detection unit. “Correction means for correcting the image of a picture” and “the correction means for correcting the picture to be corrected based on the distortion of the vertical line detected before the picture for a picture not detected by the distortion detection means” In this case, the dividing unit 401 is not an essential component.

  FIG. 6 is a flowchart illustrating an example of the operation in the distortion correction processing of the image signal processing device according to the first embodiment. First, the image signal processing apparatus acquires an input image signal Vin photographed by the camera 101 (S601). Next, the image signal processing apparatus determines whether there is an input image signal Vin to be processed, that is, whether there is an unprocessed image to be processed (S602). If there is an input image signal Vin to be processed, it is further determined whether or not the input image is the first picture or a picture not to be corrected (S603). If the result of determination in step 603 is that the input image is the first picture or a picture that is not corrected, the input image signal Vin that has been input is output as it is (S604). If the result of determination in step 603 is that the input image is neither the first picture nor the uncorrected picture, the image correction unit 5305 has already been detected by the parameter detection unit 5303 and is stored in the parameter memory 5304. The input image is corrected based on the parameters and output. Next, the parameter detection unit 5303 detects the distortion of the line in the vertical direction of the input image (S606), detects a parameter for distortion correction, and stores it in the parameter memory 5304 (S607). Thereafter, the image signal processing apparatus returns to the process of step 601 and repeats the processes of step 601 to step 607 until there is no input image to be processed.

  FIG. 7 is a block diagram showing a detailed configuration of a correction processing unit 700, which is another example of the correction processing unit 300 shown in FIG. In the image processing signal device according to the first embodiment, when many pictures are corrected using the same parameters, the background image is almost stationary, but the input image signal Vin is a moving image, so the correction is actually performed. It may be performed more excessively than the screen distortion, or the degree of correction may be insufficient. Therefore, the image processing signal device which is another example of the first embodiment includes a correction processing unit 700. The correction processing unit 700 detects the correction parameters of several pictures (for example, pictures 1, 2, and 3), accumulates them in the parameter memory, respectively, and sets the several subsequent pictures (for example, pictures When the correction is performed on 4), a temporally smoothed correction value such as an average value of parameters accumulated so far (that is, an average value of correction parameters of pictures 1, 2, and 3) is used. Thus, the correction is performed. Here, the parameter detection unit 703 corresponds to “the distortion detection unit that detects distortion in a plurality of specific pictures”, and the image correction unit 705 applies “the picture for which distortion is not detected by the distortion detection unit. The correction means corresponds to the “correction means for correcting the correction target picture by temporally smoothing the correction amount by averaging the deviation amounts detected earlier.

  The correction processing unit 700 includes a parameter detection unit 703, a parameter memory 704, an image correction unit 705, a switch 710, and a switch 711. The parameter memory 704 includes, for example, N (N is a natural number) parameter memory 1, parameter memory 2,..., Parameter memory N. The parameter detection unit 703 detects an image deformation parameter for each picture of the input image, switches the switch 711, and sets the detected image deformation parameter to the parameter memory 1, the parameter memory 2,. Store sequentially. In this case, the parameter detection unit 703 does not need to detect the image deformation parameter for each picture, and may be for every predetermined number of pictures. For example, it is assumed that the parameters of the first picture are stored in the parameter memory 1 and the second picture is corrected using the parameters stored in the parameter memory 1. Thereafter, the parameters detected in the second picture are stored in the parameter memory 2 and sequentially stored in N parameter memories. For example, assuming that correction has been performed using the parameters detected in the first picture until then, when correcting the (N + 1) -th picture, the parameter detection unit 703 sequentially switches the switch 710 in response to the switching signal. The image correction unit 705 is caused to read out the parameters stored in the parameter memory 1 to the parameter memory N. The image correction unit 705 calculates an average value of the N parameters read from the parameter memory 1 to the parameter memory N, and uses the calculated parameters to correct the vertical line distortion of the (N + 1) th picture. to correct. As a result, there is an effect that the distortion can be corrected by a more appropriate amount in accordance with the movement of the screen without causing the degree of correction to change abruptly with respect to the image that has changed little by little.

  In the present embodiment, only the image deformation parameter is detected in the first picture, and the image deformation process is not performed. Therefore, since the image deformation parameter is calculated from an image at an old time, it is not necessary to temporarily hold the input image signal Vin in a memory or the like, and image deformation processing can be performed. Therefore, for the first picture, the time from when the input image signal Vin is input to when the output image signal Vout is output is short, and there is an advantage that low delay can be realized. Further, in the first embodiment, since it is not necessary to hold the first picture, it is not necessary to provide a screen memory for image deformation processing. Therefore, even if it is necessary to provide the parameter memory 5304 for storing the image deformation parameters, the capacity of the parameter memory 5304 is smaller than the capacity of the screen memory for storing the image (one frame or one field picture). Therefore, there is an advantage that the memory capacity can be saved.

  FIG. 8 is a diagram showing an example of screen division by the dividing unit 401 for calculating the correlation in the vertical direction of the input image signal Vin. FIG. 8A shows an example of a screen for calculating the correlation. FIG. 8B shows an example in which the screen shown in FIG. 8A is divided into division units. In the figure, the screen is divided into two parts in the horizontal direction and eight parts in the vertical direction. However, the figure is only an example, and the horizontal direction may be divided into two parts or more, or the number of vertical divisions may be made eight or more. Is possible. Ideally, the vertical direction should be divided for each line. Here, the divided units are L0 to L7 and R0 to R7.

  For each divided unit, the correlation calculation unit 402 shifts the pixel values on the horizontal line by one pixel in the horizontal direction, and calculates the correlation value of the vertical pixel value with respect to each shift amount. The horizontal width for calculating the correlation value is several pixels. Specifically, for each of the divided units L0 to L7 and R0 to R7, the correlation value in the vertical direction of the values of the pixels arranged on the line is calculated between the adjacent lines in the vertical direction.

  FIG. 9 shows an example of the correlation when the vertical correlation value of the pixel value arranged on the line is calculated between the lines adjacent in the vertical direction for each of the screen division units L0 to L7 and R0 to R7. It is a graph to show. The correlation value CorD is a value obtained by moving the target line in the horizontal direction and calculating the correlation with the pixel on the line immediately above. Correlation values CorA to CorC are obtained by calculating the correlation in the vertical direction by shifting the pixels of the target line by 3 to 1 pixel in the left direction and correlation values CorE to CorG by 1 to 3 pixels in the right direction, respectively. Here, for example, for each of the division units L0 to L7 and R0 to R7, the vertical correlation of a set of two adjacent lines is calculated in order from the top. Then, the vertical correlation of all the two line sets in the division unit is calculated while shifting the two line sets downward by one line. Further, for example, a vertical correlation value is calculated for a set of all adjacent two lines in the division unit, and the sum thereof is set as the vertical correlation of the division unit. In this example, the division units L0 to L7 have the largest correlation value CorC shifted by one pixel to the left, and the division units R0 to R7 have the largest correlation value CorE shifted by one pixel to the right. This indicates that the division units L0 to L7 are shifted to the right from the vertical state, and the division units R0 to R7 are shifted to the left from the vertical state.

Now, suppose that the right direction is a positive value, the left direction is a negative value, and the correlation at the position where the division units L0 to L7 are shifted to the right by XL pixels and the division units R0 to R7 are shifted to the right by XR pixels is the largest. Further, if the number of horizontal pixels of the division units L0 to L7 and the division units R0 to R7 is W, the pixel position X of the row (if the pixel position at the left end of the row is 0, X is 0 or more and less than 2W) Pixels (XR-XL) (X-W / 2) / W + XL
By performing image correction such that the image is shifted to the right only, it can be corrected to an image having a strong correlation in the vertical direction. That is, according to the above formula, correction is performed by shifting the XL pixel at the position of X = W / 2 in the division units L0 to L7, shifting the XR pixel at the position of X = 3W / 2 in the division units R0 to R7, and at other positions. In accordance with the distance in the X direction from X = W / 2, correction is performed by proportionally distributing the deviation amount.

For example, as in the above example, the division units L0 to L7 are 1 pixel left (that is, XL is -1), and the division units R0 to R7 are 1 pixel right (that is, XR is 1). If
(1-(-1)) (X-W / 2) / W- (1) = 2X / W-2
By shifting to the right by the number of pixels, the vertical shift is eliminated. It should be noted that XL is simply the average value of the position where the vertical correlation is large in each of the division units L0 to L7, and XR is the average value of the position where the vertical correlation is large in each of the division units R0 to R7. However, it is not always necessary to calculate an average value, and a mode value (mode) or a median value (median) may be used. Further, the correction for moving the pixel at the pixel position X by the calculated shift amount not only shifts the pixel position by the number of integer pixels, but also uses an inter-pixel filter or the like as in the case of motion detection or motion compensation. It is also possible to move by a shift amount of a fractional pixel by performing interpolation.

  The parameter detection unit 5303 calculates the shift amounts XR and XL, which are the movement amounts (parameters) for correcting each pixel, from the vertical correlation values CorA to CorG calculated by the correlation calculation unit 402. The image correction unit 5305 performs image correction based on the movement amounts (parameters) XR and XL calculated by the parameter detection unit 5303 and outputs an output image signal Vout.

  Here, the correlation calculation unit 402 determines that “a pixel value of pixels that are located on at least two horizontal lines of a distortion detection picture and that have a relative positional relationship with the same shift amount in the horizontal direction. Is equivalent to a “vertical correlation calculation unit that calculates the correlation of a plurality of deviations”, and the image correction unit 5305 displays a “horizontal that maximizes the degree of correlation represented by the correlation calculated in the distortion detection picture”. This corresponds to the “correcting means for correcting the correction target picture so as to shift in the reverse direction by the amount of shift in the direction”.

  FIG. 10 is a diagram showing an example of an image obtained by the correction of the image signal processing apparatus of the present invention. As described above, an image obtained by distorting what is supposed to be vertical as shown in FIG. 24 is corrected by the image signal processing apparatus of the present invention to be corrected as shown in FIG. . In this process, the entire screen can be corrected by sequentially performing the corrected line as the line immediately above.

  Next, a description will be given of a method of correcting not on a line basis but on a screen basis. FIG. 11 is a diagram illustrating an example of a method for correcting image distortion in units of screens using XY coordinates. FIG. 11A shows an input image signal Vin to be corrected, and FIG. 11B shows a corrected image after correction. As shown in the figure, assuming that the width of one screen is 2 W and the upper left corner of the screen is (X, Y) = (0, 0), the shift XL between adjacent upper and lower lines before correction is constant in the screen. If there is, XL (Y), which is XL in the Yth row, can be assumed to be proportional to the vertical coordinate, and can be expressed as follows.

    XL (Y) = a · Y + b

  Therefore, when XL (Y + 1) which is XL of the Y + 1th row is used, a and b can be detected as follows.

a = XL (Y + 1) -XL (Y)
b = XL (Y) -Y {XL (Y + 1) -XL (Y)}

If this a and b are calculated for rows where Y is an even number, and average values a_ave and b_ave of a and b are calculated in units of screens,
XL (Y) = a_ave · Y + b_ave
Thus, XL in the Yth row can be detected.

  The same applies to XR in the Yth row.

c = XR (Y + 1) -XR (Y)
d = XR (Y) -Y {XR (Y + 1) -XR (Y)}
XR (Y) = c_ave · Y + d_ave

  When this is used, the pixel positions X and Y before correction corresponding to the corrected coordinate position (x, y) can be acquired at the following positions.

  FIG. 12 is a flowchart illustrating an example of a calculation procedure for correcting an image in units of screens. In the above calculation method, for example, the correlation calculation unit 402 calculates the correlation values CorA to CorG between the upper two lines and the lower two lines with the upper and lower three lines sandwiching the even-numbered rows (S102). The parameter detection unit 5303 calculates a position having a large vertical correlation from the amount of deviation between the upper two lines and the amount of deviation between the lower two lines (S103). The correlation calculation unit 402 and the parameter detection unit 5303 repeat the above-described processing of step S103 and step S102 in units of even lines (S101), so that the upper and lower sides sandwich the even lines for each even line sequentially from the top of the screen. Correlation values CorA to CorG between the upper two lines and the lower two lines in three lines are calculated. Then, from the correlation values CorA to CorG obtained by calculation, vertical line equations XL (Y) = a · Y + b and XR (Y) = c · Y + d are calculated, and the average of the coefficients of the straight line equations is calculated. Values a_ave, b_ave, c_ave, and d_ave are calculated (S104). As a result, the equations XL (Y) = a_ave · Y + b_ave and XR (Y) = c_ave · Y + d_ave in the vertical direction in one screen can be obtained. From the obtained equation, the image correction unit 5305 The image on the screen is corrected so that the angle of is vertical. Specifically, the image correction unit 5305 calculates the pixel value of the corrected pixel position (x, y) from the uncorrected pixel position (X, Y) using Expression A (S107). The image correcting unit 5305 corrects the image on one screen by repeatedly performing the processing in step S107 for each corrected row (S105) and one pixel at a time (S106).

  FIG. 13 is a block diagram illustrating an example of the configuration of the correlation calculation unit 402 illustrated in FIG. The input image signal is delayed by one pixel by the pixel delay units PelDelayB to PelDelayG, and delayed by one line and three pixels by the row delay unit LineDelay. Therefore, the output of the row delay device LineDelay and the output of the pixel delay device PelDelayD are shifted by one line. Similarly, the output of the row delay device LineDelay and the output of the pixel delay device PelDelayE are shifted by one pixel to one line, that is, one pixel to the left from one line. The output of the row delay device LineDelay is multiplied by the input image signal and the outputs of the pixel delay devices PelDelayB to PelDelayG by the multipliers MulA to MulG. The values multiplied by the multipliers MulA to MulG are cumulatively added for each division unit by the multipliers SumA to SumG, respectively, and become correlation values CorA to CorG. That is, the correlation values CorA to CorG are the correlation value between the first line and the second line in each division unit, the correlation value between the second line and the third line, the correlation value between the third line and the fourth line, ..., represented by the sum of correlation values between the (H-2) th line and the (H-1) th line.

  Here, the correlation calculation unit 402 corresponds to “a vertical correlation calculation unit that calculates a product sum of pixel values of pixels located on each line between the horizontal lines for each shift amount”. .

  As in the case of the correlation calculation unit 402 shown in FIG. 13, instead of multiplying the pixel values on two adjacent lines by the multipliers MulA to MulG, the difference absolute value units AbsDifA to AbsDifG are easier to calculate. A value may be calculated. FIG. 14 is a block diagram illustrating a configuration example of the correlation calculation unit 402 using difference absolute value units AbsDifA to AbsDifG instead of the multipliers MulA to MulG illustrated in FIG. 13. Similarly to the configuration shown in FIG. 13, each of the pixel delay units PelDelayB to PelDelayG delays the input image signal Vin by one pixel, and the row delay unit LineDelay delays the input image signal Vin by one line by three pixels. The difference absolute value units AbsDifA to AbsDifG output the absolute value of the difference between the input image signal Vin and the outputs of the pixel delay units PelDelayB to PelDelayG and the output of the row delay unit LineDelay. The accumulators SumA to SumG cumulatively add the absolute values of the differences for the division units. However, in this case, since the correlation increases as the correlation values CorA to CorG are smaller, the correlation is larger as the correlation values CorA to CorG are larger. In the present specification, the correlation will be described below in the case of FIG. 13 where the correlation values CorA to CorG are larger. Therefore, in the case of the correlation calculation block diagram of the present invention in FIG. Please be careful.

  Here, the correlation calculation unit 402 is “the vertical correlation calculation that calculates the sum of the absolute values of the pixel values of the pixels located on each line between the horizontal lines for each shift amount. Part.

  In FIG. 13 and FIG. 14, the amount of calculation increases in proportion to the number of deviations to be detected, but since the deviation in line units is small, the number of deviations to be detected is not so large. Further, when the vertical correlation is shifted due to the problem of the camera installation position, it is considered that the shift from the vertical position is not extremely large. Therefore, the range of the correlation value that the correlation calculation unit 402 needs to calculate ( Considering that the horizontal pixel shift range is small, the accuracy required for image correction is considered to be sufficient even if the number of shifts to be detected is sufficiently reduced and the amount of calculation of the correlation calculation unit 402 is reduced.

  FIG. 15 is a diagram illustrating examples of various patterns of straight lines included in an image. FIG. 15A shows a completely vertical pattern, and FIGS. 15B to 15E show patterns close to vertical. The range of FIG. 15A to FIG. 15E is a pattern in which the correlation can be calculated with the correlation value Cor. On the other hand, in the case of a pattern close to the horizontal in FIGS. 15 (f) to 15 (j) or a pattern different from both vertical and horizontal in FIGS. 15 (k) to 15 (p), the correlation value Cor is used as the correlation. The peak cannot be detected. When this peak cannot be detected, the maximum correlation value Cor becomes small. Therefore, when the correlation value Cor that is a peak is small, the correlation value Cor is not used for image correction, so that a pattern that originally does not include a vertical correlation is excluded from the image correction calculation. Accuracy can be improved.

  FIG. 16 is a flowchart showing the procedure of the image signal processing method in the image signal processing apparatus of the present invention. The correlation calculation unit 402 calculates a correlation value within the divided region (predetermined range) (S11), and determines whether the calculated correlation value has a peak value (S12). If there is a peak value, the deviation of the peak position from the center position is calculated (S13). The correlation value calculation and the calculation of the deviation of the peak position from the center position are performed in all the divided areas in the image. After the above processing is performed on all the divided areas in the image, an image deformation parameter for making the peak value the center of the left and right divided areas is calculated (S14). The entire image is deformed using the calculated image deformation parameter (S15), and a deformed image corrected to increase the vertical correlation is obtained.

  In the above-described embodiment, the correlation calculation unit 402 calculates the correlation values CorA to CorG in the vertical direction between vertically adjacent lines, and the amount of image shift to be corrected from the calculated peak of the correlation values CorA to CorG. Was calculated. However, the present invention is not limited to this, and image displacement may be calculated by other methods. For example, a vertical line may be detected using a conventional edge detection method, and the inclination may be corrected. For edge detection, for example, a filter that multiplies a pixel value of 3 rows and 3 columns by the following coefficient to calculate a sum of products in each column direction and each row direction is used. FIG. 17 is a diagram showing a simple example of a pixel filter used for calculation for vertical edge detection. The numbers in the circles in the figure indicate the coefficients that are multiplied by the pixel values of the pixels at the corresponding positions. Specifically, for the pixel in the first column, the pixel value at the position of (1, 1) is multiplied by (−1). For the pixel at the position (1, 2), the pixel value is multiplied by (+2). Further, the pixel value at the position (1, 3) is multiplied by (−1). When these multiplication results are added, it becomes 0 if the pixel values are the same. The same applies to the pixels in the second row and the pixels in the third row. In the column direction, any row has some value unless all pixel values are 0. That is, when there is a line having the same pixel value (luminance) in the vertical direction, if this filter is used, the product sum in the column direction has a different value for each row, but the product sum in the horizontal direction perpendicular to the line is zero. Become. By applying this operation to all the pixels in the picture, a vertical line can be detected. Also, by applying such a filter at different angles, lines with various angles can be detected. Thus, a desired image can be obtained by detecting a line that forms an angle within a certain range with respect to the vertical direction and correcting the inclination to be vertical.

  In the above embodiment, the correlation values CorA to CorG are calculated for each picture and the image is corrected. However, the present invention is not limited to this, and the correlation values CorA to CorG are calculated for each picture at a fixed interval. Based on the calculation result, the image of the subsequent picture may be corrected. For example, the correlation values CorA to CorG may be calculated every 5 to 10 minutes to calculate the deviation amount, or the deviation amount may be calculated every time the image signal processing apparatus is activated.

  Furthermore, in the above-described embodiment, it has been described that image transformation is performed on the input image signal Vin of one picture using parameters detected in one picture, but the relationship may not necessarily be one-to-one. For example, the shift amount calculated for one past picture may be proportionally distributed to a plurality of subsequent pictures to correct a plurality of images. Proportional distribution is performed by multiplying a weighting coefficient in accordance with a temporal distance from a picture in which a deviation amount is detected to a picture to be deformed. By doing so, there is an advantage that an image in which a straight line that should be vertical is corrected is displayed without giving a sense of incongruity to the user while gradually deforming the picture from which the parameter is detected. Conversely, parameters may be detected in a plurality of pictures, and the average value of these may be used to correct vertical line distortion in subsequent pictures. By doing so, it is possible to reduce the influence of a temporary change (noise) that occurs when a person crosses in front of the camera and correct image distortion more accurately.

  In the above embodiment, the correlation values CorA to CorG are calculated between adjacent lines. However, it is not always necessary to calculate the correlation values CorA to CorG between adjacent lines, for example, every other row. For example, the correlation values CorA to CorG may be calculated between the skipped lines.

(Embodiment 2)
FIG. 18 is a block diagram showing the configuration of the image signal processing apparatus 4000 according to the second embodiment of the present invention. The image signal processing apparatus 4000 according to the second embodiment calculates the correlation in the vertical direction of the input image signal Vin, and generates the output image by modifying the input image signal Vin so that the calculated correlation becomes the highest. This is the same as the image signal processing apparatus 3000 of the first embodiment. However, the input image signal Vin [i] for the i-th screen to be displayed is held in the image memory, the image deformation parameter is calculated from the held input image signal Vin [i], and This is an apparatus for applying an image deformation parameter to the held input image signal Vin [i] and outputting an output image in which vertical line distortion is corrected. The image signal processing device 4000 includes a screen memory 307, a correction processing unit 300, and a distortion detection unit 306. The screen memory 307 is a memory that holds image data in units of screens (pictures). For example, an input image signal Vin [i] that is an i-th picture is detected while processing parameters for image deformation of the picture are detected. , Mem are temporarily stored. When a processing parameter for image deformation is detected for the input image signal Vin [i] as the i-th picture, the input image signal Vin [i] stored in mem is input to the correction processing unit 300. Is done. The correction processing unit 300 applies the detected processing parameters for image deformation to correct vertical line distortion, and outputs the output image signal Vout [i] of the input image signal Vin [i] as the i-th picture. ] Is output.

(Embodiment 3)
FIG. 19 is a block diagram showing the configuration of the image signal processing apparatus according to the third embodiment of the present invention. The image signal processing apparatus according to the third embodiment includes an input unit (not shown) as an external device, and a range in which a correlation value is calculated, a degree of image deformation, and image deformation by a user-specified parameter User input from an external input unit by a user The feature of the third embodiment which is not in the first embodiment is that the presence or absence can be switched. The image signal processing apparatus includes a distortion detection unit 1910, a correction processing unit 1911, and a switch 1903. The distortion detection unit 1910 includes a division unit 401 and a correlation calculation unit 1901, and the correction processing unit 1911 includes a parameter memory 5304, a parameter detection unit 1902, and an image correction unit 1904.

  The division unit 401 divides the input image signal Vin into division units, and the correlation calculation unit 1901 calculates a correlation value. A user-specified parameter User is input to the correlation calculation unit 1901. The user-specified parameter User is a set of parameters for specifying a range for calculating a correlation value for the input image signal Vin, for example. For example, when there is a vertical pillar in the background, such as in a video conference / TV phone, the user designates this part with the user-specified parameter User, and calculates the correlation value only in the division unit including this pillar. The calculation processing can be greatly reduced, and further, since there is no influence other than the column portion, the vertical correlation of the column portion is enhanced, and the accuracy of making the column vertical is improved. Here, the correlation calculation unit 1901 accepts, via the user interface, designation of an area in the picture in which the vertical line distortion of the image is to be detected from the user, and within the received area, the vertical direction This corresponds to the above-mentioned distortion detecting means for detecting line distortion.

  Further, the parameter detection unit 1902 detects an image deformation parameter such that a region that is considered to have a large vertical correlation is vertical, and stores it in the parameter memory 5304. The parameter detection unit 1902 receives a user-specified parameter User from the outside. Correcting what is originally vertical to the original vertical is correct, but it does not transform the area with high vertical correlation completely vertically, but rather weakly (i.e., a little closer to vertical) Side effects caused by image deformation can be suppressed by performing deformation. The side effect due to image deformation is a process of erroneously deforming a non-vertical area vertically. Therefore, an instruction to weaken the degree of image deformation is given by an externally designated user parameter User. Alternatively, an instruction is given not to perform image deformation in the picture or period specified by the user-specified parameter User. In this way, it is effective if the parameter detection unit 1902 can detect an image deformation parameter adapted according to an instruction from the user-specified parameter User and can correct the parameter stored in the parameter memory 5304. Note that this processing is not limited to the case where the parameter detection unit 1902 performs the process, and the parameter detection unit 1902 only needs to store the same parameters as usual in the parameter memory 5304, and instead, the switch 1903 or the image correction unit 1904. It is good as well.

  The image correction unit 1904 performs image deformation on the input image signal Vin based on the image deformation parameter read from the parameter memory 5304, and outputs it as an output image signal Vout that has been deformed so as to increase the vertical correlation. However, the image correction unit 1904 adjusts the degree of deformation by multiplying the image deformation parameters XR and XL by a predetermined coefficient according to the degree of image deformation designated by the user with the user-specified parameter User. When this process is performed by the parameter detection unit 1902, the image correction unit 1904 ignores the user-specified parameter User. Here, the image correction unit 1904 accepts the designation of the degree of correction from the user via the user interface, and the horizontal direction of the image detected by the distortion detection unit corresponding to the received degree of correction. It corresponds to the “correction means for correcting the correction target picture” by adjusting the shift amount in the direction and shifting the adjusted shift amount in the reverse direction.

  The switch 1903 outputs the unmodified image (that is, the input image signal Vin) as it is when the image is not to be deformed according to the user-specified parameter User from the outside. Here, the switch 1903 corresponds to “the correction unit that accepts designation of a non-correction period from the user via the user interface, and the received period outputs the input picture corresponding to the correction target picture as it is”. To do. The case where the user does not want to deform is a case where the user confirms an effect due to the presence or absence of deformation, or a case where a line that is not originally vertical is corrected vertically.

  In addition, although the example which switches the range of correlation value calculation, the degree of image deformation, and presence / absence according to the user-specified parameter User has been shown, it is not necessary to switch all of these, and only a part of the user may be switched. .

  Furthermore, in order to further reduce the amount of calculation required for correlation value calculation and image deformation parameter detection, it is not necessary to perform correlation value calculation and image deformation parameter detection for all images of a moving image signal. . For example, the processing may be performed once every several images, or the correlation value calculation or the image deformation parameter detection may be performed at a timing designated by the user from the outside.

(Embodiment 4)
FIG. 20 is a block diagram showing the configuration of the image coding apparatus according to the fourth embodiment. In the figure, the portion surrounded by a broken line is the same as the configuration of the image signal processing device 3000 of the present invention shown in FIG. The image encoding unit 2001 encodes the image deformed by the image correction unit 5305 and outputs it as an image encoded stream Str.

  In the TV conference / TV phone system of the fourth embodiment, an image captured by the camera 101 is encoded and transmitted to the other party. With the image signal processing apparatus having the configuration shown in FIG. 1, an image obtained by the camera 101 is subjected to image modification so that the vertical correlation value is increased, and then encoded by the image encoding unit 2001. Therefore, the other party (reception side) can display an image that has been corrected vertically without an inclination of an object that should be vertical by simply receiving and decoding the encoded image stream Str. Here, the image encoding unit 2001 corresponds to “an encoding unit that encodes the corrected picture corrected by the correcting unit and outputs a code string”.

(Embodiment 5)
FIG. 21 is a block diagram showing the configuration of the image decoding apparatus according to the fifth embodiment. In the same figure, the part enclosed with the broken line is the same as the structure of FIG. The image decoding unit 2111 decodes the image encoded stream Str, performs image deformation by the image signal processing device in FIG. 1 so that the vertical correlation value between pixels on the line becomes large, and then displays the image on the display device 102. For this purpose, an output image signal Vout is output.

  When the transmission side of the video conference / telephone system receives the encoded image stream Str obtained by encoding the image captured by the camera 101 as it is, the reception side receives the encoded image stream Str and decodes it. By itself, objects that should be vertical are not displayed as such. Therefore, the image decoded by the image decoding unit 2111 is detected by the image signal processing apparatus in FIG. As a result, an image that is supposed to be vertical is deformed so that an object that is supposed to be vertical is correctly vertical. Therefore, an image encoded stream Str that is encoded without performing image deformation on an image captured by the camera 101 is also correctly vertical. Will be displayed after being corrected. Here, the image decoding unit 2111 corresponds to “a decoding unit that decodes an input code string”, and the correlation calculation unit 402 and the parameter detection unit 5303 are “by being decoded by the decoding unit. The obtained moving image corresponds to the “distortion detecting unit that detects distortion of a line in the vertical direction of an image in a distortion detection picture to be subjected to distortion detection”. This corresponds to the “correction means for correcting the detected vertical line distortion for the correction target picture”.

  In the above-described embodiment, it has been described that the camera is used for a TV conference or a TV phone and the inclination is corrected with respect to the vertical line included in the moving image. However, the present invention is not limited to this. For example, it is possible to correct the inclination of the line in the horizontal direction by using the same method as in the above-described embodiment by switching the processing of the row and the column.

  Further, according to the present invention described in the second embodiment, the above correction can be performed even when the picture to be subjected to distortion detection and the picture to be corrected are the same. Accordingly, not only a moving image but also a still image can be corrected similarly in the vertical direction.

(Embodiment 6)
Further, by recording a program for realizing the image signal processing method shown in each of the above embodiments on a recording medium such as a flexible disk, the processing shown in each of the above embodiments can be performed independently. It can be easily implemented in a computer system.

  FIG. 22 is an explanatory diagram when the image signal processing method of each of the above embodiments is implemented by a computer system using a program recorded on a recording medium such as a flexible disk.

  FIG. 22B shows an appearance, a cross-sectional structure, and a flexible disk as seen from the front of the flexible disk, and FIG. 22A shows an example of a physical format of the flexible disk that is a recording medium body. The flexible disk FD is built in the case F, and on the surface of the disk, a plurality of tracks Tr are formed concentrically from the outer periphery toward the inner periphery, and each track is divided into 16 sectors Se in the angular direction. ing. Therefore, in the flexible disk storing the program, the program is recorded in an area allocated on the flexible disk FD.

  FIG. 22C shows a configuration for recording and reproducing the program on the flexible disk FD. When the program for realizing the image signal processing method is recorded on the flexible disk FD, the program is written from the computer system Cs via the flexible disk drive FDD. When the image signal processing method for realizing the image signal processing method by the program in the flexible disk is constructed in the computer system, the program is read from the flexible disk by the flexible disk drive FDD and transferred to the computer system.

  In the above description, a flexible disk is used as the recording medium, but the same can be done using an optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card or a ROM cassette capable of recording a program can be similarly implemented.

  Each functional block in the block diagrams (FIGS. 1, 18, 19, 20, and 21) is typically realized as an LSI that is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them. For example, the functional blocks other than the memory may be integrated into one chip. The name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

  In addition, among the functional blocks, only the means for storing the data to be encoded or decoded may be configured separately instead of being integrated into one chip.

  As described above, in the present invention, since the image signal is deformed using only the information of the photographed image, there is no need to measure the distance between the camera and the subject or the camera direction in advance or acquire it with a sensor. The image signal can be transformed inexpensively and easily so that a portion that is supposed to be a vertical line or a horizontal line becomes a vertical line or a horizontal line.

  Therefore, even when the image cannot be taken from the front of the subject, the distortion of the taken image can be greatly reduced, and the uncomfortable feeling in the TV conference / TV phone can be reduced, and its industrial utility value is high.

1 is a block diagram illustrating a configuration of an image signal processing device according to a first embodiment. It is a block diagram which shows the structure of the distortion | strain detection part in Embodiment 1 shown in FIG. FIG. 2 is a block diagram illustrating a configuration of a correction processing unit illustrated in FIG. 1. It is a block diagram which shows the detailed structure of the correction | amendment process part shown in FIG. 6 is a diagram illustrating a relationship between a screen that is a target for detecting a parameter and a screen that is a target for deformation using the detected parameter in Embodiment 1. FIG. 6 is a flowchart illustrating an example of an operation in distortion correction processing of the image signal processing device according to the first embodiment. It is a block diagram which shows the detailed structure of the correction process part which is another example of the correction process part shown in FIG. It is a figure which shows the example of a division | segmentation of a screen for calculating the correlation of the vertical direction of the input image signal Vin. It is a graph which shows the example of the correlation at the time of calculating the correlation value of the vertical direction of the value of the pixel arrange | positioned on a line between the lines adjacent to an up-down direction about each of screen division unit L0-L7, R0-R7. is there. It is a figure which shows an example of the image obtained by correction | amendment of the image signal processing apparatus of this invention. It is a figure which shows an example of the method of correct | amending the distortion of an image per screen using XY coordinate. It is a flowchart which shows an example of the calculation procedure in the case of correct | amending an image per screen. It is a block diagram which shows an example of a structure of the correlation calculation part shown in FIG. FIG. 12 is a block diagram illustrating a configuration example of a correlation calculation unit using difference absolute value units AbsDifA to AbsDifG instead of the multipliers MulA to MulG illustrated in FIG. 11. It is a figure which shows the example of the various patterns of the straight line contained in an image. It is a flowchart which shows the procedure of the image signal processing method in the image signal processing apparatus of this invention. It is a figure which shows a simple example of the pixel filter used for the calculation for edge detection. 5 is a block diagram illustrating a configuration of an image signal processing device according to a second embodiment. FIG. FIG. 10 is a block diagram illustrating a configuration of an image signal processing device according to a third embodiment. FIG. 10 is a block diagram illustrating a configuration of an image encoding device according to a fourth embodiment. FIG. 10 is a block diagram illustrating a configuration of an image decoding device according to a fifth embodiment. It is explanatory drawing when the image signal processing method of each said embodiment is implemented by a computer system using the program recorded on recording media, such as a flexible disk. It is a figure which shows the example of installation of the display apparatus and camera of a video conference system. It is a figure which shows the example of the image image | photographed by distorting what should be perpendicular | vertical.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 102 Display apparatus 103 Column 104 Wall 300, 700, 1911 Correction processing part 306, 1910 Distortion detection part 307 Screen memory 310, 311, 710, 711, 1903 Switch 401 Dividing part 402, 1901 Correlation calculation part 1804 Screen memory 2001 Image Encoding unit 2111 Image decoding unit 3000, 4000 Image signal processing device 5303, 703, 1803, 1902 Parameter detection unit 5304, 704 Parameter memory 5305, 705, 1805, 1904 Image correction unit PelDelayA to PelDelayF Pixel delay line delay line delay unit MulA to MulG Multiplier SumA to SumG Accumulator CorA to CorG Correlation value AbsDifA to AbsDifG Difference absolute value calculator Vin Input image signal Vo t output image signal Str coded image stream User user-specified parameters Cs computer system FD flexible disk FDD flexible disk drive

Claims (14)

An image signal processing device for correcting distortion of a moving image composed of a plurality of pictures that are sequentially input,
Distortion detection means for detecting distortion of a line in the vertical direction of an image in a distortion detection picture to be subjected to distortion detection;
Correction means for correcting an image of a correction target picture input after the distortion detection picture so as to cancel a horizontal shift of the image corresponding to the distortion of the vertical line detected by the distortion detection means ; Preparation ,
The distortion detection means calculates a correlation between pixel values of pixels located on at least two horizontal lines of the distortion detection picture and having a relative positional relationship with the same shift amount in the horizontal direction. It has a vertical correlation calculator that calculates multiple deviations,
The correction means corrects the correction target picture so as to shift in the reverse direction by the shift amount in the horizontal direction where the degree of correlation represented by the correlation calculated in the distortion detection picture is maximized. An image signal processing device. The vertical correlation calculation unit calculates a product sum of pixel values of pixels located on each line between the horizontal lines for each shift amount,
It said correction means, the image signal processing apparatus according to claim 1, wherein the determining the degree of the larger the calculated value of the product sum correlation by said vertical correlation calculation unit is larger. The vertical correlation calculation unit calculates, for each shift amount, a sum of absolute values of pixel value differences between pixels located on each line between the horizontal lines,
It said correction means, the image signal processing apparatus according to claim 1, wherein the determining the absolute value the degree of the correlation smaller value of the sum of the difference calculated by said vertical correlation calculation unit is larger.   An image signal processing device for correcting distortion of a moving image composed of a plurality of pictures that are sequentially input,
  Distortion detection means for detecting distortion of a line in the vertical direction of an image in a distortion detection picture to be subjected to distortion detection;
  Correction means for correcting an image of a correction target picture input after the picture so as to cancel a horizontal shift of the image corresponding to the distortion of the vertical line detected by the distortion detection means,
  The correction means further receives a designation of a period not to be corrected from the user via the user interface, and outputs the input picture corresponding to the correction target picture as it is during the received period.
  An image signal processing apparatus. The distortion detection unit further receives, from the user, designation of an area in the picture in which a vertical line distortion of the image is to be detected via a user interface, and the vertical line is detected within the received area. the image signal processing apparatus according to any one of claims 1-4, characterized in that to detect the strain. The correction means further accepts designation of the degree of correction from the user via the user interface, and in the horizontal direction of the image detected by the distortion detection means, corresponding to the accepted degree of correction. adjust the amount of deviation, the image signal processing apparatus according to any one of claims 1-4, characterized in that for correcting the correction target picture to shift only in the reverse direction adjusted the amount of deviation. The correction means further accepts designation of a period not to be corrected from a user via a user interface, and outputs an input picture corresponding to the correction target picture as it is during the accepted period. The image signal processing device according to claim 1 , claim 2, claim 3, claim 5, or claim 6 . The distortion detection means detects distortion in a specific picture,
Said correction means, for the picture that is not detected by the distortion detecting means, according to claim 1-7, characterized by correcting the correction target picture based on the distortion of the detected vertical lines before the picture The image signal processing device according to any one of the above. The distortion detection means detects distortion in a plurality of specific pictures,
The correction unit corrects the correction target picture by temporally smoothing a correction amount by averaging the deviation amounts detected before the picture for a picture in which distortion is not detected by the distortion detection unit. The image signal processing device according to any one of claims 1 to 7, wherein The image signal processing device further includes:
Said encoded picture after correction is corrected by the correction means, the image signal processing apparatus according to any one of claims 1 to 9, characterized in that it comprises coding means for outputting a code string. The image signal processing device further includes:
Decoding means for decoding an input code string,
The distortion detection means detects distortion of a line in the vertical direction of an image in a distortion detection picture that is a distortion detection target for a moving image obtained by being decoded by the decoding means,
The image signal according to any one of claims 1 to 9 , wherein the correction unit corrects the distortion of the vertical line detected by the distortion detection unit with respect to the correction target picture. Processing equipment. An image signal processing method for correcting image distortion of a moving image composed of a plurality of sequentially input pictures,
A distortion detection step for detecting distortion of a vertical line of the image in a distortion detection picture to be subjected to distortion detection ;
A correction step of correcting an image of a correction target image input after the picture so as to cancel a shift in a horizontal direction of the image corresponding to the distortion of the vertical line detected by the distortion detection step ,
In the distortion detection step, a correlation between pixel values of pixels located on at least two horizontal lines of the distortion detection picture and having a relative positional relationship with the same shift amount in the horizontal direction. For multiple deviations,
In the correction step, the correction target picture is corrected so as to be shifted in the reverse direction by the shift amount in the horizontal direction where the degree of correlation represented by the correlation calculated in the distortion detection picture is maximized. An image signal processing method.   A computer executes the image signal processing method according to claim 12.
  program. Integrated circuit characterized in that it comprises the respective means included in the image signal processing apparatus according to any one of claims 1 to 1 1.

Images