JP2012085069A - Image processing device and controlling method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a display device display a further natural motion picture, according to the characteristics of an imaging element of a picture source and the display characteristics of the display device.SOLUTION: Scan timing information indicating the exposure timing of each line of an imaging element imaging an input picture and a frame rate of the input picture are obtained, and information indicating the scan timing of each line and the frame rate displayed by the display device are obtained. Next, according to the obtained exposure timing of each line and display timing information, correction output timing information of each line between the input picture and the display device is produced. Then, respective pixel data on a target line of a target frame to be produced which is indicated by the frame rate of the display device, are produced by referring to the correction output timing information and interpolating respective pixel data on corresponding lines of two input picture frames sandwiching the target line along a temporal axis. By executing the producing processing for the whole lines of the target frame, the image data of the target frame are produced.

Description

  The present invention relates to a technique for converting a frame rate of a moving image according to a display device.

  In recent years, with the technical development of video camera image sensors, video recording, and digitization, video with various characteristics has been mixed. For example, as a difference in characteristics, it is known that a rolling shutter phenomenon occurs in a DV (Digital Video camera) using a CMOS element, whereas this phenomenon does not occur in a DV using a CCD element. In addition, various video such as 24P / 30P / 60P are mixed in the frame rate. In addition, displays with various display methods and frame rates are mixed. For example, a PDP (Plasma Display Panel) or DMD (Digital Micromirror Device) device is a display based on PWM (Pulse Width Modulation) in the time direction, and is a system close to frame sequential. On the other hand, FED (Field Emission Display) or LCD (Liquid Crystal Display) is line-sequentially displayed. Also, the frame rate suitable for display varies depending on the display method. Under such circumstances, a frame rate conversion technique for realizing a suitable video display is important.

  Here, the following techniques are known as frame rate conversion techniques. For example, Patent Document 1 relates to IP conversion for converting interlaced video to progressive video. IP conversion is realized by generating an interpolated image by obtaining a motion vector between fields and mixing it with video in the field. . Further, Patent Document 2 performs frame rate conversion on a 60 Hz 3-2 pull-down video, obtains a 24 Hz frame image from the video, and interpolates a 50 Hz or 48 Hz video based on a motion vector.

JP-A-6-133280 Japanese Patent Laid-Open No. 7-7713

  However, these techniques assume that each frame or field line is captured or displayed at the same time. However, as described above, there are some actual video characteristics and display characteristics that do not satisfy these assumptions. For this reason, depending on the combination of the characteristics of the video and the characteristics of the display, there are cases where the movement of the video is not suitable.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide a technique in which a display device displays an image of a more natural motion according to the imaging characteristics of a video source and the display characteristics of the display device. .

In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is,
An image processing device that converts a frame rate of an input video from a video source into a frame rate of a video displayed by a display device,
First acquisition means for acquiring information indicating the exposure timing of each line of the image sensor that has captured the input video and the frame rate of the input video;
Second acquisition means for acquiring information indicating a scan timing of each line displayed by the display device and a frame rate of a display image displayed by the display device;
Generating means for generating correction output timing information of each line between the input video and the display video based on information indicating the exposure timing and scan timing of each line acquired by the first and second acquisition means; ,
Each pixel data on the target line of the frame to be displayed on the display device acquired by the second acquisition unit is referred to the correction output timing information, and the target line is sandwiched along the time axis 2 It generates by interpolating each pixel data on the corresponding line of one input video frame, and generates the image data of the frame to be displayed on the display device by executing the generation process for all lines. Image generating means.

  According to the present invention, the display device can display a more natural motion image according to the imaging characteristics of the video source and the display characteristics of the display device.

The system block diagram of embodiment. The block block diagram of the set top box of embodiment. The figure which shows an example of a user interface image. The schematic diagram which shows an example of the data structure of scan timing information, and the scan time of each line / pixel. The block block diagram of a decoder. 6 is a flowchart showing image quality correction processing. The schematic diagram which shows an example of the time relationship with respect to the line of an input image and a display image. The flowchart showing a correction | amendment video production | generation process. The flowchart showing a motion vector calculation process. The schematic diagram which shows the change of search block size. The flowchart showing the frame image generation process of a correction | amendment image | video. The schematic diagram which shows an example of the intersection of a motion vector and the scanning surface of a correction image.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing the system configuration of the first embodiment according to the present invention. It comprises a broadcast receiving antenna 101, a set top box 102, and a display device 103. The broadcast receiving antenna 101 receives terrestrial digital broadcasts, satellite digital broadcasts, and the like, and transmits broadcast signals to the set top box 102. The set-top box 102 decodes received broadcast signals and video signals recorded on media (for example, hard disk, memory card, optical disk, CF card, SD card, USB memory, etc.) The signal is transmitted to the display device 103. The display device 103 is a display (for example, a liquid crystal display, a plasma display, an organic EL display, a field emission display, etc.) that displays a video signal.

  FIG. 2 is a block diagram showing the configuration of the set top box 102 which is the image processing apparatus of the present invention.

  The media interface 202 is an interface for connecting to a medium 201 (for example, a hard disk, a memory card, a CF card, an SD card, a USB memory) taken out from the digital video camera. The encoded video signal stored in the medium 201 is once decoded by the decoder 206, subjected to frame rate conversion processing, encoded by the encoder 217, and stored in the HDD 205. The broadcast input terminal 203 is a terminal for inputting digital broadcast signals such as terrestrial digital broadcast and satellite digital broadcast. The digital tuner 204 converts the digital broadcast signal into an encoded video signal such as H.264 or MPEG2. Then, the encoded video signal is once decoded by the decoder 206, is subjected to frame rate conversion processing, is encoded by the encoder 217, and is stored in the HDD 206. Details of the frame rate conversion process will be described later. The HDD 205 is a large-capacity storage device that is used to temporarily store an encoded input video signal from the digital tuner 204 or the media I / F 202 or to store scan timing information described later. Under the control of the CPU 210, the decoder 206 decodes the specified encoded display video signal, generates an uncompressed video signal, and outputs it. The post filter 207 performs noise removal processing, edge enhancement processing, color gradation correction processing, and the like on the uncompressed display video signal. The display control unit 208 generates characters, graphics, or a user interface image, generates a video in which the user interface image is superimposed on the video signal received from the post filter 207, and transfers the video to the digital video signal terminal 209.

  The CPU 210 controls the entire apparatus via the system bus 216, and sequentially reads instructions stored in the ROM 211 and RAM 212 and executes processing. The ROM 211 and RAM 212 provide the CPU 210 with programs, data, work areas, and the like necessary for the processing. The operation unit 213 corresponds to a button, a touch panel, or the like, and receives a user instruction input via these buttons. The USB interface 214 is connected to the display device 103 via the USB terminal 215, and receives information necessary for image quality correction processing described later from the display device 103.

  When the power of the set top box 102 of the present embodiment is turned on in the above configuration, first, the video of the digital broadcast signal from the broadcast input terminal 203 is selected and decoded by the decoder 206, and the display device 103 from the digital video signal terminal 209. 3 is displayed on the display device 103, for example. In addition, a user interface image 302 for setting the operation of the set top box 102 at the time of activation is generated by the display control unit 208 and displayed on the display device 103 together with the screen 301. The user interface image 302 allows the user to select four items: video selection, image quality correction selection, display device setting update, and end.

  In the video selection, the user is allowed to select either a broadcast video from the broadcast input terminal 203 or a video stored in the medium 201 as a generation source of an image to be displayed on the display device 103. Based on the selection result, it is determined which encoded input video signal of the digital tuner 204 or the medium 201 is temporarily stored in the HDD 205. At this time, when the medium 201 is selected as the video source, the medium 201 is searched for, scan timing information (file) is searched, and stored in the HDD 205. In addition, when the digital tuner 204 is selected as the video source, since the scan timing information (file) is not broadcast at present, the default scan timing information stored in advance in the HDD 205 (all lines have the same timing). ), And use it. The same applies when there is no scan timing information in the medium 201, that is, when the search fails. If the search fails, the information may be set by the user.

  In image quality correction selection, it is selected whether or not to perform image quality correction processing on the input video. The image quality correction process is performed based on a flowchart of FIG.

  In the display device setting update, scan timing information necessary for image quality correction processing described later is received from the display device 103 via a bidirectional communication interface, for example, the USB interface 214, and stored in the RAM 212 (that is, in the embodiment, Two pieces of scan timing information are used, one from the video source and the other from the display device). This information is used in the generation of a display video described later, and has a data structure shown in the schematic diagram of FIG. At the top of the data, the number of display lines and the number of display pixels are described, followed by the frequency and period for vertical synchronization and horizontal synchronization. After this, the correspondence between the line number and the scan time is described for the number of pixel lines. The scan time here is a relative time based on the timing of vertical synchronization. An example of the scan time of each line / pixel expressed by this data structure is shown in the schematic diagram of FIG. FIG. 4B is a schematic diagram for 1024 line display, in which pixel numbers and line numbers scanned at a certain time are represented in a three-dimensional space composed of scanning times, pixel numbers, and line numbers. In this figure, t1 is the scan time of line number 1, tn is the scan time of line number n, and t1024 is the scan time of line number 1024. The line with reference number 401 represents a scan with respect to line number 1 at time t1, and it is approximated that all pixels of the line with line number 1 are scanned at time t1. The line with reference number 402 represents a scan for line number n at time tn, and the line with reference number 403 represents a scan for line number 1024 at time t1024. A line of reference number 404 is a line representing the relationship between the scan time and the line number for pixel number 1. This data is also used as scan timing information for the above-described input video. The information at the top of the data in this case is the number of imaging lines and the number of imaging pixels.

  When the user selects the end, these settings are ended and the user interface image 302 is closed. However, when the user operates the operation unit 213, it is displayed again.

  FIG. 5 shows an example of the configuration of the decoder 206. The code memory 502 temporarily stores encoded data to be decoded. The decoder 502 decodes the encoded data and outputs a decoding result. The inverse quantizer 503 inversely quantizes the input coefficient, and the inverse DCT transformer 504 performs an inverse DCT transform process. Then, the adder 505 adds the motion compensation data from the motion compensator 508, stores it as image data in the memory 506, and outputs it according to the external timing (for example, the display device 103). The motion compensator 508 outputs nothing when the target frame to be decoded is an I-frame. Therefore, the adder 505 outputs the DCT conversion result to the memory 506 as it is. On the other hand, when the frame of interest is double the P or B-frame, the motion compensator 508 operates and outputs a motion compensated prediction value.

  FIG. 6 is a flowchart showing the above-described image quality correction processing executed by the CPU 210. In step S601, scan timing information describing the frame rate of the input video and the scan time indicating the exposure timing of each line of the image sensor is acquired from the HDD 205 via the system bus 215 (first acquisition means).

  In step S602, scan timing information describing the frame rate for the display device 103 and the scan time of each line in the frame is acquired from the RAM 212 via the system bus 215 (second acquisition means).

  In step S603, based on the scan timing information acquired in steps S601 and S602, the time relationship with respect to the line of the input video and the display video is obtained from the start frame of the input video, and the time relationship is obtained from the scan time table (corrected output timing information). ) And stored in the RAM 212. The method of creating this scan time table is as follows.

  First, in order to match the number of lines and the number of pixels in one line in the scan timing information of the input video to the number of scan lines in the scan timing information of the display device and the number of pixels in one line, the scan timing information of the input video is normalized. Then, after the normalization, the corresponding scan time of the input video after normalization is subtracted from the scan time of each line of the scan timing information of the display device. That is, the difference between the scan times of each line is calculated. The input frame image data is also corrected so as to have this normalized number of lines. Simply, image data having the same number of lines as that for display is generated by performing linear interpolation processing.

  FIGS. 7A and 7B are schematic diagrams showing an example of a time relationship with respect to the lines of the input video and the display video for calculating the scan time table. FIG. 7A is a schematic diagram when an input image captured by a camera having a CMOS sensor as an image sensor is displayed on a display device of a PDP or DMD projector. Here, reference numerals 701 to 703 represent the scan timing of the input video, and reference numerals 704 to 705 represent the scan timing of the display video. Here, ti1 is a vertical scan reference relative scan time for the first line of the input video, and ti1024 is a vertical sync reference relative scan time for the input video line 1024. Tvi is a vertical synchronization period of the input video. Also, to1 is the vertical scan reference relative scan time for the first line of display video, and to1024 is the vertical sync reference relative scan time for the display video 1024th line. Tvo is a vertical synchronization period of the display video. Since the display device is a PWM display, the display image approximates that all pixels are scanned at the same time. FIG. 7B is a schematic diagram when an input image captured by a camera having a CCD sensor as an image sensor is output to a FED or LCD. Here, reference numerals 707 to 710 represent the scan timing of the input video, and reference numerals 711 and 712 represent the scan timing of the display video. Here, the variables ti1, ti1024, Tvi, and to1, to1024, and Tvo have the same meanings as those used in FIG. Since the imaging method of the input device is a CCD sensor, the input video approximates that all pixels are scanned at the same time. 7A and 7B, the dotted line indicates the scan timing of the input video, and the broken line indicates the scan timing of the display video.

  In step S604, a display video frame is generated from the input video frame based on the scan time table created in step S603. Then, the generated display video frame is encoded by the encoder 217 and stored in the HDD 205. Of course, the frame rate of the display video matches the frame rate of the display device.

  FIG. 8 is a flowchart showing the corrected video generation processing in step S604.

  In step S801, the frame number of the display video generated in this process is initialized with the start frame. In step S802, an input video frame number necessary for generating one display frame image is calculated. This calculation is performed by obtaining a set of input video frames including a scan plane formed by the display video frame in a three-dimensional space including a scan time, a pixel number, and a line number. For example, consider the case of generating a display video frame of reference number 705 in FIG. In this case, 1 to M lines of the display video frame to be generated are generated by calculating a weighted average according to the absolute value of the time difference of each line of the frames 702 and 703 with respect to the time Tv0 + t01. The 1024 lines are generated by calculating the weighted average of the frames 701 and 702. That is, pixel data on the target line in the target frame is obtained by calculating a weighted average of each pixel on the corresponding line in two input frames sandwiching the target line along the time axis (FIG. 7A ) Arrows 750, 751). If the target line is an intersection line between the display video frame 705 and the input frame 705 as shown in FIG. 7A, the Mth line on the input frame 705 is adopted as the target line. Then, image generation processing for repeating the above process for all lines is performed to generate image data of the frame of interest. In short, it can be said that the input video frames 701, 702, and 703 are necessary to generate the display video frame 705 in FIG. Further, for example, input video frames 707, 708, and 709 are necessary to generate the video display frame 711 of FIG.

  In step S803, a motion vector between images of the input video frame is calculated based on the input video frame number calculated in step S802. This process is performed based on the flowchart of FIG. 9, and details will be described later. In step S804, a display frame image is generated by inter-frame interpolation based on the motion vector and the input video. This process is performed based on the flowchart of FIG. 11, and details will be described later.

  In step S805, it is determined whether or not the frame number of the generated display frame image corresponds to the encoded intra frame. If true, the process jumps to step S806, and if false, the process jumps to step S807. In step S806, the display frame image stored in the RAM 212 is encoded, and the bit sequence is encoded and recorded on the hard disk. For encoding, H.264, MPEG2, or the like is used. In step S807, the display frame image is temporarily stored in the RAM 212. This is because the frame is a target of intra prediction encoding and cannot be encoded until an image several frames ahead is determined. In step S808, it is determined whether video generation has been performed using the final frame of the input video. If true, the process jumps to step S809, and if false, the process jumps to step S810. In step S809, audio data is acquired from the input video, added to the generated encoded display video, and hunting is performed for the generation of the corrected video. In step S810, the frame number of the display video is updated.

  FIG. 9 is a flowchart showing the motion vector calculation process in step S803. This process searches for a motion vector while gradually reducing the division size of a search block from a large scale (in the embodiment, it is three times, but is not limited to this number). FIG. 10 schematically shows changes in the search block size.

  In step S901, the block division size for performing motion vector search is initialized. In step S902, the image is scaled and divided based on the block division size. Here, bicubic interpolation is used for image scaling, but other interpolation methods may be used. The scaling rate is determined so that the divided image is reduced to a predetermined size or less.

  In step S903, the motion vector search range of each divided image is determined based on the motion vector of the previous search result based on the correlation coefficient and the motion vector at the previous search. Here, when the correlation coefficient at the previous search is low, the search range is widened, and when the correlation coefficient is high, the search range is narrowed. Further, the control is performed so that the search range is wide when the variation with the surrounding motion vector is large, and the search range is narrow when the correlation coefficient is high.

  In step S904, a motion vector having the maximum correlation coefficient is searched, and the correlation value is calculated and stored. In step S905, it is determined whether to conceal the divided image based on the surrounding motion vector and the correlation value. When it is determined that concealment has occurred, the correlation value is set to 0 and a motion vector does not exist. In step S906, it is determined whether or not a motion vector search has been performed with a predetermined block size. If true, the process ends. If false, the process jumps to step S907. In step S907, the block division size is updated to the current half size for both horizontal and vertical.

  FIG. 11 is a flowchart showing inter-frame interpolation processing of one display frame image in step S804. The flowchart is roughly divided into two loop processes, and the outline of each process is as follows. In the loop processing from step S1101 to S1103, based on the motion vector, calculate the intersection point in the three-dimensional space consisting of the image position and time axis where the image of each subject in the video is located in the display frame image Calculate with One plane used for the intersection calculation is determined from the scan timing of the display image, and is, for example, reference numbers 704, 705, and 706 in FIG. 7A and reference numbers 711 and 712 in FIG. 7B. Looking at the number of intersections of each pixel in the display frame image from the result of the intersection calculation, there are pixels where there are no intersections due to concealment of the subject and pixels where there are a plurality of intersections due to motion vector calculation errors. Therefore, as a countermeasure, the process based on the number of intersections is branched in the loop processing from step S1104 to S1110, and each pixel value is determined so as to obtain an appropriate image with no omission. The details of each step are as follows.

  In step S1101, pixels are selected from the input frame image in a predetermined order based on the input video frame number calculated in step S802. For example, when the display frame image 701 in FIG. 7A is generated, pixels are selected in an appropriate order from the input frame images 704 and 705.

In step S1102, based on the selected pixel and the motion vector corresponding to the pixel, an intersection point with the scanning plane of one display frame image is calculated. The pixel position of the corrected image at the intersection is stored by associating the selected pixel with the motion vector. A schematic diagram of this intersection calculation is shown in FIG. In FIG. 12, reference numeral 1201 represents the scan timing of the display frame image that is the object of the generation process, and is hereinafter referred to as the scan plane of the display frame image. Reference numerals 1202 to 1204 denote scan timings of the input frame image used for display frame image generation. Here, the variables ti1, ti1024, Tvi, and to1, to1024, and Tvo have the same meanings as those used in FIG. 1205 is the coordinate x of the selected pixel, and 1206 is the motion vector v corresponding to the pixel. Reference numeral 1207 denotes an intersection point y with the scanning plane of the display frame image. The dotted line represents the scan timing of the input video, and the broken line represents the scan timing of the display video. Expressing the coordinates (px, lx, tx) of the selected pixel x, px is the pixel coordinate of x, and lx is the line coordinate itself of x. On the other hand, the scan time tx is tx = m × Tvi + ti1x. If the motion vector v is expressed as (pv, lv, tv), px is the calculated pixel direction motion vector, lx is the calculated line direction motion vector, and tv = Tvi. On the other hand, the scanning plane of the display frame image is expressed as n · y = C where the normalized normal vector n is (pn, ln, tn) and the point y on the scanning plane is (py, ly, ty). it can. However, “•” is an inner product. In the case of FIG. 12, pn = 0, ln = 0, tn = 1, and C = n × Tvo + to1. The coordinates of the intersection point y are expressed using these vectors,
y = x + {(C−x · n) / (x · v)} v
Is calculated.

  In step S1103, it is determined whether or not processing has been performed on all pixels of the predetermined input frame image. If true, the process jumps to step S1104, and if false, the process jumps to step S1101. In step S1104, pixels are selected in a predetermined order in the display frame image. In step S1105, it is determined whether there are a plurality of intersections with the motion vector. If true, the process jumps to step S1106, and if false, the process jumps to step S1108. In step S1106, a motion vector to be used for correction calculation is selected from the correlation coefficient and surrounding motion vectors. This selection is controlled so that a motion vector having a larger correlation coefficient and a smaller variation from surrounding motion vectors is selected. In step S1107, based on the selected motion vector, the pixel value of the start point of the motion vector is determined as the pixel value of the display frame image and stored in the RAM 212. In step S1108, it is determined whether or not an intersection exists. If true, the process jumps to step S1106, and if false, the process jumps to step S1109. In step S1109, the pixel value at the same position in the previous frame image of the input image is determined as the pixel value of the display frame image and stored in the RAM 212. In step S1110, it is determined whether or not processing has been performed on all the pixels of the display frame image. If true, the processing ends. If false, the process jumps to step S1104.

  As a result, the HDD 205 stores an image that displays a video of a more natural motion when displayed on the display device 103. Therefore, after that, the decoder 206 may decode and display as usual. In the embodiment, an example in which the data is once encoded and stored in the HDD has been described. However, if the HDD has a sufficient capacity, the moving image may be stored in an uncompressed state. In the embodiment, the moving image data from the input source is compressed and encoded, but may be uncompressed. Furthermore, in the embodiment, an example in which the present invention is applied to a set-top box has been described. However, the present invention may be applied to other devices, or may be applied to a display device that can be mounted thereon.

  As described above, according to the present embodiment, the frame rate conversion is performed so that the scan timing of the subject matches the scan timing of the video display, considering the scan timing as the video characteristics and the display characteristics. As a result, it is possible to reproduce the ideal image display more ideally than the actual movement of the subject in the image display.

[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

An image processing device that converts a frame rate of an input video from a video source into a frame rate of a video displayed by a display device,
First acquisition means for acquiring information indicating the exposure timing of each line of the image sensor that has captured the input video and the frame rate of the input video;
Second acquisition means for acquiring information indicating a scan timing of each line displayed by the display device and a frame rate of a display image displayed by the display device;
Based on information indicating the exposure timing of each line acquired by the first acquisition unit and the scan timing acquired by the second acquisition unit, corrected output timing information of each line between the input video and the display video Generating means for generating
Each pixel data on the target line of the frame to be displayed on the display device is referred to the correction output timing information, and on the corresponding line of two input video frames sandwiching the target line along the time axis. Image generation means for generating image data of a frame to be displayed on the display device by performing the generation process for all lines by generating the pixel data of each of the image data by interpolation. An image processing apparatus.   2. The apparatus according to claim 1, further comprising means for linearly interpolating along a line of the lines of the input video so that the number of lines in the frame of the input video is the same as the number of display lines of the display device. Image processing apparatus. A decoding unit configured to input the encoded moving image data stored in the storage medium and generate the input video by decoding the encoded image data;
The first acquisition means acquires from the storage medium a file storing information indicating the exposure timing of each line of the image sensor that has captured the input video and the frame rate of the input video. The image processing apparatus according to claim 1.   The said 1st acquisition means considers that the exposure timing of each line of the image pick-up element which imaged the said input image | video is all the same, when acquisition of the said file fails. Image processing device. A communication means for bidirectional communication with the display device;
The second acquisition unit acquires information indicating a scan timing of each line displayed on the display device and a frame rate for display on the display device via the communication unit. The image processing apparatus according to claim 1.   When the second acquisition unit fails to acquire information indicating the scan timing of each line displayed by the display device and the frame rate for display of the display device via the communication unit, 6. The image processing apparatus according to claim 5, wherein each line displayed by the display device is regarded as being displayed at the same timing. Encoding means for encoding each frame generated by the image generation means to generate encoded data;
The image processing apparatus according to claim 1, further comprising: a storage unit that stores the encoded data generated by the encoding unit in a storage unit that outputs the encoded data to the display device. A control method of an image processing device for converting a frame rate of an input video from a video source into a frame rate of a video displayed by a display device,
A first acquisition unit that acquires information indicating an exposure timing of each line of an image sensor that has captured the input video and a frame rate of the input video;
A second acquisition step in which a second acquisition unit acquires information indicating a scan timing of each line displayed by the display device and a frame rate of a display video displayed by the display device;
Based on the information indicating the exposure timing of each line acquired in the first acquisition step and the scan timing acquired in the second acquisition step, the generation unit generates each line between the input image and the display image. A generation step of generating corrected output timing information;
Two input video frames in which the image generating means refers to each pixel data on the target line of the frame to be displayed on the display device with reference to the correction output timing information and sandwiches the target line along the time axis Generating by interpolating each pixel data on the corresponding line, and generating the image data of the frame to be displayed on the display device by executing the generation process for all lines; and A control method for an image processing apparatus, comprising: A program that causes a computer to function as an image processing device that converts a frame rate of an input video from a video source into a frame rate of a video displayed by a display device by being read and executed by a computer,
The computer,
First acquisition means for acquiring information indicating an exposure timing of each line of the image sensor that has captured the input video and a frame rate of the input video;
Second acquisition means for acquiring information indicating a scan timing of each line displayed by the display device and a frame rate of a display image displayed by the display device;
Based on information indicating the exposure timing of each line acquired by the first acquisition unit and the scan timing acquired by the second acquisition unit, corrected output timing information of each line between the input video and the display video Generating means for generating
Each pixel data on the target line of the frame to be displayed on the display device is referred to the correction output timing information, and on the corresponding line of two input video frames sandwiching the target line along the time axis. Each of the pixel data is interpolated, and the generation process is executed for all lines, thereby functioning as image generation means for generating image data of a frame to be displayed on the display device. Program.   A computer-readable storage medium storing the program according to claim 9.

Images