JP2011211270A - Image processor, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily and appropriately generate an image wherein a gap between dots is corrected from an image in which a display image is picked up.SOLUTION: An imaging apparatus 100 for acquiring a picked up image in which a display image displayed on a display screen where a plurality of dots are arranged in a matrix shape is picked up includes: a quadrilateral conversion processing part 5d for generating a right angle quadrilateral image by executing right angle quadrilateral conversion processing to a prescribed image area of the acquired picked up image; and a filter processing part 5g for executing gap correction processing of correcting the image data of a gap area corresponding to the gap between the dots adjacent in the prescribed direction of the display image to the generated right angle quadrilateral image.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for performing predetermined image processing on a captured image.

In an image obtained by capturing a display image displayed on a display screen of a display device such as a liquid crystal display, a gap region between a plurality of dots arranged in a matrix of the display device appears as a lattice pattern.
Therefore, a periodic pattern corresponding to the lattice pattern on the image is detected, an area corresponding to each dot is specified based on the periodic pattern, and a pixel of each pixel of new image data is determined from the pixel value of the area. An image processing apparatus that generates a value is known (for example, see Patent Document 1).

JP 2006-350440 A

However, when the display image displayed on the display screen is captured, the grid pattern appearing in the captured image is distorted unless the imaging device is accurately opposed to the display screen. . In this case, not only the periodic pattern cannot be detected accurately, but also when a captured image is processed and a pixel value of each pixel of new image data is generated, a predetermined filter is applied. It becomes impossible to cause a deterioration in processing efficiency.
Further, if a predetermined filter corresponding to the maximum number of pixels of the periodic pattern in the captured image is applied uniformly, there is a problem that the image quality of new image data is deteriorated. On the other hand, if a predetermined filter corresponding to the minimum number of pixels of the periodic pattern in the captured image is applied uniformly, the grid pattern on the display screen cannot be erased sufficiently. There is also.

  Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of easily and appropriately generating an image in which a gap between dots is corrected from an image obtained by capturing a display image. That is.

In order to solve the above problem, an image processing apparatus according to claim 1 is provided.
An acquisition unit that acquires a captured image obtained by capturing a display image displayed on a display screen in which a plurality of dots are arranged in a matrix, and a right-angled quadrilateral transformation for a predetermined image area of the captured image acquired by the acquisition unit Corresponding to a gap between dots adjacent to each other in a predetermined direction of the display image with respect to the right-angled quadrilateral image generated by the first processing means and a right-handed quadrilateral image by performing processing. And a second processing means for performing a gap correction process for correcting the image data of the gap area to be performed.

The invention according to claim 2 is the image processing apparatus according to claim 1,
In the gap correction processing based on the number of pixels constituting the gap area in the right-angled quadrilateral image generated by the first processing means and the number of pixels calculated by the calculation means. Specifying means for specifying a processing range of one pixel to be processed, wherein the second processing means uses the right-sided quadrilateral image based on the processing range of the one pixel specified by the specifying means. The gap correction processing is performed for the above.

The invention according to claim 3 is the image processing apparatus according to claim 2,
The specifying unit defines a processing range of one pixel to be processed based on the number of pixels calculated by the calculating unit, and a size of a predetermined filter used for the filter processing as the gap correction processing The second processing means performs the filtering process on the right-angled quadrilateral image using a predetermined filter having the size specified by the specifying means, and the image of the gap region in the right-angled quadrilateral image. It is characterized by correcting data.

The invention according to claim 4 is the image processing apparatus according to claim 3,
The calculation means specifies a plurality of the gap areas along a predetermined direction parallel to any one side of the right-angled quadrilateral image, and calculates the number of pixels constituting each of the plurality of gap areas. The specifying means specifies a size of a predetermined filter used for the filter processing with reference to a maximum value among the plurality of pixels calculated by the calculating means.

The invention according to claim 5 is the image processing apparatus according to claim 4,
The calculation means specifies a plurality of the gap regions along a plurality of directions parallel to any one side of the right-angled quadrilateral image, and the number of pixels constituting each of the plurality of gap regions in each direction And the specifying unit represents the number of pixels in the plurality of directions parallel to each other based on the maximum value among the plurality of pixels in each direction calculated by the calculation unit. A representative value is specified, and a size of a predetermined filter used for the filter processing is specified on the basis of the specified representative value.

The invention according to claim 6 is the image processing apparatus according to any one of claims 1 to 5,
The image processing apparatus further includes an extraction unit that performs an outline extraction process on the captured image acquired by the acquisition unit to extract the predetermined image area, and the first processing unit includes the predetermined image extracted by the extraction unit. It is characterized in that the right-angled quadrilateral transformation process is performed on the image area.

The invention according to claim 7 is the image processing apparatus according to claim 6,
The extraction means extracts a plurality of the predetermined image areas, and designates a predetermined image area that is subjected to the right-angled quadrilateral transformation process by the first processing means among the plurality of predetermined image areas. The apparatus further comprises means.

The invention according to claim 8 is the image processing apparatus according to any one of claims 1 to 7,
In the captured image acquired by the acquisition unit, the image processing apparatus further includes a correction unit that corrects distortion caused by optical characteristics of the imaging unit that captured the display image, and the first processing unit is distorted by the correction unit. The right-angled quadrangle conversion process is performed on a predetermined image area of the corrected captured image.

The invention according to claim 9 is the image processing apparatus according to any one of claims 1 to 8,
The image processing apparatus further includes a control unit that causes the recording unit to record the image after the gap correction processing by the second processing unit.

An image processing method according to the invention of claim 10 is provided.
An image processing method using an image processing apparatus including an acquisition unit that acquires a captured image obtained by capturing a display image displayed on a display screen in which a plurality of dots are arranged in a matrix. The image processing method is acquired by the acquisition unit. A process for generating a right-angled quadrilateral image by performing a right-angled quadrangle conversion process on a predetermined image region of the captured image, and dots adjacent in a predetermined direction of the display image with respect to the generated right-angled quadrilateral image And a gap correction process for correcting the image data of the gap area corresponding to the gap.

The program of the invention described in claim 11 is
A computer of an image processing apparatus including an acquisition unit that acquires a captured image obtained by capturing a display image displayed on a display screen in which a plurality of dots are arranged in a matrix. A predetermined image of the captured image acquired by the acquisition unit First processing means for generating a right-angled quadrilateral image by performing a right-angled quadrangle conversion process on the image region, and adjacent to the right-angled quadrilateral image generated by the first processing means in a predetermined direction of the display image It is characterized by functioning as second processing means for performing gap correction processing for correcting the image data of the gap area corresponding to the gap between dots.

  According to the present invention, it is possible to easily generate an image in which a gap between dots is corrected from an image obtained by capturing a display image without degrading the image quality of the image.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to gap interpolation imaging processing by the imaging apparatus of FIG. 1. FIG. 3 is a diagram for explaining an example of calculation of the number of pixels in a gap area between adjacent dots in a display image in the gap interpolation imaging process of FIG. 2. It is a figure for demonstrating an example of the gap | interval interpolation imaging process of FIG. It is a figure for demonstrating an example of the gap | interval interpolation imaging process of FIG.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.
The imaging apparatus 100 of the present embodiment captures a captured image i1 (FIG. 4B) obtained by capturing a display image S (see FIG. 4A) displayed on a display screen 201 in which a plurality of dots are arranged in a matrix. And a right-angled quadrilateral transformation process is performed on a predetermined image area (square area A1) of the captured image i1 to generate a right-angled quadrilateral image i2 (see FIG. 5A). In addition, the imaging apparatus 100 corrects the image data of the gap region C (see FIG. 3) corresponding to the gap between dots adjacent in the predetermined direction of the display image S with respect to the generated right-angled quadrilateral image i2. Perform correction processing.
Specifically, as illustrated in FIG. 1, the imaging device 100 includes an imaging unit 1, an imaging control unit 2, an image data generation unit 3, a memory 4, an image processing unit 5, and an encoding unit 6. A recording medium control unit 7, a display control unit 8, a display unit 9, an operation input unit 10, and a central control unit 11.

  The imaging unit 1 captures a subject and generates an image frame as an imaging unit. In other words, the imaging unit 1 is not illustrated, but includes a lens unit composed of a plurality of lenses such as a zoom lens and a focus lens, a diaphragm for adjusting the amount of light passing through the lens unit, and a CCD (Charge Coupled). It comprises an image sensor such as a device (CMOS) or a complementary metal-oxide semiconductor (CMOS), and includes an electronic imaging unit that converts an optical image that has passed through various lenses of the lens unit into a two-dimensional image signal.

The imaging control unit 2 controls the imaging of the subject by the imaging unit 1. That is, the imaging control unit 2 includes a timing generator, a driver, and the like, although not illustrated. Then, the imaging control unit 2 scans and drives the electronic imaging unit with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit every predetermined period, and captures the imaging area of the electronic imaging unit Image frames are read out for each screen and output to the image data generation unit 3.
In addition, the imaging control unit 2 performs adjustment control of conditions when imaging a subject such as AE (automatic exposure processing) and AWB (automatic white balance).

In addition, the imaging control unit 2 causes the imaging unit 1 to capture the display image S displayed on the display device 200 (see FIG. 4A).
Here, the display device 200 is, for example, a liquid crystal display or an organic EL display, and R (red), G (green), and B (blue) color components on the surface of the display panel (display screen 201). A plurality of dots corresponding to are arranged in a matrix.
And when the imaging part 1 images the display image S currently displayed on the predetermined position of the display screen 201 of the display apparatus 200, the imaging control part 2 is an optical image of the captured image i1 which passed the various lenses of the lens part. Is converted into a two-dimensional image signal by the electronic imaging unit at a predetermined timing, and the image frame of the captured image i1 is read from the imaging region of the electronic imaging unit and output to the image data generation unit 3.
Therefore, the image information of the captured image i1 includes not only the image information of the display image S itself but also the image information of the dots arranged in a matrix on the surface of the display panel and the gap region C between the dots. For this reason, when the captured image i1 is displayed on the display unit 9 or the like of the imaging apparatus 100, the gap region C between the plurality of dots on the display screen 201 appears as a grid pattern superimposed on the display image S itself. End up. Such a lattice-like pattern is conspicuous particularly when the display image S is captured using a high-resolution (high pixel) electronic imaging unit or a bright (F-number) lens unit.

The image data generation unit 3 appropriately adjusts the gain for each of the RGB color components with respect to the analog value signal of the image frame transferred from the electronic imaging unit, and then samples and holds it by a sample hold circuit (not shown). The digital data is converted into digital data by a D converter (not shown), and color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a luminance signal Y and a color difference signal Cb, Cr (YUV data) is generated.
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to a memory 4 used as a buffer memory via a DMA controller (not shown).

  The memory 4 is composed of, for example, a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores data processed by the image processing unit 5, the central control unit 11, and the like.

  The image processing unit 5 includes an image acquisition unit 5a, an image correction unit 5b, a contour extraction unit 5c, a quadrilateral conversion processing unit 5d, a gap pixel number calculation unit 5e, a filter size specifying unit 5f, and a filter processing unit. 5g.

The image acquisition unit 5a acquires the image data of the captured image i1 from the memory 4.
That is, the image acquisition unit 5a acquires the image data (YUV data) of the captured image i1 captured by the imaging unit 1 and output from the image data generation unit 3. Specifically, the image acquisition unit 5 a is generated and output by the image data generation unit 3 when the imaging unit 1 captures the display image S displayed at a predetermined position on the display screen 201 of the display device 200. The image data of the captured image i1 is acquired.
Here, the image acquisition unit 5a constitutes acquisition means for acquiring a captured image i1 obtained by imaging the display image S displayed on the display screen 201 in which a plurality of dots are arranged in a matrix.

The image correction unit 5b performs distortion aberration correction for correcting distortion of the captured image i1 caused by the optical characteristics of the lens unit of the imaging unit 1.
That is, the optical image of the captured image i1 differs in the magnitude of distortion caused by the position (zoom value) of the zoom lens of the lens unit, for example. Therefore, the image correction unit 5b has a distortion with a predetermined intensity with respect to the image data acquired by the image acquisition unit 5a based on the optical characteristic information related to the distortion of the optical image associated with the zoom value of the lens unit. Aberration correction is performed to correct distortion of the captured image i1 caused by the optical characteristics of the lens unit of the imaging unit 1.
Here, the image correction unit 5b constitutes a correction unit that corrects distortion caused by the optical characteristics of the imaging unit 1 that captured the display image S in the captured image i1 acquired by the image acquisition unit 5a.

The contour extraction unit 5c performs a contour extraction process on the captured image i1 to extract a predetermined image region.
That is, for example, the contour extraction unit 5c performs a straight line detection process (for example, a Hough transform process) on the luminance signal Y of the YUV data after distortion correction by the image correction unit 5b to detect a straight line in a predetermined direction. A quadrangle formed by four straight lines extending to is specified as an outline. Then, the contour extraction unit 5c extracts an image area (square area A1; see FIG. 4C) within the specified quadrangle as a predetermined image area. Here, when a plurality of quadrangles are specified, the contour extracting unit 5c extracts image regions in the plurality of quadrangles.
Thereby, the substantially rectangular (right-angled quadrilateral) image displayed on the display screen 201 of the display device 200, that is, the display image S itself, the file name of the display image S in addition to the display image S, and the like are displayed. An image of an area (so-called “window”) including various title bars and GUI (Graphical User Interface) buttons is extracted as a predetermined image area (square area A1). When the display image S is displayed on the entire display screen 201 of the display device 200, each side of the display screen 201 having a substantially rectangular shape may be specified as the outline of the square area A1.
Here, the contour extracting unit 5c constitutes an extracting unit that performs a contour extracting process on the captured image i1 acquired by the image acquiring unit 5a to extract a square region (predetermined image region) A1.

In addition, although the thing using a straight line detection process was illustrated as an outline extraction process, it is an example and is not restricted to this, Arbitrary arbitrarily if it can extract square area (predetermined image area) A1 Can be changed.
In addition, although the Hough conversion process is illustrated as the straight line detection process, it is an example, and the present invention is not limited to this, and can be arbitrarily changed as appropriate, such as an edge detection process. The Hough transform process and the edge detection process are well-known techniques, and thus detailed description thereof is omitted here.

The quadrangle conversion processing unit 5d performs a right-angled quadrangle conversion process on the square area A1 extracted by the contour extraction unit 5c to generate a right-angled quadrilateral image i2 (see FIG. 5A).
That is, an image in a distorted quadrangle that does not form a right-handed quadrilateral shape (inner angles are not right-angled) by being imaged in a state where the display screen 201 of the display device 200 and the image-capturing device 100 are not exactly facing each other. The region is extracted as a predetermined image region by the contour extracting unit 5c. Therefore, the quadrilateral conversion processing unit 5d is perpendicular to the square region (predetermined image region) A1 extracted by the contour extraction unit 5c from the luminance signal Y whose distortion is corrected by the distortion correction by the image correction unit 5b. A quadrangle conversion process (for example, a trapezoid correction process) is performed to generate a right-angled quadrilateral image i2 (for example, a rectangular image, a square image, etc.). Here, when the plurality of square areas A1 are extracted by the contour extraction unit 5c, the quadrilateral conversion processing unit 5d performs a predetermined operation of the operation input unit 10 by the user in the plurality of square areas A1. Based on the rectangular area A1 selected and specified by the central control unit 11 based on the processing, a right-angled quadrangle conversion process is performed.
Further, the aspect ratio of the right-angled quadrilateral image after the right-angle quadrature transformation processing can be calculated based on the aspect ratio calculated from the acquired display dot number of the display screen 201.
The right-angled quadrilateral image i2 generated in this way is an image having sides extending along the vertical direction and the horizontal direction, and the pitches of the plurality of dots corresponding to the display image S are made substantially uniform. Can do. Further, a lattice-like pattern (gap region C between a plurality of dots) appearing superimposed on the display image S is also extended along each of the vertical direction and the horizontal direction.

Note that the trapezoid correction process is exemplified as the right-angled quadrangle conversion process, but is not limited to this example, and can be arbitrarily changed as long as it is an oblique conversion process that converts a quadrangle into a right-angled quadrilateral shape. . Since the trapezoid correction process is a known technique, detailed description is omitted here. In this way, the quadrilateral conversion processing unit 5d is a rectangular area (predetermined value) of the captured image i1 acquired by the image acquisition unit 5a. The first processing means for generating a right-angled quadrilateral image i2 by performing a right-angled quadrangle conversion process on the image area A1.

The gap pixel number calculation unit 5e calculates a grid pattern in the right-angled quadrilateral image i2 generated by the quadrilateral conversion processing unit 5d, that is, the number of pixels constituting the gap region C between adjacent dots along a predetermined direction. calculate.
That is, the gap pixel number calculation unit 5e specifies a plurality of gap areas C along a direction substantially parallel to each of the vertical and horizontal directions of the right-angled quadrilateral image i2, and then configures each of the plurality of gap areas C. The number of pixels to be calculated is calculated. Specifically, the gap pixel number calculation unit 5e first specifies an arbitrary processing target area A2 in which pixels brighter than a predetermined threshold occupy a predetermined ratio (for example, a majority or more) in the rectangular quadrilateral image i2. . The processing target area A2 has, for example, sides that are substantially parallel to the sides of the right-angled quadrilateral image i2.
Next, the gap pixel number calculation unit 5e calculates the cumulative frequency distribution of the luminance values in the processing target area A2, and for example, assumes that a pixel having a predetermined distribution ratio or less is the gap area C, A luminance value corresponding to the distribution ratio is set as a predetermined threshold value. Then, the gap pixel number calculating unit 5e sets the processing target line along each of a plurality of directions parallel to one of the vertical direction and the horizontal direction (for example, the vertical direction) in the processing target area A2. Then, a plurality of areas each including pixels having a luminance value smaller than a predetermined threshold are specified as the gap area C by scanning each processing target line. Subsequently, the gap pixel number calculation unit 5e calculates the number of pixels (see FIG. 3) constituting each of the plurality of gap areas C for each of a plurality of processing target lines parallel to each other.
For example, as shown in FIG. 3, the pixel corresponding to the display image S itself (dot) has a large luminance value, while the pixel corresponding to the gap region C between adjacent dots has a small luminance value. The example illustrated in FIG. 3 is an example of the luminance values of a plurality of pixels along one processing target line. For example, when the processing target line is on the gap area C, only the pixels corresponding to the gap area C are displayed. That is, it is occupied only by pixels having a small luminance value. In this case, as will be described later, it is excluded from the filter size specifying process by the filter size specifying unit 5f.

In addition, the gap pixel number calculation unit 5e also performs a plurality of gap areas C in the same manner as described above in the other direction (for example, the horizontal direction) that is not the process target among the vertical direction and the horizontal direction of the process target area A2. The number of pixels constituting each of the above is calculated.
As described above, the gap pixel number calculation unit 5e configures the gap region C corresponding to the gap between dots adjacent in the predetermined direction of the display image S in the right-angled quadrilateral image i2 generated by the quadrilateral conversion processing unit 5d. The calculation means for calculating the number of pixels to be configured is configured.

The filter size specifying unit 5f specifies the size of a predetermined filter used for the filter processing for the right-angled quadrilateral image i2.
That is, the filter size specifying unit 5f is a predetermined two-dimensional filter (for example, a median filter) used for a filter process performed as a gap interpolation process for interpolating the image data of the gap area C of the rectangular quadrilateral image i2 by the filter processing unit 5g. Etc.) size (number of taps). That is, the size of the predetermined two-dimensional filter used for the filter processing defines the calculation processing range of one pixel to be processed in the rectangular quadrilateral image i2. Therefore, the filter size specifying unit 5f sets the size of a predetermined two-dimensional filter used for the filter process as the gap interpolation process based on the number of pixels constituting the gap area C calculated by the gap pixel number calculation unit 5e. This is specified as the calculation processing range of the pixel.

Specifically, the filter size specifying unit 5f first sets one of the vertical direction and the horizontal direction (for example, the vertical direction) of the processing target area A2 of the right-angled quadrilateral image i2 as the processing target direction. . Next, the filter size specifying unit 5f sets the maximum value (for example, in FIG. 3) among the plurality of pixels constituting the gap region C for each of the plurality of processing target lines parallel to the processing target direction. Calculate “2” as the maximum value. Then, the filter size specifying unit 5f is a representative representing the number of pixels in the processing target direction (the extending direction of the plurality of processing target lines parallel to each other) based on the calculated maximum value of the plurality of pixels. The value is specified, and the size of the two-dimensional filter used for the filter processing is specified based on the specified representative value.
That is, when the processing target line is set on a straight line extending in a predetermined direction constituting the gap region C having a lattice shape, most (for example, a majority or more) are formed by pixels having a small luminance value corresponding to the straight line. Therefore, the value of the number of pixels constituting the gap region C increases in the processing target line. Therefore, the filter size specifying unit 5f specifies, for example, the mode value or the average value as the representative value among the maximum values of the plurality of pixels related to the plurality of processing target lines parallel to the processing target direction. The processing target line in which the pixels having a small luminance value occupy most is excluded. Then, for example, the filter size specifying unit 5f calculates the number of taps “tap” of the median filter used for the filter processing according to the following formula (1), with the representative value specified for the processing target direction as “gap”.
tap = 2gap +1 ... Formula (1)

For example, FIG. 3 illustrates a case where the maximum value “gap” among the plurality of pixels constituting the gap region C in one processing target line is “2”. In this case, the median filter As the number of taps “tap”, “5” is calculated, and 5 × 5 pixels are specified as the filter size.
The above equation (1) is an example of an arithmetic expression for calculating the number of taps “tap” of the median filter, and is not limited to this, and the coefficients, constants, and the like can be arbitrarily changed as appropriate. Further, the arithmetic expression can be arbitrarily changed as appropriate depending on the type of filter used for the filter processing.

In addition, the filter size specifying unit 5f is also used for the filter processing in the same manner as described above for the other direction (for example, the horizontal direction) that is not the processing target among the vertical direction and the horizontal direction of the processing target area A2. Specify the size of the dimensional filter.
The filter size specifying unit 5f specifies the final filter size based on the filter size specified with each of the vertical direction and the horizontal direction of the processing target area A2 as the processing target direction. Here, when different sizes are specified depending on the direction to be processed among the vertical direction and the horizontal direction of the processing target area A2, different sizes in the vertical direction and the horizontal direction may be used as they are. The final filter size may be specified based on the maximum value.
As described above, the filter size specifying unit 5f corrects the gap interpolation process (gap correction process) for correcting the image data of the gap region C of the right-angled quadrilateral image i2 based on the number of pixels calculated by the gap pixel number calculation unit 5e. A specifying means for specifying the processing range of one pixel to be processed is configured.

The filter processing unit 5g performs a filtering process on the rectangular quadrilateral image i2 using a predetermined two-dimensional filter (for example, a median filter) having a size specified by the filter size specifying unit 5f.
That is, the filter processing unit 5g performs the filter process as a gap interpolation process for interpolating the image data of the gap area C of the right-angled quadrilateral image i2. At this time, the filter processing unit 5g specifies the filter size specified by the filter size specifying unit 5f, that is, the filter size that defines the calculation processing range of one pixel to be processed in the right-angled quadrilateral image i2 (for example, 5 Filter processing is performed on the right-angled quadrilateral image i2 on the basis of × 5 pixels). Specifically, the filter processing unit 5g performs, for example, a filtering process using a median filter having a size of 5 × 5 pixels on each pixel of the right-angled quadrilateral image i2, so that a gap area between adjacent dots is obtained. The luminance value of the pixel constituting C is replaced with the luminance value of the pixel corresponding to the display image S itself (dot) in the vicinity of the pixel. As a result, the filter processing unit 5g interpolates (corrects) the gap-interpolated image i3 (FIG. 5B) obtained by interpolating (correcting) the lattice-like pattern (gap region C between the plurality of dots) appearing superimposed on the display image S. )) Is generated.

Note that, for example, a filtering process using a median filter has been illustrated. However, the filtering process is an example, and the filtering process is not limited thereto. The filtering process can change the luminance value of the pixels constituting the gap region C more greatly. Any change can be made as appropriate. For example, a low-pass filter, a maximum value filter, or the like may be applied as the two-dimensional filter.
Further, the image data of the gap area C of the right-angled quadrilateral image i2 is corrected by the gap interpolation process (filter process), but the correction method of the gap area C is not limited to this. For example, an image that reduces the difference in pixel value between the gap area C and the surrounding area, such as a method of correcting the luminance value of each pixel in the gap area C so as to reduce the luminance difference with respect to neighboring pixels. Any method may be applied as long as it is a process.
Here, the filter processing unit 5g has the image data of the gap region C corresponding to the gap between adjacent dots in the predetermined direction of the display image S with respect to the right-angled quadrilateral image i2 generated by the quadrilateral conversion processing unit 5d. The second processing means for performing the gap interpolation process (gap correction process) for correcting.

  The encoding unit 6 encodes the image data (YUV data) of the gap-interpolated image i3 after the image processing by the image processing unit 5 in a predetermined compression format (for example, JPEG format), and records the image data for recording. Generate.

The recording medium control unit 7 is configured so that the recording medium M is detachable, and controls reading of data from the loaded recording medium M and writing of data to the recording medium M. That is, the recording medium control unit 7 records the recording image data encoded by the encoding unit 6 on the recording medium M.
Here, the recording medium control unit 7 constitutes a control unit that records the image i3 after the gap interpolation processing (filter processing) by the filter processing unit 5g on the recording medium M.

  The recording medium M is composed of, for example, a non-volatile memory (flash memory) or the like. However, the recording medium M is an example and is not limited to this, and can be arbitrarily changed as appropriate.

The display control unit 8 performs control for reading display image data temporarily stored in the memory 4 and displaying the read image data on the display unit 9.
Specifically, the display control unit 8 includes a VRAM (Video Random Access Memory), a VRAM controller, a digital video encoder, and the like. The digital video encoder reads the luminance signal Y and the color difference signals Cb and Cr read from the memory 4 and stored in the VRAM (not shown) under the control of the central control unit 11 via the VRAM controller. Are periodically read out, and a video signal is generated based on these data and output to the display unit 9.

  The display unit 9 is, for example, a liquid crystal display panel, and displays an image captured by the electronic imaging unit on the display screen based on a video signal from the display control unit 8. Specifically, the display unit 9 sequentially updates a plurality of image frames generated by subject imaging by the imaging unit 1 and the imaging control unit 2 at a predetermined frame rate in the still image capturing mode and the moving image capturing mode. While displaying the live view image. The display unit 9 displays an image (rec view image) recorded as a still image or an image being recorded as a moving image.

  The operation input unit 10 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 10 includes a shutter button related to a subject shooting instruction, a selection determination button related to a selection instruction for an imaging mode, a function, etc., a zoom button related to a zoom amount adjustment instruction, etc. Abbreviation), a predetermined operation signal is output to the central control unit 11 in accordance with the operation of these buttons.

In addition, the operation input unit 10 specifies a designation signal for the square area A1 to be processed by the quadrilateral conversion processing unit 5d among the plurality of square areas A1 extracted by the contour extraction unit 5c based on a predetermined operation by the user. Is output. That is, the operation input unit 10 selects one of the square areas A1 selected based on a predetermined operation by the user among the plurality of square areas A1 extracted by the contour extraction unit 5c and displayed on the display unit 9. A designation signal is output to the central control unit 11. When the designation signal output from the operation input unit 10 is input, the central control unit 11 designates the square area A1 to be processed by the quadrilateral transformation processing unit 5d in accordance with the designation instruction signal.
Here, the central control unit 11 and the operation input unit 10 designate a quadrangular area A1 to be subjected to the right-angled quadrangle conversion processing by the quadrangle conversion processing unit 5d among the plural quadrangular areas (predetermined image areas) A1. It constitutes the designation means.

  Note that the quadrangular area A1 to be processed by the quadrilateral conversion processing unit 5d is manually specified based on a predetermined operation of the operation input unit 10 by the user, but this is the method for specifying the quadrangular area A1. It is not limited. That is, the central control unit 11 may automatically specify and specify the square area A1 to be processed by the quadrilateral conversion processing unit 5d by image analysis or character recognition, for example.

  The central control unit 11 controls each unit of the imaging device 100. Specifically, although not shown, the central control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and includes various processing programs for the imaging apparatus 100 ( Various control operations are performed according to (not shown).

Next, the gap interpolation imaging process by the imaging apparatus 100 will be described with reference to FIG.
FIG. 2 is a flowchart illustrating an example of an operation related to the gap interpolation imaging process.
The gap interpolation imaging process is executed when a gap interpolation imaging mode is instructed from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button of the operation input unit 10 by the user. It is processing. Further, in the gap interpolation imaging process, the imaging unit 1 captures the display image S displayed at a predetermined position on the display screen 201 of the display device 200 (see FIG. 4A) as a subject.

First, the display control unit 8 performs a process of displaying a live view image on the display screen 201 of the display unit 9 based on a plurality of image frames generated by imaging of the subject by the imaging unit 1 and the imaging control unit 2 (steps). S1). At this time, the imaging control unit 2 may detect flicker or the like on the display screen 201 of the display device 200 and set the exposure condition in accordance with the detected flicker.
Then, the central control unit 11 determines whether an imaging instruction is input based on a predetermined operation of the shutter button of the operation input unit 10 by the user during the live view display process (step S2).

If it is determined that an imaging instruction has not been input (step S2; NO), the central control unit 11 continues until it is determined in step S2 that an imaging instruction has been input (step S2; YES). Shifts the process to step S2 to repeatedly perform the process of displaying the live view image.
When it is determined in step S2 that an imaging instruction has been input (step S2; YES), the imaging control unit 2 performs, for example, an exposure condition (shutter) corresponding to flicker or the like of the display screen 201 of the display device 200. The imaging conditions such as speed, aperture, amplification factor, etc.) and white balance are adjusted, and the optical image of the display image S displayed on the display device 200 is captured by the electronic imaging unit of the imaging unit 1 (step S3). Thereafter, the image data generation unit 3 generates YUV data of the captured image i1 (see FIG. 4B) transferred from the electronic imaging unit.
The generated YUV data is DMA-transferred to a memory 4 used as a buffer memory via a DMA controller (not shown).
At this time, the display control unit 8 may read the image data for display of the captured image i1 temporarily stored in the memory 4 and display it on the display unit 9.

Next, the image acquisition unit 5a of the image processing unit 5 acquires YUV data of the captured image i1 generated by the image data generation unit 3 from the memory 4 (step S4).
Subsequently, the image correction unit 5b performs distortion aberration correction with a predetermined intensity on the basis of the zoom value of the lens unit for the acquired YUV data, and is caused by the optical characteristics of the lens unit of the imaging unit 1. The distortion of the captured image i1 is corrected (step S5).

Next, the contour extraction unit 5c performs straight line detection processing (for example, Hough transform processing) on the luminance signal Y of the YUV data after distortion correction by the image correction unit 5b, and then detects a straight line. A quadrangle formed by four straight lines extending to is specified, and an image region (square region A1) in the quadrangle is extracted (step S6; see FIG. 4C).
When a plurality of square areas A1 are extracted by the contour extraction unit 5c, the display control unit 8 displays the plurality of square areas A1 as candidates for processing targets of a right-angled quadrangle conversion process (described later). 9 is displayed. That is, for example, the display control unit 8 displays a frame line along the outer edge portion of each square area A1, and displays the plurality of square areas A1 in a switchable manner. Then, the central control unit 11 sets the rectangular area A1 selected and specified based on a predetermined operation of the operation input unit 10 by the user as a processing target of the right-angled quadrangle conversion process.

Next, the quadrangle conversion processing unit 5d performs a right-angled quadrangle conversion process (for example, a trapezoid correction process) on the square area A1 extracted from the luminance signal Y by the contour extraction unit 5c, thereby generating a right-angled quadrilateral image i2. (For example, a rectangular image) is generated (step S7; see FIG. 5A).
Subsequently, the gap pixel number calculation unit 5e specifies the gap region C between adjacent dots in the right-angled quadrilateral image i2 generated by the quadrilateral conversion processing unit 5d, and determines the number of pixels constituting the gap region C. Calculate (step S8). Specifically, the gap pixel number calculation unit 5e specifies an arbitrary processing target area A2 in which pixels brighter than a predetermined threshold occupy a predetermined ratio in the rectangular quadrilateral image i2. Thereafter, the gap pixel number calculation unit 5e calculates a cumulative frequency distribution of the luminance values of each pixel in the processing target area A2, and sets the luminance value corresponding to a predetermined distribution rate as a predetermined threshold value. Subsequently, the gap pixel number calculation unit 5e processes the processing target lines along each of a plurality of directions parallel to one of the vertical direction and the horizontal direction (for example, the vertical direction) in the processing target area A2. And each of the processing target lines is scanned, and a plurality of areas each including a pixel having a luminance value smaller than a predetermined threshold are specified as gap areas C. Subsequently, the gap pixel number calculation unit 5e calculates the number of pixels constituting each of the plurality of gap areas C for each of the plurality of processing target lines parallel to each other.
In addition, the gap pixel number calculation unit 5e configures each of the plurality of gap areas C in the same manner as described above in the other direction (for example, the horizontal direction) that is not the processing target among the vertical direction and the horizontal direction. The number of pixels is calculated.

Next, the filter size specifying unit 5f specifies the size of a predetermined two-dimensional filter (for example, a median filter) used for the filtering process on the right-angled quadrilateral image i2 (step S9).
More specifically, the filter size specifying unit 5f includes a plurality of processing target lines that are parallel to the processing target direction (for example, the vertical direction) of the processing target area A2 of the rectangular quadrilateral image i2 in the vertical direction and the horizontal direction. For each of the above, a maximum value among a plurality of pixels constituting the gap region C is calculated. Then, the filter size specifying unit 5f specifies a representative value that represents the number of pixels in the processing target direction based on the maximum value of the specified number of pixels, and then specifies the specified representative value as “gap The number of taps “tap” of the median filter used for the filter processing is calculated according to the following formula (1).
tap = 2gap +1 ... Formula (1)
In addition, the filter size specifying unit 5f performs predetermined processing used for the filter processing in the same manner as described above in the other direction (for example, the horizontal direction) that is not the processing target among the vertical direction and the horizontal direction of the processing target area A2. Specifies the size of the two-dimensional filter.
Then, the filter size specifying unit 5f specifies the final filter size based on the filter size specified with each of the vertical direction and the horizontal direction of the processing target area A2 as the processing target direction.

Next, the filter processing unit 5g performs a filtering process on the rectangular quadrilateral image i2 using a predetermined two-dimensional filter (for example, a median filter) having a size specified by the filter size specifying unit 5f (step S10).
Specifically, the filter processing unit 5g performs the filtering process on each pixel of the right-angled quadrilateral image i2, thereby obtaining the luminance value of the pixel constituting the gap region C between adjacent dots, in the vicinity of the pixel. Is replaced with the luminance value of the pixel corresponding to the display image S itself (dot). As a result, the filter processing unit 5g generates image data of the gap-interpolated image i3 obtained by interpolating the grid pattern (gap area C between the plurality of dots) appearing superimposed on the display image S.

Thereafter, the encoding unit 6 encodes the image data (YUV data) of the gap-interpolated image i3 after the image processing by the image processing unit 5 in a predetermined compression format (for example, JPEG format) and the like. Data is generated (step S11).
Subsequently, the recording medium control unit 7 records the image data for recording of the gap interpolation processed image i3 encoded by the encoding unit 6 on the recording medium M (step S12). At this time, the display control unit 8 acquires the image data for display of the gap interpolation processed image i3 and causes the display unit 9 to display the acquired image data.
Thus, the gap interpolation imaging process is finished.

As described above, according to the imaging device 100 of the present embodiment, the right-angled quadrangle conversion process is performed on the predetermined image region of the captured image i1 obtained by capturing the display image S displayed on the display screen 201 of the display device 200. Since the right-angled quadrilateral image i2 is generated, it is possible to appropriately specify the gap region C corresponding to the gap between dots adjacent to each other in the predetermined direction of the display image S in the right-angled quadrilateral image i2. Thereby, the subsequent gap interpolation process (gap correction process) for the right-angled quadrilateral image i2 can be easily performed with a simple circuit configuration with a small processing load, and the processing efficiency and speed of the gap interpolation process are improved. In addition, the gap region C (lattice pattern) from the quadrilateral quadrilateral image i2 can be appropriately interpolated without causing image quality deterioration or resolution reduction of the image data after the gap interpolation processing.
In this way, a gap between adjacent dots from the right-angled quadrilateral image i2 obtained by deforming the image obtained by capturing the display image S, that is, an image corrected by interpolating information unnecessary for display of the image is generated. It can be performed simply and appropriately.

Further, since the right-angled quadrilateral image i2 is generated by performing the right-angled quadrangle conversion process, the gap region C in the right-angled quadrilateral image i2 can be appropriately specified, so the number of pixels constituting the gap region C Can be appropriately calculated.
Thereby, based on the calculated number of pixels in the gap region C, it is possible to appropriately specify the processing range of one pixel to be processed in the gap interpolation process. Specifically, the size of a predetermined two-dimensional filter used for filter processing that defines the processing range of one pixel to be processed is specified, and the filter processing is performed using the predetermined two-dimensional filter of the size. Therefore, the gap region C in the right-angled quadrilateral image i2 can be appropriately interpolated without causing the image quality deterioration of the image data after the filter processing. That is, since the filtering process for the rectangular quadrilateral image i2 can be uniformly performed by applying a predetermined two-dimensional filter of a specified size, the filtering process is easily performed with a simple circuit configuration with a small processing load. be able to. As a result, it is possible to improve the processing efficiency and speed of the filtering process, and to interpolate the gap region C from the rectangular quadrilateral image i2 without causing image quality deterioration or resolution reduction of the filtered image data. Can be performed properly.

Also, a plurality of gap areas C are specified along a plurality of directions parallel to any one side of the processing target area A2 of the rectangular quadrilateral image i2, and pixels constituting each of the plurality of gap areas C in each direction. Calculate the number. Then, based on the calculated maximum value among the plurality of pixels in each direction, a representative value representative of the plurality of pixels in a plurality of directions parallel to each other is specified, and the specified representative value is used as a reference. Since the size of the predetermined two-dimensional filter used for the filtering process is specified, the size of the predetermined two-dimensional filter can be optimized, and the luminance value of the pixels constituting the gap region C by the filtering process Can be properly replaced.
That is, the processing target line set so as to be parallel to the processing target direction in the vertical direction and the horizontal direction of the processing target area A2 is a straight line extending in a predetermined direction constituting the lattice-shaped gap region C. In this case, the calculated value of the number of pixels constituting the gap area C is increased. Therefore, based on the maximum value among the plurality of pixels corresponding to the plurality of processing target lines in the processing target direction, those related to the processing target line in which pixels having a small luminance value occupy most are excluded. A representative value of a plurality of pixels in the processing target direction is specified, and a size of a predetermined two-dimensional filter used for the filter processing is specified using the specified representative value as a reference.
As described above, the filter size can be specified in consideration of the processing target line set on the straight line extending in a predetermined direction constituting the lattice-shaped gap region C in the processing target region A2. The size of the predetermined two-dimensional filter can be optimized.

In addition, since a predetermined image region is extracted by performing contour extraction processing on the captured image i1, it is possible to automatically specify a region to be processed by the right-angled quadrangle conversion processing.
Here, even when a plurality of predetermined image areas are extracted, the predetermined image area to which the central control unit 11 is subjected to the right-angled quadrangle conversion processing based on the predetermined operation of the operation input unit 10 by the user. Therefore, a user-desired area among a plurality of predetermined image areas can be set as a processing object of the right-angled quadrangle conversion process.

  Furthermore, since the distortion of the captured image i1 caused by the optical characteristics of the lens unit of the imaging unit 1 is corrected, it is possible to appropriately perform a right-angled quadrangle conversion process for a predetermined image region of the captured image i1 thereafter. Accordingly, the gap region C in the right-angled quadrilateral image i2 can be specified more appropriately, and the number of pixels constituting the gap region C can be calculated more appropriately.

  Further, high-frequency components can be appropriately removed from the captured image i1 by interpolation of the image data of the gap region C of the right-angled quadrilateral image i2, and the encoding efficiency by a predetermined compression format is improved as compared with the conventional method. be able to. As a result, a large amount of image data can be recorded on the recording medium M on which the image after the gap interpolation processing is recorded, and the recording (storage) efficiency of the recording medium M can be improved.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the above embodiment, the filter process is exemplified as the gap correction process (gap interpolation process). However, the filter process is an example, and the present invention is not limited to this, and can be arbitrarily changed as appropriate. That is, in the present invention, before the gap correction process, a right-angled quadrilateral image i2 is generated by performing a right-angled quadrangle conversion process on a predetermined image area of the captured image i1, thereby generating a right-angled quadrilateral image i2. It is possible to appropriately specify the gap region C and simplify subsequent gap correction processing. Therefore, in order to improve the processing efficiency and speed of the gap correction processing, it is preferable to use a simple circuit configuration with a small processing load that applies a two-dimensional filter of a predetermined size uniformly as in this embodiment. However, any other configuration, for example, various conventional methods may be applied.

In the above embodiment, a processing target line is set only in a predetermined direction parallel to any one side of the processing target area A2 of the right-angled quadrilateral image i2, and a gap is formed along the processing target line. A plurality of areas C may be specified. That is, for example, image analysis is performed on the processing target region, a line in which pixels having a small luminance value occupy most of the predetermined direction is specified, and the processing target line is set so as not to overlap the line. .
In this case, the number of pixels constituting each of the plurality of gap regions C specified along the processing target line is calculated, and the filter processing is performed using the maximum value among the calculated number of pixels as a reference. By specifying the size of the predetermined two-dimensional filter used in the above, it is possible to optimize the size of the predetermined two-dimensional filter.

  Furthermore, in the above-described embodiment, the filter size specifying unit 5f sets each of the vertical direction and the horizontal direction of the processing target area A2 of the rectangular quadrilateral image i2 as the processing target direction when specifying the filter size. However, the present invention is not limited to this, and only one of the vertical direction and the horizontal direction of the processing target area A2 may be set as the processing target direction. That is, since each of the plurality of dots on the display screen 201 of the display device 200 has a substantially square shape, one of the vertical direction and the horizontal direction of the processing target area A2 is set as the processing target direction. The filter size specified in (1) may be the final filter size of a predetermined two-dimensional filter used for the filter processing.

In the above embodiment, a contour extraction process is performed on the captured image i1 to extract a predetermined image region (square region A1). However, the present invention is not limited to this. Whether or not to perform the process of extracting the region can be arbitrarily changed as appropriate.
Furthermore, in the above embodiment, the distortion caused by the optical characteristics of the lens unit of the imaging unit 1 in the captured image i1 is corrected. However, the present invention is not limited to this, and the distortion occurred in the captured image i1. Whether or not to perform the process of correcting the distortion can be arbitrarily changed as appropriate.

  In the above embodiment, the calculation of the number of gap pixels in the gap area C of the right-angled quadrilateral image i2 and the specification of the filter size for the filter process are performed on the predetermined processing target area A2 in the right-angled quadrilateral image i2. However, the present invention is not limited to this, and for example, the entire right-angled quadrilateral image i2 may be processed.

Furthermore, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto.
That is, the display image S displayed on the display device 200 is captured by the imaging device 100 to acquire the captured image i1, but for example, the captured image captured by an imaging device other than the imaging device 100 May be acquired via the recording medium M or a predetermined communication line. Further, the image data of the right-angled quadrilateral image i2 (image data of the gap-interpolated image i3) after the image processing by the image processing unit 5 is recorded on the recording medium M. May be transferred to a predetermined recording means (for example, a hard disk drive) connected via a predetermined communication line.
In addition, the imaging apparatus 100 has been exemplified as the image processing apparatus, but the configuration is not limited thereto, and any configuration may be used as long as the predetermined image processing according to the present invention can be performed.

In addition, in the above-described embodiment, the functions of the acquisition unit, the first processing unit, and the second processing unit are controlled by the central control unit 11 so that the image acquisition unit 5a of the image processing unit 5 has a quadrilateral shape. Although the configuration realized by driving the conversion processing unit 5d and the filter processing unit 5g is not limited to this, the configuration realized by executing a predetermined program or the like by the central control unit 11 Also good.
That is, a program including an image acquisition processing routine, a first processing routine, and a second processing routine is stored in a program memory (not shown) that stores the program. Then, the CPU of the central control unit 11 is caused to function as an acquisition unit that acquires the captured image i1 obtained by capturing the display image S displayed on the display screen 201 in which a plurality of dots are arranged in a matrix by the acquisition processing routine. Anyway. In addition, the CPU of the central control unit 11 performs a right-angled quadrangle conversion process on a predetermined image area of the captured image i1 acquired by the acquisition unit by the first processing routine to generate a right-angled quadrilateral image i2. You may make it function as. Further, the CPU of the central control unit 11 in the second processing routine causes the gap area corresponding to the gap between the dots adjacent in the predetermined direction of the display image S to the right-angled quadrilateral image i2 generated by the first processing means. You may make it function as a 2nd processing means which performs the clearance correction process which corrects the C image data.

  Similarly, the calculation unit, the identification unit, the extraction unit, the designation unit, and the correction unit may be realized by executing a predetermined program or the like by the CPU of the central control unit 11.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

DESCRIPTION OF SYMBOLS 100 Image pick-up device 1 Image pick-up part 2 Image pick-up control part 5 Image processing part 5a Image acquisition part 5b Image correction part 5c Outline extraction part 5d Quadrangle conversion processing part 5e Gaps pixel number calculation part 5f Filter size specific part 5g Filter processing part 7 Recording medium Control unit 10 Operation input unit 11 Central control unit M Recording medium

Claims (11)

An acquisition means for acquiring a captured image obtained by capturing a display image displayed on a display screen in which a plurality of dots are arranged in a matrix;
First processing means for generating a right-angled quadrilateral image by performing a right-angled quadrangle conversion process on a predetermined image area of the captured image acquired by the acquisition means;
A second process for performing a gap correction process for correcting the image data of the gap area corresponding to the gap between adjacent dots in the predetermined direction of the display image with respect to the right-angled quadrilateral image generated by the first processing means. Means,
An image processing apparatus comprising: Calculating means for calculating the number of pixels constituting the gap area in the right-angled quadrilateral image generated by the first processing means;
A specifying unit for specifying a processing range of one pixel to be processed in the gap correction processing based on the number of pixels calculated by the calculating unit;
The second processing means includes
The image processing apparatus according to claim 1, wherein the gap correction process is performed on the right-angled quadrilateral image with reference to a processing range of the one pixel specified by the specifying unit. The specifying unit defines a processing range of one pixel to be processed based on the number of pixels calculated by the calculating unit, and a size of a predetermined filter used for the filter processing as the gap correction processing Identify
The second processing means performs the filtering process on the right-angled quadrilateral image using a predetermined filter having a size specified by the specifying means, and corrects the image data of the gap area in the right-angled quadrilateral image. The image processing apparatus according to claim 2. The calculation means specifies a plurality of the gap areas along a predetermined direction parallel to any one side of the right-angled quadrilateral image, and calculates the number of pixels constituting each of the plurality of gap areas. ,
The said specifying means specifies the size of the predetermined filter used for the said filter process on the basis of the maximum value in the said some pixel count calculated by the said calculation means. The image processing apparatus described. The calculation means specifies a plurality of the gap regions along a plurality of directions parallel to any one side of the right-angled quadrilateral image, and the number of pixels constituting each of the plurality of gap regions in each direction To calculate
The specifying unit specifies a representative value representing the number of pixels in the plurality of parallel directions based on the maximum value among the plurality of pixels in each direction calculated by the calculating unit. The image processing apparatus according to claim 4, wherein a size of a predetermined filter used for the filter processing is specified based on the specified representative value. The image processing apparatus further includes an extraction unit that performs a contour extraction process on the captured image acquired by the acquisition unit and extracts the predetermined image region,
The first processing means includes
The image processing apparatus according to claim 1, wherein the right-angled quadrilateral transformation process is performed on the predetermined image area extracted by the extraction unit. The extraction means extracts a plurality of the predetermined image areas,
7. The specifying device according to claim 6, further comprising: a specifying unit that specifies a predetermined image region to be subjected to the right-angled quadrilateral transformation process by the first processing unit among the plurality of predetermined image regions. Image processing device. A correction unit that corrects distortion caused by optical characteristics of the imaging unit that captured the display image in the captured image acquired by the acquisition unit;
The first processing means includes
The image processing apparatus according to claim 1, wherein the right-angled quadrangle conversion process is performed on a predetermined image area of the captured image whose distortion is corrected by the correction unit.   The image processing apparatus according to claim 1, further comprising a control unit that causes the recording unit to record an image after the gap correction processing by the second processing unit. An image processing method using an image processing apparatus including an acquisition unit that acquires a captured image obtained by capturing a display image displayed on a display screen in which a plurality of dots are arranged in a matrix,
A process of generating a right-angled quadrilateral image by performing a right-angled quadrangle conversion process on a predetermined image area of the captured image acquired by the acquisition unit;
A process of performing a gap correction process for correcting the image data of a gap area corresponding to a gap between dots adjacent in a predetermined direction of the display image with respect to the generated right-angled quadrilateral image;
An image processing method comprising: A computer of an image processing apparatus including an acquisition unit that acquires a captured image obtained by capturing a display image displayed on a display screen in which a plurality of dots are arranged in a matrix.
First processing means for generating a right-angled quadrilateral image by performing a right-angled quadrangle conversion process on a predetermined image area of the captured image acquired by the acquisition means;
A second process for performing a gap correction process for correcting the image data of the gap area corresponding to the gap between adjacent dots in the predetermined direction of the display image with respect to the right-angled quadrilateral image generated by the first processing means. means,
A program characterized by functioning as