WO2020196091A1 - Image processing apparatus and image processing method - Google Patents

Image processing apparatus and image processing method Download PDF

Info

Publication number
WO2020196091A1
WO2020196091A1 PCT/JP2020/011686 JP2020011686W WO2020196091A1 WO 2020196091 A1 WO2020196091 A1 WO 2020196091A1 JP 2020011686 W JP2020011686 W JP 2020011686W WO 2020196091 A1 WO2020196091 A1 WO 2020196091A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
display
imaging
processing apparatus
image processing
Prior art date
Application number
PCT/JP2020/011686
Other languages
French (fr)
Inventor
Haruhiko Higuchi
Original Assignee
Clarion Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Clarion Co., Ltd. filed Critical Clarion Co., Ltd.
Publication of WO2020196091A1 publication Critical patent/WO2020196091A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Definitions

  • the present invention relates to an image processing apparatus and an image processing method.
  • This application claims the priority based on the Japanese Patent Application No. 2019-58469 filed on March 26, 2019. The entire contents of which are incorporated herein by reference for all purpose.
  • Patent Literature 1 discloses a technique capable of easily adjusting a position deviation when superimposing a projection image onto an image included in a projection plane.
  • Patent Literature 1 discloses “a projection apparatus including: a projection unit configured to project an image onto a projection plane; an acquisition unit configured to acquire information related to colors of an image included in the projection plane; a generation unit configured to generate an alignment image for adjusting a position deviation between the image included in the projection plane and the image projected on the projection plane based on the information acquired by the acquisition unit; and a control unit configured to control the projection unit so as to project the alignment image in an overlapped manner onto the image on the projection plane, in which the generation unit generates the alignment image that has a predetermined relationship with the colors of the image included in the projection plane and indicates an amount of the position deviation from the image projected on the projection plane based on the information acquired by the acquisition unit”.
  • the technique disclosed in Patent Literature 1 can detect a target from the captured image and correct the shape of the image so as to minimize the position deviation.
  • Patent Literature 1 JP 2018-197824 A
  • a test evaluation for a product including a display there is a method that images an image displayed on the display by a camera, and checks the captured image against an image when the product is designed to determine whether the images match. In this case, it is necessary to detect a display part to be checked from the captured image and deform the image by obtaining a position deviation from the image against which the display part is to be checked.
  • Patent Literature 1 does not disclose a method for detecting a covered part of an object to be checked.
  • the peripheral edge part of the display is not imaged in the captured image.
  • the present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus and an image processing method that are capable of estimating an imaging position of a display image, the imaging position not being included in a captured image.
  • an image processing apparatus includes an imaging control unit configured to cause a display image to be imaged; an image analysis unit configured to analyze positional information of a feature point of a captured image obtained by imaging the display image; and an estimation unit configured to estimate an imaging position of the display image, the imaging position not being included in the captured image, based on the positional information of the feature point of the captured image.
  • the present invention it is possible to estimate the imaging position of the display image, the imaging position not being included in the captured image.
  • Fig. 1 is a block diagram illustrating the configuration of an image processing apparatus according to an embodiment.
  • Fig. 2 is a perspective view illustrating the configuration of an image display system to which the image processing apparatus according to the embodiment is applied.
  • Fig. 3(a) is a diagram illustrating an example of an expected value image
  • Fig. 3(b) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when a peripheral edge part is not covered with a housing frame
  • Fig. 3(c) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when the peripheral edge part is covered with the housing frame
  • Fig. 3(d) is a diagram illustrating an imaging example of the display image of Fig.
  • Fig. 4(a) is a diagram illustrating an example of boundary coordinates when an analysis image with no black frame pattern added is split into sixteen regions
  • Fig. 4(b) is a diagram illustrating an example of five pattern images which are used in generation of 16-split images
  • Fig. 4(c) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is not covered with the housing frame
  • Fig. 4(d) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is covered with the housing frame.
  • FIG. 5(a) is a diagram illustrating an example of boundary coordinates when an analysis image with a black frame pattern added is split into sixteen regions
  • Fig. 5(b) is a diagram illustrating an example of five pattern images obtained by adding the black frame pattern to the pattern images of Fig. 4(a) when the peripheral edge part is covered with the housing frame
  • Fig. 5(c) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is not covered with the housing frame
  • Fig. 5(d) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is covered with the housing frame
  • Fig. 6(a) is a diagram illustrating binary images of the captured images of Fig.
  • Fig. 6(b) is a diagram illustrating inverse images obtained by inverting the binary images of Fig. 6(a).
  • Fig. 7 is a diagram illustrating a logical AND operation method when 16-split images are generated based on the binary images of Fig. 6(a) and the inverse images of Fig. 6(b).
  • Fig. 8 is a diagram illustrating coordinates of feature points of the 16-split images analyzed from the captured image.
  • Fig. 9 is a diagram illustrating coordinate values of the peripheral edge part estimated based on the feature points of the 16-split images analyzed from the captured image.
  • Fig. 10 is a flowchart illustrating an image processing method according to the embodiment.
  • Fig. 11 is a block diagram illustrating a hardware configuration example of the image processing apparatus of Fig. 1.
  • FIG. 1 is a block diagram illustrating the configuration of an image processing apparatus according to the embodiment.
  • an image processing apparatus 100 includes an image input unit 101, an imaging control unit 102, an image analysis unit 103, an interpolative estimation unit 104, an image output unit 105, and an image deformation unit 106.
  • the image input unit 101, the imaging control unit 102, the image analysis unit 103, the interpolative estimation unit 104, the image output unit 105, and the image deformation unit 106 are connected to each other via a bus 107.
  • the bus 107 transfers image data, control information, and analysis information which are handled by each processing unit connected to the bus 107.
  • the image processing apparatus 100 is connected to a display device 110 and an imaging device 120 via the bus 107.
  • the image processing apparatus 100 may be connected to the display device 110 and the imaging device 120 by either wired or wireless connection.
  • Fig. 1 illustrates an example in which the display device 110 and the imaging device 120 are disposed outside the image processing apparatus 100. However, the display device 110 and the imaging device 120 may be incorporated in the image processing apparatus 100.
  • the image input unit 101 processes an input image input from an external connection device such as a personal computer (PC). For example, when the input image is an analog image, the image input unit 101 quantizes the analog image by decoding to convert the analog image to a digital image handled in the subsequent process.
  • the input image input by the image input unit 101 is referred to as an expected value image.
  • the expected value image is, for example, a design image against which a captured image captured by the imaging device 120 is to be checked.
  • the imaging control unit 102 causes the imaging device 120 to perform imaging for an image analysis by the image analysis unit 103. At this time, the imaging control unit 102 can set an exposure condition and white balance of the imaging device 120 so that the image analysis by the image analysis unit 103 is appropriately performed. Further, the imaging control unit 102 controls imaging of a display image in response to switching of the display image displayed by the display device 110. For example, when the imaging control unit 102 causes the imaging device 120 to image a display image displayed on the display device 110, the imaging control unit 102 sets an imaging condition under which gradation information of the captured image does not collapse, and sets the same exposure condition and the same white balance in the subsequent imaging.
  • the image analysis unit 103 analyzes shape information and color information which are included in the captured image obtained by imaging the display image displayed by the display device 110, obtains coordinate information of a feature point having a specific feature inside the captured image, and generates information required for interpolative estimation by the interpolative estimation unit 104 and generates information required for image deformation by the image deformation unit 106.
  • the image analysis unit 103 is capable of cutting out a range in which the display image is imaged and a range in which the display image is not imaged.
  • a feature value extraction method such as image edge detection or pattern matching can be used as a method for obtaining coordinate information of the feature point.
  • the interpolative estimation unit 104 estimates an imaging position of the display image which cannot be imaged by the imaging device 120 based on information of an analysis image displayed by the display device 110 and positional information of the feature point of the captured image analyzed by the image analysis unit 103, and generates information required for deformation by the image deformation unit 106.
  • the imaging position of the display image which cannot be imaged by the imaging device 120 is, for example, the position of a part hidden behind a housing frame when imaging is performed in a state in which the peripheral edge part of a display which displays the display image is covered with the housing frame.
  • the interpolative estimation unit 104 is, for example, capable of estimating the imaging position of the display image which cannot be imaged by the imaging device 120 by extrapolating the imaging position of the display image which cannot be imaged by the imaging device 120 based on the positional information of the displays image imaged by the imaging device 120.
  • the image output unit 105 outputs an image deformed by the image deformation unit 106 to the external connection device such as a PC.
  • the image deformation unit 106 deforms the captured image input from the imaging device 120 based on information obtained by the image analysis unit 103 and the interpolative estimation unit 104. At this time, the image deformation unit 106 performs the image deformation based on an interpolatively estimated image obtained by interpolatively estimating the imaging position of the display image which cannot be imaged by the imaging device 120.
  • the image deformation unit 106 is capable of performing the image deformation by affine transformation, projective transformation, or a combination of a plurality of transformations so that the shape of the interpolatively estimated image of the captured image captured by the imaging device 120 matches the shape of the expected value image input by the image input unit 101.
  • the image deformation unit 106 may perform the image deformation by a combination of perspective transformation of the interpolatively estimated image and N-region splitting free deformation (N is an integer equal to or larger than 2).
  • N is an integer equal to or larger than 2.
  • an image deformed by the image deformation unit 106 is referred to as a geometrically corrected image.
  • the display device 110 displays the analysis display image or the expected value image input by the image input unit 101.
  • the display device 110 may be a liquid crystal display device, an organic electro luminescence (EL) display, or a display using a cathode ray tube. Further, the display device 110 may be an information processing device including a display such as a car navigation system, a smartphone, or a tablet terminal.
  • the imaging device 120 images the display image displayed by the display device 110.
  • the imaging device 120 is installed or an imaging range of the imaging device 120 is set so as to image the display image displayed by the display device 110.
  • the imaging device 120 may be a camera that captures visible light or a camera that captures infrared light.
  • the operation of the image processing apparatus 100 will be described with an example in which the peripheral edge part of the display device 110 is covered with the housing frame, and the peripheral edge part of the display image cannot be imaged by the imaging device 120.
  • Fig. 2 is a perspective view illustrating the configuration of an image display system to which the image processing apparatus according to the embodiment is applied.
  • the image display system includes the image processing apparatus 100, the display device 110, and the imaging device 120.
  • the display device 110 includes a display 201 which displays the display image.
  • the peripheral edge part of the display 201 is covered with a housing frame 202, and a part of the display image displayed on the display 201, the part being covered with the housing frame 202, cannot be imaged by the imaging device 120.
  • the image processing apparatus 100 causes the display device 110 to display the display image, and causes the imaging device 120 to image the display image. At this time, since the peripheral edge part of the display 201 is covered with the housing frame 202, the captured image captured by the imaging device 120 does not include the part of the display image displayed on the display 201, the part being covered with the housing frame 202.
  • Fig. 3(a) is a diagram illustrating an example of the expected value image.
  • Fig. 3(b) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when the peripheral edge part is not covered with the housing frame.
  • Fig. 3(c) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when the peripheral edge part is covered with the housing frame.
  • Fig. 3(d) is a diagram illustrating an imaging example of the display image of Fig. 3(b).
  • Fig. 3(e) is a diagram illustrating an imaging example of the display image of Fig. 3(c).
  • a checker image 301 is input as the expected value image to the image processing apparatus 100.
  • the image processing apparatus 100 causes the display 201 to display the checker image 301.
  • the image processing apparatus 100 causes the imaging device 120 to image the display image displayed on the display 201.
  • a captured image 301a without lack of a peripheral edge part 302 corresponding to an outer edge 203 of the checker image 301 is obtained as illustrated in Fig. 3(d).
  • a captured image 301b with lack of the peripheral edge part 302 corresponding to the outer edge 203 of the checker image 301 is obtained as illustrated in Fig. 3(e).
  • Fig. 4(a) is a diagram illustrating an example of boundary coordinates when an analysis image with no black frame pattern added is split into sixteen regions.
  • Fig. 4(b) is a diagram illustrating an example of five pattern images which are used in generation of 16-split images.
  • Fig. 4(c) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is not covered with the housing frame.
  • Fig. 4(d) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is covered with the housing frame.
  • Fig. 4(a) the display 201 is split into four in each of the horizontal direction and the vertical direction. Then, an analysis image 400 in which boundary coordinates of sixteen split regions obtained by splitting into four in each of the horizontal direction and the vertical direction are known is estimated.
  • the boundary coordinates of these sixteen split regions can be defined as coordinates (x 0 , y 0 ) to (x 4 , y 4 ) of four corners of each split region.
  • the pattern images 401 to 405 show the expected value images.
  • the pattern image 401 includes a white image set on the entire face thereof.
  • the pattern image 402 is split into two regions in the vertical direction, and includes a white image and a black image which are set on the respective split regions.
  • the pattern image 403 is split into four regions in the vertical direction, and includes white images and black images which are alternately set on the respective split regions.
  • the pattern image 404 is split into two regions in the horizontal direction, and includes a white image and a black image which are set on the respective split regions.
  • the pattern image 405 is split into four regions in the horizontal direction, and includes white images and black images which are alternately set on the respective split regions.
  • the pattern images 401 to 405 are sequentially input to the image processing apparatus 100.
  • the image processing apparatus 100 causes the display 201 to sequentially display the pattern images 401 to 405.
  • the image processing apparatus 100 causes the imaging device 120 to sequentially image the display images displayed on the display 201.
  • the image processing apparatus 100 extracts captured images of the sixteen split regions for the analysis image 400 based on logical AND operations of the five captured images 401a to 405a.
  • the image processing apparatus 100 extracts captured images of the sixteen split regions for the analysis image 400 based on logical AND operations of the five captured images 401b to 405b.
  • the image processing apparatus 100 performs geometric correction based on coordinate values of feature points of the captured images of the sixteen split regions. At this time, the image processing apparatus 100 can perform geometric correction using a spatial code in the following order.
  • the image processing apparatus 100 causes the display 201 or a projector to display a pattern image having known coordinate values of a feature point where color information changes such as black and white.
  • the image processing apparatus 100 causes the imaging device 120 to image the display image or the projector projection image, detects a feature point from the captured image, and obtains coordinate values of the feature point. Then, the image processing apparatus 100 obtains the correspondence relationship between the coordinate values of the feature point of the pattern image and the coordinate values of the feature point of the captured image.
  • the image processing apparatus 100 causes a pattern image having different coordinate values of a feature point to be displayed, and causes the imaging device 120 to image the display image or the projector projection image. Then, the image processing apparatus 100 detects a feature point from the captured image, obtains coordinate values of the feature point, and obtains a plurality of correspondence relationships between the coordinate values of the feature point of the pattern image and the coordinate values of the feature point of the captured image.
  • the image processing apparatus 100 deforms the image by affine transformation, projective transformation, or a combination of a plurality of transformations based on the correspondence relationship between the coordinate values of the feature point of the pattern image and the coordinate values of the feature point of the captured image.
  • Fig. 4 illustrates an example in which the five pattern images 401 to 405 to be displayed and imaged are used, and the correspondence relationship between coordinate values can be obtained for five points x0 to x4 in the horizontal direction and five points y0 to y4 in the vertical direction.
  • the image deformation can be executed for each smaller region, and it is possible to improve the accuracy of geometric correction with respect to nonuniform deformation distortion of the captured images 401a to 405a caused by optical distortion.
  • the image processing apparatus 100 estimates an imaging position of the display image which cannot be imaged by the imaging device 120 based on positional information of the feature point of the captured image which is captured with a part of the display image lacking, and generates information required for deformation by the image deformation unit 106.
  • Fig. 5(a) is a diagram illustrating an example of boundary coordinates when an analysis image with a black frame pattern added is split into sixteen regions.
  • Fig. 5(b) is a diagram illustrating an example of five pattern images obtained by adding the black frame pattern to the pattern images of Fig. 4(a) when the peripheral edge part is covered with the housing frame.
  • Fig. 5(c) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is not covered with the housing frame.
  • Fig. 5(d) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is covered with the housing frame.
  • the display 201 is split into four in each of the horizontal direction and the vertical direction inside the peripheral edge part having a width m. Then, an analysis image 600 in which boundary coordinates of sixteen split regions obtained by splitting into four in each of the horizontal direction and the vertical direction and the width m of the peripheral edge part are known is estimated.
  • the boundary coordinates of these split regions can be defined as coordinates (x 0 , y 0 ) to (x 4 , y 4 ) of four corners of each split region. Coordinates of four corners of the peripheral edge part having the width m can be defined as (0,0), (x 5 , 0), (0, y 5 ), (x 5 , y 5 ).
  • the pattern images 601 to 605 are used as illustrated in Fig. 5(b).
  • the pattern images 601 to 605 are obtained by adding black frame patterns K1 to K5 around the respective pattern images 401 to 405 of Fig. 4(b).
  • the black frame patterns K1 to K5 are disposed at positions where four corners inside each of the black frame patterns K1 to K5 are imaged even when the display 201 of Fig. 2 is covered with the housing frame 202.
  • the black frame patterns K1 to K5 can be used as reference patterns for detecting positional information of the feature point for interpolative estimation from the captured image.
  • the reference patterns are not necessarily limited to the black frame patterns K1 to K5, and any pattern can be used as long as positional information of the feature point for interpolative estimation can be detected from the captured image.
  • the pattern images 601 to 605 are sequentially input to the image processing apparatus 100.
  • the image processing apparatus 100 causes the display 201 to sequentially display the pattern images 601 to 605.
  • the image processing apparatus 100 causes the imaging device 120 to sequentially image the display images displayed on the display 201 to acquire captured images 601a to 605a.
  • the captured images 601a to 605a imaged patterns Z1 to Z5 of the black frame patterns K1 to K5 displayed on the display 201 are disposed inside an imaging position PK of the housing frame 202 of Fig. 2 as illustrated in Fig. 5(d).
  • the captured images 601a to 605a include the imaged patterns Z1 to Z5 corresponding to the black frame patterns K1 to K5 even when the display 201 is covered with the housing frame 202.
  • the image processing apparatus 100 estimates outer edge coordinates of the display 201 in the captured images 601a to 605a when the display 201 is not covered with the housing frame 202 based on boundary coordinates of the imaged patterns Z1 to Z5 and boundary coordinates of the peripheral edge part having the width m set on the analysis image 600.
  • the image processing apparatus 100 extracts captured images of sixteen split regions for the analysis image 600 based on logical AND operations of the captured images 601a to 605a in which the imaging position of the outer edge of the display 201 is interpolatively estimated. Then, the image processing apparatus 100 performs geometric correction of the captured image to be checked based on coordinate values of feature points of the captured images of these sixteen split regions.
  • peripheral edge part having the width m in the analysis image 600 be larger than a region in the display 201 covered with the housing frame 202.
  • Different widths may be set between up and down and right and left sides of the pattern image.
  • Fig. 6(a) is a diagram illustrating binary images of the captured images of Fig. 5(c).
  • Fig. 6(b) is a diagram illustrating inverse images obtained by inverting the binary images of Fig. 6(a).
  • the image analysis unit 103 obtains binary images 701 to 705 from the captured images 601a to 605a of Fig. 5(c). For example, when the captured images 601a to 605a have 256 levels of gradation for each of three RGB channels, the image analysis unit 103 performs the binary conversion after obtaining a brightness Y among Y, Cb, and Cr from RGB pixel values.
  • the image analysis unit 103 obtains inverse images 712 to 715 from the binary images 702 to 705 by inverting white and black. Next, the image analysis unit 103 extracts imaging regions corresponding to the sixteen split regions of the analysis image 600 of Fig. 5 by the logical AND of the binary images 701 to 705 and the inverse images 712 to 715.
  • Fig. 7 is a diagram illustrating a logical AND operation method when the 16-split images are generated based on the binary images of Fig. 6(a) and the inverse images of Fig. 6(b).
  • the image analysis unit 103 of Fig. 1 generates sixteen 16-split images 802 based on logical AND operations #1 to #16 performed by combining the binary images 701 to 705 and the inverse images 712 to 715 in 16 ways.
  • the logical AND operation #1 the logical AND of the binary image 701 of the binary image in Fig. 6 and the inverse images 712, 713, 714, 715 is obtained.
  • the logical AND operation #1 only a part having a white pixel value is extracted as white.
  • the logical AND operation #1 only the uppermost and leftmost one of the 16-split regions is extracted. Different regions are extracted by applying different images to the logical AND.
  • the imaging device 120 directly generates 16-split images 802, it is necessary to repeatedly perform imaging 16 times.
  • the 16-split images 802 are generated by the logical AND operations based on the captured images 601a to 605a of Fig. 5(c), performing imaging five times is sufficient for the generation the 16-split images 802.
  • Fig. 8 is a diagram illustrating coordinates of feature points of the 16-split images analyzed from the captured image.
  • the image analysis unit 103 of Fig. 1 generates an analyzed image 901 by extracting 25 feature points P 00 to P 44 in total based on the sixteen 16-split images 802 of Fig. 7.
  • Fig. 9 is a diagram illustrating coordinate values of the peripheral edge part estimated based on the feature points of the 16-split images analyzed from the captured image.
  • the interpolative estimation unit 104 of Fig. 1 generates an interpolatively estimated image 1001 to which 16 extrapolation points E 00 to E 44 in total are added.
  • an extrapolation point E 10 can be approximated to be located on a line connecting the feature point P 10 and the feature point P 11 of Fig. 8. Further, according to the analysis image 600 of Fig. 5, the ratio between the distance between the peripheral edge part and the feature point P 10 and the distance between the feature point P 10 and the feature point P 11 is x 0 : x 1 – x 0 .
  • coordinate values (x E10 , y E10 ) of the extrapolation point E 10 can be obtained by the following mathematical formulae using coordinate values P 10 (x 10 , y 10 ), P 11 (x 11 , y 11 ).
  • Coordinate values of the other extrapolation points can be obtained in a similar manner.
  • a general formula for projective-transforming a captured image (x, y) to an expected value image (x’, y’) with respect to a plane surrounded by four feature points on the captured image and the pattern image can be obtained in the following manner.
  • eight variables a 1 to a 8 can be obtained from an x-coordinate value and a y-coordinate value of each of the four feature points.
  • Fig. 10 is a flowchart illustrating an image processing method according to the embodiment.
  • the imaging control unit 102 of Fig. 1 causes the display device 110 to display the pattern images 601 to 605 inside the peripheral edge part covered with the housing frame 202 of Fig. 2, and causes the imaging device 120 to image the pattern images 601 to 605.
  • the image analysis unit 103 extracts the sixteen 16-split images 802 of Fig. 7 based on the logical AND operations of the captured images 601a to 605a captured by the imaging device 120.
  • the image analysis unit 103 detects four corners of each of 16 rectangular regions from the 16-split images 802 as feature points, and acquires coordinate values of the feature points. At this time, the image analysis unit 103 extracts four sides of each rectangular region by a method such as edge detection or line segment detection, and obtains coordinate values of the four corners of the rectangular region, the four corners being intersection points of the four sides. As illustrated in Fig. 8, the coordinate values P 00 (x 00 , y 00 ) to P 44 (x 44 , y 44 ) of the 25 feature points P 00 to P 44 in total are obtained by acquiring all the coordinate values of the feature points of the 16 rectangular regions of the 16-split images 802.
  • the interpolative estimation unit 104 obtains, by interpolation, the coordinate values of the peripheral edge part of the display 201 in the captured image based on the coordinate values of the 25 feature points obtained in S503 and the coordinate values of the feature points of the pattern images.
  • the coordinate values (x E00 , y E44 ) of the 16 extrapolation points E 00 to E 44 in total are obtained as the coordinate values of the peripheral edge part of the display 201 in the captured image.
  • the image deformation unit 106 calculates an image deformation parameter using the coordinate values detected in S503 and the coordinate values interpolatively estimated in S504.
  • the above embodiment it is possible to calculate the image deformation parameter after interpolatively estimating the imaging position of the display image, the imaging position not being included in the captured image.
  • the imaging position not being included in the captured image.
  • the image processing apparatus 100 of Fig. 1 may estimate the imaging position of the display image, the imaging position not being imaged, based on the geometrically corrected image obtained by deforming the captured image.
  • the image processing apparatus 100 can estimate the position of the captured image covered with the housing frame 202 of Fig. 2 and exclude a part covered with the housing frame 202 of Fig. 2 in checking of the captured image.
  • Fig. 11 is a block diagram illustrating a hardware configuration example of the image processing apparatus of Fig. 1.
  • the image processing apparatus 100 includes a processor 11, a communication control device 12, a communication interface 13, a main storage device 14, and an external storage device 15.
  • the processor 11, the communication control device 12, the communication interface 13, the main storage device 14, and the external storage device 15 are connected to each other via an internal bus 16.
  • the main storage device 14 and the external storage device 15 are accessible from the processor 11.
  • an input device 20 and an output device 21 are disposed outside the image processing apparatus 100.
  • the input device 20 and the output device 21 are connected to the internal bus 16 via an input/output interface 17.
  • the input device 20 is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device, an imaging device, or a scanner.
  • the output device 21 is, for example, a screen display device (e.g., a liquid-crystal monitor, an organic electro luminescence (EL) display, or a graphics card), a voice output device (e.g., a speaker), or a printing device.
  • a screen display device e.g., a liquid-crystal monitor, an organic electro luminescence (EL) display, or a graphics card
  • a voice output device e.g., a speaker
  • the processor 11 is hardware which controls the operation of the entire image processing apparatus 100.
  • the processor 11 may be a central processing unit (CPU) or a graphics processing unit (GPU).
  • the processor 11 may be a single-core processor or a multi-core processor.
  • the processor 11 may include a hardware circuit (e.g., a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) which performs a part or the whole of processing.
  • the processor 11 may include a neural network.
  • the main storage device 14 may include a semiconductor memory such as an SRAM or a DRAM.
  • the main storage device 14 may be provided with a work area for storing a program being executed by the processor 11 or executing a program by the processor 11.
  • the external storage device 15 is a storage device having a large storage capacity.
  • the external storage device 15 is, for example, a hard disk device or a solid state drive (SSD).
  • the external storage device 15 is capable of holding execution files of various programs and data which is used in the execution of the programs.
  • An image processing program 15A can be stored in the external storage device 15.
  • the image processing program 15A may be software installable into the image processing apparatus 100 or may be incorporated as firmware in the image processing apparatus 100.
  • the communication control device 12 is hardware having a function of controlling communication with the outside.
  • the communication control device 12 is connected to a network 19 via the communication interface 13.
  • the network 19 may be a wide area network (WAN) such as the Internet, a local area network (LAN) such as WiFi or the Ethernet (registered trademark), or a combination of WAN and LAN.
  • WAN wide area network
  • LAN local area network
  • WiFi WiFi or the Ethernet
  • the input/output interface 17 converts data input from the input device 20 to a data format which can be processed by the processor 11 or converts data output from the processor 11 to a data format which can be processed by the output device 21.
  • the processor 11 reads the image processing program 15A into the main storage device 14 and executes the image processing program 15A. Accordingly, it is possible to analyze positional information of the feature point of the captured image obtained by imaging the display image and estimate the imaging position of the display image, the imaging position not being included in the captured image, based on the positional information of the feature point of the captured image.
  • the processor 11 can implement the functions of the image input unit 101, the imaging control unit 102, the image analysis unit 103, the interpolative estimation unit 104, the image output unit 105, and the image deformation unit 106 of Fig. 1.
  • the execution of the image processing program 15A may be shared among a plurality of processors or computers.
  • the processor 11 may instruct, for example, a cloud computer, via the network 19, to execute a part or the whole of the image processing program 15A and receive a result of the execution.
  • the present invention is not limited to the above embodiment, and includes various modifications.
  • the above embodiment has been described in detail in order to describe the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations.
  • a part of the configuration of one embodiment can be replaced by the configuration of another embodiment, and the configuration of one embodiment can be added to the configuration of another embodiment.
  • a part or all of each configuration, function, processing unit, processing mean, and the like described above may also be implemented by hardware, for example, by design with an integrated circuit.
  • image processing apparatus 101 image input unit 102 imaging control unit 103 image analysis unit 104 interpolative estimation unit 105 image output unit 106 image deformation unit 107 bus 110 display device 120 imaging device

Abstract

To estimate an imaging position of a display image, the imaging position not being included in a captured image. An interpolative estimation unit 104 estimates an imaging position of a display image which cannot be imaged by an imaging device 120 based on information of the display image displayed by the display device 110 and positional information of a feature point of a captured image analyzed by an image analysis unit 103, and generates information required for deformation by an image deformation unit 106. The image deformation unit 106 performs image deformation by affine transformation, projective transformation, or a combination of a plurality of transformations so that the shape of the captured image of the display image displayed on a display of the display device 110 matches the shape of an expected value image input by an image input unit 101.

Description

IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD
　The present invention relates to an image processing apparatus and an image processing method. This application claims the priority based on the Japanese Patent Application No. 2019-58469 filed on March 26, 2019. The entire contents of which are incorporated herein by reference for all purpose.
　For example, Patent Literature 1 discloses a technique capable of easily adjusting a position deviation when superimposing a projection image onto an image included in a projection plane.
　Patent Literature 1 discloses “a projection apparatus including: a projection unit configured to project an image onto a projection plane; an acquisition unit configured to acquire information related to colors of an image included in the projection plane; a generation unit configured to generate an alignment image for adjusting a position deviation between the image included in the projection plane and the image projected on the projection plane based on the information acquired by the acquisition unit; and a control unit configured to control the projection unit so as to project the alignment image in an overlapped manner onto the image on the projection plane, in which the generation unit generates the alignment image that has a predetermined relationship with the colors of the image included in the projection plane and indicates an amount of the position deviation from the image projected on the projection plane based on the information acquired by the acquisition unit”. The technique disclosed in Patent Literature 1 can detect a target from the captured image and correct the shape of the image so as to minimize the position deviation.
　Patent Literature 1: JP 2018-197824 A
　As a test evaluation for a product including a display, there is a method that images an image displayed on the display by a camera, and checks the captured image against an image when the product is designed to determine whether the images match. In this case, it is necessary to detect a display part to be checked from the captured image and deform the image by obtaining a position deviation from the image against which the display part is to be checked.
　However, Patent Literature 1 does not disclose a method for detecting a covered part of an object to be checked. Thus, in the technique disclosed in Patent Literature 1, when a peripheral edge part of the display is covered with a housing in the product, the peripheral edge part of the display is not imaged in the captured image. Thus, it is not possible to accurately detect the display part and deform the image.
　The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus and an image processing method that are capable of estimating an imaging position of a display image, the imaging position not being included in a captured image.
　In order the achieve the above object, an image processing apparatus according to a first aspect includes an imaging control unit configured to cause a display image to be imaged; an image analysis unit configured to analyze positional information of a feature point of a captured image obtained by imaging the display image; and an estimation unit configured to estimate an imaging position of the display image, the imaging position not being included in the captured image, based on the positional information of the feature point of the captured image.
　According to the present invention, it is possible to estimate the imaging position of the display image, the imaging position not being included in the captured image.
Fig. 1 is a block diagram illustrating the configuration of an image processing apparatus according to an embodiment. Fig. 2 is a perspective view illustrating the configuration of an image display system to which the image processing apparatus according to the embodiment is applied. Fig. 3(a) is a diagram illustrating an example of an expected value image, Fig. 3(b) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when a peripheral edge part is not covered with a housing frame, Fig. 3(c) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when the peripheral edge part is covered with the housing frame, Fig. 3(d) is a diagram illustrating an imaging example of the display image of Fig. 3(b), and Fig. 3(e) is a diagram illustrating an imaging example of the display image of Fig. 3(c). Fig. 4(a) is a diagram illustrating an example of boundary coordinates when an analysis image with no black frame pattern added is split into sixteen regions, Fig. 4(b) is a diagram illustrating an example of five pattern images which are used in generation of 16-split images, Fig. 4(c) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is not covered with the housing frame, and Fig. 4(d) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is covered with the housing frame. Fig. 5(a) is a diagram illustrating an example of boundary coordinates when an analysis image with a black frame pattern added is split into sixteen regions, Fig. 5(b) is a diagram illustrating an example of five pattern images obtained by adding the black frame pattern to the pattern images of Fig. 4(a) when the peripheral edge part is covered with the housing frame, Fig. 5(c) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is not covered with the housing frame, and Fig. 5(d) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is covered with the housing frame. Fig. 6(a) is a diagram illustrating binary images of the captured images of Fig. 5(c), and Fig. 6(b) is a diagram illustrating inverse images obtained by inverting the binary images of Fig. 6(a). Fig. 7 is a diagram illustrating a logical AND operation method when 16-split images are generated based on the binary images of Fig. 6(a) and the inverse images of Fig. 6(b). Fig. 8 is a diagram illustrating coordinates of feature points of the 16-split images analyzed from the captured image. Fig. 9 is a diagram illustrating coordinate values of the peripheral edge part estimated based on the feature points of the 16-split images analyzed from the captured image. Fig. 10 is a flowchart illustrating an image processing method according to the embodiment. Fig. 11 is a block diagram illustrating a hardware configuration example of the image processing apparatus of Fig. 1.
　An embodiment will be described with reference to the drawings. It is to be noted that the embodiment described below does not limit the invention according to the claims, and all elements and combinations thereof described in the embodiment are not necessarily essential for the solution of the invention.
　Fig. 1 is a block diagram illustrating the configuration of an image processing apparatus according to the embodiment.
　In Fig. 1, an image processing apparatus 100 includes an image input unit 101, an imaging control unit 102, an image analysis unit 103, an interpolative estimation unit 104, an image output unit 105, and an image deformation unit 106. The image input unit 101, the imaging control unit 102, the image analysis unit 103, the interpolative estimation unit 104, the image output unit 105, and the image deformation unit 106 are connected to each other via a bus 107. In this case, the bus 107 transfers image data, control information, and analysis information which are handled by each processing unit connected to the bus 107.
　The image processing apparatus 100 is connected to a display device 110 and an imaging device 120 via the bus 107. The image processing apparatus 100 may be connected to the display device 110 and the imaging device 120 by either wired or wireless connection. Fig. 1 illustrates an example in which the display device 110 and the imaging device 120 are disposed outside the image processing apparatus 100. However, the display device 110 and the imaging device 120 may be incorporated in the image processing apparatus 100.
　The image input unit 101 processes an input image input from an external connection device such as a personal computer (PC). For example, when the input image is an analog image, the image input unit 101 quantizes the analog image by decoding to convert the analog image to a digital image handled in the subsequent process. Hereinbelow, the input image input by the image input unit 101 is referred to as an expected value image. The expected value image is, for example, a design image against which a captured image captured by the imaging device 120 is to be checked.
　The imaging control unit 102 causes the imaging device 120 to perform imaging for an image analysis by the image analysis unit 103. At this time, the imaging control unit 102 can set an exposure condition and white balance of the imaging device 120 so that the image analysis by the image analysis unit 103 is appropriately performed. Further, the imaging control unit 102 controls imaging of a display image in response to switching of the display image displayed by the display device 110. For example, when the imaging control unit 102 causes the imaging device 120 to image a display image displayed on the display device 110, the imaging control unit 102 sets an imaging condition under which gradation information of the captured image does not collapse, and sets the same exposure condition and the same white balance in the subsequent imaging.
　The image analysis unit 103 analyzes shape information and color information which are included in the captured image obtained by imaging the display image displayed by the display device 110, obtains coordinate information of a feature point having a specific feature inside the captured image, and generates information required for interpolative estimation by the interpolative estimation unit 104 and generates information required for image deformation by the image deformation unit 106. The image analysis unit 103 is capable of cutting out a range in which the display image is imaged and a range in which the display image is not imaged. For example, a feature value extraction method such as image edge detection or pattern matching can be used as a method for obtaining coordinate information of the feature point.
　The interpolative estimation unit 104 estimates an imaging position of the display image which cannot be imaged by the imaging device 120 based on information of an analysis image displayed by the display device 110 and positional information of the feature point of the captured image analyzed by the image analysis unit 103, and generates information required for deformation by the image deformation unit 106. The imaging position of the display image which cannot be imaged by the imaging device 120 is, for example, the position of a part hidden behind a housing frame when imaging is performed in a state in which the peripheral edge part of a display which displays the display image is covered with the housing frame.
　At this time, the interpolative estimation unit 104 is, for example, capable of estimating the imaging position of the display image which cannot be imaged by the imaging device 120 by extrapolating the imaging position of the display image which cannot be imaged by the imaging device 120 based on the positional information of the displays image imaged by the imaging device 120.
　The image output unit 105 outputs an image deformed by the image deformation unit 106 to the external connection device such as a PC.
　The image deformation unit 106 deforms the captured image input from the imaging device 120 based on information obtained by the image analysis unit 103 and the interpolative estimation unit 104. At this time, the image deformation unit 106 performs the image deformation based on an interpolatively estimated image obtained by interpolatively estimating the imaging position of the display image which cannot be imaged by the imaging device 120. For example, the image deformation unit 106 is capable of performing the image deformation by affine transformation, projective transformation, or a combination of a plurality of transformations so that the shape of the interpolatively estimated image of the captured image captured by the imaging device 120 matches the shape of the expected value image input by the image input unit 101. At this time, the image deformation unit 106 may perform the image deformation by a combination of perspective transformation of the interpolatively estimated image and N-region splitting free deformation (N is an integer equal to or larger than 2). Hereinbelow, an image deformed by the image deformation unit 106 is referred to as a geometrically corrected image.
　The display device 110 displays the analysis display image or the expected value image input by the image input unit 101. The display device 110 may be a liquid crystal display device, an organic electro luminescence (EL) display, or a display using a cathode ray tube. Further, the display device 110 may be an information processing device including a display such as a car navigation system, a smartphone, or a tablet terminal.
　The imaging device 120 images the display image displayed by the display device 110. The imaging device 120 is installed or an imaging range of the imaging device 120 is set so as to image the display image displayed by the display device 110. The imaging device 120 may be a camera that captures visible light or a camera that captures infrared light.
　Hereinbelow, the operation of the image processing apparatus 100 will be described with an example in which the peripheral edge part of the display device 110 is covered with the housing frame, and the peripheral edge part of the display image cannot be imaged by the imaging device 120.
　Fig. 2 is a perspective view illustrating the configuration of an image display system to which the image processing apparatus according to the embodiment is applied.
　In Fig. 2, the image display system includes the image processing apparatus 100, the display device 110, and the imaging device 120. The display device 110 includes a display 201 which displays the display image. The peripheral edge part of the display 201 is covered with a housing frame 202, and a part of the display image displayed on the display 201, the part being covered with the housing frame 202, cannot be imaged by the imaging device 120.
　The image processing apparatus 100 causes the display device 110 to display the display image, and causes the imaging device 120 to image the display image. At this time, since the peripheral edge part of the display 201 is covered with the housing frame 202, the captured image captured by the imaging device 120 does not include the part of the display image displayed on the display 201, the part being covered with the housing frame 202.
　Fig. 3(a) is a diagram illustrating an example of the expected value image. Fig. 3(b) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when the peripheral edge part is not covered with the housing frame. Fig. 3(c) is a diagram illustrating a display example of the expected value image of Fig. 3(a) when the peripheral edge part is covered with the housing frame. Fig. 3(d) is a diagram illustrating an imaging example of the display image of Fig. 3(b). Fig. 3(e) is a diagram illustrating an imaging example of the display image of Fig. 3(c).
　In Fig. 3(a), for example, a checker image 301 is input as the expected value image to the image processing apparatus 100. At this time, as illustrated in Figs. 3(b) and 3(c), the image processing apparatus 100 causes the display 201 to display the checker image 301. Further, as illustrated in Figs. 3(d) and 3(e), the image processing apparatus 100 causes the imaging device 120 to image the display image displayed on the display 201.
　When the peripheral edge part of the display 201 of Fig. 2 is not covered with the housing frame 202 as illustrated in Fig. 3(b), the checker image 301 of Fig. 3(a) input from the image processing apparatus 100 is displayed on the display 201 without any lack.
　Thus, when the imaging device 120 images the display image displayed on the display 201 of Fig. 3(b), a captured image 301a without lack of a peripheral edge part 302 corresponding to an outer edge 203 of the checker image 301 is obtained as illustrated in Fig. 3(d).
　On the other hand, when the peripheral edge part of the display 201 of Fig. 2 is covered with the housing frame 202 as illustrated in Fig. 3(c), the outer edge 203 of the display image displayed on the display 201 is invisible. Thus, the checker image 301 of Fig. 3(a) input from the image processing apparatus 100 is displayed on the display 201 with lack of the outer edge 203.
　Thus, when the imaging device 120 images the display image displayed on the display 201 of Fig. 3(c), a captured image 301b with lack of the peripheral edge part 302 corresponding to the outer edge 203 of the checker image 301 is obtained as illustrated in Fig. 3(e).
　Fig. 4(a) is a diagram illustrating an example of boundary coordinates when an analysis image with no black frame pattern added is split into sixteen regions. Fig. 4(b) is a diagram illustrating an example of five pattern images which are used in generation of 16-split images. Fig. 4(c) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is not covered with the housing frame. Fig. 4(d) is a diagram illustrating an imaging example of the pattern images of Fig. 4(b) when the peripheral edge part is covered with the housing frame.
　In Fig. 4(a), the display 201 is split into four in each of the horizontal direction and the vertical direction. Then, an analysis image 400 in which boundary coordinates of sixteen split regions obtained by splitting into four in each of the horizontal direction and the vertical direction are known is estimated. The boundary coordinates of these sixteen split regions can be defined as coordinates (x0, y0) to (x4, y4) of four corners of each split region.
　In order to obtain captured images corresponding to these sixteen split regions, five pattern images 401 to 405 are used as illustrated in Fig. 4(b). The pattern images 401 to 405 show the expected value images. The pattern image 401 includes a white image set on the entire face thereof. The pattern image 402 is split into two regions in the vertical direction, and includes a white image and a black image which are set on the respective split regions. The pattern image 403 is split into four regions in the vertical direction, and includes white images and black images which are alternately set on the respective split regions. The pattern image 404 is split into two regions in the horizontal direction, and includes a white image and a black image which are set on the respective split regions. The pattern image 405 is split into four regions in the horizontal direction, and includes white images and black images which are alternately set on the respective split regions.
　The pattern images 401 to 405 are sequentially input to the image processing apparatus 100. At this time, the image processing apparatus 100 causes the display 201 to sequentially display the pattern images 401 to 405. Then, as illustrated in Figs. 4(c) and 4(d), the image processing apparatus 100 causes the imaging device 120 to sequentially image the display images displayed on the display 201.
　When the peripheral edge part is not covered with the housing frame 202, five captured images 401a to 405a of Fig. 4(c) are obtained. The image processing apparatus 100 extracts captured images of the sixteen split regions for the analysis image 400 based on logical AND operations of the five captured images 401a to 405a.
　On the other hand, when the peripheral edge part is covered with the housing frame 202, five captured images 401b to 405b of Fig. 4(d) are obtained. The image processing apparatus 100 extracts captured images of the sixteen split regions for the analysis image 400 based on logical AND operations of the five captured images 401b to 405b.
　Then, the image processing apparatus 100 performs geometric correction based on coordinate values of feature points of the captured images of the sixteen split regions. At this time, the image processing apparatus 100 can perform geometric correction using a spatial code in the following order.
　For example, the image processing apparatus 100 causes the display 201 or a projector to display a pattern image having known coordinate values of a feature point where color information changes such as black and white.
　Next, the image processing apparatus 100 causes the imaging device 120 to image the display image or the projector projection image, detects a feature point from the captured image, and obtains coordinate values of the feature point. Then, the image processing apparatus 100 obtains the correspondence relationship between the coordinate values of the feature point of the pattern image and the coordinate values of the feature point of the captured image.
　Next, the image processing apparatus 100 causes a pattern image having different coordinate values of a feature point to be displayed, and causes the imaging device 120 to image the display image or the projector projection image. Then, the image processing apparatus 100 detects a feature point from the captured image, obtains coordinate values of the feature point, and obtains a plurality of correspondence relationships between the coordinate values of the feature point of the pattern image and the coordinate values of the feature point of the captured image.
　Next, the image processing apparatus 100 deforms the image by affine transformation, projective transformation, or a combination of a plurality of transformations based on the correspondence relationship between the coordinate values of the feature point of the pattern image and the coordinate values of the feature point of the captured image. Fig. 4 illustrates an example in which the five pattern images 401 to 405 to be displayed and imaged are used, and the correspondence relationship between coordinate values can be obtained for five points x0 to x4 in the horizontal direction and five points y0 to y4 in the vertical direction.
　At this time, as the number of correspondence relationships between coordinate values increases, the image deformation can be executed for each smaller region, and it is possible to improve the accuracy of geometric correction with respect to nonuniform deformation distortion of the captured images 401a to 405a caused by optical distortion.
　On the other hand, when the display image is imaged with a part thereof lacking like the captured images 401b to 405b, a deviation occurs in the coordinate values of the detected feature point. Thus, the geometrically corrected image which is a result of the image deformation becomes distorted, and the matching accuracy between the geometrically corrected image and the expected value image is reduced.
　At this time, in order to improve the matching accuracy between the geometrically corrected image and the expected value image, the image processing apparatus 100 estimates an imaging position of the display image which cannot be imaged by the imaging device 120 based on positional information of the feature point of the captured image which is captured with a part of the display image lacking, and generates information required for deformation by the image deformation unit 106.
　Fig. 5(a) is a diagram illustrating an example of boundary coordinates when an analysis image with a black frame pattern added is split into sixteen regions. Fig. 5(b) is a diagram illustrating an example of five pattern images obtained by adding the black frame pattern to the pattern images of Fig. 4(a) when the peripheral edge part is covered with the housing frame. Fig. 5(c) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is not covered with the housing frame. Fig. 5(d) is a diagram illustrating an imaging example of the pattern images of Fig. 5(b) when the peripheral edge part is covered with the housing frame.
　In Fig. 5(a), the display 201 is split into four in each of the horizontal direction and the vertical direction inside the peripheral edge part having a width m. Then, an analysis image 600 in which boundary coordinates of sixteen split regions obtained by splitting into four in each of the horizontal direction and the vertical direction and the width m of the peripheral edge part are known is estimated. The boundary coordinates of these split regions can be defined as coordinates (x0, y0) to (x4, y4) of four corners of each split region. Coordinates of four corners of the peripheral edge part having the width m can be defined as (0,0), (x5, 0), (0, y5), (x5, y5).
　In order to obtain captured images of the sixteen split regions in which the peripheral edge part having the width m is interpolatively estimated, five pattern images 601 to 605 are used as illustrated in Fig. 5(b). The pattern images 601 to 605 are obtained by adding black frame patterns K1 to K5 around the respective pattern images 401 to 405 of Fig. 4(b). The black frame patterns K1 to K5 are disposed at positions where four corners inside each of the black frame patterns K1 to K5 are imaged even when the display 201 of Fig. 2 is covered with the housing frame 202. At this time, the black frame patterns K1 to K5 can be used as reference patterns for detecting positional information of the feature point for interpolative estimation from the captured image. Note that the reference patterns are not necessarily limited to the black frame patterns K1 to K5, and any pattern can be used as long as positional information of the feature point for interpolative estimation can be detected from the captured image.
　The pattern images 601 to 605 are sequentially input to the image processing apparatus 100. At this time, the image processing apparatus 100 causes the display 201 to sequentially display the pattern images 601 to 605. Then, as illustrated in Fig. 5(c), the image processing apparatus 100 causes the imaging device 120 to sequentially image the display images displayed on the display 201 to acquire captured images 601a to 605a.
　In the captured images 601a to 605a, imaged patterns Z1 to Z5 of the black frame patterns K1 to K5 displayed on the display 201 are disposed inside an imaging position PK of the housing frame 202 of Fig. 2 as illustrated in Fig. 5(d). Thus, the captured images 601a to 605a include the imaged patterns Z1 to Z5 corresponding to the black frame patterns K1 to K5 even when the display 201 is covered with the housing frame 202.
　Then, the image processing apparatus 100 estimates outer edge coordinates of the display 201 in the captured images 601a to 605a when the display 201 is not covered with the housing frame 202 based on boundary coordinates of the imaged patterns Z1 to Z5 and boundary coordinates of the peripheral edge part having the width m set on the analysis image 600.
　The image processing apparatus 100 extracts captured images of sixteen split regions for the analysis image 600 based on logical AND operations of the captured images 601a to 605a in which the imaging position of the outer edge of the display 201 is interpolatively estimated. Then, the image processing apparatus 100 performs geometric correction of the captured image to be checked based on coordinate values of feature points of the captured images of these sixteen split regions.
　It is only required that the peripheral edge part having the width m in the analysis image 600 be larger than a region in the display 201 covered with the housing frame 202. Different widths may be set between up and down and right and left sides of the pattern image.
　Fig. 6(a) is a diagram illustrating binary images of the captured images of Fig. 5(c). Fig. 6(b) is a diagram illustrating inverse images obtained by inverting the binary images of Fig. 6(a).
　In Fig. 6(a), the image analysis unit 103 obtains binary images 701 to 705 from the captured images 601a to 605a of Fig. 5(c). For example, when the captured images 601a to 605a have 256 levels of gradation for each of three RGB channels, the image analysis unit 103 performs the binary conversion after obtaining a brightness Y among Y, Cb, and Cr from RGB pixel values.
　Further, as illustrated in Fig. 6(b), the image analysis unit 103 obtains inverse images 712 to 715 from the binary images 702 to 705 by inverting white and black. Next, the image analysis unit 103 extracts imaging regions corresponding to the sixteen split regions of the analysis image 600 of Fig. 5 by the logical AND of the binary images 701 to 705 and the inverse images 712 to 715.
　Fig. 7 is a diagram illustrating a logical AND operation method when the 16-split images are generated based on the binary images of Fig. 6(a) and the inverse images of Fig. 6(b).
　In Fig. 7, the image analysis unit 103 of Fig. 1 generates sixteen 16-split images 802 based on logical AND operations #1 to #16 performed by combining the binary images 701 to 705 and the inverse images 712 to 715 in 16 ways.
　For example, in the logical AND operation #1, the logical AND of the binary image 701 of the binary image in Fig. 6 and the inverse images 712, 713, 714, 715 is obtained. At this time, in all the images used in the logical AND, only a part having a white pixel value is extracted as white. For example, in the logical AND operation #1, only the uppermost and leftmost one of the 16-split regions is extracted. Different regions are extracted by applying different images to the logical AND.
　When the imaging device 120 directly generates 16-split images 802, it is necessary to repeatedly perform imaging 16 times. On the other hand, when the 16-split images 802 are generated by the logical AND operations based on the captured images 601a to 605a of Fig. 5(c), performing imaging five times is sufficient for the generation the 16-split images 802. Thus, it is possible to stabilize the imaging condition required for the generation of the 16-split images 802 and improve the position accuracy of feature points extracted from the 16-split images 802.
　Fig. 8 is a diagram illustrating coordinates of feature points of the 16-split images analyzed from the captured image.
　In Fig. 8, the image analysis unit 103 of Fig. 1 generates an analyzed image 901 by extracting 25 feature points P00 to P44 in total based on the sixteen 16-split images 802 of Fig. 7.
　Fig. 9 is a diagram illustrating coordinate values of the peripheral edge part estimated based on the feature points of the 16-split images analyzed from the captured image.
　In Fig. 9, the interpolative estimation unit 104 of Fig. 1 generates an interpolatively estimated image 1001 to which 16 extrapolation points E00 to E44 in total are added.
　Hereinbelow, an interpolative estimation method for the coordinate values of the peripheral edge part of the display 201 in the captured image will be described.
　For example, an extrapolation point E10 can be approximated to be located on a line connecting the feature point P10 and the feature point P11 of Fig. 8. Further, according to the analysis image 600 of Fig. 5, the ratio between the distance between the peripheral edge part and the feature point P10 and the distance between the feature point P10 and the feature point P11 is x0 : x1 – x0. Thus, coordinate values (xE10, yE10) of the extrapolation point E10 can be obtained by the following mathematical formulae using coordinate values P10 (x10, y10), P11 (x11, y11).
Mathematical Formula 1
Figure JPOXMLDOC01-appb-I000001
Mathematical Formula 2
Figure JPOXMLDOC01-appb-I000002
Mathematical Formula 3
Figure JPOXMLDOC01-appb-I000003
　Coordinate values of the other extrapolation points can be obtained in a similar manner.
　A general formula for projective-transforming a captured image (x, y) to an expected value image (x’, y’) with respect to a plane surrounded by four feature points on the captured image and the pattern image can be obtained in the following manner. In the following mathematical formulae, eight variables a1 to a8 can be obtained from an x-coordinate value and a y-coordinate value of each of the four feature points.
Mathematical Formula 4
Figure JPOXMLDOC01-appb-I000004
Mathematical Formula 5
Figure JPOXMLDOC01-appb-I000005
　Fig. 10 is a flowchart illustrating an image processing method according to the embodiment.
　In Fig. 10, in S501, the imaging control unit 102 of Fig. 1 causes the display device 110 to display the pattern images 601 to 605 inside the peripheral edge part covered with the housing frame 202 of Fig. 2, and causes the imaging device 120 to image the pattern images 601 to 605.
　Next, in S502, the image analysis unit 103 extracts the sixteen 16-split images 802 of Fig. 7 based on the logical AND operations of the captured images 601a to 605a captured by the imaging device 120.
　Next, in S503, the image analysis unit 103 detects four corners of each of 16 rectangular regions from the 16-split images 802 as feature points, and acquires coordinate values of the feature points. At this time, the image analysis unit 103 extracts four sides of each rectangular region by a method such as edge detection or line segment detection, and obtains coordinate values of the four corners of the rectangular region, the four corners being intersection points of the four sides. As illustrated in Fig. 8, the coordinate values P00 (x00, y00) to P44 (x44, y44) of the 25 feature points P00 to P44 in total are obtained by acquiring all the coordinate values of the feature points of the 16 rectangular regions of the 16-split images 802.
　Next, in S504, the interpolative estimation unit 104 obtains, by interpolation, the coordinate values of the peripheral edge part of the display 201 in the captured image based on the coordinate values of the 25 feature points obtained in S503 and the coordinate values of the feature points of the pattern images. At this time, as illustrated in Fig. 9, the coordinate values (xE00, yE44) of the 16 extrapolation points E00 to E44 in total are obtained as the coordinate values of the peripheral edge part of the display 201 in the captured image.
　Next, in S505, the image deformation unit 106 calculates an image deformation parameter using the coordinate values detected in S503 and the coordinate values interpolatively estimated in S504.
　As described above, according to the above embodiment, it is possible to calculate the image deformation parameter after interpolatively estimating the imaging position of the display image, the imaging position not being included in the captured image. Thus, even when a part of an object to be checked is not included in the captured image, it is possible to improve the accuracy of deformation of the captured image of the object to be checked.
　Note that the image processing apparatus 100 of Fig. 1 may estimate the imaging position of the display image, the imaging position not being imaged, based on the geometrically corrected image obtained by deforming the captured image. In this case, the image processing apparatus 100 can estimate the position of the captured image covered with the housing frame 202 of Fig. 2 and exclude a part covered with the housing frame 202 of Fig. 2 in checking of the captured image.
　Fig. 11 is a block diagram illustrating a hardware configuration example of the image processing apparatus of Fig. 1.
　In Fig. 11, the image processing apparatus 100 includes a processor 11, a communication control device 12, a communication interface 13, a main storage device 14, and an external storage device 15. The processor 11, the communication control device 12, the communication interface 13, the main storage device 14, and the external storage device 15 are connected to each other via an internal bus 16. The main storage device 14 and the external storage device 15 are accessible from the processor 11.
　Further, an input device 20 and an output device 21 are disposed outside the image processing apparatus 100. The input device 20 and the output device 21 are connected to the internal bus 16 via an input/output interface 17. The input device 20 is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device, an imaging device, or a scanner. The output device 21 is, for example, a screen display device (e.g., a liquid-crystal monitor, an organic electro luminescence (EL) display, or a graphics card), a voice output device (e.g., a speaker), or a printing device.
　The processor 11 is hardware which controls the operation of the entire image processing apparatus 100. The processor 11 may be a central processing unit (CPU) or a graphics processing unit (GPU). The processor 11 may be a single-core processor or a multi-core processor. The processor 11 may include a hardware circuit (e.g., a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) which performs a part or the whole of processing. The processor 11 may include a neural network.
　The main storage device 14 may include a semiconductor memory such as an SRAM or a DRAM. The main storage device 14 may be provided with a work area for storing a program being executed by the processor 11 or executing a program by the processor 11.
　The external storage device 15 is a storage device having a large storage capacity. The external storage device 15 is, for example, a hard disk device or a solid state drive (SSD). The external storage device 15 is capable of holding execution files of various programs and data which is used in the execution of the programs. An image processing program 15A can be stored in the external storage device 15. The image processing program 15A may be software installable into the image processing apparatus 100 or may be incorporated as firmware in the image processing apparatus 100.
　The communication control device 12 is hardware having a function of controlling communication with the outside. The communication control device 12 is connected to a network 19 via the communication interface 13. The network 19 may be a wide area network (WAN) such as the Internet, a local area network (LAN) such as WiFi or the Ethernet (registered trademark), or a combination of WAN and LAN.
　The input/output interface 17 converts data input from the input device 20 to a data format which can be processed by the processor 11 or converts data output from the processor 11 to a data format which can be processed by the output device 21.
　The processor 11 reads the image processing program 15A into the main storage device 14 and executes the image processing program 15A. Accordingly, it is possible to analyze positional information of the feature point of the captured image obtained by imaging the display image and estimate the imaging position of the display image, the imaging position not being included in the captured image, based on the positional information of the feature point of the captured image. At this time, the processor 11 can implement the functions of the image input unit 101, the imaging control unit 102, the image analysis unit 103, the interpolative estimation unit 104, the image output unit 105, and the image deformation unit 106 of Fig. 1.
　The execution of the image processing program 15A may be shared among a plurality of processors or computers. Alternatively, the processor 11 may instruct, for example, a cloud computer, via the network 19, to execute a part or the whole of the image processing program 15A and receive a result of the execution.
　It is to be noted that the present invention is not limited to the above embodiment, and includes various modifications. For example, the above embodiment has been described in detail in order to describe the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. In addition, a part of the configuration of one embodiment can be replaced by the configuration of another embodiment, and the configuration of one embodiment can be added to the configuration of another embodiment. Further, it is possible to perform addition, deletion, and replacement of another configuration on a part of the configuration of each embodiment. Furthermore, a part or all of each configuration, function, processing unit, processing mean, and the like described above may also be implemented by hardware, for example, by design with an integrated circuit.
　100 image processing apparatus
　101 image input unit
　102 imaging control unit
　103 image analysis unit
　104 interpolative estimation unit
　105 image output unit
　106 image deformation unit
　107 bus
　110 display device
　120 imaging device

Claims (15)

  1. An image processing apparatus comprising:
    an imaging control unit configured to cause a display image to be imaged;
    an image analysis unit configured to analyze a positional information of a feature point of a captured image obtained by imaging the display image; and
    an estimation unit configured to estimate an imaging position of the display image, the imaging position not being included in the captured image, based on the positional information of the feature point of the captured image.
  2. 　The image processing apparatus according to claim 1, wherein the estimation unit estimates a position of a peripheral edge part of a display configured to display the display image, the peripheral edge part not being included in the captured image, based on the positional information of the feature point of the captured image and positional information of the display.
  3. 　The image processing apparatus according to claim 1, wherein
    the display image includes a reference pattern displayed at a position included in the captured image, and
    the estimation unit estimates the imaging position of the display image, the imaging position not being included in the captured image, based on positional information of a feature point of an imaging result of the reference pattern and positional information of a display configured to display the display image.
  4. 　The image processing apparatus according to claim 3, wherein the imaging position of the display image estimated by the estimation unit is covered with a housing frame of a display device configured to display the display image.
  5. 　The image processing apparatus according to claim 4, wherein
    the reference pattern is a black frame pattern disposed inside the housing frame,
    the estimation unit estimates a position of a peripheral edge part of the display, the peripheral edge part being covered with the housing frame, based on an imaging position of the black frame pattern displayed on the display and positional information of the peripheral edge part of the display configured to display the display image.
  6. 　The image processing apparatus according to claim 1, further comprising an image deformation unit configured to deform the captured image based on the positional information of the feature point of the captured image.
  7. 　The image processing apparatus according to claim 6, wherein the image deformation unit performs perspective transformation and N-region splitting free deformation (N is an integer equal to or larger than 2) of the captured image.
  8. 　The image processing apparatus according to claim 7, wherein the image analysis unit extracts N-region split images of the captured image based on a logical AND operation of an imaging result of a displayed pattern image.
  9. 　The image processing apparatus according to claim 1, wherein the image analysis unit cuts out a range in which the display image is imaged and a range in which the display image is not imaged.
  10. 　The image processing apparatus according to claim 1, wherein the imaging control unit sets an imaging condition under which gradation information of the captured image does not collapse.
  11. 　The image processing apparatus according to claim 10, wherein the imaging control unit fixes imaging conditions of two or more display images.
  12. 　The image processing apparatus according to claim 6, wherein positional information of a display image not being imaged is estimated based on a geometrically corrected image obtained by deforming the captured image.
  13. 　An image processing method executed by a processor, the method comprising:
    by the processor,
    analyzing positional information of a feature point of a captured image obtained by imaging a display image; and
    estimating an imaging position of the display image, the imaging position not being included in the captured image, based on the positional information of the feature point of the captured image.
  14. 　The image processing method according to claim 13, further comprising estimating a position of a peripheral edge part of a display configured to display the display image, the peripheral edge part not being included in the captured image, based on the positional information of the feature point of the captured image and positional information of the display.
  15. 　The image processing method according to claim 14, wherein a peripheral edge part of the display image is covered with a housing frame of the display configured to display the display image,
    the display image includes a black frame pattern displayed inside the housing frame, and
    the processor estimates the position of the peripheral edge part of the display, the peripheral edge part being covered with the housing frame, based on positional information of a feature point of a captured image of the black frame pattern displayed on the display and positional information of the peripheral edge part of the display.
PCT/JP2020/011686 2019-03-26 2020-03-17 Image processing apparatus and image processing method WO2020196091A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019058469A JP2020160221A (en) 2019-03-26 2019-03-26 Image processing device and image processing method
JP2019-058469 2019-03-26

Publications (1)

Publication Number Publication Date
WO2020196091A1 true WO2020196091A1 (en) 2020-10-01

Family

ID=70166099

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/011686 WO2020196091A1 (en) 2019-03-26 2020-03-17 Image processing apparatus and image processing method

Country Status (2)

Country Link
JP (1) JP2020160221A (en)
WO (1) WO2020196091A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012154878A1 (en) * 2011-05-11 2012-11-15 Tyzx, Inc. Camera calibration using an easily produced 3d calibration pattern
EP3216005A1 (en) * 2015-03-11 2017-09-13 Huawei Technologies Co., Ltd. Image processing device and method for geometric calibration of images
JP2018197824A (en) 2017-05-24 2018-12-13 キヤノン株式会社 Projection device, information processing apparatus, and method for controlling them, and program
JP2019058469A (en) 2017-09-27 2019-04-18 富士フイルム株式会社 Endoscope holder for cleaning, cleaning device, and endoscope cleaning method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012154878A1 (en) * 2011-05-11 2012-11-15 Tyzx, Inc. Camera calibration using an easily produced 3d calibration pattern
EP3216005A1 (en) * 2015-03-11 2017-09-13 Huawei Technologies Co., Ltd. Image processing device and method for geometric calibration of images
JP2018197824A (en) 2017-05-24 2018-12-13 キヤノン株式会社 Projection device, information processing apparatus, and method for controlling them, and program
JP2019058469A (en) 2017-09-27 2019-04-18 富士フイルム株式会社 Endoscope holder for cleaning, cleaning device, and endoscope cleaning method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
KURILLO GREGORIJ ET AL: "Geometric and Color Calibration of Multiview Panoramic Cameras for Life-Size 3D Immersive Video", 2013 INTERNATIONAL CONFERENCE ON 3D VISION, IEEE, 29 June 2013 (2013-06-29), pages 374 - 381, XP032480448, DOI: 10.1109/3DV.2013.56 *
LEI TAN ET AL: "Automatic Camera Calibration Using Active Displays of a Virtual Pattern", SENSORS, vol. 17, no. 4, 1 April 2017 (2017-04-01), pages 685, XP055695950, ISSN: 1424-8220, DOI: 10.3390/s17040685 *

Also Published As

Publication number Publication date
JP2020160221A (en) 2020-10-01

Similar Documents

Publication Publication Date Title
US9946954B2 (en) Determining distance between an object and a capture device based on captured image data
JP7255718B2 (en) Information processing device, recognition support method, and computer program
CN108028900B (en) Projection apparatus, projection method, and program storage medium
TW201905573A (en) Projector system and display image calibration method thereof
US20240029273A1 (en) Information processing apparatus, control method, and program
CN108074237B (en) Image definition detection method and device, storage medium and electronic equipment
CN111083456A (en) Projection correction method, projection correction device, projector and readable storage medium
CN110619629B (en) CPU socket detection method and equipment
US11416978B2 (en) Image processing apparatus, control method and non-transitory computer-readable recording medium therefor
WO2020196091A1 (en) Image processing apparatus and image processing method
JP7114431B2 (en) Image processing method, image processing device and program
JP2009302731A (en) Image processing apparatus, image processing program, image processing method, and electronic device
US11830177B2 (en) Image processing apparatus, control method and non-transitory computer-readable recording medium therefor
US9412036B2 (en) Methods and systems for object detection based on column-wise and row-wise sums of pixel values
US10097736B2 (en) Image processing device and image processing method
JP5955003B2 (en) Image processing apparatus, image processing method, and program
JP4222013B2 (en) Image correction apparatus, character recognition method, and image correction program
JP2016072691A (en) Image processing system, control method of the same, and program
US20210012459A1 (en) Image processing method and apparatus
JP2018041169A (en) Information processing device and control method and program thereof
WO2019116975A1 (en) Image processing method, image processing device, and program
JP2017120455A (en) Information processing device, program and control method
CN111428707A (en) Method and device for identifying pattern identification code, storage medium and electronic equipment
JP2020166653A (en) Information processing device, information processing method, and program
JP2018031803A (en) Projector device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20716964

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20716964

Country of ref document: EP

Kind code of ref document: A1