JP2019114968A - Information processing device, imaging guide display program, and imaging guide display method - Google Patents

Abstract

To provide an information processing device capable of improving imaging accuracy of captured image, an imaging guide display program and an imaging guide display method.SOLUTION: When picking up an image with an imaging apparatus as a pick-up image to be matched against a template image, multiple partial images which are parts of the template image picked up at plurality of positions are generated (B1). In the pick-up image, a difference between the size of a subject included in a portion corresponding to the partial images and the size of the subject included in the partial image is calculated for each partial image (B2, B3). On the basis of the difference of each of the partial images, a movement direction of the imaging apparatus in which the difference decreases is estimated (B4, B5), and the movement direction is displayed on a display unit (B6).SELECTED DRAWING: Figure 8

Description

  The present invention relates to an apparatus, program, and display method for performing matching processing between a captured image and a template image.

  2. Description of the Related Art Conventionally, there is known a technique for comparing a captured image captured by an imaging device with a template image serving as a prototype thereof to evaluate the similarity (the degree of matching) of images. For example, there is known a technique in which an image of a person's face is registered in advance, and a score (similarity of face information) of a captured image calculated using a method of template matching is used for personal authentication of the person. . In template matching, it is automatically searched where in the captured image a part of the subject corresponding to the template image is located, and the degree of similarity and the degree of matching between the corresponding portion and the template image are calculated. Such image processing can improve the matching accuracy between the captured image and the template image (see Patent Documents 1 to 3).

JP 2008-257329 A JP, 2010-219825, A JP, 2006-114053, A

  The position, the direction, and the angle of view at which the captured image is captured do not necessarily match the capturing condition of the template image. Therefore, when a three-dimensional three-dimensional object is used as an imaging target, the size deviation between the subject included in the captured image and the subject included in the template image may be large, and the matching accuracy may be reduced. With respect to such problems, it is also conceivable to reduce the deviation between the images by performing geometrical transformation such as affine transformation or homography transformation on the captured image. However, it is not possible to reproduce the state of a portion of a three-dimensional object that falls in a shadow or a blind spot (that is, a portion that can not be seen from the imaging device) by geometrical transformation.

  As described above, depending on the imaging state, the deviation between the images can not be properly corrected, and the image matching accuracy is reduced. Therefore, when matching a captured image with a template image, a “good” captured image suitable for matching, that is, a captured image captured at an imaging position, an imaging direction, and a size (field angle) equivalent to the template image is acquired. Is desirable. Such a subject becomes remarkable in the matching process using the captured image of the solid object imaged with a hand-held imaging device (for example, small video camera, a smart phone, a tablet etc.).

  In one aspect, it is an object to improve the imaging accuracy of a captured image.

  In one embodiment, the information processing apparatus includes a generation unit configured to generate a plurality of partial images in which a part of the template image is cut out at a plurality of positions when the imaging device captures an imaged image to be compared with the template image. . The image processing apparatus further includes a calculator configured to calculate, for each of the partial images, a difference between the size of the subject included in the part corresponding to the partial image in the captured image and the size of the subject included in the partial image. The image processing apparatus further includes: an estimation unit that estimates a moving direction of the imaging device that reduces the difference based on the difference for each partial image; and a display unit that causes the display device to display the moving direction.

  In one aspect, the imaging accuracy of a captured image can be improved.

It is a perspective view which shows the state of the image test | inspection in an information processing apparatus. It is a flowchart which illustrates the flow of image inspection. It is a figure which shows the hardware constitutions of an information processing apparatus (tablet). It is a figure which shows the software configuration of a photography guide display program. (A) and (B) are template images, and (C) to (F) are partial images. (A) and (B) are captured images. It is an example of a display screen of a display apparatus. It is a flow chart which illustrates a photography guide display method. 5 is a flowchart illustrating an estimation method of a moving direction. It is an example of a display screen after adjusting an imaging position, an imaging direction, and a size. It is a figure for demonstrating the guide display as a modification. It is a figure for demonstrating the guide display as a modification. It is a flowchart which illustrates the display method of a guide. It is a figure for demonstrating the guide display as a modification.

[1. Overview]
Hereinafter, an information processing apparatus, a photographing guide display program, and a photographing guide display method according to an embodiment will be described with reference to the drawings. The information processing apparatus, the display program, and the display method perform guidance for acquiring a “good” captured image 40 suitable for matching when capturing the captured image 40 to be matched with the template image 30 with the imaging device. Provide a shooting guide function. The captured image 40 mentioned here includes not only an image captured as a single independent still image, for example, by the operation of the shutter button, but also an image displayed on the display device 12 (viewfinder frame on the screen) included. That is, not only the photographed image in a general sense but also the photographed image corresponding to one frame in the moving image is referred to as a photographed image 40.

  Further, the “good” captured image 40 mentioned here means a captured image 40 with high imaging accuracy. In other words, a good captured image 40 is a captured image 40 whose imaging position, imaging direction, and size (angle of view) are close to the template image 30 to be compared. Alternatively, it is a captured image 40 having an imaging position, an imaging direction, and a size equal to (more preferably, the same as) that of the template image 30.

  In FIG. 1, a photographer captures a target product 15 (for example, a system board of a large computer 14) using a tablet 10 (information terminal) on hand at a production site, and collates the captured image 40 with a template image 30. FIG. 1 is a schematic view of a system for image inspection. In the image inspection, an imaging device 11 (camera) built in the tablet 10 is used to check the assembled state of the target product 15 and check whether there is a defect such as a deformation or a flaw. Further, the collation result between the captured image 40 and the template image 30 is displayed on the display device 12 built in the tablet 10.

  FIG. 2 is a flowchart illustrating the flow of the above-mentioned image inspection. First, the photographer holds the tablet 10 and adjusts the imaging position of the target product 15 while referring to the template image 30 (step A1). At this time, the imaging position, imaging direction, size and the like are guided to the photographer so that the captured image 40 becomes a “good” captured image 40 suitable for matching with the template image 30. In the present embodiment, the moving direction in which the tablet 10 should be moved is presented to the photographer. In addition, the captured image 40 and the template image 30 at that time are collated with each other, and the matching degree of the entire image is calculated to guide the photographer. In response to this, the photographer changes the position and angle of the tablet 10 so as to increase the degree of coincidence along the guidance, and captures the captured image 40 (step A2).

  Geometrical transformation processing such as affine transformation and homography transformation is applied to the captured image 40 (step A3) and image correction processing is applied (step A4). The geometrical transformation transforms the captured image 40 into a shape closer to the template image 30, and the deviation between the images of the subject is reduced. Further, the lightness, tone, contrast, histogram, noise and the like of the captured image 40 are corrected by the image correction processing, and the matching accuracy of the captured image 40 with the template image 30 is improved. A known image correction method or image conversion method can be applied to a specific method of geometric conversion processing or image correction processing.

  Further, a matching process of matching the template image 30 with the captured image 40 is performed, and the matching degree of the image is calculated (step A5). As a specific index representing the degree of coincidence, coefficients such as normalized cross correlation (NCC, Normalized Cross Correlation) of the template image 30 and the captured image 40 and zero mean normalized cross correlation (ZNCC) are used. It can be used. Thereafter, the quality of the target product 15 is determined based on the matching degree of the template image 30 and the captured image 40. The quality of the target product 15 is judged by a manager or a photographer at the production site. The control configuration may be such that the quality of the target product 15 is automatically determined in accordance with the degree of coincidence between the template image 30 and the captured image 40.

[2. Hardware configuration]
FIG. 3 is a block diagram illustrating the hardware configuration of the tablet 10. The tablet 10 incorporates an imaging device 11, a display device 12, a touch panel 13, and an information processing device 20 (computer). The imaging device 11 is a digital camera attached to the back of the tablet 10, and includes an image sensor such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), a lens, a filter, and the like. Information of an image captured by the imaging device 11 and information of a captured image (captured image 40) is transmitted to the information processing device 20.

  The display device 12 is one of output devices attached to the surface of the tablet 10, and is, for example, a liquid crystal display (LCD) or an organic electro-luminescent display (OELD). In addition to the template image 30, an image of a shooting range of the imaging device 11 is displayed on the display device 12 at any time regardless of the operation state of the shutter button. That is, the display device 12 also functions as a viewfinder of the imaging device 11. The touch panel 13 is one of the input devices attached to the surface of the display device 12 in a superimposed state. When the tablet 10 is provided with a physical key, the physical key may be used instead of the touch panel 13. The touch panel 13 is used to select or input information related to imaging.

  The information processing device 20 includes a processor 21 (central processing unit), a memory 22 (main memory, main storage device), an auxiliary storage device 23, an interface device 24, a recording medium drive 25, etc. They are communicably connected to each other. The processor 21 is a central processing unit including a control unit (control circuit), an operation unit (operation circuit), a cache memory (register group), and the like. The memory 22 is a storage device in which programs and data during work are stored, and includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The auxiliary storage device 23 is a memory device in which data and firmware held for a longer period of time than the memory 22 are stored, and includes, for example, non-volatile memory such as flash memory and EEPROM.

  The interface device 24 is responsible for input and output (I / O) between the information processing device 20 and the outside. The information processing device 20 is connected to the imaging device 11, the display device 12, the touch panel 13 and the like via the interface device 24. Further, the recording medium drive 25 is a reading device (or a reading device (or a reading device) having a function of reading information recorded and stored in a recording medium 27 (removable medium) such as an optical disk or a semiconductor memory (portable flash drive conforming to Universal Serial Bus standard). Read / write device). The program executed by the information processing apparatus 20 may be recorded and stored in the memory 22 or may be recorded and stored in the auxiliary storage device 23 or the recording medium 27.

[3. Software configuration]
FIG. 4 is a block diagram showing a functional configuration of the photographing guide display program 1 executed by the information processing apparatus 20. As shown in FIG. The photographing guide display program 1 is provided with a generation unit 2, a collation unit 3, a calculation unit 4, an estimation unit 5, and a display unit 6. Each element in FIG. 4 shows functions of the photographing guide display program 1 classified for convenience, and each element may be described as an independent program, or a compound having these functions. It may be written as a program.

  The photographing guide display program 1 is recorded and stored in the memory 22 or in the auxiliary storage device 23. Alternatively, the photographing guide display program 1 is recorded and stored on the recording medium 27, and the photographing guide display program 1 written to the recording medium 27 is read into the information processing apparatus 20 via the recording medium drive 25 and executed. Ru. The photographing guide display program 1 is implemented when the photographer photographs the photographed image 40 to be collated with at least the template image 30 with the tablet 10. The photographing guide display program 1 of the present embodiment is executed at step A1 in FIG.

  The generation unit 2 generates a plurality of partial images in which a part of the template image 30 is cut out at a plurality of positions when the captured image 40 to be collated with the template image 30 is captured. The generation unit 2 generates at least two or more partial images, and preferably generates three or more partial images. For example, the first image, the second image, and the third image are generated, and the second image is different in position in the vertical direction from the first image, and the third image is different in position in the horizontal direction from the first image It shall be. That is, a pair of partial images (first image, second image) having different positions in the vertical direction and a pair of first partial images (first image, third image) having different positions in the horizontal direction are generated.

  The template image 30 of the target product 15 is illustrated in FIG. 5 (A). The generation unit 2 according to the present embodiment generates partial images corresponding to four positions of the upper position, the lower position, the right position, and the left position with respect to the center position of the template image 30. The symbols u, d, l and r in the portions indicated by the dashed box in FIG. 5B indicate the upper position, the lower position, the left position and the right position, respectively. If the image at the upper position u is the first image, the one corresponding to the second image is the image at the lower position d, the left position l and the right position r, and the one corresponding to the third image is the left position l , Right position r.

  In addition, the generation unit 2 of the present embodiment generates an enlarged deformation of each of the four partial images and a reduced deformation. For example, as shown in FIG. 5C, with the upper partial image Nu cut out from the upper position u of the template image 30 as a reference size, it is enlarged (for example + 10%) to an enlarged partial image Bu and reduced ( For example, the reduced partial image Su is generated by −10%). Similarly, with respect to the lower position d, the left position l, and the right position r, those obtained by deforming the partial images in the enlargement / reduction direction are generated (see FIGS. 5D to 5F). Thus, the generation unit 2 of the present embodiment generates a total of twelve partial images from one template image 30.

  The collation unit 3 calculates the degree of coincidence with the captured image 40 for each of the partial images generated by the generation unit 2. Here, the captured image 40 is illustrated in FIG. 6A, and positions corresponding to four positions of the upper position u, the lower position d, the left position l, and the right position r of the template image 30 are illustrated in FIG. Indicated by a dashed box. The collation unit 3 of the present embodiment calculates the degree of coincidence with the corresponding part of the captured image 40 for each of the twelve partial images.

For example, for the upper partial image Nu, the upper enlarged partial image Bu, and the upper reduced partial image Su, the coincidence with the upper position Zu in FIG. 6B is calculated. Similarly, the three partial images corresponding to the lower position u calculate the coincidence with the lower position Zd in FIG. 6 (B). The degree of coincidence calculated here can be calculated using the same method as the degree of coincidence calculated in step A5 described above. When the normalized cross correlation is used as an index of the degree of coincidence, the normalized cross correlation coefficient R _NCC may be calculated according to the following equation 1. The value range of the normalized cross correlation coefficient R _NCC calculated here is −1 ≦ R _NCC ≦ 1, and the closer the normalized cross correlation coefficient R _NCC is to 1 (the larger the value of R _NCC ), the more consistent It can be evaluated that the degree is high.

Further, in the case of using the zero mean normalized cross correlation, the normalized cross correlation coefficient R _ZNCC may be calculated according to the following Equation 2. The range of the normalized cross correlation coefficient R _ZNCC is −1 ≦ R _ZNCC ≦ 1, and the closer the normalized cross correlation coefficient R _ZNCC is to 1 (the larger the value of R _ZNCC ), the higher the degree of coincidence It can be evaluated.

  The calculation unit 4 calculates how the size of the subject included in the captured image 40 is different from that of the template image (that is, is it larger or smaller or is just right). Here, the difference (size difference) between the size of the subject included in the part corresponding to each partial image in the captured image 40 and the size of the subject included in the corresponding partial image is calculated for each partial image. The “difference” mentioned here can be calculated based on the change in the degree of matching calculated by the matching unit 3 when the partial image is deformed in the scaling direction.

  For example, when the degree of coincidence with the upper position Zu of the captured image 40 is increased when the upper partial image Nu is enlarged and deformed, the size of the subject included in the upper position Zu is larger. It can be determined that the distance from the subject to the subject is rather short. Conversely, when the degree of coincidence with the upper position Zu of the captured image 40 is increased when the upper partial image Nu is reduced and deformed, the size of the subject included in the upper position Zu is smaller. It can be determined that the distance from the upper side to the subject is rather long. By repeating such a determination for four places (Zu, Zd, Zl, Zr) of the captured image 40, the deviation of the position or the inclination of the tablet 10 can be accurately grasped.

  In order to observe the matching level while deforming the partial image in the enlargement / reduction direction, considerable computing power is required, so changes in the matching level are observed using the enlarged partial image and the reduced partial image prepared in advance. May be For example, the matching degree of each of the partial image, the enlarged partial image, and the reduced partial image as shown in FIGS. 5C to 5F is compared. If the partial image is the one with the highest degree of coincidence, it is determined that the distance from the imaging device 11 to the subject included in the partial image is appropriate. On the other hand, it is judged that the distance is short if the one with the highest degree of coincidence is the enlarged partial image, and it is judged that the distance is far when the one with the highest degree of coincidence is the reduced partial image.

The calculating unit 4 of the present embodiment calculates the difference between the size of the subject and the size of the subject based on the degree of coincidence of the twelve partial images generated by the generating unit 2. For example, among the three partial images Bu, Nu, and Su generated from the upper position u of the template image 30, the one with the highest degree of coincidence is selected with respect to the size difference in the vertical direction. Further, among the three partial images Bd, Nd, and Sd generated from the lower position d, the one with the highest degree of coincidence is selected. Here, when the combination with the highest degree of coincidence is the following, it is determined that the difference in size in the vertical direction is 0 (the distance from the tablet 10 to the subject is uniform in the vertical direction).
・ When the combination with the highest degree of coincidence is Nu and Nd ・ When the combination with the highest degree of coincidence is Bu and Bd ・ When the combination with the highest degree of coincidence is Su and Sd

On the other hand, when the combination with the highest degree of matching is the following, the size of the subject is large on the upper side of the captured image 40 and the size of the subject is small on the lower side (the lower side of the tablet 10 is inclined to move away from the subject) I will judge.
-When the combination with the highest degree of coincidence is Bu and Nd-When the combination with the highest degree of coincidence is Bu and Sd-When the combination with the highest degree of coincidence is Nu and Sd

If none of the above applies, it is determined that the size of the subject is small on the upper side of the captured image 40 and the size of the subject is large on the lower side (the upper side of the tablet 10 is tilted away from the subject). The combinations which do not correspond to any of the above are as follows.
・ When the combination with the highest degree of match is Nu and Bd ・ When the combination with the highest degree of match is Su and Bd ・ When the combination with the highest degree of match is Su and Nd

The same applies to the difference in size in the left and right direction. The conditions for determining that the difference in size in the left-right direction is 0 (the distance from the tablet 10 to the subject is uniform in the left-right direction) are as follows.
・ When the combination with the highest degree of coincidence is Nl and Nr ・ When the combination with the highest degree of coincidence is Bl and Br ・ When the combination with the highest degree of coincidence is Sl and Sr

On the other hand, conditions for determining that the size of the subject is large on the left side of the captured image 40 and the size of the subject on the right side is small (the right side of the tablet 10 is inclined to move away from the subject) are as follows.
・ When the combination with the highest degree of coincidence is Bl and Nr ・ When the combination with the highest degree of coincidence is Bl and Sr ・ When the combination with the highest degree of coincidence is Nl and Sr

If none of the above applies, it is determined that the size of the subject is small on the left side of the captured image 40 and the size of the subject is large on the right side (the left side of the tablet 10 is tilted away from the subject). The combinations which do not correspond to any of the above are as follows.
・ When the combination with the highest degree of agreement is Nl and Br ・ When the combination with the highest degree of agreement is Sl and Br ・ When the combination with the highest degree of agreement is Sl and Nr

In addition, the calculation unit 4 calculates a “total match degree” that is the overall match degree of the captured image 40 with the template image 30. When the normalized cross correlation is used as an index of the overall degree of coincidence, the normalized cross correlation coefficients R _NCC , R _{Z NCC} and the like may be calculated according to the above-described Equation 1 and Equation 2. The calculation unit 4 of the present embodiment calculates the degree of coincidence of the entire captured image 40 based on the highest degree of coincidence at each of the upper position Zu, the lower position Zd, the left position Zl, and the right position Zr. The overall degree of coincidence can be calculated, for example, from the total value or the average value of the four degrees of coincidence. The overall matching degree calculated here is displayed on the display device 12 by the display unit 6 as needed.

  The estimation unit 5 estimates the moving direction of the tablet 10 for reducing the difference based on the difference between the partial images calculated by the calculation unit 4. Here, when the imaging position and the angle are changed, it is estimated that the direction in which the part determined to be far from the subject is closer to the subject is the “moving direction of the tablet 10”. For example, when it is determined that the upper side of the tablet 10 is far from the subject, the distance to the subject becomes uniform by moving the tablet 10 upward (or tilting the upper side of the tablet 10 so as to approach the subject). sell. Therefore, the movement direction of the tablet 10 is estimated to be "upward".

  The estimation unit 5 of the present embodiment estimates the moving direction in the vertical direction of the captured image 40 and the moving direction in the horizontal direction based on the determination of the calculation unit 4. For example, when it is determined that the upper side of the tablet 10 is inclined to move away from the subject, the movement direction is estimated to be “upward”. Similarly, when it is determined that the left side of the tablet 10 is inclined to move away from the subject, the movement direction is estimated to be “left direction”.

  The display unit 6 causes the display device 12 to display the moving direction estimated by the estimation unit 5. Here, the display device 12 displays the captured image 40, a mark 50 having a shape, an arrangement, and a color representing the moving direction of the imaging device 11 (tablet 10), and a degree of coincidence information 51 representing the overall degree of coincidence. . An example of a display screen of the display device 12 is shown in FIG. In this example, the template image 30 is displayed on the left side of the screen, and the captured image 40 is displayed on the right side. On the template image 30, an enclosing frame representing a matching position is superimposed and displayed.

  On the captured image 40, an enclosing frame having a size corresponding to the size difference of the subject with respect to the template image 30 is superimposed and displayed. Further, on the right side of the captured image 40, a wedge-shaped (long and narrow triangle shape) image bar is displayed. The direction of the image bar is set to point to the moving direction (upward in FIG. 7) in the vertical direction of the captured image 40. When the movement direction estimated by the estimation unit 5 is "downward", an image bar with the tip directed downward is displayed. Similarly, an image bar is also displayed below the captured image 40. The direction of the image bar is set to point to the moving direction (left direction in FIG. 7) in the left-right direction of the captured image 40. Note that the thickness of the image bar may be displayed in correspondence with the distance to the subject.

[4. flowchart]
FIG. 8 is a flowchart illustrating the procedure of the photographing guide display method by the photographing guide display program 1. First, a plurality of partial images in which a portion of the template image 30 is cut out at a plurality of locations are generated (step B1), and enlarged partial images and reduced partial images are generated with each partial image as a reference size (step B2). The steps so far can be performed before the image inspection is started.

  Subsequently, for all partial images including the enlarged partial image and the reduced partial image, the coincidence with the corresponding part of the captured image 40 is calculated (step B3). Further, for each position at which the partial image is cut out in step B1, the one with the highest degree of coincidence is selected among the partial image, the enlarged partial image, and the reduced partial image (step B4). Thereby, the size difference of the subject at each position where the partial image is cut out in step B1 is grasped. Thereafter, based on the size difference of the subject at each position, the moving direction of the tablet 10 for reducing the size difference is estimated (step B5), and the mark 50 representing the moving direction is displayed on the display 12 Step B6).

  FIG. 9 is a flowchart illustrating a procedure for estimating the moving direction of the captured image 40 in the vertical direction among the moving directions estimated in step B5. First, among the three partial images Bu, Nu, Su generated from the upper position u, the one with the highest degree of coincidence is selected (step C1). Similarly, among the three partial images Bd, Nd, and Sd generated from the lower position d, the one with the highest degree of coincidence is selected (step C2). There are nine combinations of the partial image selected in step C1 and the partial image selected in step C2.

  Here, when the combination with the highest degree of coincidence is “Nu and Nd” or “Bu and Bd” or “Su and Sd” (step C3), the distance from the tablet 10 to the subject is uniform in the vertical direction It is determined that there is an image bar having a constant thickness in the vertical direction is displayed on the display device 12 (step C5). On the other hand, if the combination with the highest degree of coincidence is “Bu and Nd” or “Bu and Sd” or “Nu and Sd” (step C4), it is assumed that the lower side of the tablet 10 is inclined away from the subject. It is judged, and an image bar (the upper end is thick and the lower end is thin) whose tip is directed downward is displayed on the display device 12 (step C6). If the conditions in steps C3 and C4 are not satisfied, it is determined that the upper side of the tablet 10 is inclined to move away from the subject, and the image bar with the tip directed upward (the image bar with a thick lower end and a thin upper end) Is displayed on the display unit 12 (step C7).

  The display contents of the image bar are also controlled in the same procedure with respect to the moving direction of the captured image 40 in the left and right direction. The photographer holding the tablet 10 adjusts the position and orientation of the tablet 10 while looking at the image bar, and continues adjustment until the thickness of the image bar becomes constant. As a result, as shown in FIG. 10, both the image bar extending in the vertical direction and the image bar extending in the lateral direction have a constant thickness. The imaging position, imaging direction, and size at this time are equal to those of the template image 30. Therefore, by capturing the still image by operating the shutter button, the captured image 40 with high imaging accuracy can be obtained.

[5. Action, effect]
(1) In the above-described embodiment, the moving direction of the imaging device 11 (tablet 10) for reducing the difference in the size of the subject for each partial image is displayed on the display device 12. Thereby, the photographer can easily correct the deviation of the imaging position, imaging direction, and size (angle of view) of the imaging device 11 with respect to the subject. That is, a good captured image 40 suitable for matching with the template image 30 can be easily captured, and the imaging accuracy of the captured image 40 can be improved. Therefore, the inspection accuracy, the determination accuracy, and the reliability of the image inspection can be improved. In addition, since the moving direction of the imaging device 11 is displayed on the display device 12 as needed, the work of capturing a good captured image 40 becomes easy, and the work efficiency can be improved.

  (2) In the above embodiment, when the partial image is cut out from the template image 30, the cut-out position is set so that a pair having different vertical positions and a pair having different horizontal positions are included. The former pair is used to grasp how the size of the subject is changing in the vertical direction, and the latter pair is how the size of the subject is changing in the horizontal direction Used to figure out Therefore, the overall inclination and positional deviation of the captured image 40 with respect to the subject can be accurately grasped, and the imaging accuracy of the captured image 40 can be further improved.

  (3) In the above-described embodiment, the matching unit 3 calculates the degree of coincidence with the captured image 40 for each partial image generated by the generation unit 2. For example, if the matching degree of a partial image is high, the size of the subject at the position where the partial image is cut out can be regarded as appropriate (the difference is zero) in relation to the template image 30. . On the other hand, if the degree of coincidence of a partial image is low, it can be regarded that there is a difference in the size of the subject (the difference is large) at that position. Thus, the difference in the size of the subject can be evaluated using the degree of coincidence as an index, and the imaging accuracy of the captured image 40 can be further improved.

(4) As shown in Equations 1 and 2, when the normalized cross correlation coefficients R _NCC and R _ZNCC between the partial image of the template image 30 and the captured image 40 are used as the index of the coincidence, the template image It is possible to calculate the degree of coincidence highly robust against the fluctuation of the brightness of the image 30 or the captured image 40. For example, regardless of the use state of the auxiliary light incorporated in the tablet 10, a stable degree of coincidence can be calculated. Therefore, the imaging accuracy of the captured image 40 can be further improved.

  (5) Further, by tracking the change in the degree of coincidence when the partial image is deformed in the enlargement / reduction direction, it is possible to accurately calculate the difference in the size of the subject. For example, if the degree of coincidence increases when the partial image is enlarged and deformed, it is possible to determine that the subject included in the partial image before enlargement is small and the size of the subject included in the partial image after enlargement is appropriate. . Conversely, if the degree of coincidence increases when the partial image is reduced and deformed, it is determined that the subject included in the partial image before reduction is large and the size of the subject included in the reduced partial image is appropriate. it can. Therefore, the difference in the size of the subject can be accurately grasped, and the imaging accuracy of the captured image 40 can be further improved.

  (6) In the above embodiment, as shown in FIGS. 5A to 5F, enlarged partial images obtained by enlarging and deforming each of the four partial images and reduced partial images obtained by reducing and generating the partial images are created. Be done. By comparing the matching degree of each partial image, the distance from the imaging device 11 to the subject can be accurately evaluated, and the difference in the size of the subject can be accurately grasped. Therefore, the imaging accuracy of the captured image 40 can be further improved.

  (7) In the above-described embodiment, as shown in FIG. 7, the matching degree information 51 indicating the overall matching degree between the captured image 40 and the template image 30 is displayed on the display device 12. As a result, the photographer can easily grasp the necessity of changing the imaging position, imaging direction, size, and the like, and the movement of the imaging device 11 to an appropriate position and angle can be promoted. Therefore, the imaging accuracy of the captured image 40 can be further improved.

  (8) In the embodiment described above, the mark 50 representing the moving direction of the imaging device 11 (tablet 10) is displayed on the display device 12. As a result, the photographer can intuitively grasp the direction in which the imaging device 11 (tablet 10) should be moved, and can easily move the imaging device 11 in the direction in which a good captured image 40 can be expected. Therefore, the imaging accuracy of the captured image 40 can be further improved.

[5. Modified example]
The present embodiment is merely an example, and there is no intention to exclude the application of various modifications and techniques not explicitly stated in the above embodiment. That is, it is possible to variously deform and implement the above-mentioned embodiment (for example, by combining an embodiment and a modification) in the range which does not deviate from the meaning.

  For example, in the above-described embodiment, partial images corresponding to four positions of the upper position u, lower position d, left position l, and right position r than the center position of the template image 30 are generated. The number and generation position are not limited to this. By generating at least a plurality of (two or more) partial images, the size of the subject included in the part corresponding to the partial image can be evaluated, and the same effect as the above-described embodiment can be obtained.

  In the above embodiment, the moving direction of the imaging device 11 (tablet 10) is expressed by the orientation of the wedge-shaped image bar, but the shape, arrangement, and color of specific marks 50 for prompting the photographer to move Etc. are not limited to this. For example, as shown in FIG. 11, a dashed circle 53 is displayed at the center of the captured image 40. Further, a circle 52 having a diameter smaller than that of the broken line circle 53 is displayed at a position determined according to the difference in size of the subject. Furthermore, the movement direction of the tablet 10 in the above-described embodiment is represented by a single arrow, the starting point of the arrow is disposed at the center of the dashed circle 53, and the circle 52 is disposed at the tip of the arrow.

  That is, the position of the circle 52 is set to a position on the captured image 40 that is biased toward the side at which the distance to the subject at that time is far. By using such a mark 50, the tablet 10 can be moved so that the center of the dashed circle 53 and the center of the circle 52 coincide with each other, and the imaging position, imaging direction, size and the like can be easily optimized. . Further, since the direction in which the tablet 10 is inclined can be intuitively grasped, the work efficiency can be further improved.

  As shown in FIG. 12, guide bars 54 having different colors and display densities may be arranged around the captured image 40. The guide bar 54 displays the upper end side, the lower end side, the right end side, and the left end side of the captured image 40 (that is, four guide bars 54 in total). Also, the density (color) of each guide bar 54 is set according to the degree of coincidence of the four partial images corresponding to the four positions of the upper position u, the lower position d, the left position l and the right position r.

  A flowchart for setting the density of the guide bar 54 is illustrated in FIG. This flowchart is performed independently for each guide bar 54. The degree of coincidence referred to in setting the density of the guide bar 54 is the degree of coincidence of one partial image corresponding to the position of the guide bar 54. For example, the density of the guide bar 54 on the upper end side of the captured image 40 is controlled based on the matching degree of the partial image of the upper position u.

  First, the degree of coincidence of the partial image at the upper position u is calculated (step D1), and it is determined whether the degree of coincidence exceeds 80% (step D2). If this condition is satisfied, the density of the guide bar 54 on the upper end side is set to the lightest density (indicating that the distance is an appropriate distance) (step D4). On the other hand, if this condition is not satisfied, it is determined whether the degree of coincidence exceeds 50% (step D3). If this condition is satisfied, a slightly dark density (indicating that the distance is slightly different from the appropriate distance) is set (step D5). If this condition is not satisfied, the degree of coincidence is less than 50%, so the darkest density (indicating that the distance is considerably different from the appropriate distance) is set (step D6).

  As described above, by displaying the guide bar 54 with the density corresponding to the degree of coincidence, the photographer can adjust the position and the orientation of the tablet 10 so that the density of the guide bar 54 becomes generally thin, and the template The same imaging position, imaging direction, and size as those of the image 30 can be searched. Further, by capturing the still image by operating the shutter button in a state in which all the guide bars 54 have a low density, it is possible to obtain a captured image 40 with high imaging accuracy.

  In the example shown in FIG. 12, the magnitude relationship between the distance to the subject and the appropriate distance is not displayed on the display device 12. Therefore, a semi-transparent partial image may be superimposed on the captured image 40. The simplest superimposed display method is to semi-transparent the upper partial image Nu, the lower partial image Nd, the left partial image Nl, and the right partial image Nr in the above-described embodiment, and superimpose them on the captured image 40 for display. In this case, it is possible to provide the photographer with an image serving as a guide for adjusting the position and orientation of the tablet 10, and the work efficiency can be further improved.

  Further, as shown in FIG. 14, the positions corresponding to the four positions of the upper position u, the lower position d, the left position l, and the right position r are displayed by a dashed box, and the highest matching rate is obtained at each position. The partial image (one of the partial image, the enlarged partial image, and the reduced partial image) may be translucent and displayed so as to be superimposed on the captured image 40. In this case, the photographer may adjust the position and orientation of the tablet 10 so that the inside of the frame of the captured image 40 matches the semi-transparent partial image, and the working efficiency can be further improved.

[6. Appendix]
The following appendices will be disclosed regarding the embodiment including the above-described modified example.
(Appendix 1 to 8: information processing apparatus)
(Supplementary Note 1)
A generation unit configured to generate a plurality of partial images in which a part of the template image is cut out at a plurality of positions when the imaging device captures an imaged image to be compared with the template image;
A calculator configured to calculate, for each of the partial images, a difference between the size of the subject included in the part corresponding to the partial image in the captured image and the size of the subject included in the partial image;
An estimation unit configured to estimate a moving direction of the imaging device to reduce the difference based on the difference for each partial image;
A display unit for displaying the moving direction on a display device;
An information processing apparatus comprising:

(Supplementary Note 2)
The plurality of portions including the first image, the second image whose vertical position is different from the first image, and the third image whose horizontal position is different from the first image. The information processing apparatus according to appendix 1, wherein an image is generated.

(Supplementary Note 3)
The information processing apparatus according to any one of appendices 1 or 2, further comprising a collating unit that calculates a degree of coincidence of each of the partial images with the captured image.

(Supplementary Note 4)
The information processing apparatus according to claim 3, wherein the matching unit calculates a normalized cross correlation between the partial image corresponding to the degree of coincidence and the captured image.

(Supplementary Note 5)
The above-mentioned difference is characterized by calculating the above-mentioned difference based on change of the above-mentioned degree of coincidence computed by the above-mentioned collation part, when the above-mentioned calculation part is made to deform the above-mentioned partial image in the expansion and contraction direction. An information processing apparatus according to item 1.

(Supplementary Note 6)
The generation unit generates, for each of the partial images, an enlarged partial image obtained by enlarging and deforming the partial image and a reduced partial image obtained by reducing and deforming the partial image;
The matching unit calculates the matching degree of each of the partial image, the enlarged partial image, and the reduced partial image;
Among the partial image, the enlarged partial image, and the reduced partial image, the calculation unit
When the partial image is the one with the highest degree of coincidence, it is determined that the distance from the imaging device to the subject included in the partial image is appropriate;
When the one with the highest degree of coincidence is the enlarged partial image, it is determined that the distance from the imaging device to the subject included in the partial image is short.
The system according to any one of appendices 3 to 5, wherein it is determined that the distance from the imaging device to the subject included in the partial image is long when the reduced image is the one with the highest degree of coincidence. The information processing apparatus according to claim 1.

(Appendix 7)
Appendix 3. The display device causes the display device to display the overall matching score between the captured image and the template image, which is calculated based on the matching score for each of the partial images. The information processing apparatus according to any one of 6.

(Supplementary Note 8)
The information processing according to any one of Appendices 1 to 7, wherein the display unit causes the display device to display a mark having a shape, an arrangement, or a color representing the movement direction and the captured image. apparatus.

(Appendix 9 to 16: Shooting guide display program)
(Appendix 9)
A plurality of partial images in which a part of the template image is cut out at a plurality of positions when the imaging device captures an imaged image to be compared with the template image;
Calculating a difference between the size of the subject included in the part corresponding to the partial image in the captured image and the size of the subject included in the partial image for each of the partial images;
Estimating the moving direction of the imaging device to reduce the difference based on the difference for each partial image;
The imaging | photography guide display program which makes a computer perform the process which displays the said moving direction on a display apparatus.

(Supplementary Note 10)
A process of generating the plurality of partial images including a first image, a second image whose position in the vertical direction is different from the first image, and a third image whose position in the horizontal direction is different from the first image The imaging guide display program according to appendix 9, causing the computer to execute.

(Supplementary Note 11)
The imaging guide display program according to supplementary note 9 or 10, causing the computer to execute a process of calculating the degree of coincidence with the captured image for each of the partial images.

(Supplementary Note 12)
The imaging guide display program according to appendix 11, causing a computer to execute processing of calculating normalized cross-correlation of the partial image and the captured image corresponding to the degree of coincidence.

(Supplementary Note 13)
The computer according to any one of Appendices 11 or 12, causing the computer to execute the process of calculating the difference based on a change in the matching degree calculated by the matching unit when the partial image is deformed in the enlargement / reduction direction. Shooting guide display program.

(Supplementary Note 14)
Generating an enlarged partial image obtained by enlarging and deforming the partial image and a reduced partial image obtained by reducing and deforming the partial image for each of the partial images;
Calculating the matching degree of each of the partial image, the enlarged partial image, and the reduced partial image;
Of the partial image, the enlarged partial image, and the reduced partial image,
When the partial image is the one with the highest degree of coincidence, it is determined that the distance from the imaging device to the subject included in the partial image is appropriate;
When the one with the highest degree of coincidence is the enlarged partial image, it is determined that the distance from the imaging device to the subject included in the partial image is short.
The computer executes processing for determining that the distance from the imaging device to the subject included in the partial image is long when the reduced image is the one with the highest degree of coincidence, any one of appendices 11 to 13. The shooting guide display program described in the section.

(Supplementary Note 15)
The computer according to any one of Appendices 11 to 14, causing the display device to display the overall matching score between the captured image and the template image, which is calculated based on the matching score for each of the partial images. The shooting guide display program described in item 1.

(Supplementary Note 16)
The imaging guide display program according to any one of appendices 9 to 15, causing a computer to execute a process of causing the display device to display a mark having a shape, an arrangement, or a color representing the moving direction and the captured image.

(Appendix 17 to 24: Display method of shooting guide)
(Supplementary Note 17)
A plurality of partial images in which a part of the template image is cut out at a plurality of positions when the imaging device captures an imaged image to be compared with the template image;
Calculating a difference between the size of the subject included in the part corresponding to the partial image in the captured image and the size of the subject included in the partial image for each of the partial images;
Estimating the moving direction of the imaging device to reduce the difference based on the difference for each partial image;
And displaying the moving direction on a display device.

(Appendix 18)
Generating a plurality of partial images including a first image, a second image whose position in the vertical direction is different from the first image, and a third image whose position in the horizontal direction is different from the first image The photographing guide display method according to appendix 17, characterized in that:

(Appendix 19)
The photographing guide display method according to appendix 17 or 18, characterized in that the degree of coincidence with the captured image for each of the partial images is calculated.

(Supplementary Note 20)
24. The photographing guide display method according to appendix 19, wherein normalized cross correlation between the partial image corresponding to the degree of coincidence and the captured image is calculated.

(Supplementary Note 21)
20. The difference according to any one of appendices 19 or 20, wherein the difference is calculated based on a change in the matching degree calculated by the matching unit when the partial image is deformed in the enlargement / reduction direction. Shooting guide display method.

(Supplementary Note 22)
Generating an enlarged partial image obtained by enlarging and deforming the partial image and a reduced partial image obtained by reducing and deforming the partial image for each of the partial images;
Calculating the matching degree of each of the partial image, the enlarged partial image, and the reduced partial image;
Of the partial image, the enlarged partial image, and the reduced partial image,
When the partial image is the one with the highest degree of coincidence, it is determined that the distance from the imaging device to the subject included in the partial image is appropriate;
When the one with the highest degree of coincidence is the enlarged partial image, it is determined that the distance from the imaging device to the subject included in the partial image is short.
Supplementary note 19 to 21, wherein it is determined that the distance from the imaging device to the subject included in the partial image is long when the reduced image is the one with the highest degree of coincidence. The shooting guide display method described in.

(Supplementary Note 23)
Supplementary note 19 to 22, wherein the overall degree of coincidence between the captured image and the template image, which is calculated based on the degree of coincidence for each of the partial images, is displayed on the display device. Method of displaying a shooting guide described in the section.

(Supplementary Note 24)
The photographing guide display method according to any one of appendices 17 to 23, characterized in that the display device displays the mark having a shape, an arrangement, or a color representing the movement direction and the captured image.

DESCRIPTION OF SYMBOLS 1 photography guide display program 2 generation part 3 collation part 4 calculation part 5 estimation part 6 display part 10 tablet 11 imaging device 12 display device 20 information processing device (computer)
30 template image 40 captured image

Claims (10)

A generation unit configured to generate a plurality of partial images in which a part of the template image is cut out at a plurality of positions when the imaging device captures an imaged image to be compared with the template image;
A calculator configured to calculate, for each of the partial images, a difference between the size of the subject included in the part corresponding to the partial image in the captured image and the size of the subject included in the partial image;
An estimation unit configured to estimate a moving direction of the imaging device to reduce the difference based on the difference for each partial image;
A display unit for displaying the moving direction on a display device;
An information processing apparatus comprising: The plurality of portions including the first image, the second image whose vertical position is different from the first image, and the third image whose horizontal position is different from the first image. The information processing apparatus according to claim 1, which generates an image. The information processing apparatus according to claim 1, further comprising a collating unit that calculates a degree of coincidence of the partial images with the captured image. The information processing apparatus according to claim 3, wherein the matching unit calculates a normalized cross correlation between the partial image corresponding to the degree of coincidence and the captured image. 5. The method according to claim 3, wherein the calculation unit calculates the difference based on a change in the matching degree calculated by the matching unit when the partial image is deformed in a scaling direction. An information processing apparatus according to any one of the preceding items. The generation unit generates, for each of the partial images, an enlarged partial image obtained by enlarging and deforming the partial image and a reduced partial image obtained by reducing and deforming the partial image;
The matching unit calculates the matching degree of each of the partial image, the enlarged partial image, and the reduced partial image;
Among the partial image, the enlarged partial image, and the reduced partial image, the calculation unit
When the partial image is the one with the highest degree of coincidence, it is determined that the distance from the imaging device to the subject included in the partial image is appropriate;
When the one with the highest degree of coincidence is the enlarged partial image, it is determined that the distance from the imaging device to the subject included in the partial image is short,
The method according to any one of claims 3 to 5, characterized in that when the reduced partial image is the one with the highest degree of coincidence, it is determined that the distance from the imaging device to the subject included in the partial image is long. An information processing apparatus according to any one of the preceding items. 4. The display device according to claim 3, wherein the display unit displays, on the display device, the overall degree of coincidence between the captured image and the template image, which is calculated based on the degree of coincidence for each of the partial images. The information processing apparatus according to any one of to 6. The information according to any one of claims 1 to 7, wherein the display unit causes the display device to display a mark having a shape, an arrangement, or a color representing the movement direction and the captured image. Processing unit. A plurality of partial images in which a part of the template image is cut out at a plurality of positions when the imaging device captures an imaged image to be compared with the template image;
Calculating a difference between the size of the subject included in the part corresponding to the partial image in the captured image and the size of the subject included in the partial image for each of the partial images;
Estimating the moving direction of the imaging device to reduce the difference based on the difference for each partial image;
The imaging | photography guide display program which makes a computer perform the process which displays the said moving direction on a display apparatus. A plurality of partial images in which a part of the template image is cut out at a plurality of positions when the imaging device captures an imaged image to be compared with the template image;
Calculating a difference between the size of the subject included in the part corresponding to the partial image in the captured image and the size of the subject included in the partial image for each of the partial images;
Estimating the moving direction of the imaging device to reduce the difference based on the difference for each partial image;
And displaying the moving direction on a display device.

Images