JP2017028687A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device that can enhance the working efficiency when plural input frame images are positionally aligned with one another by a manual work.SOLUTION: An image processing device includes a reference frame adjusting unit 18 for displaying an image of a reference frame and a parameter used to generate a high-resolution image, updating the display content of the image of the reference frame based on adjustment of the displayed parameter, and storing the adjusted parameter as a parameter of the reference frame in a reference point position holding memory 17A and a blur PSF parameter holding memory 17B, and a reference frame adjusting unit 19 for displaying the parameter of the reference frame as a parameter of a reference frame when an image of a reference frame is displayed, updating the display content of the image of the reference frame based on adjustment of the displayed parameter, and storing the adjusted parameter as the parameter of the reference frame in the reference point position holding memory 17A and the blur PSF parameter holding memory 17B.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that generates a high-resolution image by aligning low-resolution images of a plurality of input frames each having a common partial area.

  If there are multiple input frame images taken from the same subject and each image has a low resolution and a positional shift, a high-resolution image can be generated by aligning and superimposing the positions of those images. it can. The process of generating a high resolution image from the plurality of low resolution images is called super-resolution processing.

  Here, the shooting where the image is displaced is when shooting while moving the camera with respect to a stationary subject, or when shooting with a camera with a moving subject fixed. If the subject and the camera are completely stationary, the resolution will not increase no matter how many images are superimposed.

  For image alignment (matching), it is common to obtain alignment parameters (geometric transformation matrix such as translation and rotation) using template matching or feature point matching. In addition, a reconstruction method is generally used for overlaying images. The reconstruction method is to generate an estimated image of a high-resolution image, repeat the update and evaluation of the estimated image so that the likelihood of the estimated image is maximized, and use the converged solution of the estimated image as the final output. is there. As the likelihood of the estimated image, [1] how close the low resolution image obtained by shifting the position of the estimated image and degrading the resolution is close to the original input image, and [2] the connection of pixel values in the estimated image. Whether it is natural or not.

  By the way, in the automatic matching process that automatically aligns images, a process of finding and matching a reference point is performed in the alignment of images of a plurality of input frames, but mismatch due to erroneous detection of the reference point is unavoidable. In particular, if a moving subject is photographed at a slow shutter speed, subject blurring occurs, so the possibility of mismatching increases.

  In order to reduce the possibility of mismatch due to subject blur, it is only necessary to remove subject blur in the image before performing the super-resolution processing. In order to remove subject blurring, based on a model in which a blurring PSF (Point Spread Function) is convolved with an original image without blurring, the result is a blurry image. Deconvolution) is used (see Patent Document 1). By using this method, a deblurred image close to the original image can be obtained.

JP2011-259119A

  However, even if subject blurring is removed before performing super-resolution processing, it is difficult to completely avoid the mismatch problem due to the influence of noise and the like. In the automatic matching process, if the matching cannot be performed well, an image of a frame that should originally be usable is excluded. As a result, the number of frames that can be used decreases, and a high-resolution image cannot be obtained even if super-resolution processing is performed.

  This problem can be avoided by manually aligning the images without going through the automatic matching process, but manually shifting the images of multiple input frames one by one. The alignment process is very complicated and unproductive.

  The present invention has been made in view of such points, and an object of the present invention is to provide an image processing apparatus capable of improving work efficiency when manually aligning images of a plurality of input frames. To do.

  An image processing apparatus of the present invention is an image processing apparatus that generates a high-resolution image by aligning low-resolution images of a plurality of input frames each having a common partial area, and is selected from the plurality of input frames Display one reference frame image and a parameter used when generating the high-resolution image, update display content of the reference frame image based on the adjustment of the displayed parameter, and adjust A reference frame adjustment unit that stores the obtained parameters in the memory as parameters of the reference frame, an image of a reference frame that is one of the input frames other than the reference frame, and a parameter of the reference frame as a parameter of the reference frame Display and reference frame based on adjustment of the displayed parameter A reference frame adjustment unit that updates display content of an image and stores the adjusted parameter in a memory as a parameter of the reference frame, and the input frame includes the corresponding parameter for each input frame. A blur removing unit that performs processing to remove the blur, and an image generation unit that generates a high resolution image by aligning the low resolution image of the input frame from which the blur is removed.

  According to the present invention, since the adjustment result in the base frame can be reflected on all other reference frames, the input work can be reduced and the work efficiency can be reduced when manually aligning the images of the plurality of input frames. Can be improved. As a result, the number of frames that can be used for the super-resolution processing can be increased, so that a high-resolution image can be obtained by the super-resolution processing.

1 is a block diagram showing a schematic configuration of an image processing apparatus according to Embodiment 1 of the present invention. The flowchart for demonstrating the super-resolution process made to align automatically in the image processing apparatus which concerns on Embodiment 1 of this invention. The flowchart for demonstrating the super-resolution process which performed the position alignment manually in the image processing apparatus which concerns on Embodiment 1 of this invention. Flowchart for explaining the image data reading process in the super-resolution process of the image processing apparatus according to the first embodiment of the present invention. The flowchart for demonstrating the process which selects a reference | standard frame automatically in the super-resolution process of the image processing apparatus which concerns on Embodiment 1 of this invention. The flowchart for demonstrating the process which selects a reference frame manually in the super-resolution process of the image processing apparatus which concerns on Embodiment 1 of this invention. The flowchart for demonstrating the setting process of the attention object area | region in the super-resolution process of the image processing apparatus which concerns on Embodiment 1 of this invention. Flowchart for explaining automatic alignment processing in super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. The figure which shows an example of the geometric transformation matrix used for the automatic alignment process in the super-resolution process of this invention Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. Flowchart for explaining parameter adjustment processing at a reference frame in super-resolution processing of the image processing apparatus according to Embodiment 1 of the present invention. The figure which shows an example of the user interface screen before parameter adjustment with a reference frame The figure which shows an example of the user interface screen after the parameter adjustment in a reference frame Flowchart for explaining parameter adjustment processing at a reference frame in super-resolution processing of the image processing apparatus according to Embodiment 1 of the present invention. Flowchart for explaining parameter adjustment processing at a reference frame in super-resolution processing of the image processing apparatus according to Embodiment 1 of the present invention. The figure which shows an example of the user interface screen before parameter adjustment in a reference frame The figure which shows an example of the user interface screen after parameter adjustment in a reference frame Flowchart for explaining blur removal processing in super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. Flowchart for explaining a reconstruction process in the super-resolution process of the image processing apparatus according to the first embodiment of the present invention. The figure which shows an example of the expansion conversion matrix Mz used for a reconstruction process Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the second embodiment of the present invention. Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the second embodiment of the present invention. Flowchart for explaining face feature point detection processing in super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. The figure which shows an example of the face feature point and plane patch which are displayed in the super-resolution process of the image processing apparatus which concerns on Embodiment 3 of this invention. Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. Flowchart for explaining parameter adjustment processing at a reference frame in super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. The flowchart for demonstrating the detail of the mapping process of the face feature point in the parameter adjustment process in the reference | standard frame of Embodiment 3 of this invention The flowchart for demonstrating the parameter adjustment process in the reference frame in the super-resolution process of the image processing apparatus which concerns on Embodiment 3 of this invention. The flowchart for demonstrating the parameter adjustment process in the reference frame in the super-resolution process of the image processing apparatus which concerns on Embodiment 3 of this invention. Flowchart for explaining reconstruction processing in super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the fourth embodiment of the present invention. Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the fourth embodiment of the present invention. Flowchart for explaining blur PSF estimation processing in super-resolution processing of the image processing apparatus according to the fourth embodiment of the present invention.

  Hereinafter, the present embodiment will be described in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus 1 according to Embodiment 1 of the present invention. In the figure, an image processing apparatus 1 according to the present embodiment is an apparatus capable of performing super-resolution processing for generating a high-resolution image from a plurality of low-resolution images, and includes an image data input unit 10, a frame memory, and the like. 11. Graphical user interface unit 12, reference frame selection unit 13, attention subject region selection unit 14, matching calculation unit 15, blur PSF estimation unit 16, memory unit 17, reference frame adjustment unit 18, reference frame adjustment unit 19, blur removal Unit 20, a reconstruction processing unit 21, and a super-resolution image output unit 22. The memory unit 17 includes a reference point position holding memory 17A and a blur PSF parameter holding memory 17B. The reconstruction processing unit 21 and the super-resolution image output unit 22 constitute an image generation unit 23.

  The image data input unit 10 inputs image data sequentially output in units of frames from a camera (not shown). Note that the frame-unit image data input by the image data input unit 10 is also referred to as an input frame. The frame memory 11 temporarily stores an input frame. The graphical user interface unit 12 is a user interface that provides intuitive operations using computer graphics and a pointing device such as a mouse.

  The reference frame selection unit 13 selects one of a plurality of input frames stored in the frame memory 11 and sets it as a reference frame. The selection of the input frame to be the reference frame is performed by evaluating the image quality of each of the plurality of input frames stored in the frame memory 11 and selecting the input frame having the highest evaluation, or the plurality of input frames stored in the frame memory 11. Are displayed, and the user is allowed to select one of them.

  The target subject region selection unit 14 sets a target subject region (partial region) in the reference frame selected by the reference frame selection unit 13. The target subject area is set for the reference frame by the user. That is, the user sets a target subject area using a pointing device such as a mouse.

  The matching calculation unit 15 finds a region that matches the target subject region of the reference frame for each of a plurality of input frames other than the reference frame stored in the frame memory 11.

  The blur PSF estimation unit 16 obtains the center point of the area that matches the image data of the target subject area of the reference frame in each of the plurality of input frames other than the reference frame, and further, the blur angle and length from the motion vector of the center point. These are calculated and stored as blur PSF parameters.

  The reference frame adjustment unit 18 displays the position of the partial area and the blur PSF parameter used when generating the image of the single reference frame selected from the plurality of input frames and the high-resolution image, and the displayed partial area. The display content of the image of the reference frame is updated based on the adjustment of the position and the blur PSF parameter, and the reference point position is held as the adjusted partial region position and the blur PSF parameter as the position and blur PSF parameter of the reference frame. The data is stored in the memory 17A and the blur PSF parameter holding memory 17B, and the positions and blur PSF parameters of a plurality of input frames other than the reference frame are updated, and the reference point position holding memory 17A and the blur PSF parameter holding memory 17B are updated. Further remember.

  The reference frame adjustment unit 19 displays an image of a reference frame that is one of a plurality of input frames other than the base frame, a position of a partial area of the reference frame, and a blur PSF parameter, and displays the position of the displayed partial area and the blur PSF parameter. The display content of the image of the reference frame is updated based on the adjustment of the reference point, and the reference position holding memory 17A and the blur PSF parameter are used as the position and blur PSF parameter of the reference partial area as the adjusted partial region position and blur PSF parameter. Each is stored in the holding memory 17B.

  The blur removal unit 20 performs a process of removing the blur included in the input frame for each of the plurality of input frames stored in the frame memory 11 using the corresponding blur PSF parameter. The reconstruction processing unit 21 generates a high resolution image by aligning the low resolution image of the frame from which the blur is removed. The super-resolution image output unit 22 outputs the high resolution image generated by the reconstruction processing unit 21.

  Next, the operation of the image processing apparatus 1 according to this embodiment will be described.

(Super-resolution processing)
FIG. 2 is a flowchart showing the super-resolution processing. In the figure, super-resolution processing includes reading of image data (step S1), selection of a reference frame (step S2), setting of a subject area of interest (step S3), automatic alignment (step S4), and estimation of blur PSF. (Step S5), parameter adjustment in the reference frame (Step S6), parameter adjustment in the reference frame (Step S7), blur removal (Step S8), reconstruction processing (Step S9), and super-resolution image output ( It consists of 10 steps of step S10). Note that the super-resolution processing shown in FIG. 2 is a case where automatic alignment is performed, but there is a case where alignment is performed manually. FIG. 3 is a flowchart showing a super-resolution process in which manual alignment is performed. In the case of manual alignment, the two steps of the automatic alignment in step S4 and the blur PSF estimation in step S5 are not necessary. Hereinafter, each step of the super-resolution processing when using automatic alignment will be described in detail.

(Image data reading process)
FIG. 4 is a flowchart for explaining the image data reading process. In the figure, PosC, PosP, and PosN are registers. First, an image file is opened (step S20), and seek (move) to a designated frame (step S21). Then, the frame number of the designated current input frame is set to PosC (step S22). Next, the frame data of the frame number of the current input frame set in PosC is read into the frame memory 11 (step S23). Next, “1” is subtracted from the set value of PosC, and the value is set to PosP (step S24). In other words, the frame number of the input frame immediately preceding the current input frame is set to PosP. Next, it is determined whether or not there is frame data with the frame number set in PosP (step S25). In this determination, if there is frame data of the frame number set in PosP (step S25: YES), it is determined whether or not the value obtained by subtracting the frame number set in PosP from the frame number set in PosC is equal to or less than “R”. (Step S26). Here, “R” is a maximum number of frames that can be referenced forward and backward from the current frame as a base point, and is a value set in advance by the user.

  If the value obtained by subtracting the frame number set in PosP from the frame number set in PosC is equal to or less than “R” (step S26: YES), the frame data of the frame number set in PosP is read into the frame memory 11 (step S27). ). Then, “1” is subtracted from the frame number set in PosP (step S28), and the process returns to the determination in step S25.

  On the other hand, if there is no frame data of the frame number set in PosP in step S25 (step S25: NO), or a value obtained by subtracting the frame number set in PosP from the frame number set in PosC in step S26. When larger than “R” (step S26: NO), “1” is added to the value of PosC, and the value is set to PosN (step S29). In other words, the frame number of the input frame immediately after the current input frame is set to PosN. Then, it is determined whether or not the frame data having the frame number set in PosN exists (step S30). In this determination, when there is frame data of the frame number set in PosN (step S30: YES), it is determined whether or not a value obtained by subtracting the frame number set in PosC from the frame number set in PosN is equal to or less than R (step S30). S31).

  When the value obtained by subtracting the frame number set in PosC from the frame number set in PosN is equal to or less than “R” (step S31: YES), the frame data of the frame number set in PosN is read into the frame memory 11 (step S32). ). Next, “1” is added to the frame number set in PosN (step S33), and the process returns to the determination in step S30.

  On the other hand, if there is no frame data of the frame number set to PosN in step S30 (step S30: NO), or the value obtained by subtracting the frame number set to PosC from the frame number set to PosN in step S31 is If it is not less than “R” (step S31: NO), the reading of the image data is terminated.

(Reference frame selection process)
FIG. 5 is a flowchart for explaining the reference frame selection process. In the figure, FrmR is a register. First, “1” is set in FrmR (step S40). Next, data of the FrmR-th input frame in the frame memory 11 is acquired (step S41). Then, the image quality of the acquired frame data is evaluated and an evaluation score is obtained (step S42). Next, it is determined whether or not the evaluation score obtained this time is the maximum among the evaluated input frames (step S43). If the evaluation score obtained this time is not the maximum (step S43: NO), the FrmR value is set to “ 1 "is added and the value is set to FrmR (step S44).

  On the other hand, when the evaluation score obtained this time is the maximum (step S43: YES), the FrmR-th input frame is set as the reference frame (FrmA) (step S45). Here, it is preferable to use an input frame having high image quality (not strongly compressed by the codec and having good high-frequency components) as a reference frame. For example, it is effective to convert each frame data into the frequency domain, evaluate the image quality with a large or small amount of high-frequency components, and use an input frame with the highest high-frequency components as a reference frame.

  After the process of step S45, “1” is added to the value of FrmR, and the value is set to FrmR (step S44). Next, it is determined whether or not the value of FrmR is equal to or less than “N” (the total number of input frames stored in the frame memory 11) (step S46). If it is equal to or less than “N” (step S46: YES), the process returns to step S41. If it is not less than “N” (step S46: NO), the reference frame selection process is terminated.

  The reference frame selection process is performed when the reference frame is automatically selected, but the user may manually select the reference frame. FIG. 6 is a flowchart for explaining reference frame selection processing when the user manually selects a reference frame. First, the input frame list stored in the frame memory 11 is displayed (step S40a). Then, the user is caused to select an input frame for the reference frame from the input frame list (step S41a). That is, the user is instructed to select an input frame for the reference frame from the input frame list. After issuing the instruction, it is determined whether or not the selection of the input frame with respect to the reference frame is completed (step S42a). If the selection is not completed (step S42a: NO), the process returns to step S41a and the selection is completed. (Step S42a: YES), the selected input frame is set as a reference frame (FrmA) (Step S43a). After this process, the reference frame selection process ends.

(Attention subject area setting process)
FIG. 7 is a flowchart for explaining the target subject area setting process. In the figure, first, the entire reference frame (FrmA) stored in the frame memory 11 is displayed (step S50). Next, the user is caused to set the target subject area as a reference frame (step S51). That is, the user is instructed to set the target subject area in the reference frame. In this case, on the displayed frame image, the rectangular area is selected by clicking and dragging with the mouse, or the setting window is opened, and the coordinate values of the four corners of the rectangular area are input numerically. Alternatively, the following variations can be considered by using image processing together. That is, by clicking only one point on the target subject on the displayed frame image, the vicinity of the clicked point is subjected to image processing, and the target subject region is automatically detected and set. For example, an edge is detected, a rectangle surrounding the click point is extracted, a rectangle most likely to be a license plate is selected, and a rectangular region surrounding the rectangle is set as a subject area of interest.

  Next, it is determined whether or not the setting of the target subject area for the reference frame has been completed (step S52). If the setting of the target subject area for the reference frame has not been completed (step S52: NO), the process returns to step S51 to return to the reference frame. When the setting of the subject area of interest is completed (step S52: YES), the set area is set as the subject area of interest (RoiA) (step S53). After this process, the target subject area setting process ends.

(Automatic alignment processing)
FIG. 8 is a flowchart for explaining the automatic alignment processing. In the figure, first, image data (ImgA) of the subject area of interest (RoiA) is acquired from the reference frame (FrmA) (step S60). Next, “1” is subtracted from the value of FrmA, the value is set to FrmR (step S61), and it is determined whether the value of FrmR is “1” or more (step S62). In this determination, when the value of FrmR is “1” or more (step S62: YES), the FrmR-th frame data in the frame memory 11 is acquired (ImgR) (step S63). Next, a region that matches ImgA in ImgR is found (RoiR) (step S64). When a region that matches ImgA is found in ImgR, a geometric transformation matrix (M _FrmR−FrmA ) that maps RoiR to RoiA is obtained (step S65).

Next, the RoiR and the geometric transformation matrix (M _FrmR−FrmA ) are stored in the reference point position holding memory 17A (step S66). Thereafter, “1” is subtracted from the value of FrmR, the value is set to FrmR (step S67), and the process returns to step S62.

On the other hand, if it is determined in step S62 that the value of FrmR is not “1” or more (step S62: NO), the value of FrmA is incremented by “1”, and the value is set to FrmR (step S68). It is determined whether the value is “N” or less (step S69). In this determination, when the value of FrmR is “N” or less (step S69: YES), the FrmR-th frame data in the frame memory 11 is acquired (ImgR) (step S70). Next, a region that matches ImgA in ImgR is found (RoiR) (step S71). Then, a geometric transformation matrix (M _FrmR−FrmA ) for mapping the found RoiR to RoiA is obtained (step S72). Then, RoiR and the geometric transformation matrix (M _FrmR−FrmA ) are stored in the reference point position holding memory 17A (step S73). Thereafter, “1” is added to the value of FrmR, the value is set to FrmR (step S74), and the process returns to step S69.

  If it is determined in step S69 that the value of FrmR is not equal to or less than “N” (step S69: NO), the automatic alignment process is terminated.

Here, for the matching, a known method such as template matching or feature point matching can be used. Further, the geometric transformation matrix M can be obtained by solving the following 8-ary simultaneous equations.
RoiA. C1 = M × RoiR. C1
RoiA. C2 = M × RoiR. C2
RoiA. C3 = M × RoiR. C3
RoiA. C4 = M × RoiR. C4
The geometric transformation matrix M is as shown in FIG. 9, and h1 to h8 are unknown numbers.
RoiX. Cn is a position vector ((x, y, 1) ^T , where x and y are coordinates) of the corner point n of the subject area of interest.

(Break PSF estimation process)
10 and 11 are flowcharts for explaining the blur PSF estimation processing. In this blur PSF estimation process, for each input frame, a motion vector is calculated based on the alignment result with the adjacent input frame, and the length and direction of the motion vector are set as the length and direction of subject blur in the input frame. It is.

  In these figures, first, “1” is set in FrmR (step S80). Next, RoiR is acquired from the reference point position holding memory 17A (step S81). And the center point (CtrR) of acquired RoiR is calculated | required (step S82). Next, “1” is subtracted from the value of FrmR, the value is set to FrmP, and the value of FrmR is incremented by “1”, and the value is set to FrmN (step S83). Then, it is determined whether or not the value of FrmP is “1” or more and “N” or less (step S84). If it is “1” or more and “N” or less (step S84: YES), CtrR is mapped to FrmP (CtrP). (Step S85).

The calculation of the mapping in step S85 is CtrP ← _{MFrmR → FrmP} * CtrR.
Here, M _{FrmR → FrmP} = M _{FrmA → FrmP} * M _{FrmR → FrmA} = M _{FrmP → FrmA} ⁻¹ * M _{FrmR → FrmA} =. M _{FrmR → FrmA} and M _{FrmR → FrmA} on the right side are acquired from the reference point position holding memory 17A.

  After mapping CtrR to FrmP, it is determined whether the value of FrmN is “1” or more and “N” or less (step S86). If it is “1” or more and “N” or less (step S86: YES), CtrR is set to FrmN. (CtrN) (step S87). The calculation of the mapping in step S87 is the same as that in step S85.

  Next, a value of (vector (CtrP → CtrR) + vector (CtrR → CtrN)) / 2 is set as the motion vector MvR (step S88).

  On the other hand, when it is determined in step S86 that the value of FrmN is not “1” or more and “N” or less (step S86: NO), the vector (CtrP → CtrR) is set as the motion vector MvR (step S89).

  When it is determined in step S84 that the value of FrmP is not “1” or more and “N” or less (step S84: NO), it is determined whether the value of FrmN is “1” or more and “N” or less (step S90). ), If “1” or more and “N” or less (step S90: YES), CtrR is mapped to FrmN (CtrN) (step S91). The calculation of the mapping in step S91 is the same as that in step S85.

  Next, the value of the vector (CtrR → CtrN) is set to the motion vector MvR (step S92).

  After performing any one of steps S88, S89 and S92, the angle MvR and the length of MvR made by the x-axis are set as the blur angle (BaR) and length (BlR) ( Step S93). Then, the obtained blur angle (BaR) and length (BlR) are stored in the blur PSF parameter holding memory 17B (step S94). Then, “1” is added to the value of FrmR, and the value is set to FrmR (step S95). Next, it is determined whether FrmR is “N” or less (step S96). If it is “N” or less (step S96: YES), the process returns to step S81. On the other hand, when FrmR is not equal to or less than “N” (step S96: NO), the blur PSF estimation process is terminated.

  On the other hand, when it is determined in step S90 that the value of FrmN is not “1” or more and “N” or less (step S90: NO), image data in the vicinity of RoiR is acquired from the frame memory 11 (step S104). The blur angle (BaR) and length (BlR) are estimated from the image data (step S105). Thereafter, the process proceeds to step S94.

(Parameter adjustment processing in the reference frame)
FIG. 12 is a flowchart for explaining the parameter adjustment processing in the reference frame. In the figure, first, image data of a reference frame (FrmA) is acquired from the frame memory 11 (step S110), and this is displayed (step S111). Next, the subject area (RoiA) of FrmA is acquired from the reference point position holding memory 17A (step S112), and the acquired corner points and area frame of RoiA are displayed (step S113). Further, the coordinates of the corner point of RoiA are displayed (step S114).

  Next, the blur angle (BaA) and blur length (BlA) of FrmA are acquired from the blur PSF parameter holding memory 17B (step S115), and a blur PSF image is generated from the acquired blur angle (BaA) and blur length (BlA). Are displayed (step S116). Further, numerical values of the blur angle (BaA) and the blur length (BlA) are displayed (step S117).

  Then, the user adjusts the position of RoiA being displayed and the blur PSF parameter (step S118). That is, the user is instructed to adjust the position of the RoiA being displayed and the blur PSF parameter. Accordingly, when the user adjusts the position of RoiA and the blur PSF parameter, the screen display content is updated according to the adjustment content (step S119). Then, it is determined whether or not the adjustment by the user is completed (step S120). If the adjustment is not completed (step S120: NO), the process returns to step S118, and if completed (step S120: YES), the original blurring of the reference frame is performed. A ratio (B′2B1A) between the length B1A and the adjusted blur length B1A ′ is obtained (step S121).

Next, “1” is set in FrmR (step S122). Then, the blur length (BlR) of the FrmR-th frame is obtained from the blur PSF parameter holding memory 17B (step S123), B1R * B′2B1A is calculated, and the result is set to B1R ′ (step S124). Next, B1R stored in the blur PSF parameter holding memory 17B is updated with B1R ′ (step S125). Next, the FrmR geometric transformation matrix M _{FrmR → FrmA} is acquired from the reference point position holding memory 17A (step S126), M _{FrmR → FrmA} ⁻¹ * RoiA ′ is calculated, and the result is set as RoiR ′ (step S127). ).

Here, RoiA ′: the position of RoiA when the adjustment is completed by the user. Also,
RoiR ′ ← M _{FrmR → FrmA} ⁻¹ × RoiA ′
RoiR′.C1 ← _{MFrmR → FrmA} ⁻¹ × RoiA′.C1
RoiR′.C2 ← _{MFrmR → FrmA} ⁻¹ × RoiA′.C2
RoiR′.C3 ← _{MFrmR → FrmA} ⁻¹ × RoiA′.C3
RoiR′.C4 ← _{MFrmR → FrmA} ⁻¹ × RoiA′.C4
The coordinates obtained by mapping each corner coordinate of RoiA ′ by the geometric transformation matrix M _{FrmR → FrmA} ⁻¹ are calculated. When the automatic alignment process is not performed, the calculation is performed _assuming that M _{FrmR → FrmA} ⁻¹ is a unit matrix.

  Next, the target subject area RoiR stored in the reference point position holding memory 17A is updated with RoiR '(step S128). Next, “1” is added to the value of FrmR, and the value is set to FrmR (step S129). Then, it is determined whether FrmR is “N” or less (step S130). If it is “N” or less (step S130: YES), the process returns to step S123, and if it is not “N” or less (step S130: NO), The parameter adjustment process in the reference frame is finished.

  By the processing from step S121 to step S130 (that is, a series of processing after completion of adjustment by the user), the work after the individual adjustment of the reference frame is greatly reduced.

  It should be noted that other parameters stored in the blur PSF parameter holding memory 17B (the size / shape of the out-of-focus blur, the intensity setting at the time of blur removal, etc.) may be displayed so that the setting value can be changed by the user.

  FIG. 13 is a diagram illustrating an example of a user interface screen before parameter adjustment in the reference frame. FIG. 4A is an image of a reference frame, and FIG. 4B is a blurred PSF image. In the adjustment of the reference frame, the positions of the four points (points 1 to 4) used as the reference for alignment are adjusted with the reference frame. For example, an operation of moving each point (point 1 to point 4) placed at the four corners of the target subject area in the initial state to the four corners of the license plate 100 within the target subject area is performed. In addition, before the adjustment of the blur angle and blur length, which is the blur PSF parameter, the blur PSF image is longer than the actual blur length, and if blur removal is performed as it is, overcorrection is caused and noise may be generated. It has become.

  FIG. 14 is a diagram illustrating an example of a user interface screen after parameter adjustment in the reference frame. FIG. 4A is an image of a reference frame, and FIG. 4B is a blurred PSF image. Each point (point 1 to point 4) placed at the four corners of the target subject area in the initial state is located at the four corners of the license plate 100 within the target subject area. In addition, since the blur angle and blur length, which are the blur PSF parameters, are adjusted, the blur PSF image is substantially pointed, an appropriate blur length is set, and the blur removal result is good. Yes.

(Parameter adjustment processing in the reference frame)
15 and 16 are flowcharts for explaining the parameter adjustment processing in the reference frame. In the figure, first, image data of a reference frame (FrmA) is acquired from the frame memory 11 (step S140) and displayed (step S141). Next, the subject area (RoiA) of FrmA is acquired from the reference point position holding memory 17A (step S142). Then, the corner point and the area frame of the acquired target subject area (RoiA) are displayed (step S143). Further, the coordinates of the corner point of RoiA are displayed (step S144). Further, a list of all frames stored in the frame memory 11 is displayed (step S145). Further, the frame of the subject area of interest is displayed on each frame displayed in the frame list (step S146).

  Next, one arbitrary frame is selected (step S147), and the user is caused to change the selection of the reference frame or adjust the parameters of the selected reference frame (step S148). That is, the user is instructed to change the selection of the reference frame and adjust the parameters of the reference frame being selected. Thereby, the user changes the selection of the reference frame and adjusts the parameters of the selected reference frame. Next, it is determined whether or not the reference frame selection state has been changed (step S149). If the reference frame selection state has been changed (step S149: YES), the image data of the selected reference frame (FrmR) is stored in the frame memory 11. (Step S150), and this is displayed (Step S151).

  After displaying FrmR, the target subject area (RoiR) of FrmR is acquired from the reference point position holding memory 17A (step S152). The acquired RoiR corner points and area frame are displayed (step S153), and the coordinates of the RoiR corner points are displayed (step S154). Further, the blur angle (BaR) and blur length (BlR) of FrmR are acquired from the blur PSF parameter holding memory 17B (step S155), and a blur PSF image is generated from the blur angle (BaR) and blur length (BlR) and displayed. (Step S156). Also, numerical values of the blur angle (BaR) and the blur length (BlR) are displayed (step S157). Thereafter, the process returns to step S148.

  On the other hand, if it is determined in step S149 that the selection state of the reference frame has not been changed (step S149: NO), it is determined whether or not the RoiR or blur PSF parameter has been changed (step S158). When the RoiR or blur PSF parameter is changed (step S158: YES), the screen display content is updated according to the adjustment content (step S159). Then, the set values of the reference point position holding memory 17A and the blur PSF parameter holding memory 17B for the selected reference frame (FrmR) are changed according to the adjustment contents (step S160). Thereafter, the process returns to step S148.

  If it is determined in step S158 that the RoiR or blur PSF parameter has not been changed (step S158: NO), it is determined whether or not the adjustment is completed (step S161). If the adjustment is not completed (step S161: NO), Returning to step S148, if the adjustment is completed (step S161: YES), the parameter adjustment process in the reference frame is terminated.

  FIG. 17 is a diagram illustrating an example of a user interface screen before parameter adjustment in the reference frame. FIG. 4A shows an image of the selected input frame (reference frame), and FIG. 4B shows a blurred PSF image of the selected input frame (reference frame). In the adjustment of the reference frame, the positions of four points (points 1 to 4) with respect to the reference point set in the reference frame are adjusted for all the frames before and after the reference frame. In the adjustment of the reference frame, since the reference point set in the reference frame is used, the reference frame substantially coincides with the reference frame even before the adjustment. Therefore, the positions of the four points (points 1 to 4) are substantially at the four corners of the license plate 100. To position. Thereby, the position of four points (point 1-point 4) can be located in the four corners of the license plate 100 in a short time. Of the four points before adjustment (points 1 to 4), the coordinates of point 1 are “X = 290.7500, Y = 357.2500”, and the coordinates of point 2 are “X = 313.00000, Y = 355. “5000”, the coordinates of the point 3 are “X = 291.0449, Y = 3688.3445”, and the coordinates of the point 4 are “X = 313.2500, Y = 366.7500”.

  In addition, since the blur angle and the blur length, which are blur PSF parameters, substantially coincide with the adjustment in the reference frame, the blur length is adjusted in advance to an approximately appropriate length.

  FIG. 18 is a diagram illustrating an example of a user interface screen after parameter adjustment in the reference frame. FIG. 4A shows an image of the selected input frame (reference frame), and FIG. 4B shows a blurred PSF image of the selected input frame (reference frame). Position adjustment of four points (points 1 to 4) with respect to the reference point set in the reference frame can be completed in an extremely short time. Of the four adjusted points (points 1 to 4) shown in FIG. 18, the coordinates of point 1 are “X = 290.6185, Y = 3577.3225”, and the coordinates of point 2 are “X = 313.2253, Y = 355.545 ”, the coordinates of point 3 are“ X = 291.0449, Y = 3688.3445 ”, the coordinates of point 4 are“ X = 313.6530, Y = 3666.8148 ”, and before parameter adjustment There is a slight difference from the position of each point.

  In this way, the position of the reference point for alignment in the base frame and the blur PSF parameters such as the blur length can be adjusted, and the adjustment result in the base frame can be reflected on all other reference frames. .

(Blur removal process)
FIG. 19 is a flowchart for explaining blur removal processing. In the figure, first, “1” is set to FrmR (step S170). Next, RoiR is acquired from the reference point position holding memory 17A (step S171). Next, image data (ImgR) in the vicinity of RoiR is acquired from the frame memory 11 (step S172). Further, the blur PSF parameter is acquired from the blur PSF parameter holding memory 17B (step S173).

  Then, a blurred PSF image is generated from the acquired blurred PSF parameters (step S174). Details of the generation of the blurred PSF image will be described later.

  Then, the image data (ImgR) is subjected to deconvolution with the generated blurred PSF image to obtain image data (ImgR ′) (step S175).

  A known method can be used for Deconvolution. Hereafter, a well-known example is given.

・ Inverse Filter
Using the properties of convolution and Fourier transform,
X × PSF = Y → FT (X) FT (PSF) = FT (Y) → FT (X) = Restore original image based on FT (Y) / FT (PSF).
(X: original image without blurring / blurring, PSF: point spread function of blurring / blurring, Y: observation image with blurring / blurring, *: convolution operation, FT (•): Fourier transform)

・ Wiener Filter
In the above-mentioned improved version, a random noise term is included in the model formula (X * PSF + W = Y) to suppress adverse effects (W: random noise component).

・ W. H. Richardson, "Bayesian-Based Iterative Method of Image Restoration", Journal of Optical Society of America, Vol. 62, No. 1, 1972
This is a method in which an initial solution of X is set, the next X is obtained from the values of X, PSF, and Y and the X is updated repeatedly until X converges, and does not depend on the Fourier transform.

  Then, after obtaining the image data (ImgR ′), the image data (ImgR) in the frame memory 11 is replaced with the image data (ImgR ′) (step S176). Thereafter, “1” is added to the value of FrmR, and the value is set to FrmR (step S177). Thereafter, it is determined whether or not the value of FrmR is “N” or less (step S178). If it is “N” or less (step S178: YES), the process returns to step S171, and if it is not “N” or less (step S178: NO). ), The blur removal process is terminated.

Hereinafter, an example of generating a blur PSF image from the blur PSF parameters (blur angle BaR, blur length BIR) will be described.
1. Prepare image data of a size sufficient to contain the blur locus and initialize it with zero.
2. Image data that is located at each point on a line segment (the midpoint of the line segment is equal to the origin) whose origin is the center of the prepared image, passes through the origin, has an angle with the x-axis of BaR, and has a length of BlR. Is rewritten to 1.0 (= maximum luminance level).
3. The pixel values of the entire image data are uniformly normalized so that the sum of the pixel values becomes 1.0.
Furthermore, when the blur radius BrR (a parameter used to sharpen an image that represents the blur condition of the blur locus and is out of focus) is included in the blur PSF parameter, the following step is added.
4). The perfect circle data of radius BrR is convolved with the generated image data.
True circle data: point within radius BrR from center = pixel value K, other points = pixel value 0
K: 1 / (total number of points within radius BrR from center)

(Reconfiguration process)
FIG. 20 is a flowchart for explaining the reconstruction process. In the figure, first, an initial estimated image is set to X (step S180). Note that the initial estimated image may be any enlargement method as long as the image of the target subject area of the reference frame is enlarged z times vertically and horizontally.

Next, the evaluation value S and the corrected image U are initialized (step S181), and then “1” is set to FrmR (step S182). Next, the target subject region (RoiR) and the geometric transformation matrix (M _{FrmR → FrmA} ) of the reference frame FrmR are acquired from the reference point position holding memory 17A (step S183). Further, image data in the vicinity of RoiR is acquired from the frame memory 11 (ImgR) (step S184).

Next, the image X is geometrically deformed to a position at FrmR (step S185). That is, (Mz * M _{FrmR → FrmA} ⁻¹ ) * X is calculated, and the result is set in the image Y ′ (Y ′ ← (Mz * M _{FrmR → FrmA} ⁻¹ ) * X). Here, Mz is an enlargement conversion matrix (assuming that the super-resolution enlargement magnification is z, it becomes as shown in FIG. 21).

  After geometrically deforming the image X to the position at FrmR, the resolution of the image Y ′ is degraded (Y ″) (step S186). Then, the evaluation value S and the modified image U are updated based on the difference between the image Y ″ and ImgR. (Step S187). Thereafter, “1” is added to the value of FrmR, and the value is set to FrmR (step S188). Then, it is determined whether the value of FrmR is “N” or less (step S189). If it is “N” or less (step S189: YES), the process returns to step S183, and if it is not “N” or less (step S189: NO). ), The constraint condition of the image X is evaluated, and the evaluation value S and the modified image U are updated (step S190). Then, it is determined whether or not X has sufficiently converged (step S191). If the X has sufficiently converged (step S191: YES), the image X is output (step S192), and the reconstruction process ends. On the other hand, when X is not sufficiently converged (step S191: NO), X is updated based on the corrected image U (step S193), and the process returns to step S181.

(effect)
As described above, in the present embodiment, the image processing apparatus 1 displays the reference frame image and parameters (the position of the subject region of interest and the blur PSF parameter), and displays the reference frame image based on the user's adjustment. The contents are updated and the adjusted parameters are stored as parameters of the reference frame. Then, when displaying the image of the next reference frame, the image processing apparatus 1 displays the stored reference frame parameter as the parameter of the reference frame, and displays the image of the reference frame based on the user's adjustment. The contents are updated, and the adjusted parameter is stored as the parameter of the reference frame. Thereafter, when displaying the image of the new reference frame, the image processing apparatus 1 displays the parameter of the base frame as the parameter of the new reference frame.

  As a result, the adjustment result in the base frame can be reflected in all other reference frames, so when manually aligning the images of multiple input frames, the input work is reduced and the work efficiency is improved. You can plan. As a result, the number of frames that can be used for the super-resolution processing can be increased, so that a high-resolution image can be obtained by the super-resolution processing.

(Embodiment 2)
In the second embodiment, a blur PSF estimation process different from that described in the first embodiment with reference to FIGS. 10 and 11 will be described.

(Break PSF estimation process)
22 and 23 are flowcharts for explaining the blur PSF estimation process of the image processing apparatus 1 according to the present embodiment. 22 and 23, steps common to those in FIGS. 10 and 11 are denoted by the same reference numerals as those in FIGS. 10 and 11, and description thereof is omitted.

  22 and 23, first, “1” is set to FrmR (step S80). Next, RoiR is acquired from the reference point position holding memory 17A (step S81). Then, the acquired image data in the vicinity of RoiR is acquired from the frame memory 11 (ImgR) (step S200). Next, the center point of the acquired RoiR is obtained (CtrR) (step S82). Thereafter, the value of FrmR is decremented by “1”, the value is set to FrmP, the value of FrmR is incremented by “1”, and the value is set to FrmN (step S83). Next, it is determined whether or not the value of FrmP is “1” or more and “N” or less (step S84). If it is “1” or more and “N” or less (step S84: YES), CtrR is mapped to FrmP (CtrP). (Step S85).

  After mapping CtrR to FrmP, it is determined whether the value of FrmN is “1” or more and “N” or less (step S86). If it is “1” or more and “N” or less (step S86: YES), CtrR is set to FrmN. (Step S87). Next, (vector (CtrP → CtrR) + vector (CtrR → CtrN)) / 2 is calculated, and the result is set as the motion vector MvR (step S88).

  On the other hand, when it is determined in step S86 that the value of FrmN is not “1” or more and “N” or less (step S86: NO), the vector (CtrP → CtrR) is set as the motion vector MvR (step S89).

  If the value of FrmP is not “1” or more and “N” or less in step S84 (step S84: NO), it is determined whether the value of FrmN is “1” or more and “N” or less (step S90). If it is greater than or equal to “N” (step S90: YES), CtrR is mapped to FrmN (CtrN) (step S91). Next, the value of the vector (CtrR → CtrN) is set to the motion vector MvR (step S92).

  After performing any one of Step S88, Step S89, and Step S92, the angle MvR and the length of MvR that the MvR makes with the x-axis are the blur angle (BaR) and length (MlR) ( Step S93). Next, the obtained blur angle (BaR) and length (MlR) are stored in the blur PSF parameter holding memory 17B (step S94). Next, the blur length is estimated from the image data ImgR and the blur angle BaR (BlR) (step S201). Then, the ratio between the estimated blur length BlR and the vector length MlR is calculated (B2MlR) (step S202).

  Thereafter, “1” is added to the value of FrmR, and the value is set to FrmR (step S95). Then, it is determined whether or not FrmR is “N” or less (step S96). If FrmR is “N” or less (step S96: YES), the process returns to step S81. On the other hand, if FrmR is not equal to or less than “N” (step S96: NO), the representative value of the ratio B2MlR calculated for all frames (R = 1 to N) is calculated (B2Ml) (step S203). Thereafter, “1” is set in FrmR (step S204). Next, a value obtained by multiplying the blur length MlR stored in the blur PSF parameter holding memory 17B by B2Ml is stored as the blur length BlR (step S205). Next, “1” is added to the value of FrmR, the value is set to FrmR (step S206), and it is determined whether or not FrmR is “N” or less (step S207). If FrmR is “N” or less (step S207: YES), the process returns to step S205. On the other hand, if FrmR is not equal to or less than “N” (step S207: NO), the blur PSF estimation process is terminated.

  On the other hand, if it is determined in step S90 that the value of FrmN is not “1” or more and “N” or less (step S90: NO), the angle / length of blur is estimated from the image data ImgR (BaR, BIR) ( Step S208). Thereafter, the value of BlR is set to MlR (step S209), and the process returns to step S94.

(effect)
As described above, according to the present embodiment, when the length of the blur PSF is estimated in the blur PSF estimation process, the length of the blur PSF is estimated from one piece of image data, and the length of the blur PSF thus estimated is estimated. Is calculated for each frame, the ratio is smoothed over the entire frame, and the length of each motion vector is multiplied by the smoothed ratio to calculate the length of the blur PSF of each frame. . As a result, the length of the blur PSF can be estimated in consideration of the shutter speed of the camera, so that the blur removal process can be performed using the accurate blur PSF, and a sharp image without ringing noise can be obtained. Can be obtained.

(Embodiment 3)
In the third embodiment, a case where the processing content described in the first embodiment is applied to face image detection will be described. The schematic configuration of the image processing apparatus in the present embodiment is the same as that of the image processing apparatus 1 in FIG. 1 described in the first embodiment. Further, the flow of super-resolution processing of the image processing apparatus in the present embodiment is substantially the same as that in FIGS. 2 and 3 described in the first embodiment. However, the automatic alignment process (S4) in FIG. 2 is changed to a face feature point detection process.

  The flow of the image data reading process of the image processing apparatus in the present embodiment is the same as that in FIG. 4 described in the first embodiment. Further, the flow of the reference frame selection process of the image processing apparatus in the present embodiment is the same as that in FIGS. 5 and 6 described in the first embodiment. Further, the flow of the target subject area setting process of the image processing apparatus in the present embodiment is the same as that in FIG. 7 described in the first embodiment. Further, the flow of the blur removal process of the image processing apparatus in the present embodiment is the same as that in FIG. 19 described in the first embodiment.

(Face feature point detection processing)
FIG. 24 is a flowchart for explaining the face feature point detection processing. In the figure, first, the value of FrmR is set to “1” (step S300). Next, it is determined whether the value of FrmR is “N” or less (step S301). In this determination, when the value of FrmR is “N” or less (step S301: YES), the FrmR-th frame data in the frame memory 11 is acquired (ImgR) (step S302). Next, a facial feature point is detected in ImgR (P _{FrmR, i} ) (step S303). As a face feature point detection method, a method using AAM (Active Appearance Models) is known. In this method, a face image and the positions of a plurality of feature points on the face image are set in advance, and learning data is created from a large number of sets of data (offline processing: processing performed before super-resolution processing) Next, an unknown face image is input and fitting is performed using the above learning data to estimate the position of the feature point in the input image (online processing: super-resolution processing) (Reference processing: “Active Appearance Models Revisited”, Iain Matthews and Simon Baker, International Journal of Computer Vision, Vol. 60, No. 2, pp. 135-164, November, 2004.). P _{FrmR, i} is position information of the i-th facial feature point detected in ImgR (1 ≦ i ≦ (total number of feature points in ImgR)).

Next, a region surrounding all face feature points in ImgR is calculated (RoiR) (step S304), and planar patches (see FIG. 25) formed by adjacent face feature points of ImgR are listed (S _{FrmR, *} ( S _{FrmR, k} (all of the patches in 1 ≦ k ≦ ImgR))) (step S305). In FIG. 25, a face feature point .smallcircle, each region of the S ₁₂ from S ₁ surrounded by the dotted line is a flat patch.

Then, RoiR, P _{FrmR, *} (all of P _{FrmR, i} ) and S _{FrmR, *} are stored in the reference point position holding memory 17A (step S306). Thereafter, “1” is added to the value of FrmR, the value is set to FrmR (step S307), and the process returns to step S301.

  On the other hand, if it is determined in step S301 that the value of FrmR is greater than N (step S301: NO), the value of FrmR is set to “1” (step S308), and it is determined whether the value of FrmR is “N” or less. (Step S309).

  In this determination, when the value of FrmR is equal to or smaller than “N” (step S309: YES), the patch number k is set to “1” (step S310), and it is determined whether the value of k is equal to or smaller than the total number of patches (step S310). Step S311).

In this determination, if the value of k is equal to or less than the total number of patches (step S311: YES), the position of each vertex of the patch S _{FrmR, k} of FrmR is determined as the position of each vertex of the corresponding patch S _{FrmA, k} of _FrmA. (M _{FrmR, k → FrmA, k} ) is obtained (step S312). Then, the geometric transformation matrix M _{FrmR, k → FrmA, k} is stored in the reference point position holding memory 17A (step S313). Thereafter, “1” is added to the value of k, the value is set to k (step S314), and the process returns to step S311.

  On the other hand, if it is determined in step S311 that the value of k is larger than the total number of patches (step S311: NO), the value of FrmR is incremented by “1”, and the value is set to FrmR (step S315). Return.

  If it is determined in step S309 that the value of FrmR is not equal to or less than “N” (step S309: NO), the facial feature point detection process ends.

(Breaking PSF estimation process)
26 and 27 are flowcharts for explaining the blur PSF estimation process. 26 and 27, steps common to those in FIGS. 10 and 11 are denoted by the same reference numerals as those in FIGS. 10 and 11, and description thereof is omitted.

26 and 27, first, “1” is set to FrmR (step S80). Next, P _{FrmR, *} is acquired from the reference point position holding memory 17A (step S320). Then, a representative point of the acquired P _{FrmR, *} is obtained (Ctr _FrmR ) (step S321). Thereafter, the value of FrmR is incremented by “1”, and the value is set to FrmR (step S322). Next, it is determined whether the value of FrmR is “N” or less (step S323). If it is “N” or less (step S323: YES), the process returns to step S320.

  On the other hand, if it is determined in step S323 that the value of FrmR is greater than “N” (step S323: NO), the value of FrmR is set to “1” (step S324).

  Next, “1” is subtracted from the value of FrmR, the value is set to FrmP, and the value of FrmR is incremented by “1”, and the value is set to FrmN (step S83). Then, it is determined whether the value of FrmP is “1” or more and “N” or less (step S84). If it is “1” or more and “N” or less (step S84: YES), the process proceeds to step S86.

Then, it is determined whether the value of FrmN is “1” or more and “N” or less (step S86). If it is “1” or more and “N” or less (step S86: YES), (vector (Ctr _FrmP → Ctr _FrmR )) A value of + vector (Ctr _FrmR → Ctr _FrmN )) / 2 is set as the motion vector MvR (step S325).

On the other hand, if it is determined in step S86 that the value of FrmN is not “1” or more and “N” or less (step S86: NO), the vector (Ctr _FrmP → Ctr _FrmR ) is set as the motion vector MvR (step S326). .

When it is determined in step S84 that the value of FrmP is not “1” or more and “N” or less (step S84: NO), it is determined whether the value of FrmN is “1” or more and “N” or less (step S90). ), If it is “1” or more and “N” or less (step S90: YES), the value of the vector (Ctr _FrmR → Ctr _FrmN ) is set to the motion vector MvR (step S327).

  After performing any one of steps S325, S326, and S327, the angle MvR and the length of MvR that the MvR makes with the x axis are set as the blur angle (BaR) and the length (BlR) (step S93). ). Then, the obtained blur angle (BaR) and length (BlR) are stored in the blur PSF parameter holding memory 17B (step S94). Then, “1” is added to the value of FrmR, and the value is set to FrmR (step S95). Next, it is determined whether FrmR is “N” or less (step S96). If it is “N” or less (step S96: YES), the process returns to step S83. On the other hand, when FrmR is not equal to or less than “N” (step S96: NO), the blur PSF estimation process is terminated.

  On the other hand, when it is determined in step S90 that the value of FrmN is not “1” or more and “N” or less (step S90: NO), image data in the vicinity of RoiR is acquired from the frame memory 11 (step S104). The blur angle (BaR) and length (BlR) are estimated from the image data (step S105). Thereafter, the process proceeds to step S94.

(Parameter adjustment processing in the reference frame)
FIG. 28 is a flowchart for explaining parameter adjustment processing in a reference frame. In FIG. 28, steps that are the same as in FIG. 12 are assigned the same reference numerals as in FIG.

In FIG. 28, first, image data of the reference frame (FrmA) is acquired from the frame memory 11 (step S110), and this is displayed (step S111). Next, the face feature point (P _{FrmA, *} ) is acquired from the reference point position holding memory 17A (step S330), and a mesh connecting the face feature point P _{FrmA, *} and the face feature point is displayed (step S331).

  Next, the blur angle (BaA) and blur length (BlA) of FrmA are acquired from the blur PSF parameter holding memory 17B (step S115), and a blur PSF image is generated from the acquired blur angle (BaA) and blur length (BlA). Are displayed (step S116). Further, numerical values of the blur angle (BaA) and the blur length (BlA) are displayed (step S117).

Then, the position of the facial feature point being displayed and the blur PSF parameter are adjusted by the user (step S332), and the screen display content is updated according to the adjustment content of the user (step S119). Then, it is determined whether or not the adjustment by the user is completed (step S120). If the adjustment is not completed (step S120: NO), the process returns to step S332, and if completed (step S120: YES), the original blurring of the reference frame is performed. A ratio (B′2B1A) between the length B1A and the adjusted blur length B1A ′ is obtained (step S121), and a geometric deformation matrix that maps the position of the vertex before adjustment to the position after adjustment for each patch of the reference frame is obtained (M _{FrmA, * → FrmA, * ′} ) (step S333).

Next, “1” is set in FrmR (step S122). Then, the blur length (BlR) of the FrmR-th frame is obtained from the blur PSF parameter holding memory 17B (step S123), B1R * B′2B1A is calculated, and the result is set to B1R ′ (step S124). Next, B1R stored in the blur PSF parameter holding memory 17B is updated with B1R ′ (step S125). Next, FrmR face feature points (P _{FrmR, *} ), patch information (S _{FrmR, *} ), and a geometric transformation matrix (M _{FrmR, * → FrmA, *} ) are acquired from the reference point position holding memory 17A (step S334). ), The face feature point P _{FrmR, *} is mapped (P _{FrmR, * ′} ) (see FIG. 29) (step S335).

Next, the face feature point P _{FrmR, *} stored in the reference point position holding memory 17A is updated with P _{FrmR, * ′} (step S336). Next, “1” is added to the value of FrmR, and the value is set to FrmR (step S129). Then, it is determined whether FrmR is “N” or less (step S130). If it is “N” or less (step S130: YES), the process returns to step S123, and if it is not “N” or less (step S130: NO), The parameter adjustment process in the reference frame is finished.

  By the processing from step S121 to step S130 (that is, a series of processing after completion of adjustment by the user), the work after the individual adjustment of the reference frame is greatly reduced.

  It should be noted that other parameters stored in the blur PSF parameter holding memory 17B (the size / shape of the out-of-focus blur, the intensity setting at the time of blur removal, etc.) may be displayed so that the setting value can be changed by the user.

  Next, details of the face feature point mapping processing in step S335 of FIG. 28 will be described with reference to FIG.

  In the figure, first, the patch number k is set to “1” (step S340), and it is determined whether or not the value of k is equal to or less than the total number of patches (step S341).

In this determination, if the value of k is less than or equal to the total number of patches (step S341: YES), each vertex coordinate of the patch k in the reference frame FrmR is converted by the geometric transformation matrix M _{FrmR, k → FrmA, k ′} ( P _{FrmR, * ′} ) (step S342). Thereafter, “1” is added to the value of k, the value is set to k (step S343), and the process returns to step S341.

  On the other hand, if it is determined in step S341 that the value of k is larger than the total number of patches (step S341: NO), the mapping process of the facial feature points in step S335 is terminated.

Note that since the vertex of the patch is shared with the vertex of the adjacent patch, in the flow of FIG. 29, the same vertex (that is, the facial feature point) is used in the coordinate conversion processing of one patch and the coordinate conversion processing of another patch. There may be a case where a plurality of converted coordinates for is calculated. For this reason, in the present embodiment, one post-conversion coordinate is determined from a plurality of post-conversion coordinates calculated by a predetermined method, and the determined coordinate is defined as P _{FrmR, i ′} for the face feature point i. Examples of the determination method include obtaining the center of gravity, adopting the first appearing value, or employing the last value.

(Parameter adjustment processing in the reference frame)
30 and 31 are flowcharts for explaining the parameter adjustment processing in the reference frame. 30 and 31, steps common to those in FIGS. 15 and 16 are denoted by the same reference numerals as those in FIGS. 15 and 16, and description thereof is omitted.

30 and 31, first, image data of a reference frame (FrmA) is acquired from the frame memory 11 (step S140), and this is displayed (step S141). Next, FrmA face feature points (P _{FrmA, *} ) are acquired from the reference point position holding memory 17A (step S350), and a mesh connecting the face feature points P _{FrmA, *} and the face feature points is displayed (step S351). . Further, a list of all frames stored in the frame memory 11 is displayed (step S145). Further, the frame of the subject area of interest is displayed on each frame displayed in the frame list (step S146).

  Next, one arbitrary frame is selected (step S147), and the user is caused to change the selection of the reference frame or adjust the parameters of the selected reference frame (step S148). Next, it is determined whether or not the reference frame selection state has been changed (step S149). If the reference frame selection state has been changed (step S149: YES), the image data of the selected reference frame (FrmR) is stored in the frame memory 11. (Step S150), and this is displayed (Step S151).

After displaying FrmR, the face feature point (P _{FrmR, *} ) of _FrmR is acquired from the reference point position holding memory 17A (step S352), and a mesh connecting the face feature point P _{FrmR, *} and the face feature point is displayed. (Step S353).

  Further, the blur angle (BaR) and blur length (BlR) of FrmR are acquired from the blur PSF parameter holding memory 17B (step S155), and a blur PSF image is generated from the blur angle (BaR) and blur length (BlR) and displayed. (Step S156). Also, numerical values of the blur angle (BaR) and the blur length (BlR) are displayed (step S157). Thereafter, the process returns to step S148.

  On the other hand, if it is determined in step S149 that the selection state of the reference frame has not been changed (step S149: NO), it is determined whether or not the RoiR or blur PSF parameter has been changed (step S158). When the RoiR or blur PSF parameter is changed (step S158: YES), the screen display content is updated according to the adjustment content (step S159). Then, the set values of the reference point position holding memory 17A and the blur PSF parameter holding memory 17B for the selected reference frame (FrmR) are changed according to the adjustment contents (step S160). Thereafter, the process returns to step S148.

  If it is determined in step S158 that the RoiR or blur PSF parameter has not been changed (step S158: NO), it is determined whether or not the adjustment is completed (step S161). If the adjustment is not completed (step S161: NO), Returning to step S148, if the adjustment is completed (step S161: YES), the parameter adjustment process in the reference frame is terminated.

(Reconfiguration process)
FIG. 32 is a flowchart for explaining the reconstruction process. In FIG. 32, the steps common to those in FIG. 20 are denoted by the same reference numerals as those in FIG.

  In FIG. 32, first, an initial estimated image is set to X (step S180). Next, the evaluation value S and the corrected image U are initialized (step S181), and then “1” is set to FrmR (step S182).

Next, the patch information (S _{FrmR, *} ) and the geometric transformation matrix (M _{FrmR, * → FrmA, *} ) of the reference frame FrmR are acquired from the reference point position holding memory 17A (step S360). Further, image data in the vicinity of each patch is acquired from the frame memory 11 (ImgR _{S *} ) (step S361).

Next, the image X is geometrically deformed to the position of FrmR for each patch (step S362). That is, for each k, (Mz * M _{FrmR, k → FrmA, k} ⁻¹ ) * X is calculated, and the result is set in the image Y ′ (Y ′ ← (Mz * M _{FrmR, k → FrmA, k} ⁻¹ ) * X). Here, Mz is an enlargement conversion matrix (assuming that the super-resolution enlargement magnification is z, it becomes as shown in FIG. 21).

  After geometrically deforming the image X to the position at FrmR, the resolution of the image Y ′ is degraded (Y ″) (step S186). Then, the evaluation value S and the modified image U are updated based on the difference between the image Y ″ and ImgR. (Step S187). Thereafter, “1” is added to the value of FrmR, and the value is set to FrmR (step S188). Then, it is determined whether the value of FrmR is “N” or less (step S189). If it is “N” or less (step S189: YES), the process returns to step S360, and if it is not “N” or less (step S189: NO). ), The constraint condition of the image X is evaluated, and the evaluation value S and the modified image U are updated (step S190). Then, it is determined whether or not X has sufficiently converged (step S191). If the X has sufficiently converged (step S191: YES), the image X is output (step S192), and the reconstruction process ends. On the other hand, when X is not sufficiently converged (step S191: NO), X is updated based on the corrected image U (step S193), and the process returns to step S181.

(effect)
As described above, in the present embodiment, the image processing apparatus 1 displays the image of the reference frame and the parameters (the position of the face feature points, the mesh connecting the face feature points and the blur PSF parameters) for user adjustment. Based on this, the display contents of the image of the reference frame are updated, and the adjusted parameters are stored as parameters of the reference frame. Then, when displaying the image of the next reference frame, the image processing apparatus 1 displays the stored reference frame parameter as the parameter of the reference frame, and displays the image of the reference frame based on the user's adjustment. The contents are updated, and the adjusted parameter is stored as the parameter of the reference frame. Thereafter, when displaying the image of the new reference frame, the image processing apparatus 1 displays the parameter of the base frame as the parameter of the new reference frame.

  As a result, the adjustment result in the base frame can be reflected in all other reference frames, so when manually aligning the images of multiple input frames, the input work is reduced and the work efficiency is improved. Can be planned. As a result, the number of frames that can be used for the super-resolution processing can be increased, so that a high-resolution image can be obtained by the super-resolution processing.

(Embodiment 4)
In the fourth embodiment, a blur PSF estimation process different from that described in the third embodiment with reference to FIGS. 26 and 27 will be described.

(Break PSF estimation process)
33, 34 and 35 are flowcharts for explaining the blur PSF estimation process of the image processing apparatus 1 according to the present embodiment. 33, 34, and 35, steps common to those in FIGS. 26 and 27 are denoted by the same reference numerals as those in FIGS. 26 and 27, and description thereof is omitted.

In FIG. 33, FIG. 34 and FIG. 35, first, “1” is set to FrmR (step S80). Next, P _{FrmR, *} is acquired from the reference point position holding memory 17A (step S400). Then, the acquired image data in the vicinity of RoiR is acquired from the frame memory 11 (ImgR) (step S200). Next, a representative point of the acquired P _{FrmR, *} is obtained (Ctr _FrmR ) (step S401). Thereafter, the value of FrmR is incremented by “1”, and the value is set to FrmR (step S402). Next, it is determined whether the value of FrmR is “N” or less (step S403). If it is “N” or less (step S403: YES), the process returns to step S400.

  On the other hand, when it is determined in step S403 that the value of FrmR is larger than “N” (step S403: NO), the value of FrmR is set to “1” (step S404).

  Next, “1” is subtracted from the value of FrmR, the value is set to FrmP, and the value of FrmR is incremented by “1”, and the value is set to FrmN (step S83 which is not less than or equal to step S83). Then, it is determined whether the value of FrmP is “1” or more and “N” or less (step S84). If it is “1” or more and “N” or less (step S84: YES), the process proceeds to step S86.

Then, it is determined whether the value of FrmN is “1” or more and “N” or less (step S86). If it is “1” or more and “N” or less (step S86: YES), (vector (Ctr _FrmP → Ctr _FrmR )) A value of + vector (Ctr _FrmR → Ctr _FrmN )) / 2 is set as the motion vector MvR (step S405).

On the other hand, when it is determined in step S86 that the value of FrmN is not “1” or more and “N” or less (step S86: NO), the vector (Ctr _FrmP → Ctr _FrmR ) is set as the motion vector MvR (step S406). .

When it is determined in step S84 that the value of FrmP is not “1” or more and “N” or less (step S84: NO), it is determined whether the value of FrmN is “1” or more and “N” or less (step S90). ), If it is “1” or more and “N” or less (step S90: YES), the value of the vector (Ctr _FrmR → Ctr _FrmN ) is set to the motion vector MvR (step S407).

  After performing any one of the processes in steps S405, S406, and S407, the angle MvR and the length of MvR that the MvR makes with the x-axis are set as the blur angle (BaR) and the length (MlR) (step S93). ). Then, the obtained blur angle (BaR) and length (MlR) are stored in the blur PSF parameter holding memory 17B (step S94). Next, the blur length is estimated from the image data ImgR and the blur angle BaR (BlR) (step S201). Then, the ratio between the estimated blur length BlR and the vector length MlR is calculated (B2MlR) (step S202).

  Thereafter, “1” is added to the value of FrmR, and the value is set to FrmR (step S95). Then, it is determined whether FrmR is “N” or less (step S96). If FrmR is “N” or less (step S96: YES), the process returns to step S83. On the other hand, if FrmR is not equal to or less than “N” (step S96: NO), the representative value of the ratio B2MlR calculated for all frames (R = 1 to N) is calculated (B2Ml) (step S203). Thereafter, “1” is set in FrmR (step S204). Next, a value obtained by multiplying the blur length MlR stored in the blur PSF parameter holding memory 17B by B2Ml is stored as the blur length BlR (step S205). Next, “1” is added to the value of FrmR, the value is set to FrmR (step S206), and it is determined whether or not FrmR is “N” or less (step S207). If FrmR is “N” or less (step S207: YES), the process returns to step S205. On the other hand, if FrmR is not equal to or less than “N” (step S207: NO), the blur PSF estimation process is terminated.

  On the other hand, if it is determined in step S90 that the value of FrmN is not “1” or more and “N” or less (step S90: NO), the angle / length of blur is estimated from the image data ImgR (BaR, BIR) ( Step S208). Thereafter, the value of BlR is set to MlR (step S209), and the process returns to step S94.

(effect)
As described above, according to the present embodiment, when the length of the blur PSF is estimated in the blur PSF estimation process, the length of the blur PSF is estimated from one piece of image data, and the length of the blur PSF thus estimated is estimated. Is calculated for each frame, the ratio is smoothed over the entire frame, and the length of each motion vector is multiplied by the smoothed ratio to calculate the length of the blur PSF of each frame. . As a result, the length of the blur PSF can be estimated in consideration of the shutter speed of the camera, so that the blur removal process can be performed using the accurate blur PSF, and a sharp image without ringing noise can be obtained. Can be obtained.

  The present invention is suitable for use in an image processing apparatus that processes an image taken by a surveillance camera.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Image data input part 11 Frame memory 12 Graphical user interface part 13 Reference | standard frame selection part 14 Attention object area | region selection part 15 Matching calculation part 16 Blur PSF estimation part 17 Memory part 17A Reference point position holding memory 17B Blur PSF parameter Holding memory 18 Reference frame adjustment unit 19 Reference frame adjustment unit 20 Blur removal unit 21 Reconstruction processing unit 22 Super-resolution image output unit 23 Image generation unit 100 Number plate

Claims (5)

An image processing device that generates a high-resolution image by aligning low-resolution images of a plurality of input frames each having a common partial area,
Displaying an image of one reference frame selected from the plurality of input frames and a parameter used when generating the high-resolution image, and displaying the image of the reference frame based on adjustment of the displayed parameter A reference frame adjustment unit that updates contents and stores the adjusted parameter in a memory as a parameter of the reference frame;
An image of a reference frame that is one of the input frames other than the reference frame, and a parameter of the reference frame are displayed as parameters of the reference frame, and an image of the reference frame is displayed based on the adjustment of the displayed parameter. A reference frame adjustment unit that updates display content and stores the adjusted parameter in a memory as a parameter of the reference frame;
A blur removal unit that performs a process of removing blur included in the input frame using the corresponding parameter for each input frame;
An image generation unit configured to generate a high-resolution image by aligning a low-resolution image of the input frame from which the blur is removed;
An image processing apparatus comprising: The parameter is the length of blur PSF (Point Spread Function),
For each of the input frames, a motion vector is calculated between a plurality of input frames adjacent to the input frame in the time direction, and the length and angle of the blur PSF of the input frame are estimated from the length and direction of the motion vector. It further has a blur PSF estimation unit,
The reference frame adjustment unit obtains a ratio between the length of the blur PSF of the reference frame estimated by the blur PSF estimation unit and the length of the blur PSF after adjustment, and calculates the ratio of the input frame other than the reference frame. Multiply the length of blur PSF by the above ratio,
The image processing apparatus according to claim 1. The blur PSF estimation unit estimates the length of the blur PSF from the image data of each of the input frames when estimating the length of the blur PSF, and determines the length of the blur blur PSF and the length of the motion vector. Is calculated for each frame, the ratio is smoothed over all frames, and the length of each motion vector is multiplied by the smoothed ratio to calculate the length of the blur PSF of each frame.
The image processing apparatus according to claim 2. The parameter is a plurality of reference point coordinates indicating the position of the partial area included in each input frame,
The reference frame adjustment unit displays the plurality of reference point coordinates of the reference frame, updates the plurality of reference point coordinates of the image of the reference frame based on the adjustment of the displayed plurality of reference point coordinates, Storing the adjusted plurality of reference point coordinates in a memory;
The image processing apparatus according to claim 1. The partial area is a face image;
The parameter is a position of a plurality of face feature points of the face image included in each input frame,
The reference frame adjustment unit displays a mesh connecting the position of the face feature point of the reference frame and the face feature point, and based on the adjustment of the position of the displayed face feature point, the image of the reference frame Updating the position of the face feature point and storing the adjusted face feature point in a memory;
The image processing apparatus according to claim 1.

Images