JP6771148B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing device, and more particularly to an image processing device that aligns low-resolution images of a plurality of input frames, each having a common partial region, to generate a high-resolution image.

If each image has a low resolution and there is a misalignment with respect to the images of multiple input frames of the same subject, it is possible to generate a high resolution image by aligning the positions of those images and superimposing them. it can. The process of generating a high-resolution image from the plurality of low-resolution images is called a super-resolution process.

Here, the shooting in which the image is misaligned is a case of shooting while moving the camera with respect to a stationary subject, or a case of shooting with a fixed camera of a moving subject. If both the subject and the camera are completely stationary, the resolution will not increase no matter how many images are superimposed.

Further, for image alignment (matching), it is common to obtain alignment parameters (geometric transformation matrix such as translation / rotation) by using template matching or feature point matching. In addition, a reconstruction method is generally used for superimposing images. The reconstruction method generates an estimated image of a high-resolution image, repeats updating and evaluating the estimated image so that the plausibility of the estimated image is maximized, and outputs a convergent solution of the estimated image as the final output. is there. The plausibility of the estimated image is as follows: [1] How close the low resolution image obtained by shifting the position of the estimated image and degrading the resolution is to the original input image, and [2] The connection of pixel values in the estimated image. Whether it is natural or not.

By the way, in the automatic matching process that automatically aligns images, in the alignment of images of a plurality of input frames, a process of finding and aligning a reference point is performed, but a mismatch due to erroneous detection of the reference point is unavoidable. In particular, when a moving subject is photographed at a slow shutter speed, subject blurring occurs, which increases the possibility of mismatch.

In order to reduce the possibility of mismatch due to subject blur, the subject blur of the image may be removed before the super-resolution processing is performed. In order to remove subject blur, deconvolution from the blur image and blur PSF is based on the model that the result of convolving the blur PSF (Point Spread Function) into the original image without blur is the blur image (reverse folding from the blur image and blur PSF ( A method of performing Deconvolution) is adopted (see Patent Document 1). By using this method, it is possible to obtain a blur removal image close to the original image.

Japanese Unexamined Patent Publication No. 2011-259119

However, even if the subject blur is removed before the super-resolution processing is performed, it is difficult to completely avoid the problem of mismatch due to the influence of noise and the like. In the automatic matching process, if the matching is not successful, the image of the frame that should be usable is excluded. As a result, the number of frames that can be used decreases, and a high-resolution image cannot be obtained even if super-resolution processing is performed.

This problem can be avoided by manually aligning each image without going through the automatic matching process, but it is necessary to display the images of multiple input frames one by one and shift them by hand. The work of aligning with is very complicated and unproductive.

The present invention has been made in view of such a point, and an object of the present invention is to provide an image processing apparatus capable of improving work efficiency when manually aligning images of a plurality of input frames. To do.

The image processing device of the present invention is an image processing device that generates a high-resolution image by aligning low-resolution images of a plurality of input frames, each having a common partial area, and is selected from the plurality of input frames. The image of one reference frame and the parameters used when generating the high-resolution image are displayed, and the display contents of the image of the reference frame are updated based on the adjustment of the displayed parameters, and the adjustment is performed. The reference frame adjustment unit that stores the parameters as the parameters of the reference frame in the memory, the image of the reference frame that is one of the input frames other than the reference frame, and the parameters of the reference frame as the parameters of the reference frame. A reference frame adjustment unit that displays, updates the display content of the image of the reference frame based on the adjustment of the displayed parameter, and stores the adjusted parameter in the memory as the parameter of the reference frame, and each of the inputs. A high-resolution image by aligning a blur removing unit that performs a process of removing blur included in the input frame with respect to the frame using the corresponding parameter and a low-resolution image of the input frame from which the blur has been removed. It is provided with an image generation unit for generating the above.

According to the present invention, the adjustment result in the reference frame can be reflected to all other reference frames. Therefore, when the image alignment of a plurality of input frames is manually performed, the input work is reduced and the work efficiency is reduced. Can be improved. As a result, the number of frames that can be used for the super-resolution processing can be increased, so that a high-resolution image can be obtained by the super-resolution processing.

The block diagram which shows the schematic structure of the image processing apparatus which concerns on Embodiment 1 of this invention. A flowchart for explaining super-resolution processing in which the image processing apparatus according to the first embodiment of the present invention is automatically aligned. A flowchart for explaining super-resolution processing in which the image processing apparatus according to the first embodiment of the present invention is manually aligned. A flowchart for explaining an image data reading process in the super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. A flowchart for explaining a process of automatically selecting a reference frame in the super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. A flowchart for explaining a process of manually selecting a reference frame in the super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. A flowchart for explaining the setting process of the subject area of interest in the super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. Flow chart for explaining automatic alignment processing in super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. The figure which shows an example of the geometric transformation matrix used for the automatic alignment processing in super-resolution processing of this invention. A flowchart for explaining a blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. A flowchart for explaining a blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. A flowchart for explaining a parameter adjustment process in a reference frame in the super-resolution process of the image processing apparatus according to the first embodiment of the present invention. The figure which shows an example of the user interface screen before parameter adjustment in a reference frame. The figure which shows an example of the user interface screen after parameter adjustment in a reference frame. A flowchart for explaining a parameter adjustment process in a reference frame in the super-resolution process of the image processing apparatus according to the first embodiment of the present invention. A flowchart for explaining a parameter adjustment process in a reference frame in the super-resolution process of the image processing apparatus according to the first embodiment of the present invention. The figure which shows an example of the user interface screen before parameter adjustment in a reference frame. The figure which shows an example of the user interface screen after parameter adjustment in a reference frame. A flowchart for explaining a blur removal process in the super-resolution process of the image processing apparatus according to the first embodiment of the present invention. A flowchart for explaining the reconstruction process in the super-resolution processing of the image processing apparatus according to the first embodiment of the present invention. The figure which shows an example of the expansion transformation matrix Mz used for the reconstruction process. A flowchart for explaining a blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the second embodiment of the present invention. A flowchart for explaining a blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the second embodiment of the present invention. A flowchart for explaining the face feature point detection process in the super-resolution process of the image processing apparatus according to the third embodiment of the present invention. The figure which shows an example of the face feature point and the plane patch displayed in the super-resolution processing of the image processing apparatus which concerns on Embodiment 3 of this invention. A flowchart for explaining the blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. A flowchart for explaining the blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. A flowchart for explaining a parameter adjustment process in a reference frame in the super-resolution process of the image processing apparatus according to the third embodiment of the present invention. A flowchart for explaining the details of the mapping process of facial feature points in the parameter adjustment process in the reference frame of the third embodiment of the present invention. A flowchart for explaining a parameter adjustment process in a reference frame in the super-resolution process of the image processing apparatus according to the third embodiment of the present invention. A flowchart for explaining a parameter adjustment process in a reference frame in the super-resolution process of the image processing apparatus according to the third embodiment of the present invention. A flowchart for explaining the reconstruction process in the super-resolution processing of the image processing apparatus according to the third embodiment of the present invention. A flowchart for explaining a blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the fourth embodiment of the present invention. A flowchart for explaining a blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the fourth embodiment of the present invention. A flowchart for explaining a blur PSF estimation process in the super-resolution processing of the image processing apparatus according to the fourth embodiment of the present invention.

Hereinafter, the present embodiment will be described in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a schematic configuration of an image processing device 1 according to a first embodiment of the present invention. In the figure, the image processing device 1 according to the present embodiment is a device capable of performing super-resolution processing for generating a high-resolution image from a plurality of low-resolution images, and is an image data input unit 10 and a frame memory. 11. Graphical user interface unit 12, reference frame selection unit 13, attention subject area selection unit 14, matching calculation unit 15, blur PSF estimation unit 16, memory unit 17, reference frame adjustment unit 18, reference frame adjustment unit 19, blur removal. A unit 20, a reconstruction processing unit 21, and a super-resolution image output unit 22 are provided. The memory unit 17 is composed of a reference point position holding memory 17A and a blur PSF parameter holding memory 17B. The reconstruction processing unit 21 and the super-resolution image output unit 22 constitute an image generation unit 23.

The image data input unit 10 inputs image data sequentially output from a camera (not shown) in frame units. The frame-based image data input by the image data input unit 10 is also referred to as an input frame. The frame memory 11 temporarily stores the input frame. The graphical user interface unit 12 is a user interface that provides intuitive operation by using computer graphics and a pointing device such as a mouse.

The reference frame selection unit 13 selects one of the plurality of input frames stored in the frame memory 11 and uses it as the reference frame. To select an input frame to be a reference frame, the image quality of each of the plurality of input frames stored in the frame memory 11 is evaluated, and the most evaluated input frame is selected, or the multiple input frames stored in the frame memory 11 are selected. Display a list of and let the user select one of them.

The attention subject area selection unit 14 sets the attention subject area (partial area) in the reference frame selected by the reference frame selection unit 13. The user sets the target area of interest with respect to the reference frame. That is, the user sets the subject area of interest using a pointing device such as a mouse.

The matching calculation unit 15 finds an area that matches the attention subject area of the reference frame for each of the plurality of input frames other than the reference frame stored in the frame memory 11.

The blur PSF estimation unit 16 obtains the center point of a region that matches the image data of the subject area of interest of the reference frame in each of the plurality of input frames other than the reference frame, and further, the blur angle and length are obtained from the motion vector of the center point. These are saved as blur PSF parameters.

The reference frame adjustment unit 18 displays the position and blur PSF parameter of the partial area used when generating the image of one reference frame selected from the plurality of input frames and the high resolution image, and the displayed partial area. The display content of the image of the reference frame is updated based on the adjustment of the position and blur PSF parameter of, and the position of the adjusted partial area and the blur PSF parameter are held as the position of the partial region of the reference frame and the reference point position as the blur PSF parameter. In addition to storing in the memory 17A and the blur PSF parameter holding memory 17B, respectively, the position of a partial area of a plurality of input frames other than the reference frame and the blur PSF parameter are updated in the reference point position holding memory 17A and the blur PSF parameter holding memory 17B, respectively. Remember more.

The reference frame adjustment unit 19 displays the image of the reference frame which is one of the plurality of input frames other than the reference frame, the position of the partial area of the reference frame, and the blur PSF parameter, and displays the position of the displayed partial area and the blur PSF parameter. The display content of the image of the reference frame is updated based on the adjustment of, and the adjusted partial area position and blur PSF parameter are used as the reference frame partial area position and blur PSF parameter. Reference point position holding memory 17A and blur PSF parameter Each is stored in the holding memory 17B.

The blur removing unit 20 performs a process of removing blur included in the input frame for each of the plurality of input frames stored in the frame memory 11 by using the corresponding blur PSF parameter. The reconstruction processing unit 21 aligns the low-resolution image of the frame from which the blur has been removed to generate a high-resolution image. The super-resolution image output unit 22 outputs a high-resolution image generated by the reconstruction processing unit 21.

Next, the operation of the image processing device 1 according to the present embodiment will be described.

(Super-resolution processing)
FIG. 2 is a flowchart showing the super-resolution processing. In the figure, the super-resolution processing includes reading image data (step S1), selecting a reference frame (step S2), setting a target area of interest (step S3), automatic alignment (step S4), and estimating blur PSF. (Step S5), parameter adjustment in the reference frame (step S6), parameter adjustment in the reference frame (step S7), blur removal (step S8), reconstruction process (step S9), and super-resolution image output (step S9). It consists of 10 steps of step S10). The super-resolution processing shown in FIG. 2 is a case of automatic alignment, but there is also a case of manual alignment. FIG. 3 is a flowchart showing a super-resolution process in which the positions are manually aligned. In the case of manual alignment, the two steps of automatic alignment in step S4 and estimation of blur PSF in step S5 become unnecessary. Hereinafter, each step of the super-resolution processing when automatic alignment is used will be described in detail.

(Image data reading process)
FIG. 4 is a flowchart for explaining the image data reading process. In the figure, PosC, PosP, and PosN are registers, respectively. First, the image file is opened (step S20), and seek (move) to the designated frame (step S21). Then, the frame number of the designated current input frame is set in PosC (step S22). Next, the frame data of the frame number of the current input frame set in PosC is read into the frame memory 11 (step S23). Next, the set value of PosC is subtracted by "1", and the value is set to PosP (step S24). That is, the frame number of the input frame immediately before the current input frame is set to PosP. Next, it is determined whether or not the frame data of the frame number set in PosP exists (step S25). In this determination, if the frame data of the frame number set in PosP exists (step S25: YES), it is determined whether or not the value obtained by subtracting the frame number set in PosP from the frame number set in PosC is "R" or less. (Step S26). Here, "R" is the maximum number of frames that can be referred to in the front and the rear with the current frame as the base point, and is a value set in advance by the user.

When the value obtained by subtracting the frame number set in PosP from the frame number set in PosC is "R" or less (step S26: YES), the frame data of the frame number set in PosP is read into the frame memory 11 (step S27). ). Then, the frame number set in PosP is subtracted by "1" (step S28), and the process returns to the determination in step S25.

On the other hand, in step S25, when the frame data of the frame number set in PosP does not exist (step S25: NO), or in step S26, the value obtained by subtracting the frame number set in PosP from the frame number set in PosC is If it is larger than "R" (step S26: NO), "1" is added to the value of PosC, and the value is set to PosN (step S29). That is, the frame number of the input frame immediately behind the current input frame is set to PosN. Then, it is determined whether or not the frame data of the frame number set in PosN exists (step S30). In this determination, if the frame data of the frame number set in PosN exists (step S30: YES), it is determined whether or not the value obtained by subtracting the frame number set in PosC from the frame number set in PosN is R or less (step). S31).

When the value obtained by subtracting the frame number set in PosC from the frame number set in PosN is "R" or less (step S31: YES), the frame data of the frame number set in PosN is read into the frame memory 11 (step S32). ). Next, "1" is added to the frame number set in PosN (step S33), and the process returns to the determination in step S30.

On the other hand, in step S30, when the frame data of the frame number set in PosN does not exist (step S30: NO), or in step S31, the value obtained by subtracting the frame number set in PosC from the frame number set in PosN is If it is not less than or equal to "R" (step S31: NO), the reading of the image data is finished.

(Reference frame selection process)
FIG. 5 is a flowchart for explaining the reference frame selection process. In the figure, FrmR is a register. First, "1" is set in FrmR (step S40). Next, the data of the FrmRth input frame in the frame memory 11 is acquired (step S41). Then, the image quality of the acquired frame data is evaluated, and an evaluation score is obtained (step S42). Next, it is determined whether or not the evaluation score obtained this time is the maximum among the evaluated input frames (step S43), and if the evaluation score obtained this time is not the maximum (step S43: NO), the value of FrmR is set to ". 1 ”is added and the value is set to FrmR (step S44).

On the other hand, when the evaluation score obtained this time is the maximum (step S43: YES), the FrmRth input frame is set as the reference frame (FrmA) (step S45). Here, it is preferable to use an input frame having high image quality (not strongly compressed by the codec and a good high frequency component remains) as a reference frame. For example, it is effective to convert each frame data into a frequency domain, evaluate the image quality with a large amount or a small amount of high frequency components, and use an input frame having the most high frequency components as a reference frame.

After the processing of step S45, "1" is added to the value of FrmR, and the value is set to FrmR (step S44). Next, it is determined whether or not the value of FrmR is "N" (total number of input frames stored in the frame memory 11) or less (step S46), and if it is "N" or less (step S46: YES), the process returns to step S41. If it is not "N" or less (step S46: NO), the reference frame selection process ends.

The reference frame selection process is a case where the reference frame is automatically selected, but the user may manually select the reference frame. FIG. 6 is a flowchart for explaining a reference frame selection process when a user manually selects a reference frame. First, a list of input frames stored in the frame memory 11 is displayed (step S40a). Then, the user is made to select an input frame for the reference frame from the input frame list (step S41a). That is, an instruction is given to the user to select an input frame for the reference frame from the input frame list. After issuing the instruction, it is determined whether or not the selection of the input frame for the reference frame is completed (step S42a), if the selection is not completed (step S42a: NO), the process returns to step S41a, and the selection is completed. (Step S42a: YES), the selected input frame is set as the reference frame (FrmA) (step S43a). After this process, the reference frame selection process ends.

(Featured subject area setting process)
FIG. 7 is a flowchart for explaining the attention subject area setting process. In the figure, first, the entire reference frame (FrmA) stored in the frame memory 11 is displayed (step S50). Next, the user is made to set the subject area of interest as the reference frame (step S51). That is, the user is instructed to set the subject area of interest in the reference frame. In this case, the rectangular area is selected by clicking and dragging with the mouse on the displayed frame image, or the setting window is opened and the coordinate values of the four corners of the rectangular area are input numerically. Alternatively, the following variations can be considered by using image processing together. That is, by clicking only one point on the attention subject on the displayed frame image, the vicinity of the clicked point is image-processed, and the attention subject area is automatically detected and set. For example, edge detection is performed, a quadrangle surrounding the click point is extracted, the quadrangle most likely to be a license plate is selected, and the rectangular area surrounding the quadrangle is set as the subject area of interest.

Next, it is determined whether or not the setting of the attention subject area for the reference frame is completed (step S52), and if the setting of the attention subject area for the reference frame is not completed (step S52: NO), the process returns to step S51 and the reference frame is set. When the setting of the attention subject area for the above is completed (step S52: YES), the set area is set as the attention subject area (RoiA) (step S53). After this process, the attention subject area setting process ends.

(Automatic alignment processing)
FIG. 8 is a flowchart for explaining the automatic alignment process. In the figure, first, the image data (ImgA) of the subject region of interest (RoiA) is acquired from the reference frame (FrmA) (step S60). Next, the value of FrmA is subtracted by "1", the value is set to FrmR (step S61), and it is determined whether or not the value of FrmR is "1" or more (step S62). In this determination, when the value of FrmR is "1" or more (step S62: YES), the FrmRth frame data in the frame memory 11 is acquired (ImgR) (step S63). Then, a region in ImgurR that matches ImgurA is found (RoiR) (step S64). When a region matching ImgA is found in ImgurR, a geometric transformation matrix (M _FrmR-FrmA ) that maps the RoiR to RoiA is obtained (step S65).

Next, the RoiR and the geometric transformation matrix (M _FrmR-FrmA ) are stored in the reference point position holding memory 17A (step S66). After that, the value of FrmR is subtracted by "1", the value is set to FrmR (step S67), and the process returns to step S62.

On the other hand, when it is determined in step S62 that the value of FrmR is not "1" or more (step S62: NO), the value of FrmA is added by "1", the value is set to FrmR (step S68), and the value of FrmR is set. It is determined whether the value is "N" or less (step S69). In this determination, when the value of FrmR is "N" or less (step S69: YES), the FrmRth frame data in the frame memory 11 is acquired (ImgR) (step S70). Then, a region in ImgurR that matches ImgurA is found (RoiR) (step S71). Then, a geometric transformation matrix (M _FrmR-FrmA ) that maps the found RoiR to RoiA is obtained (step S72). Then, the RoiR and the geometric transformation matrix (M _FrmR-FrmA ) are stored in the reference point position holding memory 17A (step S73). After that, "1" is added to the value of FrmR, the value is set to FrmR (step S74), and the process returns to step S69.

If it is determined in step S69 that the value of FrmR is not "N" or less (step S69: NO), the automatic alignment process ends.

Here, for matching, a known method such as template matching or feature point matching can be used. The geometric transformation matrix M can be obtained by solving the following eight-element simultaneous equations.
Roi A. C1 = M × RoiR. C1
Roi A. C2 = M × RoiR. C2
Roi A. C3 = M × RoiR. C3
Roi A. C4 = M × RoiR. C4
The geometric transformation matrix M is as shown in FIG. 9, and h1 to h8 are unknown numbers.
RoiX. Cn is a position vector of a corner point n in the subject area of interest ((x, y, 1) ^T , where x, y are the coordinates).

(Blur PSF estimation process)
10 and 11 are flowcharts for explaining the blur PSF estimation process. This blur PSF estimation process calculates a motion vector for each input frame based on the alignment result with an adjacent input frame, and sets the length and direction of the motion vector as the length and direction of subject blur in the input frame. Is.

In these figures, First, "1" is set for FrmR (step S80). Next, RoiR is acquired from the reference point position holding memory 17A (step S81). Then, the center point (CtrR) of the acquired RoiR is obtained (step S82). Next, the value of FrmR is subtracted by "1", the value is set to FrmP, the value of FrmR is added by "1", and the value is set to FrmN (step S83). Then, it is determined whether or not the value of FrmP is "1" or more and "N" or less (step S84), and when it is "1" or more and "N" or less (step S84: YES), CtrR is mapped to FrmP (CtrP). (Step S85).

The calculation of the map in step S85 is CtrP ← M _{FrmR → FrmP} * CtrR.
Here, M _{FrmR → FrmP} = M _{FrmA → FrmP} * M _{FrmR → FrmA} = M _{FrmP → FrmA} ^-1 * M _{FrmR → FrmA} =. M _{FrmR → FrmA} and M _{FrmR → FrmA} on the right side are acquired from the reference point position holding memory 17A.

After mapping CtrR to FrmP, it is determined whether the value of FrmN is "1" or more and "N" or less (step S86), and if it is "1" or more and "N" or less (step S86: YES), CtrR is set to FrmN. (CtrN) (step S87). The calculation of the mapping in step S87 is the same as that in step S85.

Next, the value of (vector (CtrP → CtrR) + vector (CtrR → CtrN)) / 2 is set in the motion vector MvR (step S88).

On the other hand, when it is determined in step S86 that the value of FrmN is not "1" or more and "N" or less (step S86: NO), the vector (CtrP → CtrR) is set in the motion vector MvR (step S89).

Further, when it is determined in step S84 that the value of FrmP is not "1" or more and "N" or less (step S84: NO), it is determined whether the value of FrmN is "1" or more and "N" or less (step S90). ), When it is "1" or more and "N" or less (step S90: YES), CtrR is mapped to FrmN (CtrN) (step S91). The calculation of the mapping in step S91 is the same as that in step S85.

Next, the value of the vector (CtrR → CtrN) is set in the motion vector MvR (step S92).

After performing any one of the processes of step S88, step S89, and step S92, the angle / MvR length formed by the MvR with the x-axis is defined as the blur angle (BaR) / length (BlR) ( Step S93). Then, the obtained blur angle (BaR) and length (BlR) are stored in the blur PSF parameter holding memory 17B (step S94). Then, "1" is added to the value of FrmR, and the value is set to FrmR (step S95). Next, it is determined whether or not FrmR is "N" or less (step S96), and if it is "N" or less (step S96: YES), the process returns to step S81. On the other hand, when FrmR is not "N" or less (step S96: NO), the blur PSF estimation process is terminated.

On the other hand, when it is determined in step S90 that the value of FrmN is not "1" or more and "N" or less (step S90: NO), the image data in the vicinity of RoiR is acquired from the frame memory 11 (step S104). The blur angle (BaR) and length (BlR) are estimated from the image data (step S105). After that, the process proceeds to step S94.

(Parameter adjustment processing in the reference frame)
FIG. 12 is a flowchart for explaining the parameter adjustment process in the reference frame. In the figure, first, the image data of the reference frame (FrmA) is acquired from the frame memory 11 (step S110), and this is displayed (step S111). Next, the attention subject area (RoiA) of FrmA is acquired from the reference point position holding memory 17A (step S112), and the corner points and the area frame of the acquired RoiA are displayed (step S113). Further, the coordinates of the corner points of RoiA are displayed (step S114).

Next, the blur angle (BaA) and blur length (BlA) of FrmA are acquired from the blur PSF parameter holding memory 17B (step S115), and a blur PSF image is generated from the acquired blur angle (BaA) and blur length (BlA). Is displayed (step S116). In addition, the numerical values of the blur angle (BaA) and the blur length (BlA) are displayed (step S117).

Then, the user is made to adjust the position of the displayed RoiA and the blur PSF parameter (step S118). That is, the user is instructed to adjust the position of the displayed RoiA and the blur PSF parameter. As a result, when the user adjusts the position of RoiA and the blur PSF parameter, the screen display content is updated according to the adjustment content (step S119). Then, it is determined whether or not the adjustment by the user is completed (step S120), if it is not completed (step S120: NO), the process returns to step S118, and if it is completed (step S120: YES), the original blurring of the reference frame is performed. The ratio (B'2BlA) of the length BlA and the adjusted blur length BlA'is obtained (step S121).

Next, "1" is set in FrmR (step S122). Then, the blur length (BlR) of the FrmR frame is acquired from the blur PSF parameter holding memory 17B (step S123), BlR * B'2BlA is calculated, and the result is set to BlR'(step S124). Next, BlR stored in the blur PSF parameter holding memory 17B is updated with BlR'(step S125). Next, the geometric transformation matrix M _{FrmR → FrmA} of _FrmR is acquired from the reference point position holding memory 17A (step S126), M _{FrmR → FrmA} ^-1 * RoiA'is calculated, and the result is RoiR'(step S127). ).

Here, RoiA': The position of RoiA at the time when the adjustment by the user is completed. Also,
RoiR'← M _{FrmR → FrmA} ^-1 × What is RoiA'?
RoiR'.C1 ← M _{FrmR → FrmA} ^-1 × RoiA'.C1
RoiR'.C2 ← M _{FrmR → FrmA} ^-1 × RoiA'.C2
RoiR'.C3 ← M _{FrmR → FrmA} ^-1 × RoiA'.C3
RoiR'.C4 ← M _{FrmR → FrmA} ^-1 × RoiA'.C4
The coordinates of each corner of _RoiA'are mapped by the geometric transformation matrix M _{FrmR → FrmA} ^-1 . When the automatic alignment process is not performed, M _{FrmR → FrmA} ^-1 is regarded as an identity matrix and calculated.

Next, the attention subject area RoiR stored in the reference point position holding memory 17A is updated with RoiR'(step S128). Next, "1" is added to the value of FrmR, and the value is set to FrmR (step S129). Then, it is determined whether or not FrmR is "N" or less (step S130), and if it is "N" or less (step S130: YES), the process returns to step S123, and if it is not "N" or less (step S130: NO). The parameter adjustment process in the reference frame ends.

The processing of steps S121 to S130 (that is, a series of processing after the adjustment by the user is completed) greatly reduces the work after the individual adjustment of the reference frame.

Other parameters (such as the size and shape of the focus blur and the intensity setting at the time of removing the blur) stored in the blur PSF parameter holding memory 17B may be displayed so that the set value can be changed by the user.

FIG. 13 is a diagram showing an example of a user interface screen before parameter adjustment in the reference frame. FIG. 6A is an image of a reference frame, and FIG. 9B is a blur PSF image. In the adjustment of the reference frame, the positions of the four points (points 1 to 4) used as the reference for alignment are adjusted by the reference frame. For example, the operation of moving each point (points 1 to 4) placed at the four corners of the subject area of interest in the initial state to the four corners of the license plate 100 in the subject area of interest is performed. Also, before adjusting the blur angle and blur length, which are the blur PSF parameters, the blur PSF image is horizontally longer than the actual blur length, and if blur removal is performed as it is, overcorrection will occur and noise may occur. It has become.

FIG. 14 is a diagram showing an example of a user interface screen after parameter adjustment in the reference frame. FIG. 6A is an image of a reference frame, and FIG. 9B is a blur PSF image. Each point (points 1 to 4) placed at the four corners of the subject area of interest in the initial state is located at the four corners of the license plate 100 in the subject area of interest. In addition, since the blur angle and blur length, which are the blur PSF parameters, are adjusted, the blur PSF image is an abbreviation, an appropriate blur length is set, and the blur removal result is in a good state. There is.

(Parameter adjustment processing in the reference frame)
15 and 16 are flowcharts for explaining the parameter adjustment process in the reference frame. In the figure, first, the image data of the reference frame (FrmA) is acquired from the frame memory 11 (step S140), and this is displayed (step S141). Next, the attention subject area (RoiA) of FrmA is acquired from the reference point position holding memory 17A (step S142). Then, the corner points and the area frame of the acquired subject area of interest (RoiA) are displayed (step S143). In addition, the coordinates of the corner points of RoiA are displayed (step S144). Further, a list of all frames stored in the frame memory 11 is displayed (step S145). In addition, the frame of the subject area of interest is displayed on each frame displayed in the frame list (step S146).

Next, any one frame is selected (step S147), and the user is made to change the selection of the reference frame and adjust the parameters of the selected reference frame (step S148). That is, the user is instructed to change the selection of the reference frame and adjust the parameters of the selected reference frame. As a result, the user changes the selection of the reference frame and adjusts the parameters of the selected reference frame. Next, it is determined whether or not the selected state of the reference frame has been changed (step S149), and when the selected state of the reference frame has been changed (step S149: YES), the image data of the selected reference frame (FrmR) is stored in the frame memory 11. (Step S150) and display this (step S151).

After displaying FrmR, the attention subject area (RoiR) of FrmR is acquired from the reference point position holding memory 17A (step S152). Then, the acquired RoiR corner point and area frame are displayed (step S153), and the coordinates of the RoiR corner point are displayed (step S154). Further, the blur angle (BaR) and blur length (BlR) of FrmR are acquired from the blur PSF parameter holding memory 17B (step S155), and a blur PSF image is generated and displayed from the blur angle (BaR) and blur length (BlR). (Step S156). In addition, the numerical values of the blur angle (BaR) and the blur length (BlR) are displayed (step S157). After that, the process returns to step S148.

On the other hand, when it is determined in step S149 that the selection state of the reference frame has not been changed (step S149: NO), it is determined whether or not the RoiR and blur PSF parameters have been changed (step S158). When the RoiR and blur PSF parameters are changed (step S158: YES), the screen display contents are updated according to the adjustment contents (step S159). Then, the set values of the reference point position holding memory 17A and the blur PSF parameter holding memory 17B are changed with respect to the selected reference frame (FrmR) according to the adjustment content (step S160). After that, the process returns to step S148.

Further, when it is determined in step S158 that the RoiR and blur PSF parameters have not been changed (step S158: NO), it is determined whether or not the adjustment is completed (step S161), and when the adjustment is not completed (step S161: NO). Returning to step S148, when the adjustment is completed (step S161: YES), the parameter adjustment process in the reference frame ends.

FIG. 17 is a diagram showing an example of a user interface screen before parameter adjustment in the reference frame. FIG. 6A is an image of the selected input frame (reference frame), and FIG. 9B is a blur PSF image of the selected input frame (reference frame). In the adjustment of the reference frame, the positions of four points (points 1 to 4) are adjusted with respect to the reference points set in the reference frame for each of the frames before and after the reference frame. Since the reference point set in the reference frame is used in the adjustment of the reference frame, the positions of the four points (points 1 to 4) are approximately at the four corners of the license plate 100 because they substantially match the reference frame even before the adjustment. To position. As a result, the positions of the four points (points 1 to 4) can be positioned at the four corners of the license plate 100 in a short time. Of the four points (points 1 to 4) before adjustment, the coordinates of point 1 are "X = 290.7500, Y = 357.2500", and the coordinates of point 2 are "X = 313.0000, Y = 355." "5000", the coordinates of the point 3 are "X = 291.0449, Y = 368.3445", and the coordinates of the point 4 are "X = 313.2500, Y = 366.7500".

Further, since the blur angle and the blur length, which are the blur PSF parameters, also substantially match the adjustment in the reference frame, the blur length is adjusted in advance to a substantially appropriate length.

FIG. 18 is a diagram showing an example of a user interface screen after parameter adjustment in the reference frame. FIG. 6A is an image of the selected input frame (reference frame), and FIG. 9B is a blur PSF image of the selected input frame (reference frame). The position adjustment of four points (points 1 to 4) with respect to the reference point set in the reference frame can be completed in an extremely short time. Of the four adjusted points (points 1 to 4) shown in FIG. 18, the coordinates of point 1 are "X = 290.6185, Y = 357.3225", and the coordinates of point 2 are "X = 313.2253,". "Y = 355.5452", the coordinates of point 3 are "X = 291.0449, Y = 368.3445", and the coordinates of point 4 are "X = 313.6530, Y = 366.8148", before parameter adjustment. The difference from the position of each point of is small.

In this way, the position of the reference point for alignment and the blur PSF parameters such as the blur length can be adjusted in the reference frame, and the adjustment result in the reference frame can be reflected in all other reference frames. ..

(Blur removal process)
FIG. 19 is a flowchart for explaining the blur removal process. In the figure, first, “1” is set for FrmR (step S170). Next, the RoiR is acquired from the reference point position holding memory 17A (step S171). Next, the image data (ImgR) in the vicinity of RoiR is acquired from the frame memory 11 (step S172). Further, the blur PSF parameter is acquired from the blur PSF parameter holding memory 17B (step S173).

Then, a blur PSF image is generated from the acquired blur PSF parameter (step S174). The details of generating the blur PSF image will be described later.

Then, the image data (ImgR) is deconvolutioned with the generated blur PSF image to obtain the image data (ImgR') (step S175).

A known method can be used for Deconvolution. Hereinafter, known examples will be given.

・ Inverse Filter
Utilizing the properties of convolution and Fourier transform
X × PSF = Y → FT (X) FT (PSF) = FT (Y) → FT (X) = FT (Y) / FT (PSF) to restore the original image.
(X: Original image without blur / blur, PSF: Point spread function with blur / blur, Y: Observation image with blur / blur, *: Convolution calculation, FT (・): Fourier transform)

・ Wiener Filter
In the above improved version, the term of random noise is included in the model formula (X * PSF + W = Y) to suppress the harmful effects (W: random noise component).

・ W. H. Richardson, `` Bayesian-Based Iterative Method of Image Restoration'', Journal of Optical Society of America, Vol. 62, NO. 1, 1972
A method in which the initial solution of X is placed, the procedure of finding the next X from the values of X, PSF, and Y and updating X is repeated until X converges, and does not depend on the Fourier transform.

Then, after obtaining the image data (ImgR'), the image data (ImgR) in the frame memory 11 is replaced with the image data (ImgR') (step S176). After that, "1" is added to the value of FrmR, and the value is set to FrmR (step S177). After that, it is determined whether or not the value of FrmR is "N" or less (step S178), and if it is "N" or less (step S178: YES), the process returns to step S171 and if it is not "N" or less (step S178: NO). ), Finish the blur removal process.

Hereinafter, an example of generating a blur PSF image from the blur PSF parameters (blur angle BaR, blur length BlR) will be given.
1. 1. Prepare image data of a size sufficient to accommodate the trajectory of the blur, and initialize it with 0.
2. Image data located at each point on a line segment (the midpoint of the line segment is equal to the origin) that passes through the origin with the center of the prepared image as the origin, the angle formed by the x-axis is BaR, and the length is BlR. The pixel value of is rewritten to 1.0 (= maximum brightness level).
3. 3. The pixel values of the entire image data are uniformly normalized so that the sum of the pixel values is 1.0.
Further, if the blur PSF parameter includes the blur radius BrR (a parameter used to sharpen an image that is out of focus and represents the degree of blurring of the blur trajectory), the following step is added.
4. Convolve the perfect circle data with radius BrR into the generated image data.
Perfect circle data: Points within a radius BrR from the center = pixel value K, other points = pixel value 0
K: 1 / (total number of points within radius BrR from the center)

(Reconstruction process)
FIG. 20 is a flowchart for explaining the reconstruction process. In the figure, first, an initial estimated image is set in X (step S180). The initial estimated image may be enlarged by any method as long as the image of the subject area of interest of the reference frame is enlarged by z times in the vertical and horizontal directions.

Next, the evaluation value S and the modified image U are initialized (step S181), and then “1” is set in FrmR (step S182). Next, the attention subject area (RoiR) of the reference frame FrmR and the geometric transformation matrix (M _{FrmR → FrmA} ) are acquired from the reference point position holding memory 17A (step S183). Further, the image data in the vicinity of RoiR is acquired from the frame memory 11 (ImgR) (step S184).

Next, the image X is geometrically deformed to the position in FrmR (step S185). That is, (Mz * M _{FrmR → FrmA} ^-1 ) * X is calculated, and the result is set in the image Y'(Y'← (Mz * M _{FrmR → FrmA} ^-1 ) * X). Here, Mz is a expansion transformation matrix (as shown in FIG. 21 when the super-resolution magnification is z).

After the image X is geometrically deformed to the position in FrmR, the resolution of the image Y'is deteriorated (Y ") (step S186), and the evaluation value S and the modified image U are updated based on the difference between the image Y" and Imgur. (Step S187). After that, "1" is added to the value of FrmR, and the value is set to FrmR (step S188). Then, it is determined whether or not the value of FrmR is "N" or less (step S189), and if it is "N" or less (step S189: YES), the process returns to step S183 and if it is not "N" or less (step S189: NO). ), The constraint condition of the image X is evaluated, and the evaluation value S and the modified image U are updated (step S190). Then, it is determined whether or not X has sufficiently converged (step S191), and if it has sufficiently converged (step S191: YES), the image X is output (step S192), and the reconstruction process is terminated. On the other hand, when X is not sufficiently converged (step S191: NO), X is updated based on the modified image U (step S193), and the process returns to step S181.

(effect)
As described above, in the present embodiment, the image processing device 1 displays the image and parameters of the reference frame (position of the subject area of interest and blur PSF parameters), and displays the image of the reference frame based on the user's adjustment. The contents are updated and the adjusted parameters are stored as the parameters of the reference frame. Then, when displaying the image of the next reference frame, the image processing device 1 displays the stored reference frame parameter as the parameter of the reference frame, and displays the image of the reference frame based on the user's adjustment. The contents are updated and the adjusted parameters are stored as the parameters of the reference frame. Hereinafter, when the image processing device 1 displays the image of the new reference frame, the parameter of the reference frame is displayed as the parameter of the new reference frame.

As a result, the adjustment result in the reference frame can be reflected to all other reference frames, so that when the images of multiple input frames are manually aligned, the input work is reduced and the work efficiency is improved. Can be planned. As a result, the number of frames that can be used for the super-resolution processing can be increased, so that a high-resolution image can be obtained by the super-resolution processing.

(Embodiment 2)
In the second embodiment, a blur PSF estimation process different from that described with reference to FIGS. 10 and 11 in the first embodiment will be described.

(Blur PSF estimation process)
22 and 23 are flowcharts for explaining the blur PSF estimation process of the image processing device 1 according to the present embodiment. In FIGS. 22 and 23, the steps common to those in FIGS. 10 and 11 are designated by the same reference numerals as those in FIGS. 10 and 11, and the description thereof will be omitted.

In FIGS. 22 and 23, “1” is first set in FrmR (step S80). Next, RoiR is acquired from the reference point position holding memory 17A (step S81). Then, the acquired image data in the vicinity of RoiR is acquired from the frame memory 11 (ImgR) (step S200). Next, the center point of the acquired RoiR is obtained (CtrR) (step S82). After that, the value of FrmR is subtracted by "1", the value is set to FrmP, the value of FrmR is added by "1", and the value is set to FrmN (step S83). Next, it is determined whether or not the value of FrmP is "1" or more and "N" or less (step S84), and when it is "1" or more and "N" or less (step S84: YES), CtrR is mapped to FrmP (CtrP). (Step S85).

After mapping CtrR to FrmP, it is determined whether the value of FrmN is "1" or more and "N" or less (step S86), and if it is "1" or more and "N" or less (step S86: YES), CtrR is set to FrmN. (Step S87). Next, (vector (CtrP → CtrR) + vector (CtrR → CtrN)) / 2 is calculated, and the result is set in the motion vector MvR (step S88).

On the other hand, when it is determined in step S86 that the value of FrmN is not "1" or more and "N" or less (step S86: NO), the vector (CtrP → CtrR) is set in the motion vector MvR (step S89).

Further, when the value of FrmP is not "1" or more and "N" or less in step S84 (step S84: NO), it is determined whether the value of FrmN is "1" or more and "N" or less (step S90), and "1". When it is equal to or greater than or equal to “N” (step S90: YES), CtrR is mapped to FrmN (CtrN) (step S91). Next, the value of the vector (CtrR → CtrN) is set in the motion vector MvR (step S92).

After performing any one of the processes of step S88, step S89, and step S92, the angle / MvR length formed by the MvR with the x-axis is defined as the blur angle (BaR) / length (MlR) ( Step S93). Next, the obtained blur angle (BaR) and length (MlR) are stored in the blur PSF parameter holding memory 17B (step S94). Next, the length of the blur is estimated from the image data ImgurR and the blur angle BaR (BlR) (step S201). Then, the ratio of the estimated blur length BlR and the vector length MlR is calculated (B2MlR) (step S202).

After that, "1" is added to the value of FrmR, and the value is set to FrmR (step S95). Then, it is determined whether or not FrmR is "N" or less (step S96), and if FrmR is "N" or less (step S96: YES), the process returns to step S81. On the other hand, when FrmR is not "N" or less (step S96: NO), the representative value of the ratio B2MlR calculated in all frames (R = 1 to N) is calculated (B2Ml) (step S203). After that, “1” is set in FrmR (step S204). Next, the value obtained by multiplying the blur length MlR stored in the blur PSF parameter holding memory 17B by B2Ml is stored as the blur length BlR (step S205). Next, "1" is added to the value of FrmR, the value is set to FrmR (step S206), and it is determined whether or not FrmR is "N" or less (step S207). When FrmR is "N" or less (step S207: YES), the process returns to step S205. On the other hand, when FrmR is not "N" or less (step S207: NO), the blur PSF estimation process is terminated.

On the other hand, when it is determined in step S90 that the value of FrmN is not "1" or more and "N" or less (step S90: NO), the angle and length of the blur are estimated from the image data Imgur (BaR, BlR) ( Step S208). After that, the value of BlR is set to MlR (step S209), and the process returns to step S94.

(effect)
As described above, according to the present embodiment, when estimating the length of the blur PSF in the blur PSF estimation process, the length of the blur PSF is estimated from one image data, and the estimated length of the blur PSF is estimated. The ratio of the motion vector to the length of the motion vector is calculated for each frame, the ratio is smoothed over all frames, and the length of the blur PSF of each frame is calculated by multiplying the length of each motion vector by the smoothed ratio. .. As a result, the length of the blur PSF can be estimated in consideration of the shutter speed of the camera, so that the blur removal process can be performed using an accurate blur PSF, and a sharp image without ringing noise can be obtained. Obtainable.

(Embodiment 3)
In the third embodiment, a case where the processing content described in the first embodiment is applied to the detection of the face image will be described. The schematic configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus 1 of FIG. 1 described in the first embodiment. Further, the flow of the super-resolution processing of the image processing apparatus in the present embodiment is substantially the same as that of FIGS. 2 and 3 described in the first embodiment. However, the automatic alignment process (S4) in FIG. 2 is changed to the face feature point detection process.

Further, the flow of the image data reading process of the image processing apparatus in the present embodiment is the same as that of FIG. 4 described in the first embodiment. Further, the flow of the reference frame selection process of the image processing apparatus in the present embodiment is the same as that of FIGS. 5 and 6 described in the first embodiment. Further, the flow of the attention subject area setting process of the image processing apparatus in the present embodiment is the same as that of FIG. 7 described in the first embodiment. Further, the flow of the blur removal processing of the image processing apparatus in the present embodiment is the same as that of FIG. 19 described in the first embodiment.

(Face feature point detection process)
FIG. 24 is a flowchart for explaining the face feature point detection process. In the figure, first, the value of FrmR is set to "1" (step S300). Next, it is determined whether or not the value of FrmR is "N" or less (step S301). In this determination, when the value of FrmR is "N" or less (step S301: YES), the FrmRth frame data in the frame memory 11 is acquired (ImgR) (step S302). Next, the facial feature points are detected in _Imgur (P _{FrmR, i} ) (step S303). A method using AAM (Active Appearance Models) is known as a method for detecting facial feature points. In this method, a face image and the positions of a plurality of feature points on the face image are set in advance to create training data from a large number of sets of data (offline processing: processing performed before super-resolution processing). ), Then, by inputting an unknown face image and performing fitting using the above training data, a process of estimating the position of a feature point in the input image (online processing: super-resolution processing). This is a method of performing (processing performed for each input frame) (Reference: "Active Appearance Models Revisited", Iain Matthews and Simon Baker, International Journal of Computer Vision, Vol. 60, No. 2, pp. 135 --164, November, 2004.). In addition, P _{FrmR, i} is the position information of the i-th facial feature point detected in _Imgur (1 ≦ i ≦ (total number of feature points in Imgur)).

Next, the region surrounding all the facial feature points in ImgurR is calculated (RoiR) (step S304), and the planar patches (see FIG. 25) formed by the adjacent facial feature points of _ImgurR are listed (S _{FrmR, *} (S _{FrmR, *} ). S _{FrmR, k} (all of the total number of patches in 1 ≦ k ≦ _Imgur ))) (step S305). In FIG. 25, the circles are facial feature points, and the regions S ₁ to S ₁₂ surrounded by the dotted lines are flat patches.

Then, RoiR, P _{FrmR, *} (all of P _{FrmR, i} ) and S _{FrmR, *} are stored in the reference point position holding memory 17A (step S306). After that, the value of FrmR is added by "1", the value is set to FrmR (step S307), and the process returns to step S301.

On the other hand, when it is determined in step S301 that the value of FrmR is larger than N (step S301: NO), the value of FrmR is set to "1" (step S308), and it is determined whether the value of FrmR is "N" or less. (Step S309).

In this determination, when the value of FrmR is "N" or less (step S309: YES), the patch number k is set to "1" (step S310), and it is determined whether the value of k is equal to or less than the total number of patches (step S309: YES). Step S311).

In this determination, when the value of k is equal to or less than the total number of patches (step S311: YES), the position of each vertex of patches S _{FrmR, k} of FrmR is set to the position of each vertex of the corresponding patches S _{FrmA, k} in _FrmA. The geometric transformation matrix mapped to is _obtained (M _{FrmR, k → FrmA, k} ) (step S312). Then, the geometric transformation matrix M _{FrmR, k → FrmA, k} is stored in the reference point position holding memory 17A (step S313). After that, "1" is added to the value of k, the value is set to k (step S314), and the process returns to step S311.

On the other hand, when it is determined in step S311 that the value of k is larger than the total number of patches (step S311: NO), the value of FrmR is added by "1", the value is set to FrmR (step S315), and in step S309. Return.

If it is determined in step S309 that the value of FrmR is not "N" or less (step S309: NO), the face feature point detection process is terminated.

(Blur PSF estimation processing)
26 and 27 are flowcharts for explaining the blur PSF estimation process. In FIGS. 26 and 27, the steps common to those in FIGS. 10 and 11 are designated by the same reference numerals as those in FIGS. 10 and 11, and the description thereof will be omitted.

In FIGS. 26 and 27, first, “1” is set in FrmR (step S80). Next, P _{FrmR, *} is acquired from the reference point position holding memory 17A (step S320). Then, the representative points of the acquired P _{FrmR, *} are obtained (Ctr _FrmR ) (step S321). After that, the value of FrmR is added by "1", and the value is set to FrmR (step S322). Next, it is determined whether or not the value of FrmR is "N" or less (step S323), and if it is "N" or less (step S323: YES), the process returns to step S320.

On the other hand, when it is determined in step S323 that the value of FrmR is larger than "N" (step S323: NO), the value of FrmR is set to "1" (step S324).

Next, the value of FrmR is subtracted by "1", the value is set to FrmP, the value of FrmR is added by "1", and the value is set to FrmN (step S83). Then, it is determined whether or not the value of FrmP is "1" or more and "N" or less (step S84), and if it is "1" or more and "N" or less (step S84: YES), the process proceeds to step S86.

Then, it is determined whether or not the value of FrmN is "1" or more and "N" or less (step S86), and when it is "1" or more and "N" or less (step S86: YES), (vector (Ctr _FrmP → Ctr _FrmR )). The value of + vector (Ctr _FrmR → Ctr _FrmN )) ÷ 2 is set in the motion vector MvR (step S325).

On the other hand, when it is determined in step S86 that the value of FrmN is not "1" or more and "N" or less (step S86: NO), the vector (Ctr _FrmP → Ctr _FrmR ) is set in the motion vector MvR (step S326). ..

Further, when it is determined in step S84 that the value of FrmP is not "1" or more and "N" or less (step S84: NO), it is determined whether the value of FrmN is "1" or more and "N" or less (step S90). ), When it is "1" or more and "N" or less (step S90: YES), the value of the vector (Ctr _FrmR → Ctr _FrmN ) is set in the motion vector MvR (step S327).

After performing any one of the processes of steps S325, S326, and S327, the angle and length of MvR formed by MvR with the x-axis are set to the blur angle (BaR) and length (BlR) (step S93). ). Then, the obtained blur angle (BaR) and length (BlR) are stored in the blur PSF parameter holding memory 17B (step S94). Then, "1" is added to the value of FrmR, and the value is set to FrmR (step S95). Next, it is determined whether or not FrmR is "N" or less (step S96), and if it is "N" or less (step S96: YES), the process returns to step S83. On the other hand, when FrmR is not "N" or less (step S96: NO), the blur PSF estimation process is terminated.

On the other hand, when it is determined in step S90 that the value of FrmN is not "1" or more and "N" or less (step S90: NO), the image data in the vicinity of RoiR is acquired from the frame memory 11 (step S104). The blur angle (BaR) and length (BlR) are estimated from the image data (step S105). After that, the process proceeds to step S94.

(Parameter adjustment processing in the reference frame)
FIG. 28 is a flowchart for explaining the parameter adjustment process in the reference frame. In FIG. 28, the steps common to FIG. 12 are designated by the same reference numerals as those in FIG. 12, and the description thereof will be omitted.

In FIG. 28, first, the image data of the reference frame (FrmA) is acquired from the frame memory 11 (step S110), and this is displayed (step S111). Next, the face feature points (P _{FrmA, *} ) are acquired from the reference point position holding memory 17A (step S330), and the mesh connecting the face feature points P _{FrmA, *} and the face feature points is displayed (step S331).

Next, the blur angle (BaA) and blur length (BlA) of FrmA are acquired from the blur PSF parameter holding memory 17B (step S115), and a blur PSF image is generated from the acquired blur angle (BaA) and blur length (BlA). Is displayed (step S116). In addition, the numerical values of the blur angle (BaA) and the blur length (BlA) are displayed (step S117).

Then, the user is made to adjust the position of the face feature point and the blur PSF parameter being displayed (step S332), and the screen display content is updated according to the adjustment content of the user (step S119). Then, it is determined whether or not the adjustment by the user is completed (step S120), if it is not completed (step S120: NO), the process returns to step S332, and if it is completed (step S120: YES), the original blurring of the reference frame is performed. The ratio (B'2BlA) of the length BlA and the adjusted blur length BlA'is obtained (step S121), and the geometric deformation matrix that maps the vertex position before adjustment to the adjusted position for each patch of the reference frame is obtained (M). _{FrmA, * → FrmA, *'} ) (step S333).

Next, "1" is set in FrmR (step S122). Then, the blur length (BlR) of the FrmR frame is acquired from the blur PSF parameter holding memory 17B (step S123), BlR * B'2BlA is calculated, and the result is set to BlR'(step S124). Next, BlR stored in the blur PSF parameter holding memory 17B is updated with BlR'(step S125). Next, the face feature point (P _{FrmR, *} ) of _FrmR, the patch information (S _{FrmR, *} ) and the geometric conversion matrix (M _{FrmR, * → FrmA, *} ) are acquired from the reference point position holding memory 17A (step S334). ), The facial feature points P _{FrmR, *} are mapped (P _{FrmR, *'} ) (see FIG. 29) (step S335).

Next, the face feature points P _{FrmR, *} stored in the reference point position holding memory 17A are updated with P _{FrmR, *'} (step S336). Next, "1" is added to the value of FrmR, and the value is set to FrmR (step S129). Then, it is determined whether or not FrmR is "N" or less (step S130), and if it is "N" or less (step S130: YES), the process returns to step S123, and if it is not "N" or less (step S130: NO). The parameter adjustment process in the reference frame ends.

The processing of steps S121 to S130 (that is, a series of processing after the adjustment by the user is completed) greatly reduces the work after the individual adjustment of the reference frame.

Other parameters (such as the size and shape of the focus blur and the intensity setting at the time of removing the blur) stored in the blur PSF parameter holding memory 17B may be displayed so that the set value can be changed by the user.

Next, the details of the mapping process of the facial feature points in step S335 of FIG. 28 will be described with reference to FIG. 29.

In the figure, first, the patch number k is set to "1" (step S340), and it is determined whether or not the value of k is equal to or less than the total number of patches (step S341).

In this determination, when the value of k is equal to or less than the total number of patches (step S341: YES), the coordinates of each vertex of patch k in the reference frame FrmR are transformed by the geometric transformation matrix M _{FrmR, k → FrmA, k'} (step S341: YES). P _{FrmR, *'} ) (step S342). After that, "1" is added to the value of k, the value is set to k (step S343), and the process returns to step S341.

On the other hand, when it is determined in step S341 that the value of k is larger than the total number of patches (step S341: NO), the mapping process of the facial feature points in step S335 ends.

Since the vertices of the patch are shared with the vertices of the adjacent patches, in the flow of FIG. 29, the same vertices (that is, face feature points) are used in the coordinate conversion process of one patch and the coordinate conversion process of another patch. There may be a case where a plurality of converted coordinates for are calculated. Therefore, in the present embodiment, one post-conversion coordinate is determined from a plurality of calculated post-conversion coordinates by a predetermined method, and the determined coordinates are P _{FrmR, i'with} respect to the face feature point i. As a method of determination, the center of gravity is obtained, the first value is adopted, or the last value is adopted.

(Parameter adjustment processing in the reference frame)
30 and 31 are flowcharts for explaining the parameter adjustment process in the reference frame. In FIGS. 30 and 31, the steps common to those in FIGS. 15 and 16 are designated by the same reference numerals as those in FIGS. 15 and 16 and the description thereof will be omitted.

In FIGS. 30 and 31, first, the image data of the reference frame (FrmA) is acquired from the frame memory 11 (step S140), and this is displayed (step S141). Next, the face feature points (P _{FrmA, *} ) of _FrmA are acquired from the reference point position holding memory 17A (step S350), and the mesh connecting the face feature points P _{FrmA, *} and the face feature points is displayed (step S351). .. Further, a list of all frames stored in the frame memory 11 is displayed (step S145). In addition, the frame of the subject area of interest is displayed on each frame displayed in the frame list (step S146).

Next, any one frame is selected (step S147), and the user is made to change the selection of the reference frame and adjust the parameters of the selected reference frame (step S148). Next, it is determined whether or not the selected state of the reference frame has been changed (step S149), and when the selected state of the reference frame has been changed (step S149: YES), the image data of the selected reference frame (FrmR) is stored in the frame memory 11. (Step S150) and display this (step S151).

After displaying FrmR, the face feature points (P _{FrmR, *} ) of _FrmR are acquired from the reference point position holding memory 17A (step S352), and the face feature points P _{FrmR, *} and the mesh connecting the face feature points are displayed. (Step S353).

Further, the blur angle (BaR) and blur length (BlR) of FrmR are acquired from the blur PSF parameter holding memory 17B (step S155), and a blur PSF image is generated and displayed from the blur angle (BaR) and blur length (BlR). (Step S156). In addition, the numerical values of the blur angle (BaR) and the blur length (BlR) are displayed (step S157). After that, the process returns to step S148.

On the other hand, when it is determined in step S149 that the selection state of the reference frame has not been changed (step S149: NO), it is determined whether or not the RoiR and blur PSF parameters have been changed (step S158). When the RoiR and blur PSF parameters are changed (step S158: YES), the screen display contents are updated according to the adjustment contents (step S159). Then, the set values of the reference point position holding memory 17A and the blur PSF parameter holding memory 17B are changed with respect to the selected reference frame (FrmR) according to the adjustment content (step S160). After that, the process returns to step S148.

Further, when it is determined in step S158 that the RoiR and blur PSF parameters have not been changed (step S158: NO), it is determined whether or not the adjustment is completed (step S161), and when the adjustment is not completed (step S161: NO). Returning to step S148, when the adjustment is completed (step S161: YES), the parameter adjustment process in the reference frame ends.

(Reconstruction process)
FIG. 32 is a flowchart for explaining the reconstruction process. In FIG. 32, the steps common to those in FIG. 20 are designated by the same reference numerals as those in FIG. 20, and the description thereof will be omitted.

In FIG. 32, first, the initial estimated image is set in X (step S180). Next, the evaluation value S and the modified image U are initialized (step S181), and then “1” is set in FrmR (step S182).

Next, the patch information (S _{FrmR, *} ) and the geometric conversion matrix (M _{FrmR, * → FrmA, *} ) of the reference frame FrmR are acquired from the reference point position holding memory 17A (step S360). Further, the image data in the vicinity of each patch is acquired from the frame memory 11 (ImgRS _* ) (step S361).

Next, the image X is geometrically deformed to the position in FrmR for each patch (step S362). That is, for each k, (Mz * M _{FrmR, k → FrmA, k} ^-1 ) * X is calculated, and the result is set in the image Y'(Y'← (Mz * M _{FrmR, k → FrmA,). k} ^-1 ) * X). Here, Mz is a expansion transformation matrix (as shown in FIG. 21 when the super-resolution magnification is z).

After the image X is geometrically deformed to the position in FrmR, the resolution of the image Y'is deteriorated (Y ") (step S186), and the evaluation value S and the modified image U are updated based on the difference between the image Y" and Imgur. (Step S187). After that, "1" is added to the value of FrmR, and the value is set to FrmR (step S188). Then, it is determined whether or not the value of FrmR is "N" or less (step S189), and if it is "N" or less (step S189: YES), the process returns to step S360 and if it is not "N" or less (step S189: NO). ), The constraint condition of the image X is evaluated, and the evaluation value S and the modified image U are updated (step S190). Then, it is determined whether or not X has sufficiently converged (step S191), and if it has sufficiently converged (step S191: YES), the image X is output (step S192), and the reconstruction process is terminated. On the other hand, when X is not sufficiently converged (step S191: NO), X is updated based on the modified image U (step S193), and the process returns to step S181.

(effect)
As described above, in the present embodiment, the image processing device 1 displays the image and parameters of the reference frame (positions of face feature points, mesh and blur PSF parameters connecting the face feature points), and is used for user adjustment. Based on this, the display content of the image of the reference frame is updated, and the adjusted parameters are stored as the parameters of the reference frame. Then, when displaying the image of the next reference frame, the image processing device 1 displays the stored reference frame parameter as the parameter of the reference frame, and displays the image of the reference frame based on the user's adjustment. The contents are updated and the adjusted parameters are stored as the parameters of the reference frame. Hereinafter, when the image processing device 1 displays the image of the new reference frame, the parameter of the reference frame is displayed as the parameter of the new reference frame.

As a result, the adjustment result in the reference frame can be reflected to all other reference frames, so that when the images of multiple input frames are manually aligned, the input work is reduced and the work efficiency is improved. Can be planned. As a result, the number of frames that can be used for the super-resolution processing can be increased, so that a high-resolution image can be obtained by the super-resolution processing.

(Embodiment 4)
In the fourth embodiment, the blur PSF estimation process different from that described with reference to FIGS. 26 and 27 in the third embodiment will be described.

(Blur PSF estimation process)
33, 34, and 35 are flowcharts for explaining the blur PSF estimation process of the image processing device 1 according to the present embodiment. In FIGS. 33, 34 and 35, the steps common to those in FIGS. 26 and 27 are designated by the same reference numerals as those in FIGS. 26 and 27, and the description thereof will be omitted.

In FIGS. 33, 34 and 35, first, “1” is set in FrmR (step S80). Next, P _{FrmR, *} is acquired from the reference point position holding memory 17A (step S400). Then, the acquired image data in the vicinity of RoiR is acquired from the frame memory 11 (ImgR) (step S200). Next, the representative points of the acquired P _{FrmR, *} are obtained (Ctr _FrmR ) (step S401). After that, the value of FrmR is added by "1", and the value is set to FrmR (step S402). Next, it is determined whether or not the value of FrmR is "N" or less (step S403), and if it is "N" or less (step S403: YES), the process returns to step S400.

On the other hand, when it is determined in step S403 that the value of FrmR is larger than "N" (step S403: NO), the value of FrmR is set to "1" (step S404).

Next, the value of FrmR is subtracted by "1" and the value is set to FrmP, the value of FrmR is added by "1", and the value is set to FrmN (Tep S83 which is not less than or equal to S). Then, it is determined whether or not the value of FrmP is "1" or more and "N" or less (step S84), and if it is "1" or more and "N" or less (step S84: YES), the process proceeds to step S86.

Then, it is determined whether or not the value of FrmN is "1" or more and "N" or less (step S86), and when it is "1" or more and "N" or less (step S86: YES), (vector (Ctr _FrmP → Ctr _FrmR )). The value of + vector (Ctr _FrmR → Ctr _FrmN )) / 2 is set in the motion vector MvR (step S405).

On the other hand, when it is determined in step S86 that the value of FrmN is not "1" or more and "N" or less (step S86: NO), the vector (Ctr _FrmP → Ctr _FrmR ) is set in the motion vector MvR (step S406). ..

Further, when it is determined in step S84 that the value of FrmP is not "1" or more and "N" or less (step S84: NO), it is determined whether the value of FrmN is "1" or more and "N" or less (step S90). ), When it is "1" or more and "N" or less (step S90: YES), the value of the vector (Ctr _FrmR → Ctr _FrmN ) is set in the motion vector MvR (step S407).

After performing any one of the processes of steps S405, S406, and S407, the angle and length of MvR formed by MvR with the x-axis are set to the blur angle (BaR) and length (MlR) (step S93). ). Then, the obtained blur angle (BaR) and length (MlR) are stored in the blur PSF parameter holding memory 17B (step S94). Next, the length of the blur is estimated from the image data ImgurR and the blur angle BaR (BlR) (step S201). Then, the ratio of the estimated blur length BlR and the vector length MlR is calculated (B2MlR) (step S202).

After that, "1" is added to the value of FrmR, and the value is set to FrmR (step S95). Then, it is determined whether or not FrmR is "N" or less (step S96), and if FrmR is "N" or less (step S96: YES), the process returns to step S83. On the other hand, when FrmR is not "N" or less (step S96: NO), the representative value of the ratio B2MlR calculated in all frames (R = 1 to N) is calculated (B2Ml) (step S203). After that, “1” is set in FrmR (step S204). Next, the value obtained by multiplying the blur length MlR stored in the blur PSF parameter holding memory 17B by B2Ml is stored as the blur length BlR (step S205). Next, "1" is added to the value of FrmR, the value is set to FrmR (step S206), and it is determined whether or not FrmR is "N" or less (step S207). When FrmR is "N" or less (step S207: YES), the process returns to step S205. On the other hand, when FrmR is not "N" or less (step S207: NO), the blur PSF estimation process is terminated.

On the other hand, when it is determined in step S90 that the value of FrmN is not "1" or more and "N" or less (step S90: NO), the angle and length of the blur are estimated from the image data Imgur (BaR, BlR) ( Step S208). After that, the value of BlR is set to MlR (step S209), and the process returns to step S94.

(effect)
As described above, according to the present embodiment, when estimating the length of the blur PSF in the blur PSF estimation process, the length of the blur PSF is estimated from one image data, and the estimated length of the blur PSF is estimated. The ratio of the motion vector to the length of the motion vector is calculated for each frame, the ratio is smoothed over all frames, and the length of the blur PSF of each frame is calculated by multiplying the length of each motion vector by the smoothed ratio. .. As a result, the length of the blur PSF can be estimated in consideration of the shutter speed of the camera, so that the blur removal process can be performed using an accurate blur PSF, and a sharp image without ringing noise can be obtained. Obtainable.

The present invention is suitable for use in an image processing apparatus that processes an image taken by a surveillance camera.

1 Image processing device 10 Image data input unit 11 Frame memory 12 Graphical user interface unit 13 Reference frame selection unit 14 Focused subject area selection unit 15 Matching calculation unit 16 Blur PSF estimation unit 17 Memory unit 17A Reference point position holding memory 17B Blur PSF parameter Retention memory 18 Reference frame adjustment unit 19 Reference frame adjustment unit 20 Blur removal unit 21 Reconstruction processing unit 22 Super-resolution image output unit 23 Image generation unit 100 Number plate

Claims (6)

An image processing device that aligns low-resolution images of a plurality of input frames, each having a common partial area, to generate a high-resolution image.
A reference frame selection unit that selects one of the plurality of input frames and uses it as a reference frame,
A partial area selection unit that sets a partial area for the selected reference frame, and
In each of the plurality of input frames other than the reference frame, the center point of the region matching the image data of the partial region of the reference frame is obtained, and the blur angle and length are obtained from the motion vector of the center point, and the blur PSF (Point Spread Function) A blur PSF estimation unit that is stored in memory as a parameter,
The position of the partial region of the reference frame and the blur PSF parameter used when generating the image of one reference frame selected from the plurality of input frames and the high resolution image are displayed, and the displayed portion is displayed. The display content of the image of the reference frame is updated based on the adjustment of the position of the region, the position of the adjusted partial region and the blur PSF parameter are stored in the memory as the parameter of the reference frame, and other than the reference frame . A reference frame adjustment unit that updates the position and blur PSF parameters of the partial areas of multiple input frames and stores them in the memory, respectively.
The image of the reference frame which is one of the input frames other than the reference frame, the position of the partial area of the reference frame, and the blur PSF parameter are displayed as the parameters of the reference frame, and the position of the displayed partial area is adjusted. Based on this, the reference frame adjustment unit that updates the display content of the image of the reference frame and stores the position of the adjusted partial area and the blur PSF parameter in the memory as the parameter of the reference frame.
A blur removing unit that performs processing for removing blur included in a partial image of the input frame using corresponding parameters for each input frame.
An image generator that aligns the low-resolution image of the input frame from which the blur has been removed to generate a high-resolution image, and
An image processing device comprising. The blur PSF estimation unit calculates a motion vector between the input frame and an input frame adjacent to the input frame in the time direction for each of the plurality of input frames, and the blur PSF of the input frame is calculated from the length and direction of the motion vector. Estimate the length and angle of
The reference frame adjusting unit obtains a ratio of the length of the blur PSF of the reference frame estimated by the blur PSF estimation unit to the length of the adjusted PSF of the blur PSF, and obtains the ratio of the input frame other than the reference frame. Multiply the length of the blur PSF by the above ratio,
The image processing apparatus according to claim 1. When estimating the length of the blur PSF, the blur PSF estimation unit estimates the length of the blur PSF from the image data of each of the input frames, and determines the length of the estimated blur PSF and the length of the motion vector. The length of the blur PSF of each frame is calculated by obtaining the ratio of each frame, smoothing the ratio over all frames, and multiplying the length of each motion vector by the smoothed ratio.
The image processing apparatus according to claim 2. The parameter is a plurality of reference point coordinates indicating the position of the partial region included in each input frame.
The reference frame adjusting unit displays the plurality of reference point coordinates of the reference frame, updates the plurality of reference point coordinates of the image of the reference frame based on the adjustment of the displayed plurality of reference point coordinates, and updates the plurality of reference point coordinates. The coordinated plurality of reference point coordinates are stored in the memory.
The image processing apparatus according to any one of claims 1 to 3. The partial area is a license plate included in each of the input frames.
The alignment reference points are adjusted to the positions of the four corners of the license plate included in each input frame.
The image processing apparatus according to any one of claims 1 to 3. The partial area is a facial image and
The position of the partial region is the position of a plurality of facial feature points of the face image included in each input frame.
The reference frame adjusting unit displays the position of the face feature point of the reference frame and the mesh connecting the face feature points, and based on the adjustment of the position of the displayed face feature point, the image of the reference frame. The position of the facial feature point is updated, and the adjusted position of the facial feature point is stored in the memory.
The image processing apparatus according to any one of claims 1 to 3.

Images