JP2009188891A - Image processor, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a higher definition still image by extracting a still image which is closer to a still image actually desired by a user and has quality worth viewing as a still image, when extracting a still image from a moving image. <P>SOLUTION: An image processor includes: an accumulated image sequence managing unit 102 for accumulating input image sequences for the predetermined number of pieces; a forward inter-image analyzer 103 which obtains an image sequence to be analyzed, performs forward inter-image analysis, and generates forward evaluation information; a backward inter-image analyzer 104 which obtains an image sequence to be analyzed, performs backward inter-image analysis, and generates backward evaluation information; a determining unit 105 which obtains forward and backward evaluation information and determines which inter-image analysis is effective; a controller 106 which extracts at least one still image from the accumulated image sequence managing unit 102 based on the result of the determination; and a super-resolving and expanding unit 107 which performs predetermined super-resolving and expanding processing of the extracted still image and generates a high-definition still image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing apparatus for extracting a desired still image from a moving image and performing image processing for generating a still image with higher definition than the still image extracted by super-resolution processing. The present invention relates to an image processing program.

When a desired still image is extracted from a moving image, a predetermined super-resolution process for converting to a higher resolution is performed using a plurality of image frames present before and after the frame position of the desired still image Thus, there are many techniques for increasing the resolution of a desired still image. In order to fully demonstrate the effects of super-resolution processing, it is necessary to find an image of a scene in which the photometric detection level suitable for super-resolution is stable and the white balance detection level is stable from moving images. is there. In order to easily identify an image suitable for such super-resolution processing, it is determined whether a predetermined condition relating to super-resolution processing is satisfied, and a flag corresponding to the determination result is added to each image frame. However, there are some that determine the suitability of the super-resolution processing based on the presence or absence of this flag. (For example, refer to Patent Document 1).
Refer to JP 2007-151080 A

  By the way, when there is a still image acquisition request from the user during moving image shooting, the still image desired by the user is not necessarily the time at which the user requested (the shooting device acquires a still image from the user). It is not necessarily the past image frame at the same time or closest to the time when the request was detected. In other words, when shooting a moving image, the user recognizes a scene or subject that the user likes and then makes a still image acquisition request such as pressing the shutter button. Due to what happens. Since such a temporal difference has individual differences for each user, it is difficult to uniquely correct the time of the still image acquisition request.

  Similarly, when there is a still image acquisition request from the user during playback of a moving image, the still image desired by the user is not necessarily the time when the user requests it (the playback device receives a request from the user). The image frame is not necessarily the same time or the closest image frame to the time when the still image acquisition request is detected.

  Here, it is considered that super-resolution processing is performed using a plurality of image frames around the still image frame position desired by the user. In the above-described conventional method, flag information corresponding to the suitability of super-resolution processing is added to each image frame of a moving image, and whether or not super-resolution processing is possible is determined based on the flag information. However, flag information indicating that the frame position of the still image desired by the user and the frame positions around the still image are suitable for super-resolution processing are not always added. In such a case, it is necessary to determine in some way whether to search the past side or the future side from the specified frame position. In addition, it is necessary to identify an image frame to which flag information indicating that it is suitable for super-resolution processing in the determined direction is added. However, the image frame specified in this way is not necessarily close to the still image desired by the user.

  Further, in the conventional method, when flags determined to be suitable for the super-resolution processing are continuous, some index is required for determining the number of image frames used for the super-resolution processing. Also, when trying to increase the speed by limiting the number of reference frames to be used, the index that determines which frame should be used is ambiguous, and usually a predetermined range is set and it becomes a processing target. Super-resolution processing is performed according to a predetermined number of frames out of a plurality of frames having a predetermined positional relationship with a still image. Furthermore, the number of image frames used for super-resolution processing is not necessarily better. Therefore, some information on the order of application and the number of applications is required.

  Therefore, when extracting a still image from a moving image, the present invention extracts a still image that is closer to the actually desired still image and has a quality that can be appreciated as a still image. An object is to provide an image processing apparatus, an image processing method, and an image processing program capable of generating a still image.

  To achieve the above object, the image processing apparatus, the image processing method, and the image processing program of the present invention accumulate a predetermined number of input image sequences, specify and supply the analysis target image sequence, and Image sequence is acquired, time analysis is performed between images in the forward direction, and evaluation information is generated based on the analysis between images in the forward direction, while the image sequence to be analyzed is acquired and analysis between images in the reverse direction is performed. And generating evaluation information based on reverse inter-image analysis, based on the evaluation information based on forward inter-image analysis and the evaluation information based on reverse inter-image analysis. In which direction the inter-image analysis is effective, and at least one still image is extracted from the analysis target image sequence accumulated based on the determination result, and the extracted still image is included in the extracted still image. Over the prescribed Generating a high-definition still image performs image enlargement process.

  In the present invention, a predetermined number of input image sequences are accumulated, analysis target image sequences are specified and supplied, analysis between images in the forward direction and reverse direction is performed on the analysis target image sequence, Based on the evaluation information based on the analysis and the evaluation information based on the inter-image analysis in the reverse direction, it is determined whether the inter-image analysis in which direction is effective, and based on the determination result, the super solution is determined from the analysis target image sequence. By extracting a still image suitable for image processing and performing predetermined super-resolution processing on the still image, it is possible to obtain a super-resolution enlarged still image with higher quality than before.

Embodiment 1.
FIG. 1 shows a configuration example of an image processing apparatus to which Embodiment 1 of the present invention is applied. Here, for the sake of simplicity, let us consider a case where a request for obtaining a still image with a higher definition than a still image normally obtained from a user is made during moving image shooting.

  In FIG. 1, the image processing apparatus according to the first embodiment includes at least an accumulated image sequence manager 102, a forward inter-image analyzer 103, a reverse inter-image analyzer 104, a determiner 105, a controller 106, and a super-resolution magnifier. 107.

  The stored image sequence manager 102 has a function of acquiring control information from the controller 106. In accordance with control information from the controller 106, it has a function of sequentially acquiring image frames taken as moving images as the image sequence input 101. It has a function of accumulating a predetermined number of image frames, managing time information at the time of accumulation, and the accumulated image frames. When the number of stored image frames satisfies the predetermined number, when an image frame is newly stored, the oldest image frame in the stored image sequence is discarded and a new image sequence is deleted. It is preferable to have a function for managing the storage state of an image sequence by using a FIFO (First In-First Out) method in which an image frame acquired as an input is added to the last position of the stored image sequence. .

  The accumulated image sequence management unit 102 has a function of managing the state of the image sequence stored based on the accumulated image sequence management information, and stores the accumulated image sequence management information with respect to the controller 106 as necessary. The function of supplying Here, the accumulated image sequence management information grasps the progress of processing, information on the image sequence to be processed next, the frame position of the representative image that is currently an output candidate, and the current processing It is preferable that the information of the hierarchy, the determination information in the forward direction and the reverse direction selected by the determination unit 105, and the like are included. Further, it is preferable to construct the accumulated image sequence management information so that the dependency relationship between the image sequences obtained in the process of the inter-image analysis process and the selection history indicating which image has been selected can be managed in a tree structure. Furthermore, the frame position corresponding to the time of the first still image acquisition request actually requested by the user, and the predetermined time, the time specified in advance by the user, or the time specified from the learning result from this position. The time point that goes back in the past is used as the second still image acquisition request from the user, and the analysis start position or the like with the latest moving image frame as the analysis start position is included at that time. Become. Further, in order to reduce the processing amount of the inter-image analysis, it is further improved if image cropping area information is included.

  The accumulated image sequence manager 102 identifies an image sequence to be supplied based on the control information and the accumulated image sequence management information, and if necessary, the forward image analyzer 103 and the backward image. The inter-analyzer 104 has a function of supplying at least two specified image sequences. Here, for the forward inter-image analyzer 103, an image at a frame position on the past side in time is a reference image, and an image at a frame position on the future side in time is a target image. It has the function to supply as. For the backward image analyzer 104, the image at the future frame position in time is supplied as the reference image, and the image at the past frame position in time is supplied as the target image. It has a function. Here, in order to reduce the processing amount of the inter-image analysis, in accordance with the control information from the controller 106, an image clipping region is specified based on the cropping region information of the image for the supplied image sequence, and the clipping is performed. It is preferable to provide a function to supply the forward direction inter-image analyzer 103 and the reverse direction inter-image analyzer 104 after processing. The accumulated image sequence manager 102 has a function of supplying an image to be output as an image sequence output 107 based on the control information and accumulated image sequence management information. Further, image sequence range information indicating from which range of image sequences in the moving image the representative image that is currently output as a candidate based on the request is extracted from the controller 106. And a function for outputting range information of the image sequence together with the output of the image sequence of the representative image.

  Further, the stored image sequence manager 102 determines the order of the image sequences obtained from the analysis of the tree structure indicating the dependency relationship of the image sequences included in the stored image sequence management information in accordance with the control information from the controller 106. According to the information, the low-resolution image before super-resolution processing, which is the requested number of observation images, is identified, and the requested number of observation images together with the still image that is the super-resolution processing target is super-resolution magnifier 107 has a function to supply to 107. Further, if necessary, it has a function of supplying the super-resolution enlarger 107 with information on the enlargement ratio in the super-resolution processing acquired as control information from the controller 106. Furthermore, the stored image sequence manager 102 has a function of acquiring and managing the super-resolution image generated by the super-resolution enlarger 107 and supplying the stored super-resolution image as necessary.

  The forward image inter-analyzer 103 acquires at least two image sequences to be analyzed from the accumulated image sequence manager 102 as reference images and target images, and performs predetermined processing on the acquired reference images and target images. It has a function of performing inter-image analysis and evaluating similarity between images. In addition, it has a function of supplying similarity information, which is an evaluation result, to the determiner 105.

  The backward image-to-image analyzer 104 acquires at least two image sequences to be analyzed from the accumulated image sequence manager 102 as reference images and target images, and performs predetermined processing on the acquired reference images and target images. It has a function of performing inter-image analysis and evaluating similarity between images. In addition, it has a function of supplying similarity information, which is an evaluation result, to the determiner 105.

  The determination unit 105 acquires forward similarity information and reverse similarity information supplied from the forward image analyzer 103 and the backward image analyzer 104, and the acquired forward similarity information and It has a function of comparing the similarity information in the reverse direction and determining whether the inter-image analysis in the forward direction or the reverse direction was effective. It has a function of supplying determination information, which is a determination result of similarity information, to the controller 106. Here, the determination information may be information that can specify whether the forward direction is selected or the reverse direction is selected. For example, it is desirable that the forward direction and the reverse direction are represented by flag information such as + and-, or 1 and 0, respectively.

  The controller 106 has a function of supplying control information to the accumulated image sequence manager 102. Here, it is preferable that the control information includes at least information regarding control of image sequence input, control of image sequence output, and update control of stored image sequence management information for the accumulated image sequence manager 102. Here, the controller 106 determines the time of the first still image acquisition request that the user actually requested based on the time when the first still image acquisition request that is a direct analysis start request from the user was made. The time that goes back in the past from the time when the first still image acquisition request was made by the amount of time difference from the time including the still image that the user actually desires is the second still image from the user. It has a function of specifying the analysis start position as an acquisition request.

  For example, as shown in FIG. 2, it is assumed that at a certain time during moving image shooting, the user presses the shutter button of the camera in order to acquire a still image. However, the frame position 1401 at the time when the user wanted to acquire a still image and the frame position 1402 at the time when the user actually pressed the shutter button are recognized until the still image is recognized and the shutter button is actually operated. The time difference is not small. In order to make the time difference until this operation as close as possible to the position of 1401 by correcting the arrow 1403 in FIG. 2, the frame position 1402 corresponding to the time of the first still image acquisition request actually requested by the user, that is, the user From the position where the shutter button is actually pressed by the second time from the user, a time point that goes back in the past by a predetermined time, a time designated in advance by the user, or a time specified from the learning result. As a still image acquisition request, an analysis start position 1404 with the latest moving image frame at that time as the analysis start position is obtained.

  In this way, the controller 106 may have a function of specifying the analysis start position. By specifying such an analysis start position 1404 and specifying it by a predetermined inter-image analysis as indicated by reference numeral 1405 in FIG. 2, it is close to the image 1401 that is the position desired by the user, and can be viewed as a still image. An image 1406 with acceptable quality is identified. In other words, by outputting the specified image 1406 as a still image 1410, a still image 1408 obtained as an output corresponding to the frame position 1402 at the time when the first still image acquisition request was made, or an analysis start position 1404 Compared to a still image 1409 obtained by outputting an image corresponding to 1), it is possible to obtain an image with less information, noise, camera shake, motion blur, and the like, and a larger amount of information.

  In addition to such a function, the controller 106 has a function of supplying information relating to the specified analysis start position to the stored image sequence manager 102 as control information. It has a function of acquiring determination information from the determiner 105. Further, it has a function of notifying the stored image sequence manager 102 of the acquired determination information as control information. The controller 106 acquires accumulated image sequence management information from the accumulated image sequence manager 102, grasps the current storage status of the image sequence in the accumulated image sequence manager 102, and determines the operation status of the image processing apparatus of the present invention. It has a function to control. That is, the controller 106 grasps the processing progress status from the accumulated image sequence management information, specifies the image sequence to be processed next, the frame position of the representative image that is the current output candidate, It has a function of grasping the hierarchy, the determination information in the forward direction and the reverse direction selected by the determination unit 105, and the like. Further, the range information of the image sequence indicating from which range of image sequences in the moving image the representative image that is currently an output candidate is specified, and the accumulated image sequence manager 102 as necessary. It has a better structure if it has a function of supplying control information for requesting to output range information of an image sequence together with output of the image sequence. In addition, the controller 106 reduces the processing amount of the inter-image analysis as necessary, when the accumulated image sequence management unit 102 supplies the image sequence based on the cropping area. In order to perform control so as to be supplied after the screen clipping process is performed, it is preferable to have a function of supplying control information with information on the cropping area to the accumulated image sequence manager 102.

  Here, a predetermined area may be set in advance for the cropping area information that is a cut-out area. Further, for example, as in 1501, 1502, and 1503 in FIG. 3, the size and position of the cropping area may be changed in conjunction with camera parameters such as focus. Here, 1501 in FIG. 3 indicates the entire accumulated image, 1502 indicates the cropping area, and 1503 indicates the focus position.

  Further, the relationship between the focus position and the cropping area may be such that the focus position is at the center of the cropping area, such as 1502 and 1503 in FIG. Further, as in 1504, 1505, and 1506 in FIG. 3, the relationship between the focus position and the cropping region may be a predetermined relationship. Here, 1504 in FIG. 3 indicates the entire accumulated image, 1505 indicates the cropping area, and 1506 indicates the focus position.

  The controller 106 acquires the stored image sequence management information from the stored image sequence manager 102, performs a predetermined tree structure analysis process on the tree structure indicating the dependency relationship between the image sequences, and obtains the obtained image sequence order information. It has a function of supplying to the stored image sequence manager 102. Here, the image sequence order information may be added to the accumulated image sequence management information and then supplied to the accumulated image sequence manager 102. Further, the controller 106 has a function of supplying control information for controlling a predetermined super-resolution enlargement process in the super-resolution enlarger 107 to the accumulated image sequence manager 102. Here, the control information may include information on the number of low-resolution images before the super-resolution processing, which is an observation image used in the super-resolution processing, and information on the enlargement ratio in the super-resolution processing. . The information regarding the number of observation images may be a predetermined number or may be based on a request from the user. In addition, the calculation amount of the image processing apparatus of the present embodiment is monitored, the number of observation images is adjusted according to the increase or decrease of the calculation load, the time required for the super-resolution enlargement process is measured, and the processing is performed within a predetermined time. By adjusting the number of observation images so that the process ends, a better configuration can be obtained. The information regarding the enlargement ratio in the super-resolution processing may be predetermined enlargement ratio information or may be based on a request from the user.

  The super-resolution magnifier 107 acquires a still image that is a target for super-resolution processing and a low-resolution image before super-resolution processing that is at least one observation image from the accumulated image sequence manager 102, A predetermined super-resolution enlargement process is performed using the acquired still image and the observed image, and a super-resolution image that is a higher-definition still image than the acquired still image is generated and supplied to the accumulated image sequence manager 102 It has the function to do.

  Next, the operation of the configuration of the image processing apparatus shown in FIG. 1 will be described with reference to FIG.

  FIG. 4 is a flowchart showing the operation of the image processing apparatus shown in FIG.

  First, the controller 106 supplies control information for starting accumulation of an image sequence to the accumulated image sequence manager 102. When shooting of a moving image is started, an image sequence sequentially captured as an image sequence input 101 is supplied to the accumulated image sequence manager 102.

  The accumulated image sequence manager 102 acquires each image frame, which is an image sequence supplied as the image sequence input 101, and accumulates it until a predetermined number is reached. When the number exceeds the predetermined number, the image frame stored at the oldest time in the FIFO method is discarded, and the newly acquired image frame is added at the end. Through such processing, the input image sequence is accumulated (step S101).

  The controller 106 monitors the presence or absence of an analysis start request from the user, for example, caused by an operation such as pressing the shutter button when the user wants to record a still image while shooting a moving image. If there is an analysis start request from the user (YES in step S102), the process proceeds to step S103. When there is no analysis start request from the user (NO in step S102), the process returns to step S101 and the image sequence is continuously accumulated.

  When the controller 106 confirms that there is an analysis start request from the user, the controller 106 supplies control information for interrupting the accumulation of the image sequence to the accumulated image sequence manager 102, and the accumulated image sequence manager 102 interrupts the accumulation of the image sequence in accordance with the acquired control information, and prepares the analysis process by initializing the accumulated image sequence management information. Further, the controller 106, based on the time when there is a first still image acquisition request that is a direct analysis start request from the user, the time of the first still image acquisition request that the user actually requested, The second still image acquisition from the user is performed for a time that goes back in the past from the time when the first still image acquisition request has been made, by the time difference from the time including the still image that the user actually desires. The analysis start position is specified as a request. The controller 106 supplies the identified analysis start position to the accumulated image sequence manager 102, and the accumulated image sequence manager 102 reflects the acquired analysis start position in the accumulated image sequence management information. Accordingly, the analysis can be started after the analysis start position is brought close to the time position of the still image actually desired by the user.

  Next, the controller 106 acquires accumulated image sequence management information from the accumulated image sequence manager 102 in order to control execution of analysis processing, and specifies an image sequence to be processed next. Control information is supplied to the stored image sequence manager 102 based on the specified image sequence information.

  The accumulated image sequence manager 102 acquires control information regarding the next image sequence to be processed from the controller 106, and specifies at least two image frames to be analyzed as image sequences. The identified image sequence is supplied to the forward inter-image analyzer 103 and the reverse inter-image analyzer 104. When supplying the specified image sequence, the forward inter-image analyzer 103 is configured to supply the image of the frame position on the past side of the image sequence to be supplied as the reference image, and temporally on the future side. An image at the frame position is supplied as a target image. Also, for the backward image analyzer 104, an image at a future frame position in time is a reference image and an image at a past frame position in time is a target image in an image sequence to be supplied. Supply.

  Thereafter, the forward inter-image analyzer 103 and the reverse inter-image analyzer 104 acquire the analysis target image sequence by acquiring the reference image and the target image from the accumulated image sequence management unit 102, respectively (step S103). Subsequent Step S104 and Step S105 are configured to perform a parallel operation, but it is also possible to configure these to perform processing by sequential operation, and the configuration is not particularly limited by these configurations. Be careful.

  When the analysis target image sequence is acquired, the forward inter-image analyzer 103 performs a predetermined forward inter-image analysis on the acquired reference image and target image (step S104), and evaluates the similarity between the images. Then, the similarity information that is the evaluation result is supplied to the determination unit 105.

  Similarly, the backward inter-image analyzer 104 performs a predetermined forward inter-image analysis on the acquired reference image and target image (step S104), evaluates the similarity between the images, and determines the similarity as an evaluation result. The degree information is supplied to the determination unit 105.

  Thereafter, the determiner 105 acquires the forward similarity information and the backward similarity information from the forward inter-image analyzer 103 and the backward inter-image analyzer 104, compares the similarity information, and forwards the similarity information. It is determined whether the inter-image analysis in the direction or the reverse direction was effective. By this determination, an effective image frame is selected from the image sequence that has been analyzed (step S106), and determination information that is a determination result of similarity information is supplied to the controller 106.

  Based on the determination information acquired from the determiner 105, the controller 106 supplies determination information of the image sequence that has been the analysis target to the accumulated image sequence manager 102 as control information.

  Based on the control information related to the acquired determination information, the stored image sequence manager 102 reflects the determination information on the stored image sequence management information, and updates the position information of the image that is currently the output target (step S107). ). Thereafter, in order to prepare for the next analysis, the management information of the image sequence is updated by updating the accumulated image sequence management information (step S108).

  Next, the controller 106 acquires accumulated image sequence management information from the accumulated image sequence manager 102, and determines whether or not there is a next analysis target image sequence (step S109). If the next analysis target image sequence exists (YES in step S109), the process proceeds to step S103, and the controller 106 continues the analysis process.

  On the other hand, when the next analysis target image sequence does not exist (NO in step S109), the controller 106 specifies the frame position of the representative image that is an output candidate from the accumulated image sequence management information, and displays the representative image sequence. The range information of the image sequence indicating which range of the image sequence in the moving image is extracted is specified. The image sequence belonging to the range information of the image sequence is used as an observation image of the super-resolution processing performed by the super-resolution enlarger 107. In order to determine the priority order in advance for the image sequence used as the observation image, the controller 106 determines a predetermined structure with respect to the tree structure indicating the dependency of the image sequence included in the accumulated image sequence management information. A predetermined tree structure analysis process is performed based on the search mode, the cost of the path constituting the tree structure, and the cost of the node constituting the tree structure (step S110), and the generated rank information is reflected in the accumulated image sequence management information.

  Thereafter, the controller 106 supplies control information for controlling a predetermined super-resolution enlargement process in the super-resolution enlarger 107 to the accumulated image sequence manager 102. This control information may include information on the number of observation images and information on the enlargement ratio in the super-resolution processing.

  Next, the accumulated image sequence manager 102 obtains control information related to the super-resolution enlargement processing from the controller 106, and is obtained from the analysis of the tree structure indicating the dependency relationship of the image sequences included in the accumulated image sequence management information. The requested number of observation images (low-resolution image before super-resolution processing) is identified according to the order information of the image sequence, and the requested number of observation images together with the still image that is the super-resolution processing target Is supplied to the super-resolution magnifier 107. Further, if necessary, information on the enlargement ratio in the super-resolution processing is also supplied to the super-resolution enlarger 107.

  Thereafter, the super-resolution magnifier 107 acquires a still image that is a super-resolution processing target and at least one observation image from the accumulated image sequence manager 102. Further, if information regarding the enlargement ratio in the super-resolution processing is necessary, the information may be acquired from the accumulated image sequence manager 102. The super-resolution enlarger 107 performs a predetermined super-resolution enlargement process using the acquired still image and observation image (step S111), and a super-resolution image that is a still image having a higher resolution than the acquired still image. Is supplied to the stored image sequence manager 102.

  When the super-resolution enlargement process is completed, the controller 106 supplies control information for controlling image sequence output to the accumulated image sequence manager 102 in order to output the generated super-resolution image. The image sequence manager 102 acquires control information related to image output from the controller 106, thereby specifying an image to be output and outputting the image (step S110). Also, the range information of the image sequence is output as necessary. Here, the super-resolution image that has been increased in resolution by the super-resolution processing is specified as an image to be output, and the image is output.

  By performing the image processing according to the present invention based on the steps as described above, there is a still image actually desired by the user as necessary from the time when a higher resolution still image acquisition request is received from the user. A still image with a quality that can be appreciated as a still image by correcting the deviation to the time, specifying the analysis start position, performing analysis processing between predetermined images, and being closer to the actually desired still image by the user It is possible to generate a high-resolution super-resolution image by performing super-resolution processing after extracting.

  Next, a predetermined tree structure analysis process in this apparatus will be described with reference to FIGS.

  FIG. 5 is a flowchart showing a detailed processing procedure of a predetermined tree structure analysis process executed by the control unit 106 in step 110 of FIG.

  That is, first, the control unit 106 configures a tree structure indicating the dependency relationship between the image sequences, for example, as illustrated in FIG. 6 by inter-image analysis (step S301).

  When configuring this tree structure, first, at least one node is assigned to each image. Here, since this is the first time, each node is assigned as a node belonging to the 0th hierarchy. The image selected in the process of the inter-image analysis process is a representative image that becomes the next output candidate, and becomes an analysis target in the next hierarchy. Therefore, a new node is newly added at a position lower by one layer at the same frame position, and this node is associated with a node having a dependency higher by one layer by a path.

  For example, when attention is paid to the nodes N00 and N01, these nodes belong to the 0th hierarchy. When the image at frame position 1 is selected by the inter-image analysis, a node N11 that is one level lower at the same frame position as the node N01 corresponding to frame position 1 is added.

  The node N11 assumes that there is a dependency between the nodes N00 and N01, and associates the nodes N11 and N00 and the nodes N11 and N01 with paths P00 and P01, respectively. A tree structure is configured by performing such association until a still image to be super-resolution processed is specified by inter-image analysis.

  In FIG. 6, since the image at the frame position 3 is specified as the super-resolution processing target, the node N33 belonging to the frame position 3 and −3 hierarchy is a reference node that is the apex of the tree structure. Although the configuration of the tree structure as described above is performed in step S301 in FIG. 5, the configuration processing of the tree structure may be configured in advance in the process of inter-image analysis.

  When the tree structure indicating the dependency of the image sequence is determined, the control unit 106 classifies the intermediate node and the candidate node for each node constituting the tree structure indicating the dependency of the image sequence (step S302). In FIG. 6, nodes N23, N13, N14, N01, N03, N04, and N07, which are indicated by gray squares, are intermediate nodes, and nodes N24, N11, N17, which are indicated by white squares. N00, N02, N05, and N06 are set as candidate nodes. In this node classification, attention is paid to nodes belonging to the same frame position, and the node having the lowest hierarchy is classified as a candidate node. A node that belongs to the same frame position as this candidate node and is higher than the hierarchy of candidate nodes is classified as an intermediate node.

  Thereafter, the control unit 106 specifies a tree structure search mode (step S303).

  FIG. 7 shows an example of a tree structure search mode.

  Here, as the search mode, for example, at least a mode as shown in FIG. 7 may be provided. In FIG. 7, search modes are classified by type and auxiliary condition. The type represents what kind of criteria is used for the search when the tree structure is searched. The auxiliary condition is for determining a search rule based on a finer criterion in the type condition.

  In FIG. 7, the search mode 0 is a mode in which the type is a distance priority and the auxiliary condition is a search with a lower hierarchy priority. Here, distance priority refers to a node having a short inter-reference frame distance, which is a distance between the reference frame position and the frame position to which the search target node belongs, with the frame position to which the reference node belongs as a reference frame position. It means to search with priority. Further, the lower hierarchy priority means that when there are a plurality of nodes having the same distance between the reference frames, a search with a lower hierarchy to which the node belongs is preferentially searched.

  The search mode 1 is a mode in which the search is performed with the priority given to the distance and the auxiliary condition given the higher hierarchy. Here, higher hierarchy priority means that when there are a plurality of nodes having the same distance between the reference frames, a higher hierarchy to which the node belongs is preferentially searched.

  Search mode 2 is a mode in which search is performed with priority given to hierarchy and priority given to short distance priority. Here, hierarchy priority means preferentially searching for a node having a short distance between the reference hierarchy and the hierarchy to which the node belongs, with the hierarchy to which the reference node belongs as the reference hierarchy. The short distance priority means that when there are a plurality of nodes having the same inter-layer distance from the reference layer, a node having a short reference frame distance is preferentially searched.

  Search mode 3 is a mode in which the type is hierarchical priority and the auxiliary condition is a long distance priority search. Here, the long distance priority means that when there are a plurality of nodes having the same inter-layer distance from the reference layer, a node having a long reference inter-frame distance is preferentially searched.

  The search mode 4 is a mode in which a search is performed when the type is cost priority. Here, the cost is a value indicating the ease of selection of a path or a node, and it is easy to select if this value is small, and difficult to select if it is large. When the type is cost-first, it means that attention is paid to a route from the node to the reference node, and a path in the route and a node having a small total cost of the node are preferentially searched.

  The search mode 5 is a mode in which search is performed based on the fact that the type is priority on limited cost and the auxiliary condition is d. Here, d is a value for limiting the search range, and represents a reference interframe distance. In addition, with limited cost priority, when searching with cost priority, a search is made for a node whose distance from the reference frame position is within the range up to d, and then a range up to a distance greater than d is searched. Means that.

  Search mode 6 is a mode in which search is performed based on the type having N-stage limited cost priority and the auxiliary conditions being d1, d2,..., DN. Here, d1, d2,..., DN are values for limiting the search range to N stages, each representing a reference interframe distance. In addition, N-level restricted cost priority means that when searching with cost priority, a search is made for a node whose distance from the reference frame position is in the range up to d1, and then included in the range from d1 to d2. In the same manner, the search range is sequentially expanded to search for nodes included in the range from d2 to d3, d4 to d5,..., DN−1 and larger than dN. This means searching for a range up to a large distance.

  Search mode 7 is a mode in which search is performed with priority given to search by type and priority given to distance 0 as auxiliary conditions. Here, “distance 0 priority” means that a path having a distance between frames of 0 is preferentially searched.

  The search mode 8 is a mode in which the search is performed with the search priority given to the type and the non-distance priority given to the auxiliary condition. Here, the distance non-zero priority means that a path whose interframe distance is not 0 is preferentially searched.

  After specifying the search mode as described above, the control unit 106 specifies the node at the search start position and sets it as the reference node (step S304).

  Here, the node of the search start position is specified based on the frame position to which the super-resolution target image specified by the inter-image analysis of this embodiment belongs, the type of search mode, and the auxiliary condition, and is set as the reference node. Good. For example, in the search mode 0, since the distance priority and the lower hierarchy priority are given, it is desirable to set the node N33 as a reference node. In search mode 1, since distance priority and higher hierarchy priority are given, it is desirable to make node N03 the reference node. In other search modes, it is desirable to use the node N33 as a reference node.

  Next, the control unit 106 calculates the cost of the path according to the search mode (step S305).

  FIG. 8 is a diagram illustrating an example of a path cost according to the search mode.

  Here, for example, as shown in FIG. 8, the path cost is calculated based on a predetermined distance cost coefficient D and a hierarchy cost coefficient L corresponding to the search mode according to the hierarchy and the interframe distance. can do.

  FIG. 9 is a diagram illustrating another example of the path cost according to the search mode.

  For example, as shown in FIG. 9, the distance cost coefficient and the hierarchy cost coefficient are different distance cost coefficients D0, D1, D2, D3,... The cost coefficients L0, L1, L2,... May be specified by the search mode, and the path cost may be calculated.

  After that, the path from the reference node to each node is identified based on the cost of each path and the cost of each node, and the path and node costs included in this path are summed up to obtain the path from the reference node to each node. Cost is calculated (step S306).

  Once the cost to each node is obtained, the ranking process is performed by prioritizing each candidate node while searching the tree structure based on the path and the cost of the node, starting from the reference node. (Step S307), the tree structure analysis process ends when the priority order is determined for all candidate nodes.

  Next, an example of ranking processing in search modes 0 and 1 will be described with reference to FIG.

  In the search mode 0, since the auxiliary condition is lower layer priority, N33 is set as the reference node. Further, FIG. 8 is applied to simplify the description. Here, it is assumed that the distance cost coefficient D = 2, the hierarchical cost coefficient L = 1, and the cost of each node itself is 0.

  FIG. 11 is a diagram showing which frame distance and hierarchy each path belongs to in the tree structure of FIG.

  It is assumed that the cost from the reference node to each node is calculated using FIG. 11 and FIG.

  FIG. 12 is a diagram showing to which reference frame distance each node belongs and which hierarchy each node belongs to.

  In search mode 0, priority is given to a node having a lower hierarchy from among nodes having a short reference frame distance in FIG. That is, since the start position is N33 and (N23), (N13), and (N03) are intermediate nodes, N24 is the first node. Thereafter, ranking is performed in the order of N02, N11, and N05. Thereafter, ranking is performed for nodes having the same distance between the reference frames, such as N00 and N06. Here, the present invention may be configured such that images belonging to both nodes are used as observation images for super-resolution processing by assigning the same ranking to such nodes. In addition, the present invention can be configured so that these nodes are ranked in ascending order of cost. Here, since the cost of N00 is smaller, the order is N00 and N06. Thereafter, when the order of N17 is determined, the order of candidate nodes is determined, and the ranking process is completed.

  In the search mode 1, since the auxiliary condition is higher-order priority, N03 is set as the reference node. Further, FIG. 8 is applied to simplify the description. Here, it is assumed that the distance cost coefficient D = 2, the hierarchical cost coefficient L = 1, and the cost of each node itself is 0. In search mode 1, priority is given to a node having a higher hierarchy from among the nodes having a short distance between the reference frames in FIG. That is, since the start position is (N03) and (N13) and (N23) are intermediate nodes, N33 is the first node, but the image at the frame position to which the node of N33 belongs is a super-resolution target. Therefore, the next node is searched instead of the first one because it is necessary to exclude it from the observed image.

  The corresponding node is N02, (N04). Since (N04) is an intermediate node, N02 is the first node. Thereafter, ranking is performed in the order of N24, N05, and N11. Thereafter, ranking is performed for nodes having the same distance between the reference frames, such as N00 and N06. Here, the present invention may be configured such that images belonging to both nodes are used as observation images for super-resolution processing by assigning the same ranking to such nodes. In addition, the present invention can be configured so that these nodes are ranked in ascending order of cost. Here, since the cost of N00 is smaller, the order is N00 and N06. Thereafter, when the order of N17 is determined, the order of candidate nodes is determined, and the ranking process is completed.

  Next, an example of the ranking processing in search modes 2 and 3 will be shown using FIG.

  In the search mode 2, N33 is set as the reference node. Further, FIG. 8 is applied to simplify the description. Here, it is assumed that the distance cost coefficient D = 2, the hierarchical cost coefficient L = 1, and the cost of each node itself is 0. In addition, it is assumed that the cost from the reference node to each node is calculated using FIGS. 11 and 8.

  FIG. 14 is a diagram showing to which hierarchy each node belongs and to which reference interframe distance.

  In search mode 2, priority is given to a node having a short distance between the reference frames among the nodes having a lower hierarchy in FIG. That is, since the start position is N33 and (N23) is an intermediate node, N24 is the first node. Thereafter, ranking is performed in the order of N11, N17, N02, and N05. Thereafter, ranking is performed for nodes having the same hierarchy, such as N00 and N06. Here, the present invention may be configured such that images belonging to both nodes are used as observation images for super-resolution processing by assigning the same ranking to such nodes. In addition, the present invention can be configured so that these nodes are ranked in ascending order of cost. Here, since the cost of N00 is smaller, the order is N00 and N06. At this point, the ranks of candidate nodes are determined, and the ranking process is completed.

  In search mode 3, N33 is set as the reference node. Further, FIG. 8 is applied to simplify the description. Here, it is assumed that the distance cost coefficient D = 2, the hierarchical cost coefficient L = 1, and the cost of each node itself is 0. In addition, it is assumed that the cost from the reference node to each node is calculated using FIGS. 11 and 8. In search mode 3, priority is given to a node having a long reference inter-frame distance among the nodes having a lower hierarchy in FIG. That is, the start position is N33, and N24 is the first node. Thereafter, ranking is performed in the order of N17 and N11. Thereafter, ranking is performed for nodes having the same hierarchy, such as N00 and N06. Here, the present invention may be configured such that images belonging to both nodes are used as observation images for super-resolution processing by assigning the same ranking to such nodes. In addition, the present invention can be configured so that these nodes are ranked in ascending order of cost. Here, since the cost of N00 is smaller, the order is N00 and N06. Thereafter, when the rankings of N05 and N02 are established, the ranking of candidate nodes is established, and the ranking processing is completed.

  Next, an example of ranking processing in search modes 4, 5, and 6 will be shown using FIG.

  In search mode 4, N33 is set as the reference node. Further, FIG. 8 is applied to simplify the description. Here, it is assumed that the distance cost coefficient D = 2, the hierarchical cost coefficient L = 1, and the cost of each node itself is 0. FIG. 11 shows to which frame distance and hierarchy each path belongs in the tree structure of FIG. It is assumed that the cost from the reference node to each node is calculated using FIG. 11 and FIG. By searching a tree structure in which the cost of each node is determined, the cost of each candidate node can be specified, and the present invention can be configured so as to be ranked in ascending order of cost. Since the cost of the candidate nodes is N00 = 9, N02 = 5, N05 = 7, N06 = 13, N11 = 6, N17 = 10, N24 = 3, the priority order of each node is N24, N02, N11. , N05, N00, N17, N06. By ranking in this way, the ranking of candidate nodes is determined, and the ranking process is completed.

  In the case of the search mode 5, since the type is a mode in which the search is based on the limited cost priority and the auxiliary condition is d, the auxiliary condition d = 1 is assumed here. That is, as indicated by a range 1 in FIG. 15, a node included in a range where the distance from the reference frame position is up to d = 1 is searched, and then a range up to a distance greater than d is searched. The process until the cost of the candidate node is specified is the same as in the search mode 4, and the description is omitted. Since the costs of the candidate nodes included in the range 1 in FIG. 15 are N02 = 5 and N24 = 3, ranking is performed as N24 and N02. Thereafter, the costs of the candidate nodes included in the distance larger than the range 1 in FIG. 15 are N00 = 9, N05 = 7, N06 = 13, N11 = 6, and N17 = 10, and thus N11, N05, N00, and N17. , N06, the ranking of candidate nodes is determined, and the ranking process is completed.

  In the case of the search mode 6, since the type is a mode in which priority is given to N-stage limited cost and the auxiliary conditions are d1, d2,..., DN, the auxiliary condition is temporarily assumed here. d1 = 1 and d2 = 3. That is, as shown by range 1 and range 2 in FIG. 15, a node included in the range from the reference frame position to the distance d = 1 is searched, and thereafter, the node included in the range greater than d1 and d2 Then, a range up to a distance larger than d2 is searched. The process until the cost of the candidate node is specified is the same as in the search mode 4, and the description is omitted. Since the costs of the candidate nodes included in the range 1 in FIG. 15 are N02 = 5 and N24 = 3, ranking is performed as N24 and N02. After that, the costs of the candidate nodes that are larger than the range 1 in FIG. 15 and included in the distance of the range 2 are N00 = 9, N05 = 7, N06 = 13, and N11 = 6, and thus N11, N05, N00, and N06. The ranking is performed as follows. Thereafter, since the only candidate node included in the distance larger than the range 2 in FIG. 15 is N17, the ranking of the candidate nodes is determined by ranking and the ranking process is completed.

  Next, an example of the ranking process in the search mode 7 is shown using FIG.

  In the search mode 7, the type is a search priority, and the auxiliary condition is a mode in which a search is performed with a distance 0 priority. That is, a search is performed preferentially for a path having a distance between frames of 0, and ranking is performed for candidate nodes that have passed through the search path for the first time. Note that even if the passing node is a candidate node, ranking is not performed if it is the second or subsequent pass.

  In the example of FIG. 16, assuming that the start position of the all search route is N33, (N23) → (N13) → (N03) → (N13) → N02 → (N13) → (N23) → N11 → (N01) → <N11> → N00 → <N11> → (N23) → (N33) → N24 → (N14) → (N04) → (N14) → N05 → (N14) → <N24> → N17 → (N07) → <N17 > → N06 → <N17> → <N24> → (N33). Here, (Nxx) represents an intermediate node, and <Nxx> represents a candidate node passed after the second time. Here, N33 is also expressed as an intermediate node. The search end condition is determined by the absence of a path that has not passed through N33, which is the node at the start position. By performing the search as described above, the candidate nodes are ranked by ordering the candidate nodes such as N02, N11, N00, N24, N05, N17, and N06, and the ranking process is completed.

  Next, an example of the ranking process in search mode 8 will be described with reference to FIG.

  In the search mode 8, the type is search priority, and the auxiliary condition is a mode in which search is performed with distance non-zero priority. That is, a search is made with priority given to a path whose inter-frame distance is not 0, and ranking is performed for candidate nodes that have passed for the first time in the search route. Note that even if the passing node is a candidate node, ranking is not performed if it is the second or subsequent pass. In the example of FIG. 17, assuming that the start position is N33 in all search routes, N24 → N17 → N06 → <N17> → (N07) → <N17> → <N24> → (N14) → N05 → (N14) → (N04) → (N14) → <N24> → (N33) → (N23) → N11 → N00 → <N11> → (N01) → <N11> → (N23) → (N13) → N02 → (N13) → (N03) → (N13) → (N23) → (N33). Here, (Nxx) represents an intermediate node, and <Nxx> represents a candidate node passed after the second time. Here, N33 is also expressed as an intermediate node. The search end condition is determined by the absence of a path that has not passed through N33, which is the node at the start position. By performing the search as described above, the candidate nodes are ranked by ordering the candidate nodes such as N24, N17, N06, N05, N11, N00, and N02, and the ranking process is completed.

  Next, a predetermined super-resolution enlargement process by the super-resolution enlarger 107 of this apparatus will be described.

  In general, high resolution of an image by super-resolution processing is performed by estimating one high resolution image from a plurality of low resolution images having relatively high correlation between images, such as a positional shift. It has been reported in many studies in recent years. For example, “Sung C.P., Min K.P. Co-author, Super-Resolution Image Reconstruction: A Technical Overview, IEEE SignalProc. Magazine, Vol. 26, No. 3, p.21-36, 2003”.

  Also, "BCTom and AKKatsaggelos, Reconstruction of a high-resolution image by simultaneous registration, restoration, and interpolation of low-resolution images, Proc. IEEE Int. Conf. Image Processing, Volume 2, p.539-542. The ML (Maximum-likelihood) method disclosed in 1995, has been proposed. In the ML method, a square error between a pixel value of a low-resolution image estimated from a high-resolution image and a pixel value actually observed is used as an evaluation function, and a high-resolution image that minimizes the evaluation function is used as an estimation image. . This method is a method for performing super-resolution processing based on the principle of maximum likelihood estimation.

  Also, MAP (Maximum A Posterior) disclosed in “RR Schulz, RL Stevenson, Extraction of high-resolution frames from video sequences, IEEE Trans. Image Processing, Vol. 5, p. 996-1011, 1996”. A law has been proposed. In the MAP method, a high-resolution image is estimated so as to minimize an evaluation function obtained by adding probability information of a high-resolution image to a square error. This method is a super-resolution processing method that estimates a high-resolution image as an optimization problem that maximizes the posterior probability using some foresight information for the high-resolution image.

  Furthermore, "H. Stark, P. Oskoui, High resolution image recovery from image-plane arrays, using convex projections, J.Opt. Soc. Am. A, Vol. 6, p.1715-1726, 1989" A disclosed POCS (Projection Onto Convex Sets) method has been proposed. This POCS method is a super-resolution processing method for obtaining a high-resolution image by creating simultaneous equations for pixel values of a high-resolution image and a low-resolution image and sequentially solving the equations.

  Here, in the present embodiment, as the predetermined super-resolution enlargement process, for example, the super-resolution process as described above is applied, and the priority order when using the inter-image analysis process and the tree structure analysis process is used. The images of the present invention are used so that a predetermined number of observation images with high priority are added in order from the observation image with the highest priority, or a required number is used as a low-resolution observation image used in the super-resolution enlargement process. A processing device may be configured.

  Hereinafter, the detailed configuration and operation of the super-resolution magnifier 107 according to the present embodiment will be described with reference to FIGS. 18, 19, and 20.

  18 is a block diagram showing a detailed configuration example of the super-resolution expander 107 of the present embodiment shown in FIG. 1, and FIG. 19 is a conceptual diagram showing the state of super-resolution processing by the super-resolution expander 107. FIG. 20 is a flowchart showing an example of super-resolution processing by the super-resolution magnifier 107.

  In FIG. 18, the super-resolution enlarger 107 according to the present embodiment illustrated in FIG. 1 includes a position aligner 402, an interpolator 403, an estimated image generator 404, and an iterative determiner 405.

  The aligner 402 has a function of acquiring a reference image serving as a reference when performing super-resolution enlargement processing and at least one observation image as shown in FIG. Here, it is assumed that the reference image, the observation image 1 and the observation image 2 are acquired as shown in FIG. The aligner 402 has a function of aligning the acquired reference image and pixel positions with a resolution higher than the resolution of the observation images 1 and 2 and generating a non-uniform high resolution image sampled at unequal intervals. . In addition, it has a function of supplying the generated non-uniform high resolution image to the interpolator 403 and the iterative determiner 405. Here, for example, as shown in FIG. 19B, the generated non-uniform high-resolution image aligns the observed image with reference to each pixel of the reference image, and the correlation with the reference image is It is desirable to arrange at the highest position. In addition, it is desirable that the resolution applied in the alignment is higher than the resolution obtained after the super-resolution enlargement process.

  The interpolator 403 obtains the generated non-uniform high resolution image, performs a predetermined non-uniform interpolation process from the obtained non-uniform high resolution image, generates an interpolated image having a desired high resolution, and estimates the estimated image. It has a function to supply to the generator 404. In addition, the interpolator 403 has a function of acquiring control information for continuing processing from the iterative determiner 405. Here, the control information for continuing the processing may include a non-uniform estimated image generated by the estimated image generator 404. In addition, when the interpolator 403 repeatedly performs processing according to the control information, the non-uniform estimated image is used according to a predetermined update method to update the non-uniform high resolution image acquired from the aligner 402, and It is preferable to have a function of correcting each interpolation pixel.

  The estimated image generator 404 acquires the generated interpolation image, performs a predetermined restoration process in consideration of a point spread function (PSF function) obtained from a predetermined camera model, and performs a predetermined noise removal process if necessary. For example, an estimated image having a desired resolution as shown in FIG. 19C is generated, and resample processing is performed on the estimated image, so that each pixel of the non-uniform high-resolution image is handled. A non-uniform estimation image is generated and supplied to the iterative determination unit 405 repeatedly. Further, the estimated image generator 404 has a function of acquiring control information from the iterative determiner 405 and outputting the generated estimated image having a desired resolution to the outside as a high resolution image according to the control information. .

  The iterative determiner 405 acquires a non-uniform high-resolution image from the aligner 402, acquires a non-uniform estimated image from the estimated image generator 404, and uses each acquired image according to a predetermined determination method. A function of determining whether or not the super-resolution enlargement process needs to be repeated, and supplying control information for continuing the process to the interpolator 403 when the determination is necessary as a result of the determination by a predetermined determination method. Have. In addition, as a result of the determination by the predetermined determination method, when it is not necessary to repeat, it has a function of supplying control information for outputting the result to the estimated image generator 404.

  Here, when it is necessary to perform a higher-speed super-resolution enlargement process, for example, a method for increasing the speed of the super-resolution process as disclosed in Japanese Patent No. 3837575 is used. When applied to the above, a better configuration can be realized.

  Next, the operation of the super-resolution magnifier 107 shown in FIG. 18 will be described using the flowchart of FIG.

  First, the aligner 402 acquires a low-resolution reference image as shown in FIG. 19A and at least one observation image (step S401), and the desired height as shown in FIG. 19B. Each acquired image is aligned with a resolution higher than the resolution (step S402). At that time, after arranging each pixel of the reference image at the corresponding position, each pixel of the observation image is arranged so that the correlation with this reference image is the highest, so that the non-uniformly sampled non-uniformly sampled Generate a resolution image.

  Thereafter, the interpolator 403 acquires the generated non-uniform high-resolution image, and performs pixel interpolation by performing predetermined non-uniform interpolation processing (step S403). Thereby, an interpolation image having a desired high resolution is generated.

  Next, the estimated image generator 404 acquires an interpolated image, performs a predetermined restoration process, and performs a predetermined noise removal process if necessary, thereby obtaining a desired resolution as shown in FIG. An estimated image is generated (step S404). In addition, by performing a resample process on the estimated image, a non-uniform estimated image corresponding to each pixel of the non-uniform high-resolution image is generated.

  Next, the iterative determiner 405 acquires a non-uniform high resolution image from the aligner 402, acquires a non-uniform estimated image from the estimated image generator 404, and uses each acquired image to obtain a predetermined value. It is determined whether it is necessary to repeat the super-resolution enlargement process according to the determination method (step S405).

  If it is determined that the super-resolution enlargement process needs to be repeated (YES in step S405), the repetition determiner 405 supplies control information for continuing the process to the interpolator 403.

  On the other hand, when it is determined that it is not necessary to repeat the super-resolution enlargement process (NO in step S405), the repetition determination unit 405 supplies control information for outputting the result to the estimated image generator 404, The estimated image generator 404 outputs the generated estimated image having the desired resolution as a high-resolution image obtained by the super-resolution enlargement process according to the control information (step S406). Thereby, the super-resolution enlargement process of the first embodiment is completed.

  As described above, according to the image processing apparatus of the first embodiment, a predetermined number of input image sequences are accumulated, the analysis target image sequence is specified and supplied, and the forward direction with respect to the analysis target image sequence is set. Perform reverse image analysis and determine whether the inter-image analysis was effective based on the evaluation information based on the forward image analysis and the evaluation information based on the reverse image analysis. Based on the determination result, a still image suitable for the super-resolution processing is extracted from the analysis target image sequence, and the super-resolution that is at least one observation image from a plurality of continuous image sequences with the still image as a reference. Predetermined super-resolution processing is performed using a low-resolution image before processing, so that when a still image is extracted from a moving image, it is closer to the user's desired still image and is viewed as a still image The quality that can withstand Still image in terms of extracting the can than conventional obtaining a still image subjected to high-quality super-resolution enlargement processing.

  Further, in the first embodiment, the second still image acquisition from the user is performed at a time point that goes back in the past by the time designated in advance by the user from the time of the first still image acquisition request actually requested by the user. By making a request and the latest moving image frame at that time as the analysis start position, it is possible to acquire a content close to what the user originally wanted to acquire as a still image and generate a higher-definition still image. .

  Here, in the first embodiment, when determining the second still image acquisition request, the time of the first still image acquisition request actually requested by the user and the still image actually desired by the user are obtained. Since the time difference from the included time is set, individual differences for each user can be eliminated, and the analysis start position can be brought closer to the time position of the still image actually desired by the user. By analyzing the surroundings of the start position according to the present invention, a still image that is closer to the actually desired still image and that can be appreciated as a still image is extracted, and a higher-definition still image is generated. can do.

  In the first embodiment, when the second still image acquisition request is determined, the time of the first still image acquisition request actually requested by the user and the still image actually desired by the user are included. By learning the time difference from the time for each user, it is possible to eliminate individual differences for each user and bring the analysis start position closer to the time position of the still image actually desired by the user. By analyzing the surroundings of the analysis start position, a still image closer to the actually desired still image and extracted with a quality that can be appreciated as a still image is extracted, and a higher-definition still image is generated. be able to.

  Further, in the first embodiment, in the inter-image analysis process performed using the analysis start position in consideration of a still image acquisition request from the user, two analysis target image sequences of moving images to be analyzed are set as one set. An image sequence of non-uniformly spaced moving images that are not necessarily equally spaced as the analysis processing level is lowered while selecting an image at one frame position of the paired image as the analysis target of the next layer. The image sequence to be analyzed in the moving image is controlled so that the two image sequences to be analyzed are determined as to which frame position image should be analyzed in the next layer. In addition, forward image analysis that evaluates image similarity by prediction from the past to the future and reverse image analysis that evaluates image similarity by prediction of the past direction from the future are performed. past Evaluation of similarity in forward and backward direction for moving image frames, and selection of still image candidates to be extracted is compared with a reference image to be compared with a target value below a certain value. By terminating the process when it is evaluated that the degree of similarity with the image is low, it is possible to output the moving image frame selected when the process is terminated as a still image to be extracted.

  In the first embodiment, the dependency of the image sequence obtained in the still image extraction process is managed by the tree structure, and the tree is determined based on the predetermined search mode and the cost given to the paths and nodes constituting the tree structure. Analyze the structure, order the analysis target image sequence, and based on the order of the analysis target image sequence, together with a still image that identifies a low-resolution image before super-resolution processing, which is a predetermined number of observation images By performing predetermined super-resolution processing, when limiting the number of observation images used in predetermined super-resolution processing as required, it is effective for super-resolution processing from the image sequence that becomes the observation image. The required number of sheets can be supplied in order of priority. As a result, when super-resolution processing is performed, all of the target image sequence is used as an observation image, so that a large amount of computation is not required and high-resolution stillness is achieved by super-resolution processing. When generating an image, it is possible to reduce the influence of limiting the number of observation images.

  As a result of the above, according to the first embodiment, a still image closer to the actually desired still image and having a quality that can be appreciated as a still image is extracted to generate a higher-definition still image. be able to.

Embodiment 2. FIG.
In the second embodiment, details of the forward inter-image analyzer 103 and the reverse inter-image analyzer 104 of the first embodiment shown in FIG. 1 are described in detail, and the forward inter-image analyzer 243 and the reverse inter-image analyzer shown in FIG. It is configured in detail like 244. Other configurations are the same as those of the first embodiment shown in FIG.

  In FIG. 21, the forward inter-image analyzer 243 includes a first image sequence acquisition unit 203, a first ME unit 204, a first MC unit 205, a first A LPF unit 206, a subtractor 207, a first A noise remover 208, and a first B. It has an LPF unit 209, a subtractor 210, a first B noise remover 211, a first L component similarity evaluator 212, and a first H component similarity evaluator 213.

  Similarly, in FIG. 21, the backward inter-image analyzer 244 includes a second image sequence acquisition unit 214, a second ME unit 215, a second MC unit 216, a second A LPF unit 217, a difference unit 218, and a second A noise removal unit 219. , A second B LPF unit 220, a difference unit 221, a second B noise remover 222, a second L component similarity evaluator 223, and a second H component similarity evaluator 224.

  Here, the difference unit 207 in FIG. 21 is replaced by a first A HPF unit, the difference unit 210 is replaced by a first B HPF unit, the difference unit 218 is replaced by a second A HPF unit, and the difference unit 221 is replaced by a second B HPF unit. It does not matter. Both of these are for extracting high frequency components from the supplied image information and generating H component information. These constituent elements in the second embodiment may be constituent elements for generating the H component information, and in the following description, the description will be made assuming that the constituent elements are configured by the differentiator shown in FIG. Further, the first A noise remover 208, the first B noise remover 211, the second A noise remover 219, and the second B noise remover 222 in FIG. 21 do not perform predetermined noise removal processing as necessary, or You may comprise so that it may not have each noise remover.

  Hereinafter, the description regarding the details of this embodiment will be made based on FIG. In addition, the description is omitted when it has the same function as the component of FIG.

  The controller 226 in FIG. 21 supplies a control information for acquiring an image sequence to each of the first image sequence acquiring unit 203 and the second image sequence acquiring unit 214 in order to newly control operations related to the first image sequence acquiring unit 203 and the second image sequence acquiring unit 214 Have

  The accumulated image sequence manager 202 obtains the first image sequence acquisition unit 203 and the second image sequence instead of supplying the image sequence to the forward direction inter-image analyzer 103 and the reverse direction inter-image analyzer 104 in FIG. A function of supplying an image sequence to be processed to the device 214.

  The first image sequence acquisition unit 203 acquires, from the accumulated image sequence management unit 202, a reference image for performing inter-image analysis processing in the forward direction and an image sequence of the target image in accordance with control information from the controller 226. It has a function. That is, in the acquired image sequence, the frame position of the reference image is obtained in the past with respect to the frame position of the target image. A function of supplying the acquired reference image and target image to the first ME unit 204; The first image sequence acquisition unit 203 has a function of supplying a reference image to the first MC unit 205. The acquired target image has a function of supplying the first B LPF unit 209 and the subtractor 210.

  The first ME unit 204 has a function of acquiring a reference image and a target image from the first image sequence acquisition unit 203, performing a predetermined motion estimation, and generating motion vector information. In this predetermined motion estimation, for example, the target image is used in a predetermined manner without duplication, such as used in moving image encoding using correlation between images such as MPEG-1, MPEG-2, MPEG-4, and AVC / SVC. It may be based on a motion estimation method that divides into rectangular regions and identifies the corresponding motion vector information from the reference image. In addition, when the target image is segmented, if the segmentation of a predetermined arbitrary shape is performed and then the motion vector information of the corresponding arbitrary shape region is specified from the reference image, the motion estimation method is used. Further, the configuration is better. The first ME device 204 has a function of supplying the generated motion vector information to the first MC device 205.

  The first MC unit 205 acquires a reference image from the first image sequence acquisition unit 203, acquires motion vector information from the first ME unit 204, and performs predetermined motion compensation based on the acquired reference image and motion vector information. And has a function of generating a predicted image. In this predetermined motion compensation, as in motion estimation, for example, motion such as that used in moving image coding using inter-image correlation such as MPEG-1, MPEG-2, MPEG-4, and AVC / SVC is used. It may be based on a motion compensation method in which a predicted image is generated using an area specified from a reference image using vector information. Also, motion compensation that identifies a corresponding arbitrary shape region from a reference image based on region division information of an arbitrary shape and motion vector information corresponding to the arbitrary shape region, and generates a predicted image using such a region. If it is based on a method, it will become a still better structure. Further, the region division information may be configured to be supplied from the first ME unit 204. Alternatively, the target image is acquired from the first image sequence acquisition unit 203, and the region dividing process having an arbitrary shape is performed again. It may be generated. The first MC unit 205 has a function of supplying the generated predicted image to the first A LPF unit 206 and the difference unit 207.

  The first A LPF unit 206 acquires a predicted image from the first MC unit 205, performs a filtering process on the predicted image using a filter having a predetermined low frequency band pass characteristic, and performs low frequency component information (L Component information). The first A LPF unit 206 has a function of supplying the L component information of the generated predicted image to the difference unit 207 and the first L component similarity evaluator 212.

  The differentiator 207 acquires the predicted image from the first MC unit 205, acquires the L component information of the predicted image from the first A LPF unit 206, and subtracts the L component information of the predicted image from the predicted image to the predicted image. It has a function of generating included high frequency component information (H component information). The subtractor 207 performs a filtering process on the predicted image using a filter having a predetermined high frequency band pass characteristic, and substitutes it with a component having a function of generating H component information of the predicted image. It can also be configured. The differentiator 207 has a function of supplying H component information of the generated predicted image to the first A noise remover 208. Here, the H component information of the generated predicted image may be supplied to the first H component similarity evaluator 213 without causing the first A noise remover 208 to function.

  The first A noise remover 208 acquires the H component information of the predicted image from the differentiator 207, performs a predetermined noise removal process, and then uses the H component information of the predicted image after noise removal as a first H component similarity evaluation. A function of supplying to the device 213. Here, it is desirable that the noise removal processing can remove minute noise included in the H component information with a predetermined range as a threshold.

  The first B LPF unit 209 acquires the target image from the first image sequence acquisition unit 203, performs a filtering process on the target image using a filter having a predetermined low frequency band pass characteristic, and performs L component information on the target image. It has the function to generate. The first B LPF unit 209 has a function of supplying the L component information of the generated target image to the difference unit 210 and the first L component similarity evaluator 212.

  The subtractor 210 acquires the target image from the first image sequence acquisition unit 203, acquires the L component information of the target image from the first B LPF unit 209, and subtracts the L component information of the target image from the target image. A function of generating H component information included in the target image; The subtractor 210 performs a filtering process on the target image using a filter having a predetermined high frequency band pass characteristic, and substitutes it with a component having a function of generating H component information of the target image. It can also be configured. The differentiator 210 has a function of supplying the H component information of the generated target image to the first B noise remover 211. Here, the H component information of the generated target image may be supplied to the first H component similarity evaluator 213 without causing the first B noise remover 211 to function.

  The first B noise remover 211 acquires the H component information of the target image from the subtractor 210, performs a predetermined noise removal process, and then uses the H component information of the target image after noise removal to evaluate the first H component similarity. A function of supplying to the device 213. Here, it is desirable that the noise removal processing can remove minute noise included in the H component information with a predetermined range as a threshold.

  The first L component similarity evaluator 212 acquires the L component information of the predicted image from the first A LPF unit 206, acquires the L component information of the target image from the first B LPF unit 209, and acquires the L component of the acquired predicted image. The information and the L component information of the target image are generated based on a predetermined similarity evaluation method, and the similarity information of the L component information in the forward direction (forward L component similarity information) is supplied to the determiner 225. It has the function to do. Here, as a predetermined similarity evaluation method, for example, the evaluation may be performed based on a root mean squared error (RMSE) of a difference value dL between the L component information of the predicted image and the L component information of the target image. . This is an arithmetic mean obtained by squaring the difference value dL and taking a square root. The greater this value, the lower the similarity.

  Further, when it is necessary to simplify the calculation, the evaluation may be performed by sum of absolute differences (SAD). Also in SAD, the greater the value, the lower the similarity. It is a good configuration if these error evaluation values are used as similarity information. Further, if necessary, evaluation is performed based on RMSE or SAD of the L component information of the predicted image and the L component information of the target image, and the result is supplied to the determiner 225 as forward predicted L component error information and forward target L component error information. By doing so, the configuration becomes even better.

  Here, when evaluation is performed by RMSE or SAD of the L component information of the predicted image and the L component information of the target image, an average image of the L component information of the predicted image and the L component information of the target image is generated. It is more preferable that the difference image between the image and the L component information of the predicted image is evaluated by RMSE or SAD, and the forward predicted L component average error information is generated and supplied to the determiner 225. Similarly, a difference image between the average image and the L component information of the target image is evaluated by RMSE or SAD, and forward target L component average error information is generated and supplied to the determiner 225. It becomes composition.

  Here, an example showing details regarding each component information when generating similarity information of L component information will be described with reference to FIG.

  FIG. 22A shows an example of L component information of a predicted image. In the example shown in FIG. 22A, a circular moving object is included in the image sequence to be analyzed. In FIG. 22A, when a predicted image is generated, motion compensation is performed based on motion vector information, so that a spatial phase difference in the screen with the target image is suppressed. However, in this example, motion blur has occurred in the contour of the object in the reference image, so that the contour is unclear as in the portion 801.

  FIG. 22B shows an example of L component information of the target image. In the example shown in FIG. 22B, the object in the target image has a contour with little blur like the portion 803. When a difference is performed between the L component information of the predicted image and the L component information of the target image, a state as illustrated in FIG.

  The difference image as shown in FIG. 22C is used when an evaluation by RMSE or SAD is performed.

  In order to confirm the transition of the component information in FIGS. 22 (a), 22 (b), and 22 (c), the component information of the same line in the regions 808, 810, and 812 that are the same spatial positions is shown. FIG. 22D, FIG. 22E, and FIG.

  The component information 809 in FIG. 22D is for one line in the region 808 in the L component information of the predicted image, and the component information 811 in FIG. 22E is 1 in the region 810 in the L component information of the target image. The component information 813 in FIG. 22F represents one line in the region 812 in the difference image.

  Here, since the component information 809 in FIG. 22D includes a large amount of motion blur and the like, the signal around the contour is not steep, and the signal component is low overall. In addition, since the component information 811 in FIG. 22E has less motion blur, the signal of the contour portion is relatively steep and the signal component is also relatively high. By performing a difference between such component information 809 and component information 811, component information 813 shown in FIG. 22 (f) is obtained.

  In the component information 813, as shown in FIG. 22F, if the component information is a closer value, the absolute value of the obtained component information of the difference is also reduced. As in this example, if the difference between the reference image and the component information is degraded due to motion compensation failure due to motion blur, the deformation of the moving object or the appearance / disappearance of a new object, the difference The absolute value of the component information increases.

  Therefore, by performing similarity evaluation between the L component information of the predicted image and the L component information of the target image, an in-screen generated due to camera shake, motion blur, deformation of a moving object, appearance / disappearance of a new object, and the like. Enables assessment of global changes. If it is determined in the similarity evaluation of the L component information that the similarity between the images to be analyzed is not so high, the forward prediction L component error information or the forward direction is determined from the L component information of the predicted image. Based on the predicted L component average error information, the L component information of the target image, and the forward target L component error information or the forward target L component average error information, it is important as the next analysis target from the image sequence being analyzed It is desirable to select an image to become.

  The first H component similarity evaluator 213 obtains the H component information of the predicted image after noise removal from the first A noise remover 208 and the H component information of the target image after noise removal from the first B noise remover 211. Based on a predetermined similarity evaluation method, the similarity information of the H component information in the forward direction is obtained based on the H component information of the obtained predicted image after noise removal and the H component information of the target image after noise removal. (Forward H component similarity information) is generated and supplied to the determiner 225.

  Here, as a predetermined similarity evaluation method, for example, the root mean squared error (RMSE) of the difference value dH between the H component information of the predicted image after noise removal and the H component information of the target image after noise removal is used. ) To evaluate. This is an arithmetic mean obtained by squaring the difference value dH and taking a square root. The greater this value, the lower the similarity.

  Further, when it is necessary to simplify the calculation, the evaluation may be performed by sum of absolute differences (SAD). Also in SAD, the greater the value, the lower the similarity. It is a good configuration if these error evaluation values are used as similarity information. If necessary, the H component information of the predicted image and the RMS component or SAD of the H component information of the target image are evaluated and supplied to the determiner 225 as forward predicted H component error information and forward target H component error information. By doing so, the configuration becomes even better.

  Here, when evaluation is performed by RMSE or SAD of the H component information of the predicted image and the H component information of the target image, an average image of the H component information of the predicted image and the H component information of the target image is generated. A difference image between the image and the H component information of the predicted image may be evaluated by RMSE or SAD, and forward predicted H component average error information may be generated and supplied to the determiner 225. Similarly, a difference image between the average image and the H component information of the target image is evaluated by RMSE or SAD, and forward target H component average error information is generated and supplied to the determiner 225. Absent.

  Here, an example showing details regarding each component information when generating similarity information of H component information will be described with reference to FIG.

  FIG. 23A shows an example of H component information of a predicted image. In the example shown in FIG. 23A, a circular moving object is included in the image sequence to be analyzed. In FIG. 23A, when a predicted image is generated, motion compensation is performed based on motion vector information, so that a spatial phase difference in the screen with the target image is suppressed. However, in this example, motion blur has occurred in the contour of the object in the reference image, so that the contour is unclear as in the portion 901.

  FIG. 23B shows an example of H component information of the target image. In the example shown in FIG. 23B, the object in the target image has a contour with little blur like the portion 903. When a difference is performed between the H component information of the predicted image and the H component information of the target image, a state as illustrated in FIG.

  The difference image as shown in FIG. 23C is used when an evaluation by RMSE or SAD is performed. In order to confirm the transition of the component information in FIGS. 23 (a), 23 (b), and 23 (c), the component information of the same line in the regions 908, 910, and 912 that are the same spatial positions is shown. FIG. 23D, FIG. 23E, and FIG.

  The component information 909 in FIG. 23D is for one line in the region 908 in the H component information of the predicted image, and the component information 911 in FIG. 23E is 1 in the region 910 in the H component information of the target image. The component information 913 in FIG. 23F represents one line in the area 912 in the difference image.

  Since the component information 909 in FIG. 23D includes many motion blurs and the like, the signal around the contour is not steep, and the signal component is widely distributed around the contour. In addition, since the component information 911 in FIG. 23E has less motion blur, the signal of the contour portion is relatively steep, the signal component is large, and the signal is concentrated on the contour portion. By performing a difference between such component information 909 and component information 911, component information 913 shown in FIG.

  In the component information 913 shown in FIG. 23 (f), as shown in the figure, if the component information is a closer value, the absolute value of the obtained difference component information is also reduced. As in this example, if the difference between the reference image and the component information is degraded due to motion compensation failure due to motion blur, the deformation of the moving object or the appearance / disappearance of a new object, the difference The absolute value of the component information increases.

  In particular, if the position of the maximum value of the component information 909 and the maximum value of the component information 911 is shifted as in this example, the signal component is not sufficiently canceled when the difference is performed, and the absolute value of the component information of the obtained difference is obtained. The value will be larger. In addition, if the contour portion is unclear due to motion blur or the like, there is a strong tendency to become H component information such as component information 909. In the component information 913 obtained by the difference from such component information, a large value tends to be distributed over a relatively wide range. From such characteristics, it is possible to determine how much camera shake, motion blur, and texture information in the screen changes between images by evaluating the similarity to the H component information. .

  Therefore, by evaluating the similarity between the H component information of the predicted image and the H component information of the target image, the contour portion generated due to camera shake, motion blur, deformation of the moving object, appearance / disappearance of a new object, or the like. This makes it possible to evaluate local changes in the screen, such as displacement and changes in the animal body. If it is determined that the similarity between the images to be analyzed is not very high in the similarity evaluation of the H component information, the forward prediction H component error information or the forward direction is determined from the H component information of the predicted image. Based on the predicted H component average error information, the H component information of the target image, the forward target H component error information, or the forward target H component average error information, the analysis target image sequence is important as the next analysis target. It is desirable to select an image to become. Further, when selecting an important image, selecting an image with clearer contour information or texture information in the screen leads to selecting a higher quality image as a still image. Therefore, for example, when it is necessary to select an image by comparing the forward prediction H component error information and the forward target H component error information, it is preferable that the higher one of these values be selected. It becomes.

  Returning to FIG. 21, the second image sequence acquisition unit 214 has the same function as the first image sequence acquisition unit 203. Further, the second image sequence acquisition unit 214 receives the reference image and the image sequence of the target image for performing the inter-image analysis processing in the reverse direction from the accumulated image sequence management unit 202 according to the control information from the controller 226. Has a function to acquire. In other words, in the acquired image sequence, the frame position of the reference image is a future image with respect to the frame position of the target image. It has a function of supplying the acquired reference image and target image to the second ME device 215. Further, the second image sequence acquisition unit 214 has a function of supplying a reference image to the second MC unit 216. In addition, it has a function of supplying the acquired target image to the second B LPF unit 220 and the differentiator 221.

  The second ME device 215 has the same function as the first ME device 204. The second ME unit 215 has a function of acquiring the reference image and the target image from the second image sequence acquiring unit 214 and generating motion vector information by performing predetermined motion estimation. In addition, it has a function of supplying the generated motion vector information to the second MC device 216.

  The second MC device 216 has a function equivalent to that of the first MC device 205. The second MC unit 216 has a function of acquiring a reference image from the second image sequence acquisition unit 214, acquiring motion vector information from the second ME unit 215, and generating a predicted image by performing predetermined motion compensation. . Further, it has a function of supplying the generated predicted image to the second A LPF unit 217 and the differentiator 218.

  The second A LPF unit 217 has a function equivalent to that of the first A LPF unit 206. The second A LPF unit 217 has a function of acquiring the predicted image from the second MC unit 216 and generating L component information of the predicted image by performing a predetermined filtering process. In addition, it has a function of supplying the generated L component information to the differentiator 218 and the second L component similarity evaluator 223.

  The differencer 218 has a function equivalent to that of the differencer 207. The differentiator 218 has a function of acquiring a predicted image from the second MC unit 216 and acquiring L component information of the predicted image from the second A LPF unit 217. In addition, by subtracting the L component information of the predicted image from the acquired predicted image, the H component information of the predicted image is generated and supplied to the second A noise remover 219. Here, the H component information of the generated predicted image may be supplied to the second H component similarity evaluator 224 without causing the second A noise remover 219 to function.

  The second A noise remover 219 has a function equivalent to that of the first A noise remover 208. After obtaining the H component information of the predicted image from the differentiator 218 and performing a predetermined noise removal process, the H component information of the predicted image after noise removal is supplied to the second H component similarity determination unit 224.

  The second B LPF unit 220 has a function equivalent to that of the first B LPF unit 209. The second B LPF unit 220 has a function of acquiring the target image from the second image sequence acquiring unit 214 and generating L component information of the target image by performing a predetermined filtering process. Further, it has a function of supplying the L component information of the generated target image to the differentiator 221 and the second L component similarity evaluator 223.

  The differentiator 221 has a function equivalent to that of the differentiator 210. The differentiator 221 has a function of acquiring the target image from the second image sequence acquisition unit 214 and acquiring L component information of the target image from the second B LPF unit 220. Further, it has a function of generating the H component information of the target image by subtracting the L component information of the target image from the acquired target image and supplying it to the second B noise remover 222. Here, the H component information of the generated target image may be supplied to the second H component similarity evaluator 223 without causing the second B noise remover 222 to function.

  The second B noise remover 222 has a function equivalent to that of the first B noise remover 211. The second B noise remover 222 acquires the H component information of the target image from the subtractor 221, performs a predetermined noise removal process, and then uses the H component information of the target image after noise removal as a second H component similarity evaluator. 224 is provided.

  The second L component similarity evaluator 223 has a function equivalent to that of the first L component similarity evaluator 212. The second L component similarity evaluator 223 has a function of acquiring the L component information of the predicted image from the second A LPF unit 217 and the L component information of the target image from the second B LPF unit 220. Further, the L component information of the predicted image and the L component information of the target image, based on a predetermined similarity evaluation method, the similarity information of the L component information in the reverse direction (reverse L component similarity information) Is generated and supplied to the determiner 225.

  The second H component similarity evaluator 224 has a function equivalent to that of the first H component similarity evaluator 213. The second H component similarity evaluator 224 obtains the H component information of the predicted image after noise removal from the second A noise remover 219 and the H component information of the target image after noise removal from the second B noise remover 222. Has a function to acquire. Further, the H component information of the predicted image after noise removal and the H component information of the target image after noise removal are based on similarity information of H component information in the reverse direction based on a predetermined similarity evaluation method ( The reverse direction H component similarity information) is generated and supplied to the determiner 225.

  The determiner 225 receives forward L component similarity information from the first L component similarity evaluator 212, forward H component similarity information from the first H component similarity evaluator 213, and reverse direction from the second L component similarity evaluator 223. It has a function of acquiring backward component H similarity information from the L component similarity information and the second H component similarity evaluator 224. In addition, the determiner 225 receives the forward prediction L component error information or the forward prediction L component average error information, the forward target L component error information, or the forward target L from the first L component similarity evaluator 212 as necessary. It has a function to acquire component average error information. Further, as necessary, the determiner 225 receives the forward prediction H component error information or the forward prediction H component average error information, the forward target H component error information, or the forward target H from the first H component similarity evaluator 213. It has a function to acquire component average error information. Further, as necessary, the determiner 225 receives the backward prediction L component error information or the backward prediction L component average error information, the backward target L component error information, or the backward target L from the second L component similarity evaluator 223. It has a function to acquire component average error information. Further, the backward prediction H component error information or the backward prediction H component average error information, the backward target H component error information or the backward target H component average error information is obtained from the second H component similarity evaluator 224 as necessary. Has a function to acquire.

  The determiner 225 compares the acquired forward L component and H component similarity information with the backward L component and H component similarity information, and performs forward and backward image analysis. It has a function of determining which one is effective and supplying determination information which is a determination result of the similarity information to the controller 226.

  Here, for example, the determination method generates the similarity information for each direction by multiplying the L component similarity information and the H component similarity information after multiplying by a predetermined weighting coefficient, and then generates the similarity information. It is preferable that the effective direction is determined by comparing the similarity information in the forward direction and the similarity information in the reverse direction.

  The determination information may be information that can specify whether the forward direction is selected or the reverse direction is selected. For example, it is desirable that the forward direction and the reverse direction are represented by flag information such as + and-, or 1 and 0, respectively. In addition, when comparing the similarity information in the forward direction and the backward direction using the generated similarity information, if it is determined that the similarity is not high compared to a predetermined threshold, it is acquired as necessary. The forward prediction L component error information or the forward prediction L component average error information, the forward target L component error information or the forward target L component average error information, and the forward prediction obtained from the processing on the forward direction side. H component error information or forward prediction H component average error information, forward target H component error information or forward target H component average error information, and reverse prediction L component error information or reverse obtained from processing on the reverse side Direction prediction L component average error information, reverse direction target L component error information or reverse direction target L component average error information, and reverse direction prediction H component error information or reverse direction prediction H component average error information, The direction target H component error information or the reverse direction target H component average error information is multiplied by a predetermined weighting coefficient and then added to generate direction-specific error information or average error information. It is preferable that the effective direction is determined by comparing the error information or average error information of the direction with the error information or average error information of the reverse direction.

  Next, the operation of each component corresponding to the forward direction inter-image analyzer 243 and the reverse direction inter-image analyzer 244 of the second embodiment illustrated in FIG. 21 will be described.

  FIG. 24 is a flowchart for explaining the operation when the inter-image analysis is performed in FIG.

  In the following description, operations related to forward inter-image analysis will be described. Regarding the operation related to the inter-image analysis in the reverse direction, the reference image, which is the image sequence acquired by the second image sequence acquiring unit 214, is different from the target image, but the other processing operations are the same as those in the forward inter-image analysis. Therefore, the description is omitted.

  First, the controller 226 acquires the accumulated image sequence management information from the accumulated image sequence manager 202 and specifies the image sequence to be processed next in order to control the execution of the analysis process. Control information is supplied to the stored image sequence manager 202 based on the specified image sequence information. Also, the control of the operation start is performed by supplying control information to the first image sequence acquisition unit 203 and the second image sequence acquisition unit 214.

  The accumulated image sequence manager 202 acquires control information regarding the next image sequence to be processed from the controller 226, and specifies at least two image frames to be analyzed as image sequences. The identified image sequence is supplied to the first image sequence acquisition unit 203 and the second image sequence acquisition unit 214. When supplying the specified image sequence, the first image sequence acquisition unit 203 that is in the forward direction, among the image sequences to be supplied, the image of the frame position on the past side in time is referred to as the reference image, temporal The image at the future frame position is supplied as the target image. Also, for the second image sequence acquisition unit 214 in the reverse direction, the image at the future frame position in terms of time of the image sequence to be supplied is the reference image, and the image at the frame position of the past side in terms of time. Is supplied as a target image.

  Thereafter, the first image sequence acquisition unit 203 acquires a reference image and a target image from the accumulated image sequence management unit 202 in response to a request from the controller 226 (step S201). The acquired reference image and target image are supplied to the first ME device 204. Further, the acquired reference image is supplied to at least the first MC unit 205. Further, the acquired target image is supplied to the 1B LPF unit 209 and the difference unit 210.

  Next, when the first ME unit 204 acquires the reference image and the target image from the first image sequence acquisition unit 203, motion vector information is generated by performing predetermined motion estimation using these image sequences (step S202). And supplied to the first MC unit 205.

  The first MC unit 205 acquires at least a reference image from the first image sequence acquisition unit 203 and acquires motion vector information from the first ME unit 204. Predicted images are generated by performing predetermined motion compensation using the acquired reference image and motion vector information (step S203), and supplied to the first A LPF unit 206 and the difference unit 207.

  When the generation of the predicted image ends, the first A LPF unit 206 acquires the predicted image from the first MC unit 205. A predetermined filtering process is performed on the acquired predicted image to generate L component information of the predicted image, thereby extracting a low frequency component of the predicted image (step S204). Thereafter, the L component information of the generated predicted image is supplied to the differentiator 207 and the first L component similarity evaluator 212.

  Thereafter, the differentiator 207 acquires the prediction image from the first MC unit 205 and also acquires the L component information of the prediction image from the first A LPF unit 206. The high frequency component of the analysis target image sequence is extracted by subtracting the L component information of the predicted image from the acquired predicted image to generate H component information of the predicted image (step S207). Thereafter, the H component information of the generated predicted image is supplied to the first noise remover 208.

  Next, the first A noise remover 208 acquires the H component information of the predicted image and performs predetermined noise removal (step S208), thereby generating H component information of the predicted image after noise removal. This is supplied to the component similarity evaluator 213.

  In parallel with the process for the predicted image as described above, the following process for the target image is performed.

  First, in the 1B LPF unit 209, the target image is acquired from the first image sequence acquisition unit 203, and L component information of the target image is generated by performing a predetermined filtering process, whereby the low frequency component of the analysis target image sequence is generated. Is extracted (step S205). Thereafter, the generated L component information of the target component is supplied to the subtractor 210 and the first L component similarity evaluator 212.

  Thereafter, the differentiator 210 acquires the target image from the first image sequence acquisition unit 203 and acquires the L component information of the target image from the first B LPF unit 209. By subtracting the L component information of the target image from the acquired target image and generating H component information of the target image, the high frequency component of the analysis target image sequence is extracted (step S209). Thereafter, the H component information of the generated target image is supplied to the 1B noise remover 211.

  Next, the first B noise remover 211 acquires the H component information of the target image and performs predetermined noise removal (step S210), thereby generating H component information of the target image after noise removal, and the first H This is supplied to the component similarity evaluator 213.

  When the respective L component information is generated, the first L component similarity evaluator 212 acquires the L component information of the predicted image from the first A LPF unit 206 and the L component information of the target image from the first B LPF unit 209. And the similarity of the low frequency component is evaluated based on a predetermined similarity evaluation method (step S206). Thereafter, the forward direction L component similarity information, which is the evaluation result, is supplied to the determiner 225.

  Similarly, when each H component information is generated, the first H component similarity evaluator 213 obtains the H component information of the predicted image after the noise removal from the first A noise remover 208 and also removes the first B noise. The H component information of the target image after noise removal is acquired from the device 211, and the similarity of the high frequency component is evaluated based on a predetermined similarity determination method (step S211). Thereafter, the forward direction H component similarity information, which is the evaluation result, is supplied to the determiner 225.

  By performing the inter-image analysis process as described above, similarity information in the forward direction is generated. Similarly, by performing inter-image analysis in the reverse direction, similarity information in the reverse direction can be generated. Based on the forward direction and reverse direction similarity information, the determiner 225 performs a predetermined determination to identify which side of the image sequence being analyzed has higher correlation between images. It is possible to determine which image should be analyzed later.

  Next, in order to describe in more detail a procedure for specifying a representative image before super-resolution as an output candidate by the image processing method of the present invention, a description will be given with reference to FIGS. 25, 26, and 27.

  Reference numeral 501 in FIG. 25 is an example of the state of the image sequence held by the stored image sequence manager 202 in FIG.

  The accumulated image sequence manager 202 includes a frame buffer capable of storing a predetermined number of image sequences as means for accumulating image sequence inputs, and accumulates input image sequences in accordance with the control of the controller 226. . Here, the frame buffer manages the stored image based on the information (frame position information or simply the frame position) regarding the frame position for specifying the storage position in the frame buffer.

  In FIG. 251, for convenience, as the stored time advances, the image corresponding to the analysis start position is set to frame position 0 and the previous image before this image is set to frame position 1. Assume that the numbers of the frame positions are sequentially increased.

  In addition to the management based on the frame position, the stored image is assumed to be associated with time information related to the time when the image was generated. That is, in the accumulated image sequence manager 202, the input image sequence is accumulated together with each time information. In 501 of FIG. 25, first, an image at a frame position corresponding to the analysis start position is set as a representative image candidate.

  In FIG. 25, it is represented by a black frame. Here, the image at frame position 0 is selected as a representative image candidate. Next, the concept of hierarchy is introduced in the time direction in the stored image sequence. For example, it is assumed that the currently stored image sequence is taken at a frame rate of 30 frames per second (30 fps). The hierarchy of the time interval of 30 fps is assumed to be 0 hierarchy here. For such an image sequence, when two images are treated as one set as indicated by 501 in FIG. 25 and one image is expressed at this time interval, it is expressed at a time interval of 15 fps. It will be. Such a hierarchy of time intervals is set to -1. That is, in this case, configuring a set of image sequences so that the frame rate is halved every time one layer is lowered is referred to as a dyadic hierarchical structure in the time direction.

  In the inter-image analysis in the image processing apparatus according to the second embodiment, since it is necessary to specify at least two image sequences to be analyzed, first, with respect to the image sequence held in the frame buffer, − Consider a set of image sequences so as to be one layer. The two image sequences to be analyzed first are frame position 0 and frame position 1 in 502 in FIG. In the case of performing an inter-image analysis in the forward direction in the image sequence to be analyzed, an image at frame position 1 that is a past frame is used as a reference image, and an image at frame position 0 that is a future frame. Is the target image. Then, by performing forward motion estimation (ME) and motion compensation (MC) on these forward image sequences, component information such as 503a in FIG. 25 is obtained. Here, in order to make it easy to specify the analysis direction, a “+” sign indicating a forward direction is attached to the target image. Further, when performing inter-image analysis in the reverse direction, an image at frame position 0, which is a future frame, is used as a reference image, and an image at frame position 1, which is a past frame, is used as a target image.

  And component information like 503b of FIG. 25 is obtained by performing reverse ME and MC with respect to these reverse image sequences. Here, in order to easily specify the analysis direction, a “−” symbol indicating the reverse direction is attached to the target image.

  Thereafter, an image sequence in a direction including an effective image as a representative image to be an output candidate is selected by predetermined similarity evaluation and determination with respect to 503a and 503b in FIG. In 504 of FIG. 25, it is determined that the forward inter-image analysis is effective.

  Thereafter, the image at frame position 1 set as the reference image is set as the next image to be analyzed. Here, the frame position on the reference image side is set as the next analysis target image. This is because the reference image side which is considered to contain a larger amount of information because the similarity with the target image is sufficiently high in the similarity comparison with the prediction image obtained by performing MC on the reference image. The image is selected at the frame position.

  However, the configuration may be such that the frame position on the target image side that is not affected by the MC is emphasized, and the image at the frame position on the target image side is selected. Further, since the current set includes frame positions that are candidates for representative images, the representative image candidate positions are updated. Here, frame position 1 is newly set as a new representative image candidate position. Similar processing is performed for the image strings 506, 510, and 514 in FIG. Here, when the operation is controlled so that an image sequence other than 506 adjacent to 502 in FIG. 25 needs to be subjected to an inter-image analysis with the image at the representative image candidate position, processing is performed. A better configuration without waste. Here, in order to simplify the description, assuming that all the analysis processing of the same layer is performed, an image sequence of −1 layer such as 505, 509, 513, and 517 in FIG. 25 is obtained.

  FIG. 26 is a diagram for illustrating a process for obtaining an image sequence of the −2 hierarchy.

  In the -2 hierarchy, the image sequence obtained as the -1 hierarchy image sequence is handled as a set of 2 images, so the 2 image sequences in the set are selected from the 4 image sequences in the 0 hierarchy. The resulting image sequence. This is indicated by reference numeral 601 in FIG. That is, the image sequence at the frame position 1 and the frame position 2 is selected from the four image sequences from the frame position 0 to the frame position 3, and the range of the image sequence is the frame position 1 and the frame position 2. It can be considered as a group having a high influence range of an image sequence, that is, a similarity.

  The process for obtaining the -2 layer image sequence is the same as that in FIG. 25. In the -1 layer image sequence, two images are handled as one set, and the analysis target image sequence as indicated by 602 in FIG. . In 602 of FIG. 26, the image sequences at the frame position 1 and the frame position 2 are to be analyzed. Thereafter, the same inter-image analysis is performed, the analysis result in the forward direction is selected, and the image at the frame position corresponding to the reference image in the forward direction is selected. Further, since the current set includes frame positions that are candidates for representative images, the representative image candidate positions are updated. Here, frame position 2 is newly set as a new representative image candidate position. Similar processing is performed for the image sequence 606 in FIG.

  FIG. 27 is a diagram illustrating a process for obtaining an image sequence of −3 layers. In the −3 hierarchy, the image sequence obtained as the −2 hierarchy image sequence is handled as a set of 2 images, so that the 2 image sequences in the set are selected from the 8 image sequences in the 0 hierarchy. The resulting image sequence. This is indicated by reference numeral 701 in FIG. In other words, the image sequence at the frame position 2 and the frame position 7 is selected from the eight image sequences from the frame position 0 to the frame position 7, and the range of the image sequence is the range of the frame position 2 and the frame position 7. It can be considered as a group having a high influence range of an image sequence, that is, a similarity.

  The process for obtaining the -3 layer image sequence is the same as in FIG. 25 and FIG. 26. In the -2 layer image sequence, two images are treated as one set, and the analysis target image as indicated by 702 in FIG. Identify the column. In 702 in FIG. 27, the image sequences at the frame position 2 and the frame position 7 are to be analyzed. Thereafter, the same inter-image analysis is performed, the analysis result in the reverse direction is selected, and the image at the frame position corresponding to the reference image in the reverse direction is selected. Further, since the current set includes frame positions that are candidates for representative images, the representative image candidate positions are updated. Here, since the representative image candidate position in the previous hierarchy, that is, the −2 hierarchy here is the same as the frame position 2, this representative image candidate position is maintained.

  In the inter-image analysis processing by the image processing apparatus according to the second embodiment, the inter-image analysis is performed on the image sequence in the frame buffer, and at least when one representative image is identified, the identified representative image is identified. If necessary, information on the range of the image sequence that is the influence range of the representative image is output. In FIG. 27, the representative image is an image at frame position 2. Further, the range of the image sequence is one level higher than the current level −3, that is, the level −2 in this case, and the range of the image sequence to which the frame position 2 belongs is specified. It is good to output.

  As indicated by reference numeral 706 in FIG. 27, the frame position of the representative image that is finally output is shifted from the analysis start position, which is the frame position that was initially set. This is because the image at the first analysis start position is determined not to have sufficient quality as a still image by the inter-image analysis processing of the image processing apparatus according to the second embodiment, and is more suitable as a still image for viewing as a still image. It can be said that the image of the position of the frame 2 is extracted as a still image having satisfactory quality.

  Therefore, according to the image processing apparatus of the second embodiment, as in the first embodiment, a still image having a quality that is closer to the actually desired still image and that can be appreciated as a still image is extracted. Therefore, a still image with higher definition can be generated.

  In particular, in the second embodiment, when performing inter-image analysis in each of the forward direction and the reverse direction, a predetermined motion vector search is performed in the reference image based on the target image from the analysis target image sequence, and the predicted image And generating the low-frequency component information of the predicted image from the generated predicted image, and subtracting the low-frequency component information of the predicted image from the generated predicted image to generate the high-frequency component information of the predicted image. In addition to generating the low-frequency component information of the target image, the low-frequency component information of the target image generated from the analysis target image sequence is subtracted to generate the high-frequency component information of the target image. Noise processing is performed, the similarity between the predicted image low frequency component information and the target image low frequency component information is evaluated to generate low frequency component similarity evaluation information, and the predicted image Generating high frequency component similarity evaluation information that evaluates the similarity between the frequency component information and the target image high frequency component information, and generating the similarity evaluation information of the low frequency component in the forward direction and the similarity of the high frequency component Based on the evaluation information and the similarity evaluation information of the low frequency component in the reverse direction and the similarity evaluation information of the high frequency component, it is determined whether the inter-image analysis in the forward direction or the reverse direction is effective. As a result, it is possible to easily evaluate how much noise, camera shake, motion blur, etc. are included, and by selecting images as extraction candidates based on these evaluation results, the user can It is possible to extract a still image that is closer to the desired still image and has a quality that can be appreciated as a still image, and generate a higher-definition still image.

Embodiment 3. FIG.
In the third embodiment, the functions of the image processing apparatuses in the first and second embodiments described above are realized by the central processing control devices 1203 and 1303 which are computers shown in FIGS. 28 and 29 by executing a program. .

  FIG. 28 is a configuration diagram of an example of an information processing apparatus that operates according to the image processing program of the third embodiment.

  In the figure, an information processing device 1200 includes an input device 1201 for inputting various types of information, an output device 1202 for outputting various types of information, and a central processing control device 1203 that operates according to the image processing program of the present invention. A two-way bus 1207 is connected to the external storage device 1204, a temporary storage device 1205 used for a work area in arithmetic processing by the central processing control device 1203, and a communication device 1206 for communicating with the outside. It is configured.

  The central processing control device 1203 captures an image processing program by the communication device 1206 via a recording medium or a communication network, and corresponds to the accumulated image sequence management means 1208 corresponding to the accumulated image sequence manager 102 and the forward direction inter-image analyzer 103. Forward image inter-analysis means 1209, reverse image inter-analysis means 1210 corresponding to the reverse image analyzer 104, determination means 1211 corresponding to the determiner 105, control means 1212 equivalent to the controller 106, super-resolution It has at least each function of the super-resolution enlarging means 1213 corresponding to the enlarging device 107, and executes the same operation as that of the image processing apparatus shown in FIG. 1 by software processing.

  FIG. 29 is a configuration diagram of another example of an information processing apparatus that operates according to the image processing program of the third embodiment.

  In the figure, an information processing device 1300 includes an input device 1301 for inputting various types of information, an output device 1302 for outputting various types of information, and a central processing control device 1303 that operates according to the image processing program of the present invention. And an external storage device 1304, a temporary storage device 1305 used as a work area for arithmetic processing by the central processing control device 1303, and a communication device 1306 for communicating with the outside are connected by a bidirectional bus 1307. It is configured.

  The central processing control device 1303 fetches the image processing program by the communication device 1306 via a recording medium or a communication network, and the accumulated image sequence management unit 1308 corresponding to the accumulated image sequence manager 202 and the determination unit 1311 corresponding to the determiner 225. Corresponding to the first image sequence acquisition unit 203, which includes a control unit 1312 corresponding to the controller 226 and a super-resolution expansion unit 1323 corresponding to the super-resolution expander 227, and further constitutes a forward image analysis unit 1309. Second image sequence acquisition unit 1322 corresponding to the second image sequence acquisition unit 214 constituting the first image acquisition unit 1321 and the reverse direction image analysis unit 1310, and the forward direction image analysis unit 1309 and the reverse direction image analysis unit. ME means 1313 for replacing the first ME unit 204 and the second ME unit 215 constituting the 1310, the first MC unit 205, MC unit 1314 for substituting the second MC unit 216, 1A LPF unit 206, 1B LPF unit 209, 2A LPF unit 217, LPF unit 1315 for substituting the 2B LPF unit 220, difference Substitute 207, subtractor 210, subtractor 218, HPF means 1316 or subtractor 1317 for substituting subtractor 221, first A noise remover 208, first B noise remover 211, second A noise remover 219, second B Noise removing means 1318 for substituting the noise remover 222, first L component similarity evaluator 212, L component similarity evaluating means 1319 for substituting the second L component similarity evaluator 223, and first H component similarity evaluation Each function of the H component similarity evaluation means 1320 for substituting the second H component similarity evaluator 224 is reduced. The operation similar to that of the image processing apparatus shown in FIG. 21 is executed by software processing.

  Therefore, the image processing apparatus according to the third embodiment also generates the similarity information in the forward direction by performing the inter-image analysis process similar to the first and second embodiments by executing the program, and similarly reverses the same. By performing the inter-image analysis of the direction, the similarity information in the reverse direction is generated, and the determinator performs a predetermined determination based on the similarity information in the forward direction and the reverse direction. It is possible to identify which side of the image sequence from which the correlation between the images is high and to determine which image should be the later analysis target.

1 is a block diagram illustrating a basic configuration of an image processing apparatus to which an embodiment of the present invention is applied. It is a conceptual diagram which shows the derivation | leading-out example of the analysis start position in the image processing apparatus of this invention. It is a conceptual diagram which shows the example of a setting of the cropping area | region in the image processing apparatus of this invention. 2 is a flowchart showing the operation of the image processing apparatus of the present invention shown in FIG. It is a flowchart which shows the operation | movement of the tree structure analysis process of this invention shown in FIG. It is a conceptual diagram which shows the tree structure used by the tree structure analysis process in the image processing apparatus of this invention. An example of a tree structure search mode is shown. It is a figure which shows an example of the cost of the path | pass according to search mode. It is a figure which shows the other example of the cost of the path | pass according to search mode. It is a conceptual diagram which shows the example of the tree structure used by the ranking process in the image processing apparatus of this invention. FIG. 7 is a diagram showing which interframe distance and hierarchy each path belongs to in the tree structure of FIG. 6. It is a figure which shows to which hierarchy each node belongs to which reference | standard frame distance each belongs to which hierarchy. It is a conceptual diagram which shows another example of the tree structure used by the ranking process in the image processing apparatus of this invention. It is a figure which shows to which hierarchy each node each belongs, and which reference | standard frame distance belongs. FIG. 6 is a conceptual diagram illustrating another example of a tree structure used in ranking processing in the image processing apparatus according to the first embodiment. FIG. 6 is a conceptual diagram illustrating another example of a tree structure used in ranking processing in the image processing apparatus according to the first embodiment. FIG. 6 is a conceptual diagram illustrating another example of a tree structure used in ranking processing in the image processing apparatus according to the first embodiment. 2 is a block diagram illustrating a detailed configuration of a super-resolution magnifier according to Embodiment 1. FIG. FIG. 3 is a conceptual diagram for illustrating a state of super-resolution processing in the first embodiment. It is a flowchart which shows the operation | movement of the super-resolution expansion process of this invention shown in FIG. It is a figure which shows the structural example of the forward direction image analyzer and reverse direction image analyzer in the image processing apparatus of Embodiment 2. FIG. It is a conceptual diagram which shows the change of the low frequency component in the image processing apparatus image processing method of Embodiment 2. It is a conceptual diagram which shows the change of the high frequency component in the image processing method of Embodiment 2. 10 is a flowchart illustrating an operation of the image processing apparatus according to the second embodiment. FIG. 6 is a conceptual diagram (part 1) illustrating an image processing method according to a second embodiment. FIG. 10 is a conceptual diagram (part 2) illustrating an image processing method according to the second embodiment. FIG. 9 is a conceptual diagram (part 3) illustrating an image processing method according to the second embodiment. It is a block diagram which shows an example of the information processing apparatus of the information Embodiment 3 which performs an image processing program. It is a block diagram which shows another example of the information processing apparatus of Embodiment 3 which performs an image processing program.

Explanation of symbols

102, 202 Accumulated image sequence manager 103, 243 Forward inter-image analyzer 104, 244 Reverse inter-image analyzer 105, 225 Judgment unit 106, 226 Controller 107, 227 Super-resolution enlarger

Claims (8)

An image processing apparatus for extracting at least one still image from a moving image composed of a plurality of continuous image sequences,
Accumulated image sequence management means for accumulating a predetermined number of input image sequences and specifying and supplying an analysis target image sequence;
A forward inter-image analysis unit that acquires the analysis target image sequence supplied from the accumulated image sequence management unit, performs time-to-forward image analysis, and generates evaluation information based on the forward image analysis; ,
Obtaining the analysis target image sequence supplied from the accumulated image sequence management means, performing an inter-image analysis in a reverse direction, and generating evaluation information based on the reverse inter-image analysis; ,
Based on the evaluation information based on the forward image analysis from the forward image analysis means and the evaluation information based on the reverse image analysis from the backward image analysis means, the forward direction or A determination means for determining whether the inter-image analysis in the opposite direction is effective;
Control means for extracting at least one still image from the analysis target image sequence stored in the stored image sequence management means based on the determination result of the determination means;
Super-resolution enlargement means for generating a high-definition still image by performing predetermined super-resolution enlargement processing on the still image extracted by the control means;
An image processing apparatus. The image processing apparatus according to claim 1.
The control means further includes
A time point that goes back in the past by a predetermined time interval from the time of the first still image acquisition request that the user actually requested is the second still image acquisition request from the user, and the moving image closest to that time When specifying the position of the frame as the analysis start position, the predetermined time interval is an analysis start position using either a predetermined initial value, a time interval specified by the user, or a time interval specified from the learning result. Image processing for controlling the supply of the analysis target image sequence from the accumulated image sequence management unit to the forward image inter-analysis unit and the forward image inter-analysis unit based on the identified analysis start position apparatus. The image processing apparatus according to claim 1 or 2,
The control means further includes
Based on a cropping area which is a predetermined designated area, the analysis target image sequence is cut out, and the analysis target image sequence subjected to the cut processing is analyzed between the forward image analysis unit and the reverse image analysis. An image processing apparatus for controlling the stored image sequence managing means so as to be supplied to the means. In the image processing device according to any one of claims 1 to 3,
The control means further includes
A set of image sequences is configured so that the frame rate is halved every time a hierarchy based on time is reduced by one with respect to the image sequence stored in the stored image sequence management means, and the moving image to be analyzed An image sequence is handled as a set of two images, and an image at one frame position of the paired image is selected as an analysis target of the next layer, and a moving image having unequal intervals as the analysis processing layer is lowered. Specifying the analysis target image sequence to be processed by controlling the accumulated image sequence management means to specify the analysis target image sequence to be an image sequence;
The dependency relation of the identified image sequence to be analyzed is managed by a tree structure, and the tree structure is analyzed based on a predetermined search mode, a cost of a path constituting the tree structure, and a cost of a node constituting the tree structure. To rank the image sequence to be analyzed,
An image processing apparatus that controls the accumulated image sequence management unit so as to supply a predetermined number of observed images together with the specified still image to the super-resolution enlargement unit based on the order of the analysis target image sequence. In the image processing apparatus according to any one of claims 1 to 4,
The super-resolution enlargement means includes
Among the still images extracted by the control means, a reference image serving as a reference when performing super-resolution enlargement processing and at least one observation image are acquired, and the acquired reference image and the observation image An aligner that performs alignment to pixel positions with a resolution higher than the resolution and generates non-uniform high-resolution images sampled at unequal intervals;
An interpolator that acquires the non-uniform high-resolution image generated by the aligner and performs a predetermined non-uniform interpolation process from the acquired non-uniform high-resolution image to generate an interpolated image having a desired high resolution;
Obtaining the interpolated image generated by the interpolator, performing a predetermined restoration process in consideration of a point spread function obtained from a predetermined camera model, generating an estimated image having a desired resolution, and generating the estimated image An estimated image generator that generates a non-uniform estimated image by performing a re-sampling process on
The non-uniform resolution image is acquired from the aligner, the non-uniform estimated image is acquired from the estimated image generator, and super-resolution expansion processing is performed according to a predetermined determination method using each acquired image If the result of determination is that repetition is necessary, the interpolator continues processing, whereas if repetition is not necessary, the result is output to the estimated image generator. A repeater for outputting
An image processing apparatus. In the image processing device according to any one of claims 1 to 5,
Further, the forward inter-image analysis means further performs temporal inter-image analysis as a reference image for an image on the past side in time, and an image on the future side in terms of time as a target image. While having a first image sequence acquisition unit for acquiring an image sequence to be analyzed from the stored image sequence management means,
The backward direction image analysis means further performs an image analysis in the reverse direction in time as a target image, and an image in the future side in time as a reference image, A second image sequence acquisition unit for acquiring an image sequence to be analyzed from the stored image sequence management means;
The forward direction image analyzing unit and the backward direction image analyzing unit are respectively
A reference image based on the target image from the temporally forward or reverse analysis target image sequence supplied from the stored image sequence manager means via the first image sequence acquisition unit or the second image sequence acquisition unit A motion estimator that performs a predetermined motion vector search and generates motion vector information;
A motion compensator that generates a predicted image from the reference image and the motion vector information generated by the motion estimation unit;
A predicted image low frequency component generator that generates low frequency component information of the predicted image from the predicted image generated by the motion estimator;
A subtractor that subtracts low frequency component information of the predicted image from the predicted image generated by the motion estimator to generate high frequency component information of the predicted image;
A predicted image high frequency component noise remover that performs a predetermined noise removal process on the high frequency component information of the predicted image;
Low frequency component information of the target image from the temporally forward or reverse analysis target image sequence supplied from the stored image sequence manager means via the first image sequence acquisition unit or the second image sequence acquisition unit A target image low frequency component information generator for generating
Low frequency components of the target image from the temporally forward or reverse analysis target image sequence supplied from the stored image sequence manager means via the first image sequence acquisition unit or the second image sequence acquisition unit A subtractor that subtracts information to generate high frequency component information of the target image;
A target image high frequency component noise remover that performs a predetermined noise removal process on the high frequency component information of the target image;
A low frequency component evaluation information generating unit that evaluates the predicted image low frequency component information and the target image low frequency component information and generates similarity evaluation information of the low frequency component;
A high frequency component evaluation information generating unit that evaluates the predicted image high frequency component information and the target image high frequency component information and generates similarity evaluation information of the high frequency component;
The determination means includes
The low frequency component similarity evaluation information and the high frequency component similarity evaluation information in the forward direction, and the low frequency component similarity evaluation information and the high frequency component similarity evaluation information in the reverse direction. An image processing apparatus that determines whether the inter-image analysis in the forward direction or the reverse direction is effective based on the image. An image processing method for extracting at least one still image from a moving image composed of a plurality of continuous image sequences,
Accumulating a predetermined number of input image sequences, specifying and supplying an analysis target image sequence;
Obtaining the analysis object image sequence, performing time-to-image analysis in the forward direction, and generating evaluation information based on the image analysis in the forward direction;
Obtaining the analysis target image sequence, performing an inter-image analysis in a reverse direction, and generating evaluation information based on the reverse inter-image analysis;
Based on the evaluation information based on the forward inter-image analysis and the evaluation information based on the reverse inter-image analysis, whether the inter-image analysis in the forward direction or the reverse direction is effective. A determining step;
Extracting at least one still image from the analysis target image sequence stored in the stored image sequence management means based on the determination result;
Performing a predetermined super-resolution enlargement process on the extracted still image to generate a high-definition still image;
An image processing method. An image processing program for causing a computer to implement an image processing method for extracting at least one still image from a moving image composed of a plurality of continuous image sequences,
Accumulating a predetermined number of input image sequences, specifying and supplying an analysis target image sequence;
Obtaining the analysis object image sequence, performing time-to-image analysis in the forward direction, and generating evaluation information based on the image analysis in the forward direction;
Obtaining the analysis target image sequence, performing an inter-image analysis in a reverse direction, and generating evaluation information based on the reverse inter-image analysis;
Based on the evaluation information based on the forward inter-image analysis and the evaluation information based on the reverse inter-image analysis, whether the inter-image analysis in the forward direction or the reverse direction is effective. A determining step;
Extracting at least one still image from the analysis target image sequence stored in the stored image sequence management means based on the determination result;
Performing a predetermined super-resolution enlargement process on the extracted still image to generate a high-definition still image;
An image processing program for realizing a computer.

Images