JP5696560B2 - Signal processing apparatus, signal processing method, and signal processing program - Google Patents

Description

  The present invention relates to a signal processing device, a signal processing method, and a signal processing program, and more particularly, to a signal processing device and a signal processing method that improve signal quality by performing super-resolution processing.

  In general, an image signal obtained by an imaging apparatus includes a considerable amount of unintended noise. For this reason, the image signal obtained by the imaging device is displayed, stored, encoded, and transmitted after various noise removal processes. In addition, the encoded image signal is acquired and decoded from a recording medium, a recording device, a storage device, or a transmission device to which the encoded image signal is transmitted. The obtained decoded image signal lacks frequency component information due to quantization specific to encoding. As a result, the decoded image signal is accompanied by coding deterioration such as block noise and ringing. In order to reduce the influence of encoding degradation in the decoded image signal, there are deblocking processing for reducing block noise on the decoded image, edge enhancement processing for increasing image sharpness, and various image restoration processing. Used. In recent years, a technique has been proposed in which a deteriorated image is brought closer to an original image by applying a super-resolution technique (for example, Patent Document 1).

  Also, in conventional super-resolution processing, for images containing noise, signals in the high frequency band can be obtained by performing super-resolution processing after removing various noises that exist in the spatial and temporal directions in advance. There has been proposed a technique for reducing image quality deterioration due to excessive emphasis on the image (for example, Patent Document 2).

JP 2006-127241 A JP 2009-066553

  In the various noise removal processes, deblocking processes, and edge enhancement processes that are the above-described conventional techniques, noise component information is generally removed, and at the same time, surrounding signal components are affected. For this reason, the processed image does not necessarily approach the original image quality. Further, the above conventional art image restoration process is, it is necessary to obtain in advance what the noise and characteristics deterioration as prior information. For this reason, it does not necessarily work effectively for a signal with noise or deterioration different from the characteristics of the prior information. Furthermore, in the conventional super-resolution technique, when the image quality differs between the image to be processed and the image to be used as a reference due to noise characteristics, shape, image quality degradation, etc. In some cases, image signal components from unintended positions can be used for super-resolution processing without grasping the image characteristics. In this case, an image far from the characteristic of the image that should be originally generated is generated.

  In general, when attention is paid to a frequency band including a noise component in an image including noise, a signal component having a characteristic as an original image included in the image includes a signal component affected by noise and a noise component. Signal components that are not significantly affected. The image is composed of a mixture of these signal components. For this reason, when such an image is perceived, it is considered that the image is recognized as being similar to the original image but being degraded by noise. When general noise removal processing is performed on an image including such noise, a signal component that is not significantly affected by noise is also affected. As a result, it becomes difficult to effectively use signal components included in the image. This is generally due to the fact that the noise component included in the image is unknown. Therefore, when various kinds of noise are removed in advance when super-resolution processing is performed on an image including noise, signal components effective for super-resolution processing are deteriorated. This hinders the effect of recovering the signal component due to the signal reconstruction processing accompanied by the iterative processing in the super-resolution processing.

  Furthermore, if noise removal is performed on an image to be processed and an image used as a reference, the influence of noise removal may differ between the processing target side and the reference side. When trying to perform super-resolution processing using these images, when performing motion search and alignment processing performed in super-resolution processing, more reliable signal components that should be originally specified are to be identified from the reference side. It cannot be specified, and super-resolution processing is performed using a signal component at a position different from the original position. As a result, an image signal component from an unintended position is used for the super-resolution processing without grasping the feature of the image, and an image far from the original image feature is generated.

  The present invention has been made in view of such circumstances, and its purpose is to provide the characteristics of an image that should originally exist even when the image quality differs between the image to be processed and the image to be used as a reference. It is an object of the present invention to provide a technique capable of generating an image close to.

First component is separated from the signal to be processed using the component extraction process of Jo Tokoro, the non-extraction and component information is extracted component information and other signal is a signal according to a frequency included in a predetermined frequency pass band An extraction unit, a second component extraction unit that generates extracted component information from a signal to be referenced using a predetermined component extraction process, and a reference-side extracted component information corresponding to a region of the extracted component information on the processing target side Based on the position information generated by the similar component specifying unit that performs processing for generating position information for specifying the region, and the position information generated by the similar component specifying unit, the region of the extracted component information on the processing target side and the extracted component information on the reference side And a reference-side signal paste position with higher accuracy than the position information specified by the similar component specifying unit in the signal to be processed, and included in the reference-side extracted component information area Belief By performing component pasting and signal synthesis, an alignment synthesis unit that generates composite component information, and predetermined band suppression processing and resampling processing are performed on the composite component information generated by the alignment synthesis unit. Thus, the band suppressing unit that generates band-limited combined component information, and the band-limited combined component information generated by the band suppressing unit are subjected to new processing performed by the similar component specifying unit, and An integrated control unit that performs control so as to repeat the reconfiguration processing, the band-limited combined component information generated by the band suppression unit, and the non-extracted component information on the processing target side separated by the first component extraction unit A first synthesizing unit that generates a reconstructed signal by a predetermined synthesizing process, wherein the first component extracting unit uses the reconstructed signal generated by the synthesizing unit as a new processing target. The control unit controls the first component extraction unit to generate extraction component information while switching the frequency passband included in the extraction component information separated by the first component extraction unit, and repeats the signal reconstruction process. A signal processing apparatus is provided.
Also, a first component that is separated from a signal to be processed using predetermined component extraction processing into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that. A component extraction unit; a second component extraction unit that generates extracted component information from a signal to be referenced using a predetermined component extraction process; and reference-side extracted component information corresponding to an extraction component information region on the processing target side A similar component specifying unit that performs processing for generating position information for specifying the region, and a region of the extracted component information on the processing target side and an extracted component on the reference side based on the position information generated by the similar component specifying unit The information area is specified, and the reference-side signal paste position is specified with higher accuracy than the position information specified by the similar component specifying section in the signal to be processed, and the reference-side extracted component information area is specified. Included A registration component that generates composite component information by pasting signal components and synthesizing signals, and predetermined band suppression processing and resampling processing on the composite component information generated by the alignment synthesis unit The band suppressing unit that generates band-limited combined component information and the band-limited combined component information generated by the band suppressing unit are targets for new processing performed by the similar component specifying unit, The integrated control unit that controls to repeat the signal reconstruction process, the first search unit that specifies the region of the extracted component information on the reference side corresponding to the region of the extracted component information on the processing target side, and the processing target A second search unit that specifies a paste position of a signal on the reference side with higher accuracy than the position information specified by the similar component specifying unit in the signal, and the alignment combining unit includes: The original signal before component extraction and the original signal before component extraction to be referred to are acquired, and the similar component specifying unit is configured to generate the original signal based on the position information generated by the first search unit. In the signal, the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side are specified, and the second search unit specifies the position specified by the similar component specifying unit in the signal to be processed The reference signal original signal pasting position is further specified with higher accuracy than the information, and the alignment combining unit is configured to determine the reference side original signal relative to the pasting position identified by the second search unit. The signal processing apparatus is characterized in that the composite component information is generated by pasting the signal components included in the region and synthesizing the signals.

First component is separated from the signal to be processed using the component extraction process of Jo Tokoro, the non-extraction and component information is extracted component information and other signal is a signal according to a frequency included in a predetermined frequency pass band An extraction step; a second component extraction step for generating extracted component information from a signal to be referenced using a predetermined component extraction process; and a reference-side extracted component information corresponding to a region of the extracted component information on the processing target side A similar component specifying step for performing processing for generating position information for specifying a region, and a region of extracted component information on the processing target side and a region of extracted component information on the reference side are specified based on the generated position information. In the signal to be processed, the reference-side signal paste position is specified with higher accuracy than the generated position information, and the signal component included in the reference-side extracted component information area is pasted. By performing the synthesis of the fine signal, and the positioning synthesis step of generating a composite component information, by a predetermined band suppression processing and resampling processing on the generated composite component information, synthetic component information is band-limited a band suppression step of generating, the generated band-limited synthesis components information, the object of the new process in the similar component specifying step, the integration control step of controlling so as to repeat the reconstruction processing of the signal, the band Synthesizing that generates the reconstructed signal by a predetermined synthesis process by obtaining the band-limited synthesis component information generated in the suppression step and the non-extraction component information on the processing target side separated in the first component extraction step step and a flop, said first component extraction step, the reconstructed signal generated by said combining step as a new processing object, the integrated control Step is that wherein the while switching frequency passband included in the extracted component information separated in the first component extraction step to generate extract component information, and controls so as to repeat the reconstruction process of the signal A signal processing method is provided.
Also, a first component that is separated from a signal to be processed using predetermined component extraction processing into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that. A component extraction step, a second component extraction step for generating extracted component information from a signal to be referenced using a predetermined component extraction process, and reference-side extracted component information corresponding to the region of the extracted component information on the processing target side A similar component specifying step for performing processing for generating position information for specifying the region of the target, and specifying the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side based on the generated position information In the signal to be processed, the reference-side signal paste position is specified with higher accuracy than the generated position information, and the signal component included in the reference-side extracted component information area is pasted. And combining the signal and the signal, and performing the alignment synthesis step for generating the synthesis component information, and performing the predetermined band suppression processing and resampling processing on the generated synthesis component information, the band-limited synthesis component A band suppression step for generating information, an integrated control step for controlling the generated band-limited synthesized component information as a target of new processing in the similar component specifying step, and repeating signal reconstruction processing; A first search step for specifying a reference-side extracted component information area corresponding to a target-side extracted component information area, and higher accuracy than the position information specified in the similar component specifying step in the signal to be processed And a second search step for specifying a reference-side signal pasting position, and the alignment synthesis step is a processing target. The original signal before the minute extraction and the original signal before the extraction of the component to be referred to are obtained, and the similar component specifying step is based on the position information generated by the first search step. A region of extracted component information on the processing target side and a region of extracted component information on the reference side in the signal are specified, and the second search step includes the positional information specified by the similar component specifying step in the signal to be processed The reference signal original signal pasting position is further specified with higher accuracy than the reference side original signal pasting position relative to the pasting position identified by the second search step. Provided is a signal processing method characterized by generating synthesized component information by pasting signal components included in a region and synthesizing signals.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  According to the present invention, even if the image quality is different between the image to be processed and the image to be used as a reference, it is possible to generate an image that is close to the characteristics of the image that should originally exist.

It is a conceptual block diagram which shows Embodiment 1-9 of the signal processing apparatus which concerns on embodiment of this invention. It is a flowchart for demonstrating operation | movement of Embodiment 1 of the signal processing apparatus which concerns on embodiment of this invention. It is the conceptual diagram showing the example of the frequency component information extracted in the component extraction process which concerns on embodiment of this invention. It is the conceptual diagram showing the example of the frequency component information extracted in the component extraction process which concerns on embodiment of this invention. It is the conceptual diagram showing the example of the frequency component information extracted in the component extraction process which concerns on embodiment of this invention. It is the conceptual diagram showing the example of the frequency component information extracted in the component extraction process which concerns on embodiment of this invention. It is the conceptual diagram showing the example of the frequency component information extracted in the component extraction process which concerns on embodiment of this invention. It is the conceptual diagram which showed an example of the data structure of extraction algorithm discrimination | determination information and extraction auxiliary information supplied as management information in the signal processing apparatus which concerns on embodiment of this invention. It is the conceptual diagram which showed an example which specifies the positional information by the 1st search part in the similar component specific | specification part in the signal processing apparatus which concerns on embodiment of this invention. It is the conceptual diagram which showed another example which specifies the positional information by the 1st search part in the similar component specific | specification part in the signal processing apparatus which concerns on embodiment of this invention. It is the conceptual diagram which showed the example of the template used with the search part in the similar component specific | specification part and the alignment synthetic | combination part in the signal processing apparatus which concerns on embodiment of this invention. It is the conceptual diagram which showed the example of the highly accurate alignment synthetic | combination process by the 2nd search part in the alignment synthetic | combination part in the signal processing apparatus which concerns on embodiment of this invention. It is a conceptual diagram which shows an example of the mode of the virtual grid and signal arrangement | positioning handled with the signal processing apparatus which concerns on embodiment of this invention. It is a conceptual diagram for showing an example of the state of the alignment composition processing in the signal processing device according to the embodiment of the present invention. It is a conceptual diagram for showing an example of the state of resampling accompanied by the band suppression process in the signal processing apparatus according to the embodiment of the present invention.

  The signal processing apparatus of the present invention may be configured as a signal processing apparatus 100 as shown in FIG.

  The signal processing apparatus 100 in FIG. 1 acquires a signal to be processed as the first input 101 from the imaging apparatus 131, the transmission apparatus 132, the recording apparatus 133, the recording medium 134, the storage apparatus 135, and the like. Similarly, a signal to be referred to is acquired as the second input 102. Here, these signals are assumed to be image signals, but other signals such as audio signals may be used. Hereinafter, although super-resolution processing will be mainly described as image signal processing, image encoding processing may be used.

  The signal processing apparatus 100 performs the signal reconfiguration processing in the signal processing apparatus 100 according to the embodiment in what order and for what signal component with respect to the acquired signal of the first input 101. It is desirable to configure so that the operation can be controlled based on predetermined management information. Further, it is possible to further control the timing at which the signal of the second input 102, which is a signal on the reference side, is switched.

  The management information used in the signal processing apparatus 100 according to the embodiment includes at least information (extraction algorithm discrimination information) for specifying an algorithm for extracting signal component information to be processed from a signal. Supplementary information (extraction supplemental information) for the signal extraction algorithm to identify the signal component to be extracted from the signal, and the extracted component after reconstruction processing obtained by reconstruction processing on the extracted signal (extracted component information) For information, information for specifying an algorithm for suppressing the frequency band component of the signal (band suppression algorithm discrimination information), what frequency band component of the signal the specified band suppression algorithm should suppress It is preferable to include auxiliary information (band suppression auxiliary information) for specifying the frequency.

  Such management information may be stored in the signal processing apparatus 100 in advance as predetermined management information. Further, it may be configured to be supplied and updated by an external input means not shown. Furthermore, if at least one of these pieces of management information is output to an external output means (not shown) and can be stored, the configuration is further improved.

  The signal processing apparatus 100 specifies an algorithm for extracting signal component information and a signal component to be extracted based on management information for the acquired signal of the first input 101 that is a signal to be processed. . The signal processing apparatus 100 generates extracted component information to be processed from the signal component to be extracted using the specified algorithm, and generates components other than the extracted component information as non-extracted component information. Similarly, the signal processing apparatus 100 generates at least extraction component information to be referred to based on the management information for the acquired signal of the second input 102 that is a reference signal.

  Thereafter, the signal processing apparatus 100 performs reconstruction processing of the extracted component information. First, in a predetermined processing unit, it is specified which part of the extracted component information on the reference side corresponds to the signal component most similar to the extracted component information to be processed.

  Next, the signal processing apparatus 100 converts the extracted reference-side extracted component information into signal components with a fineness (sampling interval is equal or short) that is equal to or greater than the sampling interval of the extracted component information to be processed. Align the position. As a result of the alignment, the signal processing apparatus 100 synthesizes the extracted component information on the processing target side and the extracted component information on the reference side, and combines the extracted component information (hereinafter, “synthesized extracted component information”). Is generated). Here, the sampling interval of the combined extracted component information may be configured to be finer than the sampling interval of the extracted component information that has been a processing target.

  Thereafter, the signal processing apparatus 100 specifies an algorithm for suppressing the frequency band component of the signal, and specifies what frequency band component of the signal should be suppressed. Further, the signal processing device 100 generates band-limited combined extracted component information by suppressing the frequency band of the signal with respect to the combined extracted component information. The above processing is the extraction component information reconstruction processing. Here, the frequency band limitation specified based on the management information may be configured to limit the frequency component band that can be extracted when the extracted component information is generated from the first input 101 or the second input 102. Although desirable, it is not necessary to limit the bandwidth in this way. For example, the resampling process may be performed after using a predetermined band limiting filter so as to be equal to the sampling interval of the signal of the original first input 101. The band limiting filter may be a filter designed to remove aliasing, or may be a filter designed to include aliasing. Furthermore, in the resampling process, the sample may be decimated so that it is equal to or smaller than the sampling interval of the signal of the first input 101, or the decimation is performed in the last process, otherwise It is also possible to configure the processing to be continued while the sampling interval is fine.

  The signal processing apparatus 100 reconstructs the extracted component information while sequentially updating the composite extracted component information as a new processing target until the predetermined repetitive condition is satisfied based on the management information. The combined extracted component information after band limitation determined by repeating the configuration process is obtained. Here, if necessary, the sample is decimated so as to be equal to or smaller than the sampling interval of the signal of the first input 101.

  Thereafter, the signal processing apparatus 100 completes the reconfiguration processing of the signal to be processed in a series by combining the combined extracted component information after the band limitation that has been determined and the non-extracted component information, obtain. Here, the signal processing apparatus 100 determines whether to repeat the reconstruction processing of the reconstruction signal while sequentially updating the reconstruction signal as a new processing target based on the management information. If it is determined that the process needs to be repeated, the signal processing apparatus 100 updates the signal of the second input 102, which is a signal referred to based on the management information, if necessary, based on the management information. An algorithm for extracting signal component information and a signal component to be extracted are identified again, and the process is repeated to obtain a reconstructed signal. Here, the signal processing apparatus 100 sets the newly generated reconstruction signal as a new signal to be processed in the repetition of the reconstruction processing of the extracted component information performed when the reconstruction processing of the reconstruction signal is repeated. When new extracted component information is generated from the signal, the extracted component information is generated while switching a predetermined frequency pass band included in the extracted component information as necessary, and this is used as an initial signal in the reconstruction processing of the extracted component information. As described above, the extraction component information reconstruction process may be repeated.

  If it is determined that it is not necessary to repeat the processing based on the management information, the obtained reconstruction signal is output as a transmission device 141, a recording device 142, a recording medium 143, a storage device 144, and a display / reproduction device 145. Supplied against etc. As a result, the signal processing apparatus 100 becomes a high-quality signal that is closer to a signal having the characteristics of an image that should be originally obtained by transmitting, storing, displaying, and reproducing the signal. Can be processed and supplied.

  Detailed embodiments of the present invention will be described below.

(Embodiment 1)
FIG. 1 is a conceptual block diagram for illustrating a configuration example of a signal processing device 100 according to Embodiment 1 of the present invention. FIG. 1 shows a basic connection relationship between each part. FIG. 1 shows a functional configuration for realizing the signal processing apparatus 100 according to the embodiment, and other configurations are omitted. In FIG. 1, each element described as a functional block for performing various processes can be configured by a CPU (Central Processing Unit), a main memory, and other LSI (Large Scale Integration) in terms of hardware. In terms of software, it is realized by a signal processing program or the like loaded in the main memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  In FIG. 1, the signal processing apparatus 100 according to the first embodiment includes at least a first component extraction unit 105, a second component extraction unit 106, a first control unit 107, a similar component identification unit 109, an alignment synthesis unit 110, a band. A suppression unit 111, a synthesis unit 113, a second control unit 114, and an integrated control unit 116 are included. Further, it is preferable to further include switches 103, 104, 108, 112, and 115 for controlling connection with each unit.

  The first component extraction unit 105 acquires the signal to be processed, which is the first input 101, via the switch 103. The first component extraction unit 105 also extracts from the first control unit 107 extraction algorithm discrimination information that is information for specifying an algorithm for extracting signal component information to be processed from the signal, and the specified extraction Extraction auxiliary information, which is auxiliary information for specifying a signal component to be extracted from the signal by the algorithm, is acquired. The first component extraction unit 105 further acquires a reconstruction signal from the synthesis unit 113 via the switch 115. Further, the first component extraction unit 105 extracts signal component information from the acquired first input 101 signal, which is a signal to be processed, based on extraction algorithm discrimination information and extraction auxiliary information that are management information. The component extraction process and the signal component to be extracted are specified, the extracted component information to be processed is generated, and the components other than the extracted component information are generated as the non-extracted component information. Here, the method of generating the non-extracted component information is a method of generating the non-extracted component information by subtracting the generated extracted component information to be processed from the signal to be processed that is the first input 101. However, it is not particularly limited to this method. For example, when generating the extracted component information, the signal component is converted from the time domain (in the case of an image, a normal image representation) to the frequency domain based on a predetermined conversion method, and the extracted component information is generated in the frequency domain. The subtracted component information in the frequency domain is generated by subtracting it from the signal to be processed, which is the first input 101 in the frequency domain. Thereafter, the out-of-extraction component information generated in the frequency domain is inversely transformed from the frequency domain to the time domain by using an inverse transformation method paired with a predetermined transformation method so as to generate out-of-extraction component information in the time domain. It doesn't matter. In addition, in the time domain, the extracted component information is generated by performing a convolution operation using a frequency band pass filter or a frequency band selection filter designed in advance in the time domain, and becomes the processing target that is the first input 101. It may be obtained by subtracting from the signal. The first component extraction unit 105 supplies the generated extracted component information to the similar component identification unit 109 and the alignment synthesis unit 110 via the switch 108 and also to the second control unit 114. The first component extraction unit 105 supplies the generated non-extraction component information to the synthesis unit 113.

  The second component extraction unit 106 acquires a reference target signal that is the second input 102 via the switch 104. Similarly to the first component extraction unit 105, the second component extraction unit 106 acquires extraction algorithm discrimination information and extraction auxiliary information that are management information from the first control unit 107. The second component extraction unit 106 extracts an algorithm for extracting signal component information from the acquired second input 102 signal, which is a referenced signal, based on extraction algorithm discrimination information and extraction auxiliary information that are management information. Then, the signal component to be extracted is specified. The second component extraction unit 106 generates extracted component information to be processed based on the specified algorithm and signal component. Here, the extraction algorithm discriminating information and the extraction auxiliary information may be configured so that the same information as that of the first component extraction unit 105 is supplied, or different in order to generate different extraction component information by different extraction algorithms. The extraction algorithm discrimination information and the extraction auxiliary information may be supplied. Here, in order to simplify the explanation, it is assumed that the same thing is supplied. The second component extraction unit 106 supplies the generated extracted component information to the similar component identification unit 109 and the alignment synthesis unit 110. Further, the second component extraction unit 106 may generate the non-extraction component information in the same manner as the first component extraction unit 105.

  The component extraction processing in the first component extraction unit 105 and the second component extraction unit 106 will be described later.

  The first control unit 107 acquires the state of each unit according to the control of the integrated control unit 116, supplies management information necessary for initialization processing of each unit, and connects each unit to control the operation of each unit Control the state. The first control unit 107 controls connection of the switches 103 and 104 in order to control incoming signals as the first input 101 and the second input 102. In addition, the first control unit 107 controls connection of the switch 115 in order to control whether to continue the signal reconstruction process or to output the generated reconstruction signal. The first control unit 107 supplies the extraction algorithm discrimination information and the extraction auxiliary information to the first component extraction unit 105 and the second component extraction unit 106 as management information. Moreover, the 1st control part 107 is the band suppression algorithm discriminating information which is the information for specifying the algorithm for suppressing the frequency band component of a signal to the band suppression part 111 as management information, and the specified band suppression The algorithm supplies band suppression auxiliary information that is auxiliary information for specifying what frequency band component of the signal should be suppressed. Here, the signal processing apparatus 100 according to the present embodiment is configured to share the processing of each unit, but the function of the first control unit 107 may be configured to be substituted for the integrated control unit 116. .

  The similar component identification unit 109 acquires extracted component information to be processed via the switch 108. In addition, the similar component specifying unit 109 acquires extracted component information to be referred to from the second component extracting unit 106. The first search unit in the similar component specifying unit 109 performs a predetermined similar component specifying process in a predetermined processing unit using the acquired extracted component information to be processed and the extracted component information to be referred to. As a result, it is specified which part of the extracted component information on the reference side corresponds to the signal component most similar to the extracted component information to be processed, and position information representing the correspondence is generated. The similar component identification unit 109 supplies the generated position information to the alignment synthesis unit 110.

  The similar component specifying process in the similar component specifying unit 109 will be described later.

  The alignment synthesis unit 110, via the switch 108, extracts component information to be processed, position information generated from the similar component identification unit 109, and extracted component information to be referred to from the second component extraction unit 106. To get. The second search unit in the alignment synthesizing unit 110 extracts the extracted component information on the processing target side by performing predetermined alignment using the acquired extracted component information to be processed and referenced and the positional information. Is associated with the extracted component information on the reference side. The alignment synthesis unit 110 synthesizes the extracted component information on the processing target side with the extracted component information on the reference side, and combines the extracted component information (hereinafter, “synthesized extracted component information”, or simply "Combined component information"). The sampling interval of the combined extracted component information may be configured to be finer than the sampling interval of the extracted component information that has been a processing target. The alignment composition processing in the alignment composition unit 110 will be described later. Further, the alignment synthesis unit 110 supplies the generated synthesis extraction component information to the band suppression unit 111.

  The band suppression unit 111 acquires band suppression algorithm determination information and band suppression auxiliary information as management information from the first control unit 107. In addition, the band suppression unit 111 acquires the synthesis component information from the alignment synthesis unit 110. The band suppression unit 111 identifies the band suppression process based on the acquired band suppression algorithm determination information. The band suppression unit 111 further performs band suppression processing specified based on the band suppression auxiliary information, thereby suppressing the frequency band of the signal with respect to the combined extracted component information, and combining the band-limited combined extracted component information. Generate. Further, the band suppression unit 111 supplies the generated band-limited combined extraction component information to the combining unit 113 via the switch 112 or to the similar component specifying unit 109 via the switches 112 and 108, and This is supplied to the second control unit 114. Here, the band suppression processing specified based on the management information may be configured to limit the frequency component band that can be extracted when generating the extracted component information from the first input 101 or the second input 102. Although desirable, it is not necessary to limit the bandwidth in this way. For example, the resampling process may be performed using a predetermined band limiting filter so that the sampling interval of the signal of the original first input 101 is simply equal. The band limiting filter may be a filter designed to remove aliasing, or may be a filter designed to include aliasing. Further, in the resampling process, the sample may be decimated so that the sampling interval is equal to or less than the sampling interval of the signal of the first input 101. The decimation is performed only in the last process, Other than that, it is also possible to configure the processing to be continued while the sampling interval is fine. Here, for the sake of simplicity, the specified band suppression process uses a predetermined band limiting filter so that it is simply equal to the sampling interval of the signal of the original first input 101, and then the resampling process. Shall be

  The synthesizing unit 113 acquires from the first component extraction unit 105 the non-extraction component information that is a component other than the extraction component information to be processed, and the combined extraction component information from the band suppression unit 111 via the switch 112. The synthesizing unit 113 generates a reconstructed signal by synthesizing the determined combined extracted component information after the band limitation and the non-extracted component information by a predetermined synthesizing process, and the first component extracting unit 105 via the switch 115. Alternatively, the output 117 is supplied to, for example, an external transmission device 141, a recording device 142, a recording medium 143, a storage device 144, a display / playback device 145, and the like. The predetermined synthesis process in the synthesis unit 113 is a synthesis process corresponding to the generation method of the extracted component information and the non-extracted component information in the first component extraction unit 105. For example, the method of generating the non-extracted component information in the first component extracting unit 105 subtracts the extracted extracted component information to be processed from the signal to be processed that is the first input 101 to obtain the non-extracted component information. In the case of the generation method, the combining unit 113 may be configured to perform the combining process by adding the combined extracted component information after the band limitation that has been determined and the non-extracted component information. In addition, when the generation method of the non-extracted component information in the first component extraction unit 105 is accompanied by the conversion process to the frequency domain by a predetermined conversion method, the synthesis unit 113 performs synthesis extraction after the determined band limitation. It is preferable that the component information is subjected to the same conversion process to the frequency domain, then added to the non-extracted component information, and the composite process is performed again by performing the reverse conversion process to the time domain by a predetermined reverse conversion method. Further, the management information includes synthesis process specifying information for specifying the synthesis process in the synthesis unit 113, and the first control unit 107 supplies the synthesis process specifying information as management information to the synthesis unit 113. If the composition processing is specified based on this management information, the configuration becomes even better.

  The second control unit 114 acquires the state of each unit based on the control of the integrated control unit 116, and supplies the operation of each unit and necessary management information. The second control unit 114 acquires at least extracted component information from the first component extraction unit 105 as information necessary for controlling the operation of each unit, and acquires combined extracted component information from the band suppression unit 111. The second control unit 114 repeats the reconstruction process of the combined extracted component information until at least a predetermined repetition condition is satisfied based on at least the extracted component information, the combined extracted component information, and the management information. Determine whether to supply to. The second control unit 114 controls the switches 108 and 112 in order to control connection of each unit based on the determination result. Here, the signal processing apparatus 100 according to the present embodiment is configured to share the processing of each unit, but the function of the second control unit 114 may be configured to be substituted for the integrated control unit 116. Absent.

  The integrated control unit 116 acquires the state of each unit constituting the signal processing apparatus 100 according to the present embodiment, controls the operation of each unit based on the control signal, and outputs a signal to be processed and a reference target. Is obtained as an input signal. The integrated control unit 116 further performs the above-described signal processing and controls each unit to output and supply a reconstructed signal that is a processing result. Further, the integrated control unit 116 detects a signal processing start request signal generated by a start operation by a user or a predetermined start condition such as a predetermined set time, and supplies a processing start command to each unit. In the present embodiment, in order to simplify the talk, the overall operation can be controlled by controlling the operations of the first control unit 107 and the second control unit 114, but the integrated control unit 116 may substitute for each function of the 1st control part 107 and the 2nd control part 114, and you may comprise so that each part of the signal processing apparatus 100 which concerns on this Embodiment may be controlled. Further, the integrated control unit 116 may substitute the functions of the first control unit 107 and the second control unit 114 to control the entire units of the signal processing device 100 according to the embodiment. Further, the integrated control unit 116 controls the order of input / output of each information to each unit, the operation order of each unit, the generation order of each information, the type of management information supplied to each unit, and the supply sequence. If integrated management information is held and the overall operation of the signal processing apparatus 100 of the present invention is controlled based on this information, the configuration is further improved. Here, the integrated management information includes state transition information in which all the states that can be taken by the above-described processing are described, internal state information for specifying in which process the current processing is in the overall processing, It is preferable to include condition information that is information related to the initial condition and the end determination condition of the process. For example, the condition information may include information related to the number of repetitions of each process or the repetition end determination condition. Further, the predetermined integrated management information may be set in the integrated control unit 116 in advance, and based on the acquired integrated management information by including in the integrated control unit 116 a function for acquiring the integrated management information from the outside. Thus, different processing operations may be performed. Here, for the sake of simplicity, it is assumed that predetermined integrated management information is set in advance, but is not particularly limited.

  In this integrated control unit 116, each control information is appropriately managed, and operation control is instructed to each unit, so that the signal component included in the region to be processed is simply signaled from the reference side. Rather than specifying the one with the smallest error between components, the position of a region including a more probable signal component to approximate the original image is specified by searching. The integrated control unit 116 uses the result obtained by reconfiguring the signal to search again or reconfigure the signal, such as reconfiguring the signal while reconfiguring the signal. By repeating the operation, the operation can be controlled to realize the super-resolution processing. For example, the integrated control unit 116 applies a frequency band as shown in FIG. 3 to FIG. 7 to be described later while switching as necessary in the repetition process of the reconstruction process, and is included in the frequency band. Control to extract the signal component information to be extracted. As a result, the integrated control unit 116 generates extracted component information from the input image signal or reconstructed signal, performs repetitive control of the reconstruction process on the extracted component information, and the reconstruction process on the generated reconstructed signal thereafter. By repeatedly controlling, it is possible to improve the image quality of the entire reconstructed signal that is the final output.

  Next, details of the component extraction processing in the first component extraction unit 105 and the second component extraction unit 106 will be described below. 3 to 7 are conceptual diagrams showing examples of frequency component information extracted in the component extraction processing according to the embodiment.

  For example, it is assumed that the signal to be processed which is the first input 101 is a signal including frequency component information in the region 301, that is, the maximum frequency fmax or less, as shown in FIG. Here, the component extraction processing specified based on the extraction algorithm discrimination information acquired as management information can be completely reconfigured in signal separation and synthesis as shown in FIG. 3B. The discussion will proceed assuming that the band division processing is performed by a QMF (Quadrature Mirror Filter) filter designed to be. The component extraction process specified based on the extraction algorithm discrimination information is specified from at least one component extraction process based on different band division processes. For example, the component extraction according to the present embodiment is possible if the band division processing is performed by a low-pass filter having a predetermined low-pass characteristic or a filter that can be used instead thereof, that is, a general Gaussian filter, Lanczos filter, wavelet filter, or the like. Can be used for processing. Further, for example, when generating the extracted component information, the signal component is expressed in the time domain (in the case of an image, based on a predetermined frequency conversion method such as Fourier transform, sine / cosine transform, Hadamard transform, and wavelet transform). Normal image representation) to the frequency domain, and extract component information is generated in the frequency domain. Subtracting the generated non-extracted component information from the signal to be processed, which is the first input 101 in the frequency domain, to generate non-extracted component information in the frequency domain, and then based on an inverse transform method that is paired with a predetermined transform method Thus, the process of generating the non-extracted component information in the time domain by performing inverse transformation from the frequency domain to the time domain can also be used for the component extraction process according to the present embodiment. Further, in the example of FIG. 3, the description proceeds with specifying that frequency component information included in frequencies f0 to f1 is extracted from the extraction auxiliary information acquired as management information.

  FIG. 3B shows a state where frequency component band division processing is performed using a QMF filter designed to have a cutoff frequency f1c, assuming that f1 = fmax / 2. Here, the low-pass band characteristic 321 is the frequency band characteristic of the low-pass filter in the QMF filter, and the high-pass band characteristic 322 is the frequency band characteristic of the corresponding high-pass filter. By performing a filtering process based on such a filter, the input signal is divided into frequency bands and separated into a low-frequency signal as shown in FIG. 3C and a high-frequency signal as shown in FIG.

  Here, it is assumed that f1 = fmax / 2 and the frequency to be subjected to band division by the QMF filter is fmax / 2. If the frequency to be band-divided by the QMF filter is different from the frequency to be band-divided for the input signal, the input signal is subjected to interpolation processing, decimation processing, or a combination thereof. Thus, the number of samples of the input signal is changed so that the frequency to be divided is equal to the frequency to be divided, thereby generating a temporary input signal. Here, when frequency shifting is performed by changing the number of samples for such an input signal, if there are restrictions on the number of samples that can be expressed by interpolation processing or decimation processing, The number of samples of the input signal may be changed so as to be closest to a certain frequency. This temporary input signal is divided into frequency bands, and then the original input signal is processed by interpolation processing, decimation processing, or a combination thereof, which is the reverse of when the temporary input signal was generated. Therefore, it is preferable to obtain a low-frequency signal and a high-frequency signal after performing the band division processing. Shifting the frequency to be subjected to band division by performing interpolation processing, decimation processing, or a combination of these on the input signal is also useful for other bandpass filters.

  By performing such processing, the frequency component information 331 included in the frequencies f0 to f1 is extracted and used as the extracted component information in the present embodiment. Further, the frequency component information 332 included from f1 to fmax is extracted and used as non-extracted component information in the present invention.

  Here, when generating the extracted component information and the non-extracted component information, in general, in order to consider the influence of aliasing components such as the frequency region of the region 312 in FIG. It is desirable to use a high pass filter. However, in order to simplify the description here, the frequency attenuation characteristic of the filter used for band division is very steep, and the discussion proceeds with almost no influence of aliasing components. Therefore, the non-extracted component information according to the embodiment is generated by subtracting from the input signal after generating the extracted component information that includes the frequency domain component such as the region 331 in FIG. The discussion proceeds as a component information generator.

  In FIG. 3, the frequency to be divided into frequency bands for the input signal is assumed to be f1 = fmax / 2. However, based on the extraction auxiliary information, the frequency band is divided into the frequency bands as shown in FIG. Information may be acquired. For example, as shown in FIG. 4B, the filtering is performed based on a filter that can generate frequency component information included in f0 to f1 by the frequency band characteristic 421 of the low-pass filter. The signal is divided into frequency bands and separated into a low frequency signal including frequency component information such as a region 431 in FIG. 4E and a high frequency signal including frequency component information such as a region 441 in FIG. The extracted component information and the non-extracted component information may be acquired, respectively. Similarly, when it is specified that the frequency to be band-divided is f2, filtering processing by a filter having a low-pass band characteristic 423 from f0 to f2 is used to perform the processing shown in FIGS. 4 (f) and 4 (i). When the frequency to be divided is specified as f3, the filter having the low pass band characteristic 425 is used as shown in FIG. Thus, the extracted component information and the non-extracted component information may be acquired, respectively.

  Further, as shown in FIG. 5, the band division may be repeated to extract signal component information of a necessary frequency band. For example, in the area 511 in FIG. 5A, after obtaining FIG. 4F, the extracted component information is obtained by removing the frequency component information included in FIG. 4E by band division in FIG. 4B. It is also possible to acquire information including frequency component information other than that as out-extraction component information. Similarly, FIGS. 5B and 5E, FIGS. 5C and 5F, FIGS. 6A and 6D, FIGS. 6B and 6E, and FIGS. The extracted component information including the frequency component information such as f), etc. and the non-extracted component information may be acquired.

  Further, as shown in FIG. 7, the signal component information may be extracted so as to include frequency component information of a plurality of frequency bands. For example, the area 711 in FIG. 7A is extracted by the same method as in FIG. 4E, and the area 712 is extracted by the same method as in FIG. What is included may be acquired as extracted component information, and other information may be acquired as non-extracted component information. Similarly, frequency component information such as FIGS. 7B and 7G, FIGS. 7C and 7H, FIGS. 7D and 7I, FIGS. 7E and 7J, and the like. The extracted component information and the non-extracted component information may be acquired.

  By extracting the signal component information as shown in FIG. 3 to FIG. 7 and generating the extracted component information and the non-extracted component information, it is referred to in the subsequent similar component specifying process and alignment composition process. The effect of making it easier to specify the position of a region including more probable signal components to approximate the original image from the side signal is obtained.

  For example, as a feature of the signal component included in the input image to be processed, the signal component included in the low frequency band side has many errors, but many of them are similar to the signal component included in the original image. Consider a case where the signal component included on the high frequency band side includes a large number of signal components greatly deteriorated due to noise or the like. In such a case, when a search for specifying a similar component is performed using an image including a signal component of the entire frequency, the image including the signal component of the entire frequency is affected by the deteriorated signal component. The image is different from the feature. For this reason, as a result of the search, an area at another position including the one closest to the signal component included in the image having different characteristics is specified from the reference side. In order to avoid the identification of different areas by such a search, in this example, the signals included in the respective frequency bands are separated into the signals included in the respective areas 331 like the areas 331 and 332 in FIG. By using the component as the extracted component information and searching on the reference side as the signal component included in the region 331 as well, it is possible to search by excluding the signal component included in the region 332 that had a large influence during the search. Become. As a result, it is possible to specify the position of a region containing more probable signal components by searching from the reference side instead of simply specifying the one with the least error between the signal components. It becomes.

  In this example, it is assumed that the low frequency band side includes some errors, but the extracted component information reconstruction process, that is, the similar component identification process, the alignment synthesis process, and the band suppression process are performed. By generating limited synthetic component information and using this as new extracted component information, this reconstruction process is repeated, so that the quality of the extracted component information can be improved each time it is repeated. This is because the quality of the extracted component information is improved, the accuracy of the search result is increased, and a more reliable signal component can be specified. As a result, the quality of the extracted component information can be further improved. It is to become. By repeating this until a predetermined condition is satisfied, composite component information that is an output result of this processing loop and that has improved image quality in the low-frequency component is obtained. Here, it is further preferable that the position information finally converged is stored as predicted position information so that it can be reused.

  Further, for example, a reconstructed signal is generated by synthesizing the composite component information with improved image quality and the non-extracted component information including the signal components included in the frequency band of the processing target region 332, Consider repeating the reconstruction process of the reconstruction signal using the composition signal as a new initial input signal. In such an example, the reconstructed signal, which is a new initial input signal, has an improved signal component quality in the frequency band of the region 331 in FIG. It becomes easy to specify an ingredient. Furthermore, it is possible to use predicted position information, which is finally converged position information, by repeating the reconstruction process of the previous extracted component information.

  Therefore, for example, by setting the search range at the time of search narrower starting from this predicted position information, the frequency band of the region 331 in FIG. 3 is slightly expanded to the high frequency component side and extracted from the reconstructed signal. Even if a signal component that causes image quality degradation is included in the extracted component information, a position that is completely different from the originally required position can be obtained by searching for the predicted position information converged with a relatively reliable signal component as a starting point. A necessary region can be specified from the reference side without detection. By repeating this reconstruction processing, it is possible to improve the quality of the extracted component information while bringing the captured signal component that causes image quality degradation close to the original signal component.

  Further, for example, when repeating the reconstruction processing of the reconstructed signal, when extracting extracted component information from the reconstructed signal, the frequency band of the region 331 in FIG. The extracted component information is generated by expanding the image quality to the image quality, and the image quality can be improved while taking in the signal component that gradually deteriorates the image quality from the high frequency component side.

  That is, when the reconstruction process of the reconstructed signal is repeated, for example, as shown in FIG. 4, the frequency band is first divided as shown in FIG. 4B, and the signal component included in the region 431 in FIG. As the extracted component information, the extracted component information is reconstructed using the predicted position information as necessary to generate a reconstructed signal. In the next iteration, the signal component included in the region 432 in FIG. 4 is used as the extracted component information, and the extracted component information is reconstructed using the predicted position information as necessary to generate a reconstructed signal. . Further, in the next iteration, the extraction component information is generated while adaptively switching the predetermined frequency band, such as using the region 433 in FIG. 4, and the reconstructed signal is generated. Image quality can be improved while suppressing the number of repetitions.

  In this case, the extracted component information is generated by extracting the signal component included in the frequency band by applying the region 431, the region 432, and the region 433 in this order, but the order of application is not particularly limited. For example, in FIG. 4, the signal component included in the frequency band between f1 and f2 is relatively greatly deteriorated, and the signal component included in the frequency band between f2 and f3 is relatively less deteriorated. In the case where a lot of signal components representing the characteristics of the original image are included, the region 433 is applied first after the region 431 is applied, and then the region 432 is applied if necessary. Thus, it is possible to suppress the use of signal components from different positions and maintain the reconstruction process from more likely positions. As a result, the image quality of the entire reconstructed signal can be improved.

  Further, as shown in FIG. 5, for example, when the reconstruction process of the reconstructed signal is repeated, a signal component different from the signal component of the frequency band whose image quality has been improved is extracted from the reconstructed signal to obtain the extracted component information. The extracted component information is reconstructed using the predicted position information as necessary, and a reconstructed signal is generated. This makes it possible to improve the image quality of the entire reconstructed signal while suppressing the adverse effect on the signal components in the frequency band whose image quality has already been improved when reconstructing the signal component that causes image quality degradation. it can.

  That is, for example, as shown in FIG. 5, after the first reconstructed signal is generated, the signal component included in the frequency band of the region 511 is extracted to generate the extracted component information, and the predicted position information is changed as necessary. While being used, the extraction component information is reconstructed to generate a reconstructed signal. In the next iteration, the signal components included in the 512 frequency band are extracted to generate extracted component information, and the extracted component information is reconstructed while using the predicted position information as necessary. Generate a signal. Further, in the next iteration, a signal component included in the frequency band of the region 513 is extracted to generate extracted component information, and the extracted component information is reconstructed using the predicted position information as necessary. A reconstruction signal is generated. Thereby, it is possible to improve the image quality of the entire reconstructed signal while suppressing adverse effects on the signal components whose image quality has already been improved.

  Here, the extracted component information is generated by extracting the signal component included in the frequency band by applying the region 511, the region 512, and the region 513 in this order. However, the order is not particularly limited. For example, when the signal component included in the frequency band of the region 511 is relatively greatly deteriorated, the region 511 may be applied after the region 512 and the region 513 are applied.

  Further, for example, among the signal components included in the frequency band shown in FIG. 6, f0 to f1 include a lot of signal components indicating the characteristics of the original image, and then f1 to f2, f3 to fmax, Assume that f2 to f3 contain the most signal components with the most deterioration. For example, extraction component information is first generated by extracting from signal components included in the frequency band from f0 to f1, and a reconstruction signal is generated after reconstructing the extraction component information. Thereafter, the region 613 may be applied after the region 611 and the region 612 are applied and the reconstructed signal is similarly generated. In this way, when the image quality of the signal component included in the frequency band from f2 to f3 with the most deterioration is improved, the reconstruction process is performed with the signal component included from f1 to f2. Since the position specified at the time of the search is different from the case where the reconstruction process is performed with the signal components included from f3 to fmax, the reconstruction process is repeated by combining these, and each has a characteristic. Different image quality improvements will be seen. As a result, a better effect can be obtained in improving the image quality of the signal component accompanied by deterioration.

  Further, for example, as shown in FIG. 7, it is possible to extract signal components included in a plurality of frequency bands to generate extracted component information, and to perform a reconstruction process. In this way, the predicted position information can be converged more quickly by incorporating into the extracted component information a signal component that includes many features of the original image that makes it easier to specify a more probable position during the search. it can. Furthermore, it is possible to improve the image quality better by reconstructing a signal component with relatively little deterioration from a more likely position.

  Further, for example, when repeating the reconstruction processing of the reconstructed signal, a plurality of frequency bands to be selected are changed every time the reconstructed signal is repeated, and signal components included in the plurality of frequency bands are extracted from the reconstructed signal and extracted component information The extracted component information is reconstructed using the predicted position information as necessary, and a new reconstructed signal is generated. As a result, it is possible to improve the image quality while determining the signal component with little deterioration and incorporating the signal component including the deterioration into the extracted component information. Furthermore, by using a signal component at a more probable position and performing signal component reconstruction processing including degradation, the image quality can be improved with respect to the entire reconstructed signal.

  For example, signal components included in a plurality of frequency bands such as region 711 and region 712, region 713 and region 714, region 715 and region 716, region 717 and region 718, region 719 and region 720 in FIG. It may be configured to repeat extraction of signal components included in a plurality of frequency bands from the reconstructed signal to generate extracted component information, reconstructing the extracted component information, and generating a reconstructed signal. Absent. Here, the order of application is not particularly limited.

  The extraction algorithm discrimination information and the extraction auxiliary information supplied as management information to the first component extraction unit 105 and the second component extraction unit 106 may have a data structure as shown in FIG. 8, for example. . For example, the extraction algorithm discrimination information may have a data structure such that a unique identification number is given as shown in the table 801 so that the data structure can be identified and the extraction algorithm can be specified based on the subsequent Index information. The association between the index information and the actual extraction algorithm may be defined by a table such as the table 802. Further, the extraction auxiliary information is preferably configured to have a different data structure according to the algorithm type specified by the extraction algorithm discrimination information. For example, if the index of the table 801 is 1, the type of the algorithm is A1. In such a case, the paired extraction auxiliary information is given a unique identification number as shown in the table 811 so that the data structure can be identified, and the frequency band to be extracted can be specified based on the subsequent Index information. The data structure should be good. The association between the index information and the information related to the actual frequency band may be defined by a table like a table 812. If the index of the table 801 is 33, the algorithm type is B1. The extraction auxiliary information that is paired in such a case may have a structure like the table 821. The structure of this table 821 includes information on the number of frequency bands included in the frequency band to be extracted and information on the lower limit value and the upper limit value of each frequency band as many as the number of included frequency bands. It is good to make it.

  In the information stored in the extraction auxiliary information, the data structure shown in FIG. 8 is merely an example, and the frequency component information and the frequency band to be extracted by the algorithm specified by the extraction algorithm discrimination information can be specified. If it is, it will not specifically limit. For example, if the algorithm uses a general Gaussian filter or a Lanczos filter, the data structure may further include parameters for controlling the frequency window and parameters for controlling the phase of the filter. . In addition, if the algorithm uses a wavelet filter, the data structure further includes information for specifying the octave division or subband division rank for the frequency component and the subband to be processed. good.

  Further, the expression format of the extraction algorithm discrimination information and the extraction auxiliary information uses a fixed-length code as shown in FIG. 8, but these are not particularly limited, and are expressed by other variable-length codes. It doesn't matter. Also, regarding the identification method of the data structure, another different identification method may be adopted.

  Next, details of the similar component specifying process in the similar component specifying unit 109 will be described below. FIG. 9 shows that the signal component most similar to the signal included in the region currently being processed in the signal currently being processed by the first search unit in the similar component specifying unit 109 is currently It is a conceptual diagram which shows an example of a mode that the part which respond | corresponds in the signal used as a reference object is specified by search, and the positional information showing a correspondence is produced | generated.

  In FIG. 9A, a region 901 is a signal that is currently processed, and a region 902 represents a region that is currently processed in the signal. Here, the region 902 is represented by a rectangle, but may be a region having a predetermined shape. For example, the signal of the rectangular area 901 may be divided into regions by a predetermined segmentation unit or a dividing unit in advance, and the region currently being processed may be specified for each region in a predetermined order. Here, the signal of the area 901 is divided into a predetermined rectangular shape without a gap, and the talk proceeds as if the area to be processed is specified in order from the upper left in the raster order as indicated by the arrow in FIG. .

  In FIG. 9B, a region 903 is a signal that is currently a reference target, and a region 904 represents a region at the same in-plane space position as the region 902 in the in-plane space of the region 903. This position is set as a reference position for search. Further, a predetermined search area such as the area 905 is set based on this reference position.

  The similar component specifying process in the similar component specifying unit 109 may be performed, for example, as shown in the conceptual diagram of FIG. Within the search area 905 of the signal 903 that is the reference object, the position of the reference area is changed in order as in the area 906, and the signal component included in the area 906 and the area 902 that is the processing target are included. Are sequentially compared with each other, and the region of the position including the most similar signal component is specified. Here, the signal components included in the reference region at the position 907 are most similar. When a similar component is specified by obtaining the distance of the in-plane space position between the area 907 specified in this way and the reference position 904 and using it as position information indicating the correspondence between the area 902 and the area 907. good.

  Here, the determination of the similarity of the signal components in the similar component specifying process may use the result of the method of measuring the distance between the signals. For example, the absolute value sum (SAD) of the differences between the signals, The sum of squares of differences (SSD), the sum of absolute values of differences between signals after frequency conversion (SATD), mean square error (MSE), mean square square error (RMSE), etc. may be used. By such a method, the error between the signal component included in the reference side region and the signal component included in the processing target region is measured while changing the position of the reference side region, and the error is minimized. The position of the area to be specified is specified.

  In the similar component specifying process, the position accuracy when performing such a search is illustrated here as a search with one pixel accuracy, but is not particularly limited, and is 1/2, 1/4. The search may be made with one pixel accuracy or less such as 1/8. Further, in order to reduce the processing amount, a hierarchical search may be performed in which a rough search is performed and then the periphery of the area specified with a fine accuracy is searched again.

  Furthermore, as another example, the determination of the distance to the similar region and the similarity of the signal component in the similar component specifying process may be performed by, for example, optical flow detection or a gradient method that is a representative method thereof. It can also be configured. For example, using the Lucas-Kanade Method as a representative method of the gradient method, the signal component information change state (gradient distribution) around the target area is detected, and the position most similar to this gradient distribution is searched. You may comprise so that it may identify by.

  Such a position search by the gradient method is generally more effective when the signal to be searched is a continuous and relatively smooth signal. Therefore, by adopting such a position search by the gradient method in the present embodiment, when generating extracted component information, a predetermined value is set so as to include a continuous and relatively smooth signal component. Set the frequency band. As a result, the position can be specified with high accuracy, and the image quality can be improved with respect to the entire reconstructed signal.

  Here, Lucas-Kanade Method is given as an example. However, the present invention is not particularly limited to this method, and other search methods using the amount of change in signal component information around the target region may be adopted. .

  Next, details of the alignment composition processing in the alignment composition unit 110 will be described below. FIG. 12 is a conceptual diagram for explaining an example of the alignment process.

  In FIG. 12, an area 902 is an area that is the current processing target of the signal 901 that is the current processing target, and an area 907 is the position information specified by the similar component specifying process according to the embodiment. Based on the positional relationship of the in-plane spatial position with the region 902, the region includes a signal component most similar to the signal component included in the region 902.

  The second search unit in the alignment synthesizer 110 searches for a position where the signal included in the region 907 matches the signal component around the pasted position with high accuracy. . As a result, the most suitable position in the area 902 and the surrounding area is specified, and the signal component is pasted. In this way, it is preferable to paste the signal component with signal alignment. Here, as in a region 1201 in FIG. 12, it is preferable to set a search range when searching for a paste position with high accuracy. After that, the signal component included in the pasted signal component and the signal component included in the region 902 and its peripheral region is subjected to a predetermined alignment synthesis process, and the synthesis is expressed with a fineness greater than the sampling interval of the original signal. It is preferable to generate component information.

  In the alignment synthesis processing according to the present embodiment, the positional accuracy when performing high-precision pasting is, for example, as shown in FIG. 13, the signal sampling interval is finer than the original signal sampling interval. By defining the grid represented in this way (hereinafter, the virtually defined grid is referred to as “virtual grid”) and making the interval between the virtual grids finer, higher positional accuracy can be realized. It is good to do so. FIG. 13A shows an example in which the sampling interval is 1. Here, the sampling interval of 1 means that the sampling interval of the signal serving as the reference of the sampling interval is equal to 1, and in the example of FIG. It means that it is the same interval. Sampling 1301 is each sample of the signal component included in region 902, and if the interval between each sample is 1, each sample of sampling 1301 is on a virtual grid 1302 with a sample interval of 1. , The sampling 1303 is arranged. As another example, as shown in FIG. 13B, when the virtual grid 1304 is set so that the sampling interval is 1/4, each sample of the sampling 1301 is sampled on the virtual grid 1304. 1305 is arranged. Here, the discussion proceeds as a virtual grid and arrangement such as sampling 1305, but by setting a virtual grid with a finer interval such as 1/8 or 1/16, a higher-precision alignment process is performed. Is possible.

  FIG. 14 is a conceptual diagram showing a process of generating synthesized component information by aligning a signal to be processed with a signal on the reference side. The sampling 1401 is each sample of the signal component included in the specified region 907 on the reference side. In addition, the sampling 1305 is included in the region 907 specified on the reference side, which is arranged on a virtual grid in which each sample of the signal component included in the region 902 currently being processed is set to a sampling interval of 1/4. Each sample of the signal component.

  Since the interval between samples of the sampling 1401 is equal to the interval between samples of the signal to be processed, it is assumed that the signal has the same interval as the interval between samples such as the sampling 1305, and the virtual on the sampling 1305 On the grid, the second search unit in the alignment synthesis unit 110 searches the paste position with high accuracy, and pastes the sampling 1401 of the reference-side signal onto the virtual grid like the sampling 1402. .

  Here, the highly accurate pasting position search process by the second search unit in the alignment synthesis unit 110 may be performed as follows, for example. First, like the sampling 1305, an interpolated signal component is generated in advance by a predetermined interpolation process for the signal component between the arranged samples. The predetermined interpolation process may be any signal interpolation process using a general low-pass filter, and may be an interpolation process using a filter based on a sinc function or a Lanczos function, for example. Also, other interpolation processing, for example, FIR and IIR filters designed so that the frequency response has a flat and steep cutoff frequency characteristic over the entire band may be used. Further, the processing may be reduced by using a filter based on the Nearest-neighbor method, Bilinear method, Bicubic method, Spline method, or the like. In this way, the interpolation signal component generated from the signal on the processing target side is arranged on the virtual grid in advance, and the second search unit in the alignment synthesis unit 110 is the signal most similar to the signal component on the reference side. It is preferable to identify the position where the component exists and paste the sampling 1401.

  After that, a re-sampling process of a predetermined signal component is performed on a non-uniformly spaced signal component such as a sampling 1402 while considering the influence between each sample based on a predetermined point spread function (PSF), For example, a sampling interval signal such as the synthesis component information 1403 is synthesized. Here, the signal is synthesized with the sampling interval being ½, but the sampling interval may be 1 or less, and such a signal is used as the synthesis component information 1403.

  Here, the position where the reference-side signal component is affixed is not particularly limited to the above example as long as the position suitable for the original signal can be specified as the affixing position. For example, in order to specify the pasting position with higher accuracy, the frequency component such as Fourier transform is performed on the signal component included in the region 902 and the signal component included in the region 907, thereby obtaining the frequency spectrum information and the phase. Ask for information. After matching the phase of the signal on the reference side with the phase of the signal on the processing target side, change the phase of the signal component on the reference side so that it becomes the signal component at the position you want to place on the virtual grid, and perform inverse Fourier transform, etc. Perform inverse frequency conversion. A sample of the signal component at the position to be arranged may be generated and arranged by interpolation processing and resampling processing of a predetermined signal component. If necessary, these processes may be selected so that the processes can be switched for each process.

  FIG. 15 is a conceptual diagram showing how the band suppression unit 111 suppresses the frequency band of a signal expressed with a finer sampling interval to generate band-limited combined extraction component information. .

  The band suppression unit 111 generates a band-limited signal 1501 by performing a filtering process on the composite component information 1403 using a predetermined low-pass filter. Here, as a filter to be used, for example, a filter based on a sinc function or a Lanczos function, or an FIR or IIR filter designed so that the frequency response has a flat and steep cutoff frequency characteristic over the entire band may be used. Further, the processing may be reduced by using a filter based on the Nearest-neighbor method, Bilinear method, Bicubic method, Spline method, or the like.

  Such band-suppressed composite component information 1501 is generated, and a signal including the composite component information is set as a new processing target signal. Similar processing is repeated until the fluctuation of the signal component of the composite component information falls within a predetermined range, so that super-resolution processing is performed to reconstruct the signal.

  In the present embodiment, each process in such super-resolution processing may be replaced with similar processing performed in other super-resolution processing. In other super-resolution processing, in general, increasing the resolution of an image by super-resolution processing is simply a single high-resolution image from a plurality of low-resolution images with relatively high correlation between images, such as a positional shift. This is done by estimating the spatial resolution image. These techniques have been reported in numerous studies in recent years.

  For example, an ML (Maximum-likelihood) method has been proposed. In this ML method, a square error between a pixel value of a low spatial resolution image estimated from a high spatial resolution image and a pixel value actually observed is used as an evaluation function, and a high spatial resolution image that minimizes the evaluation function is estimated. It is an image. This method is a method for performing super-resolution processing based on the principle of maximum likelihood estimation.

  Also, a MAP (Maximum A Posterior) method has been proposed. In the MAP method, a high spatial resolution image is estimated that minimizes an evaluation function obtained by adding probability information of a high spatial resolution image to a square error. This method is a super-resolution processing method for estimating a high spatial resolution image as an optimization problem that maximizes the posterior probability using some foresight information for the high spatial resolution image.

  Furthermore, a POCS (Projection Onto Convex Sets) method has been proposed. This POCS method is a super-resolution processing method for obtaining a high spatial resolution image by creating simultaneous equations regarding pixel values of a high spatial resolution image and a low spatial resolution image and sequentially solving the equations. By applying the super-resolution processing as described above to the present invention, a better configuration can be obtained.

  Next, the basic operation of the signal processing apparatus 100 of the first embodiment shown in FIG. 1 will be described with reference to the flowchart of FIG.

  First, upon receiving a signal processing start request signal from a user or the like, the integrated control unit 116 sends a signal processing start command to each unit.

  The first control unit 107 controls the input signal and the output signal by connecting the switches 103 and 104 and disconnecting the switch 115 in accordance with a command from the integrated control unit 116 (step S101). Then, it branches to the 1st process and the 2nd process as parallel processing. Here, the processing is described as being performed by parallel processing. For example, each step of the second processing after the first processing, the first processing after the second processing, or the first processing and the second processing is performed. It is also possible to operate so as to perform sequential processing such as alternately executing.

  The first process is a process from step S102 to step S106. First, the first control unit 107 supplies extraction algorithm discrimination information and extraction auxiliary information, which are information for controlling the extraction of frequency components, to the first component extraction unit 105, and updates the stored information. (Step S102).

  Next, since the first component extraction unit 105 acquires or updates the input signal to be processed, when the switch 103 is connected, the first component extraction unit 105 determines the processing target that is the first input 101 via the switch 103. To obtain an input signal. When the switch 103 is disconnected and the switch 115 is connected, a reconfiguration signal is acquired from the synthesis unit 113 via the switch 115 and updated as an input signal to be processed next (step S103).

  Next, the first component extraction unit 105 specifies component extraction processing based on the acquired extraction algorithm discrimination information. The first component extraction unit 105 specifies the signal component to be extracted based on the extraction assistance information, and then performs the specified component extraction process on the acquired input signal to be processed (step S104). By this process, the extracted component information of the input signal to be processed is generated, and the components other than the extracted component information are generated as the non-extracted component information, thereby separating the signal components (step S105).

  When the component extraction process for the input signal to be processed is completed, the first control unit 107 disconnects the switch 103 (step S106). Thereafter, the process waits until the first process and the second process are completed.

  The second process is performed from step S107 to step S111. First, the first control unit 107 supplies the extraction algorithm discrimination information and the extraction auxiliary information, which are information for controlling the frequency component extraction, to the second component extraction unit 106, and updates the stored information. (Step S107).

  Next, since the second component extraction unit 106 acquires or updates an input signal to be referred to, when the switch 104 is connected, the second component extraction unit 106 receives the reference object that is the second input 102 via the switch 104. An input signal is acquired or updated (step S108).

  Next, the second component extraction unit 106 specifies component extraction processing based on the acquired extraction algorithm discrimination information, and specifies signal components to be extracted based on extraction auxiliary information. Thereafter, the specified component extraction process is performed on the acquired input signal to be referenced (step S109). By this processing, the second component extraction unit 106 generates extracted component information of the input signal to be referenced, and generates a component other than the extracted component information as non-extracted component information, thereby separating the signal component. (Step S110).

  When the component extraction process for the input signal to be referred to is completed, the first control unit 107 disconnects the switch 104 (step S111). Thereafter, the first control unit 107 waits until the first process and the second process are completed.

  When the first process and the second process are completed, the first control unit 107 supplies the band suppression algorithm determination information and the band suppression auxiliary information to the band suppression unit 111, and updates the band suppression information (step S112). ).

  Thereafter, the integrated control unit 116 sends a process start command to the second control unit 114.

  The second control unit 114 connects the switch 108 to the terminal a in response to a command from the integrated control unit 116 (step S113).

  Next, the similar component identification unit 109 acquires the extraction component information to be processed from the first component extraction unit 105 via the switch 108, and acquires the extraction component information to be the reference target from the second component extraction unit 106. (Step S114).

  When the extracted component information is supplied to the similar component specifying unit 109, the second control unit 114 connects the switch 108 to the terminal b and the switch 112 to the terminal a (step S115).

  Next, the similar component specifying unit 109 updates the extracted component information to be processed (step S116). Here, when the processing of the band suppressing unit 111 is completed and the newly extracted band-limited combined extracted component information is generated, the similar component specifying unit 109 includes the switch 112 and the terminal b connected to the terminal a. And is updated as extracted component information to be processed. When the newly extracted band-limited combined extracted component information is not generated, the similar component specifying unit 109 updates the extracted component information acquired from the switch 108 connected to the terminal a as the extracted component information to be processed. To do.

  After that, the similar component specifying unit 109 acquires the extracted component information to be referred to from the second component extracting unit 106, and performs a predetermined similar component specifying process in a predetermined processing unit (step S117). It identifies which part of the extracted component information on the reference side corresponds to the signal component that is most similar to the extracted component information that is the object, and generates position information that represents the correspondence.

  Thereafter, the alignment synthesis unit 110 acquires the extracted component information to be processed via the switch 108 and acquires the extracted component information to be the reference target from the second component extraction unit 106. In addition, the alignment synthesis unit 110 acquires position information representing the correspondence between the extracted component information from the similar component specifying unit 109. The alignment synthesis unit 110 performs a predetermined alignment synthesis process using the acquired extracted component information to be processed and referenced and the position information (step S118), thereby associating the processing target The extracted component information on the side and the extracted component information on the reference side are combined to generate combined extracted component information.

  Next, the band suppressing unit 111 acquires the combined extracted component information generated from the alignment combining unit 110. And while specifying a band suppression process based on the acquired band suppression algorithm discrimination | determination information and performing the band suppression process specified based on band suppression auxiliary information (step S119), with respect to synthetic | combination extraction component information The frequency band of the signal is suppressed to generate band-limited combined extraction component information.

  Thereafter, the second control unit 114 acquires the combined extracted component information whose band is limited from the band suppressing unit 111 and acquires the extracted component information that is the processing target from the first component extracting unit 105. Then, based on at least the acquired extracted component information, the combined extracted component information, and the management information, the second control unit 114 determines whether the iterative process has been completed (step S120), thereby setting a predetermined repetition condition. Until it is satisfied, it is determined whether to repeat the reconstruction process of the combined extracted component information or supply the result to each unit.

  When it is determined that the iterative process has been completed (YES in step S120), the second control unit 114 connects the switch 112 to the terminal b (step S121). If it is determined that the repetition process has not been completed (NO in step S120), the process returns to step S116.

  When it is determined in step S120 that the iterative process has been completed and the switch is switched in step S121, the synthesis unit 113 acquires the synthesis extraction component information from the band suppression unit 111 via the switch 112 connected to the terminal b, and The extracted component information after the processing is obtained, and the non-extracted component information is acquired from the first component extraction unit 105. Then, the acquired extracted component information and non-extracted component information are combined by a predetermined combining process (step S122), thereby generating a reconstructed signal.

  Thereafter, the integrated control unit 116 determines whether or not the reconfiguration processing according to the present embodiment has been completed based on the integrated management information (step S123).

  If it is determined that the reconstruction process has been completed (YES in step S123), the process proceeds to step S124. If it is determined that the reconstruction process has not been completed (NO in step S123), the process returns to the first process and the second process, which are parallel processes performed after step 101.

  After determining that the reconfiguration processing has been completed, the integrated control unit 116 determines whether to use a different reference signal based on the integrated management information (step S124).

  If it is determined that a different reference signal is to be used (YES in step S124), the integrated control unit 116 sends a command to the first control unit 107, and the first control unit 107 connects the switch 104 (step S125). The process returns to the first process and the second process, which are parallel processes performed after step 101. If it is determined not to use a different reference signal (NO in step S124), the integrated control unit 116 sends a command to the first control unit 107, connects the switch 115 to the terminal b, and outputs the result (step S126). ).

  A series of basic operations according to the present embodiment is completed through the above steps.

  In this way, the initial reference signal when starting the predetermined super-resolution processing is extracted from the signal included in the region currently being processed (extracted with the frequency included in the predetermined frequency pass band). Component information) and other signals (extracted component information). The separated extracted component information is used as a new initial reference signal, and a signal that is more effectively pasted from the reference side for each included signal component by reconfiguring the signal by performing a predetermined super-resolution process Is specified by a predetermined search unit. By applying higher-precision alignment and synthesis processing to the specified signal to be pasted, signal components are prevented from being mixed from unintended positions due to the inclusion of other signal components. Each time, it is possible to generate an image close to the characteristics of the image that should originally exist.

  In addition, by performing predetermined super-resolution processing, extracted component information after reconstruction of the extracted component information to be processed is generated, and then synthesized with the non-extracted component information on the processing target side. Since the processing of the present embodiment is repeated as a signal to be processed, more probable signal components increase each time the number of repetitions is repeated. As a result, in a predetermined search or registration / composition process, it is possible to further suppress the mixing of signal components from unintended positions, and to generate an image close to the characteristics of the image that should originally exist.

(Embodiment 2)
In the first embodiment of the present invention, mainly by changing the control method of each part of the first control unit 107, the second control unit 114, and the integrated control unit 116, the similar component specifying unit 109, the alignment synthesis unit 110, and the like. Embodiment 2 can be achieved by changing the operation of each unit of the above and preventing the combining unit 113 from functioning.

  Based on the control of the first control unit 107, the second control unit 114, and the integrated control unit 116, the similar component specifying unit 109 is based on the extracted component information of the signal to be processed and the extracted component information on the reference side. The first search unit in the similar component specifying unit 109 obtains position information representing the correspondence relationship. The alignment synthesis unit 110 acquires a signal to be processed via the switch 103 in advance, acquires a signal to be referred to via the switch 104, and generates position information generated from the similar component specifying unit 109. To get. Based on this position information, the alignment synthesis unit 110 identifies the correspondence between the signal processing area that is the current processing target and the original signal processing target area, By using the pasting position specified by the second search unit in the alignment synthesis unit 110 for the signal component included in the identified original signal target region, the predetermined alignment synthesis process, The signal is reconstructed by combining with the original signal component to be processed and generating combined extracted component information whose band is suppressed by the band suppressing unit 111. The final output signal is generated by repeating the processing according to the present embodiment using this as a new signal to be processed.

  In this way, component extraction processing is performed on the signal to be processed and the reference target to generate extracted component information, and the first search unit in the similar component specifying unit 109 uses the extracted component information on the processing target side. The corresponding reference-side extracted component information region is identified, and the second search unit in the alignment synthesis unit 110 determines the high-precision pasting position of the identified reference-side signal in the signal to be processed. Identify. Based on the more correct position information specified by the extracted component information, the signal is reconstructed using the original signal included in the specified reference side area for the specified pasting position. It is possible to paste the signal components included in the signal area that is specified as a reference target at a more probable position, suppressing the mixing of signal components from unintended positions, and An image close to the feature can be generated.

(Embodiment 3)
In the first embodiment of the present invention, mainly by changing the control method of each part of the first control unit 107, the second control unit 114, and the integrated control unit 116, the similar component specifying unit 109, the alignment synthesis unit 110, and the like. Embodiment 3 can be obtained by changing the operation of each part of the above and preventing the combining unit 113 from functioning. The second embodiment is different from the second embodiment in that the position information is specified based on the extracted component information after reconstruction.

  First, a signal to be processed is separated into extracted component information and non-extracted component information, and extracted component information is extracted from the signal to be referenced. Next, the similar component specifying unit 109 uses the extracted component information of the signal to be processed and the extracted component information on the reference side, so that the first search unit in the similar component specifying unit 109 represents the positional information indicating the correspondence relationship. Ask for. After that, the alignment synthesis unit 110 acquires position information, and based on this position information, an extraction component information area on the processing target side and an extraction component information area on the reference side that are currently processed And a signal component included in the identified reference-side extracted component information area is determined by using a pasting position identified by the second search unit in the alignment synthesis unit 110, using a predetermined position. By combining with the signal component of the extracted component information on the processing target side by the alignment combining process, and generating the combined extracted component information band-suppressed by the band suppressing unit 111, the extracted component information on the processing target side is reconstructed. . By repeating the process of reconstructing the extraction component information on the processing target side so far until a predetermined condition is satisfied, the extraction component information on the processing target side after the reconstruction is generated, and the extraction component information on the reference side is again generated. The first search unit in the predetermined similar component specifying unit 109 specifies the position information.

  In the alignment synthesis unit 110, a signal to be processed is acquired in advance via the switch 103, and a signal to be referred to is acquired through the switch 104, and is reconfigured by the similar component specifying unit 109. Position information generated based on the extracted component information on the processing target side is acquired. Based on this position information, the alignment synthesis unit 110 identifies the correspondence between the signal processing area that is the current processing target and the original signal processing target area, The signal component included in the area of the signal that is the original reference target that has been identified is synthesized with the signal component that is the target of the original processing by a predetermined alignment synthesis process, and the band extraction is performed by the band suppression unit 111. The signal is reconstructed by generating component information. The final output signal is generated by repeating the processing of the present invention using this as a new signal to be processed.

  As described above, the component extraction process is performed on the signal to be processed and the reference target to generate the extracted component information, and the predetermined super-resolution process is performed to reconstruct the extracted component information to be processed. Later extracted component information is generated. Thereafter, component extraction processing is performed on the signal to be referenced to generate extracted component information, and the first search unit in the similar component specifying unit 109 corresponds to the extracted component information after reconstruction on the processing target side The region of the reference-side extracted component information to be identified is identified, and the second search unit in the alignment synthesis unit 110 identifies the highly accurate pasting position of the identified reference-side signal in the signal to be processed Then, the signal is reconstructed using the original signal included in the specified reference-side region with respect to the specified pasting position. This makes it possible to specify more correct position information based on the reconstructed extracted component information on the processing target side, and to specify a reference target signal area specified based on this more reliable position information. It is possible to paste the signal components included in the image in more probable positions, suppress the mixing of signal components from unintended positions, and generate an image close to the characteristics of the image that should originally exist. Become.

(Embodiment 4)
Further, as the fourth embodiment, one of the predetermined first, second, or all search units uses, as a template, an area that includes at least a part or all of the area currently being processed. You may comprise so that it may do.

  For example, as shown in FIG. 9, the first search unit in the similar component specifying unit 109 sets the search target area as 902, and uses this area as a template, and includes it in the template from the signal on the reference side. It is preferable to specify a region including a signal component that is most similar to the received signal component.

  In this way, by using, as a template, a region that includes at least a part or all of the region currently being processed as a template, the signal component included in the region to be processed can be faithfully used as a reliable reference. By performing a search process, it is possible to specify a region including the most similar signal component from the reference side.

(Embodiment 5)
Also, as Embodiment 5, unlike Embodiment 4, the predetermined first, second, or all search units do not include the current processing target area, An area that is adjacent to the existing area or that is separated by a predetermined area may be used as a template for the search.

  For example, as shown in FIG. 10, a region 1001 around a region 902 that is the current processing target is used as a template. The in-plane position relationship between the template and the target region is a relationship that can be easily specified. In this example, an area 902 to be searched is included inside the template 1001, and when the position of the template 1001 is determined, the area inside the area is also obtained at the same time. The region corresponding to the template on the reference side that is most similar to the signal component included in such a template is specified. The template being searched is a region 1002, and the template region including the most similar signal component is the region 1003 in FIG. The area corresponding to the area 902 to be identified is the area 907, and the position information 908 can be generated from the relationship between the in-plane spatial positions of the area 902 and the area 907.

  The shape of the template region may be, for example, a shape as shown in FIG. Also, in FIG. 11, a plurality of template areas, for example, all four areas on the top, bottom, left and right of the area 902 to be searched may be used as a template as shown in FIG. The template may be configured to use a part of the template. Further, each area of the template may be adjacent to the search target, or an area separated by a predetermined area may be used as the template.

  In this way, the search unit uses the template, specifies the region corresponding to this template region from the reference side, and then, based on the positional relationship with the template, the region on the processing target side to be actually obtained and the region on the reference side Even if the quality of the signal components in the search target area is not sufficient by obtaining the position information for identifying the correspondence relationship with the template area, the correspondence between the template areas is determined by the signal component in the template area. From the relationship, it is possible to specify the correspondence between the processing target side and the reference side area that are actually desired.

(Embodiment 6)
Further, as Embodiment 6, unlike Embodiment 5, the predetermined first, second, or all search units include at least a part or all of the region currently being processed. You may comprise so that the predetermined feature-value between an area | region and a periphery area | region may be utilized for a search.

  Here, the predetermined feature amount is, for example, an area 902 to be processed in FIG. 11 (j) and its peripheral areas, that is, eight areas of upper and lower left and right, upper left, upper right, lower right, and lower left. In 1110, by performing predetermined frequency conversion on each area, the frequency characteristics of each area and the frequency characteristics between the area currently being processed and its surrounding areas are displayed. It is preferable that the position information can be generated by obtaining the relationship of the distributions of these and specifying the position most similar to the positional relationship of the frequency distribution from the reference side. Here, when searching, the function of specifying the corresponding positional relationship while performing the same predetermined frequency conversion is used as the first search unit in the similar component specifying unit 109 and the function in the alignment synthesizing unit 110. The two search units may be configured to be provided.

  In addition, edge detection is performed for each region, the distribution of edge components is quantified, and the position that is most similar to the positional relationship of the edge components can be identified from the reference side so that position information can be generated. May be. Here, when searching, a function for specifying the corresponding positional relationship while performing similar edge detection is performed by using the first search unit in the similar component specifying unit 109 and the second search unit in the alignment synthesis unit 110. It is preferable that the similar component specifying unit 109 and the alignment combining unit 110 are configured so as to be used in the search unit.

  Furthermore, it is possible to generate position information by specifying the edge direction for each region, quantifying the edge direction, and specifying the position most similar to the positional relationship of the edge direction from the reference side. Anyway. Here, when searching, the function of specifying the corresponding positional relationship while performing the process of specifying the same edge direction is performed in the first search unit in the similar component specifying unit 109 and the alignment combining unit 110. The similar component specifying unit 109 and the alignment synthesizing unit 110 may be configured so as to be used in the second search unit. The edge direction can be specified by, for example, performing a Hough transform on each area, detecting a straight line included in the signal component of each area, and quantifying the direction of the representative straight line. It is good to be able to identify. Here, as an example, the method based on the direction detection by the Hough transform has been described. However, the direction of the straight line included by another different method may be quantified.

  Further, the position information can be generated by obtaining a histogram of the signal of each region or its representative signal and representative color, and specifying the position most similar to the positional relationship of this distribution from the reference side. . Here, when searching, a function for specifying a corresponding positional relationship while performing processing for obtaining a similar histogram, or a representative signal or a representative color thereof, the first search unit in the similar component specifying unit 109, and The similar component specifying unit 109 and the alignment synthesizing unit 110 may be configured so as to be used by the second search unit in the alignment synthesizing unit 110.

  As described above, the predetermined first, second, or all search units are predetermined features between a region including at least a part or all of the region currently being processed and a peripheral region. By configuring the amount to be used for search, even if the quality of the signal component in the region currently being processed or any of its surrounding regions is not sufficient, From the relationship between the feature amounts in the region, it is possible to specify the correspondence between the processing target side and the reference side region that are actually desired.

(Embodiment 7)
Further, as Embodiment 7, unlike Embodiment 6, the predetermined first, second, or all search units include at least a part or all of the region currently being processed. A predetermined feature amount between the region and the peripheral region is obtained, and a change amount of the obtained feature amount is further obtained between the region on the processing target side and the peripheral region, and used for the search. You may comprise.

  For example, in the positional relationship of the regions as shown in FIG. 11 (j), the DC component of the frequency component is obtained, the amount of change between the DC component in the region currently being processed and the DC component in the surrounding region is obtained, The position information may be generated by specifying the position most similar to the positional relationship of the change amounts from the reference side. Here, when searching, the function of identifying the corresponding positional relationship while detecting the same amount of change is the same as the first search unit in the similar component identification unit 109 and the second function in the alignment synthesis unit 110. The search unit may be configured to include.

  In addition, here, as an example, the configuration in which the amount of change between the DC components of the frequency components in each region is used for the search, but the amount of change in other feature amounts as described in the sixth embodiment is quantified, You may comprise so that it may utilize for a search.

  In this way, by obtaining a predetermined amount of change of a predetermined feature amount and performing a predetermined search, it is possible to perform a search using a global change between the surrounding area.

(Embodiment 8)
Furthermore, as an eighth embodiment, any one of the predetermined first, second, or all search units as described above may be used while switching for each region to be processed based on predetermined management information. It is also possible to configure.

  By configuring in this way, for example, in the first several iterations where the quality of the signal component in the region to be processed is not yet sufficient, a search based on the amount of change in the obtained feature value or a search based on a template And then use other searches to improve the quality of the signal components in the final processing target while determining the correct signal components little by little. It is possible to specify the correspondence between the side and the reference side area, and further improve the quality of the signal component in the area to be processed.

(Embodiment 9)
Further, as the ninth embodiment, the similar component specifying unit 109 and the alignment synthesizing unit 110 are provided so that the search unit can use a search function based on a predetermined feature amount change amount. Each time the signal reconstruction process is repeated, different information is supplied to the extraction algorithm discrimination information, which is management information supplied to the component extraction unit 105 and the second component extraction unit 106, and the extraction auxiliary information, which is auxiliary information. By setting management information so as to control the frequency passband when extracting signal components so that the signal is reconstructed in order from the more probable signal components, images from unintended positions The use of the signal component for the super-resolution processing can be suppressed, and an image close to the characteristic of the image that should originally be can be generated.

  For example, auxiliary information is supplied such that the frequency pass band is initially a band that allows low frequency components to pass as shown in FIG. 3C, and extracted component information including low frequency components is generated. Like that. After that, in the search, the DC component of the frequency component of the signal included in the region is obtained as a feature amount, and the change amount between the region to be processed and the surrounding region is obtained, and the search is performed based on this change amount. By processing, position information is generated. After that, the signal reconstruction processing is performed by performing the alignment synthesis processing and the band suppression processing, and by repeating these series of processing, the signal component that is closer to the signal component included in the original signal is obtained, and is low after the reconstruction. Extraction component information including frequency components is determined. Then, the auxiliary component information is sequentially supplied so that the frequency pass band becomes a band that allows higher frequency components to pass, and the same process is repeated to gradually determine the extracted component information. It is preferable that the process is controlled so as to approach an image close to the characteristics of the image that should be originally obtained.

  Here, as an example, a configuration has been described in which the control can be performed so that the signal is determined in order from the lower frequency component, but the extracted component information is sequentially determined in other different frequency passbands. However, the processing may be controlled so as to be close to an image close to the characteristics of the image that should originally be.

(Embodiment 10)
In the first to ninth embodiments, the signal processing device 100 is illustrated as a block diagram and configured as hardware. However, the present invention is not limited thereto, and an information processing device that is a computer is configured. Also good. For example, a central processing control device typified by a CPU (Central Processing Unit), a recording medium or a communication device, a temporary storage device typified by a memory, or an external device typified by an HDD (Hard Disc Drive), for example. Of course, the functions of the first to ninth embodiments may be achieved by software processing using a signal processing program stored in the storage device. In addition, the signal processing apparatus 100 and the signal processing method according to the embodiment may be applied to any apparatus, method, program, system, or the like that reconfigures a signal when the signal is acquired and supplied. There is no particular limitation. For example, pre-processing and post-processing of a broadcasting device typified by a TV, high-quality image processing, DVD (Digital Versatile Disc) -R / RW, BD (Blu-ray) in a mobile phone, a video conference device, and a monitoring device Disc; registered trademark) -R / RW, HDD, SD, recording / playback apparatus using a rewritable and rewritable recording medium, imaging recording / playback apparatus such as a digital camera or camcorder, and recording editing apparatus such as authoring The present invention can be applied to high image quality processing at the time of recording and reproduction in a moving image distribution device or the like. Also, a signal after local decoding, a reference signal, and a prediction signal are generated in pre-processing and post-processing and encoding processing and decoding processing when encoding and decoding images, moving images, music, and the like. It can be applied to signal improvement processing and restoration processing at the time. Furthermore, the present invention can also be applied to error tolerance processing at the time of communication in storage devices, distribution devices, reception devices, and transfer devices for images, moving images, music, and the like.

  The signal processing device 100, the signal processing method, and the signal processing program according to the embodiment include, for example, a monitor such as a TV, a digital camera, a camcorder, a recorder, a monitoring device, and an editing device that records / reproduces / displays / transmits images and sounds. It can be used for an apparatus, a moving image distribution server, and the like.

  The above image processing can be realized as a storage / reception device using hardware, as well as firmware stored in a ROM (Read Only Memory) or flash memory, a computer, etc. It can also be realized by software. The firmware program and software program can be recorded on a computer-readable recording medium, provided from a server through a wired or wireless network, or provided as a data broadcast of terrestrial or satellite digital broadcasting Is also possible.

  DESCRIPTION OF SYMBOLS 100 Signal processing apparatus, 103, 104, 108, 112, 115 switch, 105 1st component extraction part, 106 2nd component extraction part, 107 1st control part, 109 Similar component specific | specification part, 110 Position alignment composition part, 111 band Suppression unit, 113 synthesis unit, 114 second control unit, 116 integrated control unit.

Claims (14)

First component extraction that separates a signal to be processed using a predetermined component extraction process into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that. And
A second component extraction unit that generates extracted component information from a signal to be referenced using a predetermined component extraction process;
A similar component identifying unit that performs processing for generating position information for identifying a region of extracted component information on the reference side corresponding to a region of extracted component information on the processing target side;
Based on the position information generated by the similar component specifying unit, the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side are specified, and the similar component specifying unit specifies the signal to be processed The component position of the reference side is specified with higher accuracy than the position information, and the component component information is generated by pasting the signal component included in the reference-side extracted component information area and synthesizing the signal. An alignment composition unit,
A band suppression unit that generates band-limited synthesis component information by performing predetermined band suppression processing and resampling processing on the synthesis component information generated by the alignment synthesis unit;
An integrated control unit that controls the combined component information generated by the band suppression unit to be subjected to new processing performed by the similar component specifying unit, and repeats signal reconstruction processing ;
Synthesizing the band-limited combined component information generated by the band suppressing unit and the non-extracted component information on the processing target side separated by the first component extracting unit, and generating a reconstructed signal by a predetermined combining process Part and
Prepared,
The first component extraction unit is for processing the reconstructed signal generated by the synthesis unit as a new processing target,
The integrated control unit causes the first component extraction unit to generate extraction component information while switching a frequency pass band included in the extraction component information separated by the first component extraction unit, and repeats signal reconstruction processing. A signal processing device characterized by controlling. First component extraction that separates a signal to be processed using a predetermined component extraction process into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that. And
A second component extraction unit that generates extracted component information from a signal to be referenced using a predetermined component extraction process;
A similar component identifying unit that performs processing for generating position information for identifying a region of extracted component information on the reference side corresponding to a region of extracted component information on the processing target side;
Based on the position information generated by the similar component specifying unit, the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side are specified, and the similar component specifying unit specifies the signal to be processed The component position of the reference side is specified with higher accuracy than the position information, and the component component information is generated by pasting the signal component included in the reference-side extracted component information area and synthesizing the signal. An alignment composition unit,
A band suppression unit that generates band-limited synthesis component information by performing predetermined band suppression processing and resampling processing on the synthesis component information generated by the alignment synthesis unit;
An integrated control unit that controls the combined component information generated by the band suppression unit to be subjected to new processing performed by the similar component specifying unit, and repeats signal reconstruction processing ;
A first search unit that identifies a region of extracted component information on the reference side corresponding to a region of extracted component information on the processing target side;
A second search unit for specifying the reference signal paste position with higher accuracy than the position information specified by the similar component specifying unit in the signal to be processed;
With
The alignment synthesis unit acquires an original signal before component extraction to be processed and an original signal before component extraction to be a reference target,
The similar component identification unit identifies an extraction component information region on the processing target side and an extraction component information region on the reference side in these original signals based on the position information generated by the first search unit. And
The second search unit further specifies the reference signal original signal paste position with higher accuracy than the position information specified by the similar component specification unit in the signal to be processed,
The alignment synthesizing unit pastes the signal component included in the reference-side original signal region and synthesizes the signal to the pasting position specified by the second search unit, thereby synthesizing the component. A signal processing device that generates information.   Synthesizing the band-limited combined component information generated by the band suppressing unit and the non-extracted component information on the processing target side separated by the first component extracting unit, and generating a reconstructed signal by a predetermined combining process Part
Prepared,
  The first component extraction unit is for processing the reconstructed signal generated by the synthesis unit as a new processing target,
  The integrated control unit causes the first component extraction unit to generate extraction component information while switching a frequency pass band included in the extraction component information separated by the first component extraction unit, and repeats signal reconstruction processing. The signal processing apparatus according to claim 2, wherein the signal processing apparatus is controlled. The second search unit specifies a region of reference-side extracted component information corresponding to a region of extracted component information on the processing target side based on the extracted component information after reconstruction generated by the alignment synthesis unit. 4. The signal processing apparatus according to claim 2 , wherein position information for specifying the position information is specified. At least one of the first search unit and the second search unit uses, as a template, a region including at least a part or all of the region currently being processed as a template. The signal processing apparatus according to claim 2 or 3 . At least one of the first search unit and the second search unit does not include the region currently being processed and is adjacent to the region currently being processed or separated by a predetermined region The signal processing apparatus according to claim 2 , wherein the region is used for searching as a template. At least one of the first search unit and the second search unit includes a predetermined feature amount between a region including at least a part or all of the region currently being processed and a peripheral region. The signal processing apparatus according to claim 2 , wherein the signal processing apparatus is used for searching. At least one of the first search unit and the second search unit includes a predetermined feature amount between a region including at least a part or all of the region currently being processed and a peripheral region. 4. The signal processing apparatus according to claim 2 , wherein the signal processing device is used for searching after further obtaining a change amount of the obtained feature amount between the obtained region to be processed and the surrounding region. . The integrated control unit controls at least one of the first search unit and the second search unit to be used for a search while switching to the search unit according to any one of claims 6 to 9. The signal processing apparatus according to claim 2 , wherein the signal processing apparatus is a signal processing apparatus.   The alignment composition unit holds the predicted position information while sequentially updating the position information specified by the search when repeating the reconstruction process of the extracted component information,
  The integrated control unit controls to repeat the signal reconstruction process using the predicted position information held by the alignment synthesis unit as a base point when repeating the reconstruction process of the reconstructed signal. The signal processing device according to any one of claims 1 to 9. First component extraction that separates a signal to be processed using a predetermined component extraction process into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that . Steps,
A second component extraction step of generating extracted component information from a signal to be referenced using a predetermined component extraction process;
A similar component specifying step for performing processing for generating position information for specifying a region of extracted component information on the reference side corresponding to a region of extracted component information on the processing target side;
Based on the generated position information, the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side are specified, and in the signal to be processed, the reference-side information is detected with higher accuracy than the generated position information. A positioning and synthesizing step for generating a synthesized component information by identifying a pasting position of the signal, pasting the signal component included in the extracted component information area on the reference side, and synthesizing the signal;
A band suppression step for generating band-limited composite component information by performing predetermined band suppression processing and resampling processing on the generated composite component information;
An integrated control step for controlling the generated band-limited synthesized component information as a target of new processing in the similar component specifying step and repeating signal reconstruction processing;
Obtains the band-limited combined component information generated in the band suppressing step and the non-extracted component information on the processing target side separated in the first component extracting step, and generates a reconstructed signal by a predetermined combining process synthesis step and a flop that,
The first component extraction step sets the reconstructed signal generated in the synthesis step as a new processing target,
In the integrated control step, the extraction component information is generated while switching the frequency passband included in the extraction component information separated in the first component extraction step, and control is performed to repeat the signal reconstruction process. signal processing method characterized by. First component extraction that separates a signal to be processed using a predetermined component extraction process into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that . Steps,
A second component extraction step of generating extracted component information from a signal to be referenced using a predetermined component extraction process;
A similar component specifying step for performing processing for generating position information for specifying a region of extracted component information on the reference side corresponding to a region of extracted component information on the processing target side;
Based on the generated position information, the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side are specified, and in the signal to be processed, the reference-side information is detected with higher accuracy than the generated position information. A positioning and synthesizing step for generating a synthesized component information by identifying a pasting position of the signal, pasting the signal component included in the extracted component information area on the reference side, and synthesizing the signal;
A band suppression step for generating band-limited composite component information by performing predetermined band suppression processing and resampling processing on the generated composite component information;
An integrated control step for controlling the generated band-limited synthesized component information as a target of new processing in the similar component specifying step and repeating signal reconstruction processing;
A first search step for identifying a region of extracted component information on the reference side corresponding to a region of extracted component information on the processing target side;
A second search step for specifying a reference-side signal paste position with higher accuracy than the position information specified in the similar component specifying step in the signal to be processed;
Have
The alignment synthesis step acquires an original signal before component extraction to be processed and an original signal before component extraction to be a reference target,
The similar component specifying step specifies the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side in these original signals based on the position information generated by the first search step. And
The second search step further specifies the original signal paste position on the reference side with higher accuracy than the position information specified by the similar component specifying step in the signal to be processed;
The alignment synthesis step performs the synthesis component by pasting the signal component included in the original signal region on the reference side and the signal synthesis to the pasting position specified by the second search step. A signal processing method characterized by generating information . First component extraction that separates a signal to be processed using a predetermined component extraction process into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that . Function and
A second component extraction function for generating extracted component information from a signal to be referenced using a predetermined component extraction process;
A similar component specifying function for performing processing to generate position information for specifying the region of the extracted component information on the reference side corresponding to the region of the extracted component information on the processing target side;
Based on the generated position information, the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side are specified, and in the signal to be processed, the reference-side information is detected with higher accuracy than the generated position information. A positioning and synthesizing function for generating composite component information by identifying a signal pasting position, pasting a signal component included in the reference-side extracted component information area, and synthesizing the signal,
A band suppression function for generating band-limited composite component information by performing predetermined band suppression processing and resampling processing on the generated composite component information;
An integrated control function for controlling the generated band-limited synthesized component information as a target of new processing of the similar component specifying function and repeating signal reconstruction processing ;
Synthesizing the band-limited combined component information generated by the band suppressing function and the non-extracted component information on the processing target side separated by the first component extracting function, and generating a reconstructed signal by a predetermined combining process Function and
Is realized on a computer,
The first component extraction function uses the reconstructed signal generated by the synthesis function as a new processing target,
The integrated control function causes the first component extraction function to generate extraction component information while switching a frequency pass band included in the extraction component information separated by the first component extraction function, and repeats signal reconstruction processing. A signal processing program characterized by controlling . First component extraction that separates a signal to be processed using a predetermined component extraction process into extracted component information that is a signal having a frequency included in a predetermined frequency passband and non-extracted component information that is a signal other than that . Function and
A second component extraction function for generating extracted component information from a signal to be referenced using a predetermined component extraction process;
A similar component specifying function for performing processing to generate position information for specifying the region of the extracted component information on the reference side corresponding to the region of the extracted component information on the processing target side;
Based on the generated position information, the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side are specified. A positioning and synthesizing function for generating composite component information by identifying a signal pasting position, pasting a signal component included in the reference-side extracted component information area, and synthesizing the signal,
A band suppression function for generating band-limited composite component information by performing predetermined band suppression processing and resampling processing on the generated composite component information;
An integrated control function for controlling the generated band-limited synthesized component information as a target of new processing of the similar component specifying function and repeating signal reconstruction processing ;
A first search function for specifying a region of extracted component information on the reference side corresponding to a region of extracted component information on the processing target side;
A second search function for specifying a reference-side signal paste position with higher accuracy than the position information specified by the similar component specifying function in a signal to be processed;
Is realized on a computer,
The alignment synthesis function acquires an original signal before component extraction to be processed and an original signal before component extraction to be a reference target,
The similar component specifying function specifies the region of the extracted component information on the processing target side and the region of the extracted component information on the reference side in these original signals based on the position information generated by the first search function. And
The second search function further specifies the original signal paste position on the reference side with higher accuracy than the position information specified by the similar component specifying function in the signal to be processed,
The alignment synthesis function performs the synthesis component by pasting the signal component included in the original signal area on the reference side and synthesizing the signal to the pasting position specified by the second search function. A signal processing program for generating information .

Images