JP5243833B2 - Image signal processing circuit, image display device, and image signal processing method - Google Patents

Description

  The present invention relates to a technique for increasing the resolution of an image and effectively causing a stereoscopic effect and perspective.

  Recent television receivers have become larger in screen size, and do not display image signals input from broadcasting, communication, storage media, etc. as they are, but display them by increasing the number of pixels in the horizontal and vertical directions by digital signal processing. It is generally done. At this time, the resolution cannot be increased only by increasing the number of pixels by an interpolation low-pass filter using a generally known sinc function, a spline function, or the like. Conventional high-resolution technology discloses a technology that increases the number of pixels while increasing the resolution by combining a plurality of input image frames (hereinafter abbreviated as frames) into one frame. (For example, refer to Patent Document 1 and Non-Patent Document 1).

In the high resolution technology (conventional method) described in Patent Document 1 and Non-Patent Document 1, high resolution is achieved by three processes: (1) position estimation, (2) wideband interpolation, and (3) weighted sum. Do. Here, (1) position estimation is to estimate a difference in sampling phase (sampling position) of each image data using each image data of a plurality of input image frames. (2) Wideband interpolation is performed by interpolating using a wide-band low-pass filter that transmits all the high-frequency components included in the original signal to increase the number of pixels (sampling points) of each image data. This is to increase the resolution of the number of pixels. (3) The weighted sum is applied to each high-resolution image data by adding a weighting factor corresponding to the sampling phase and canceling out the aliasing components generated during pixel sampling. In addition, the high frequency component of the original signal is restored.
According to the above-described conventional method (hereinafter referred to as super-resolution processing), it is possible to increase the resolution of the display image.

FIG. 13 is a diagram for explaining the procedure of super-resolution processing. FIG. 13A is a diagram showing an original image to be processed, FIG. 13B is a diagram showing a spatial frequency spectrum of the original image, and FIG. 13C is a processed image after the super-resolution processing. (D) is a figure which shows the spatial frequency spectrum of the image after a process.
An original image 1001 shown in FIG. 13A is configured by the number of pixels of (horizontal) Nx × (vertical) Ny. The original image 1001 can be displayed as an image having a bandwidth of a frequency f that is half the sampling frequency of the imaging system or the display system. Therefore, as shown in FIG. 13B, the spatial frequency spectrum of the original image 1001 is a mixture of frequency components higher than the frequency f as aliasing components.
In the super-resolution processing, the frequency bandwidth of the displayed image is widened by restoring the aliasing component (high-frequency component) using a single image or a plurality of images. As a result, as shown in FIG. 13C, a processed image 1002 (high resolution image) having the number of pixels of (horizontal) 2Nx × (vertical) 2Ny is obtained. The spatial frequency spectrum of this processed image 1002 has a bandwidth 2f that is twice that of the original image 1001, as shown in FIG.
Therefore, in the super-resolution process, it is possible to increase the resolution of the display image.
JP-A-9-69755 Shin Aoki, “Super-resolution processing using multiple digital image data”, Ricoh Technical Report, NOVEMBER, 1998, No.24, p.19-25

However, when the resolution of the image is increased by the super-resolution processing, the blur included in the original image is reduced, and thus there is a problem that the increased resolution causes a reduction in stereoscopic effect and perspective. To do.
The present invention has been made in view of the problems as described above, and provides a technique that achieves both high resolution of an image by the super-resolution processing and effective action of stereoscopic effect and perspective. The purpose is to do.

In order to solve the above-described problem, the present invention provides a plurality of sections divided according to the subject distance related to the depth of the subject in the image, and depth information that associates the sections with the type of image processing. A storage unit for storing, calculating a subject distance of a subject in the input image, referring to the storage unit, identifying the section to which the calculated subject distance belongs, and further, to the specified section An image signal processing circuit that identifies the type of the image processing associated with the image processing and outputs a processed image by performing arithmetic processing on the input image using the image processing of the identified type, type of image processing is at least, from the composite image using a single frame or multiple frames of the input image, when widening the spatial frequency bandwidth restoring the aliasing components of the spatial frequency spectrum Resolution conversion process of increasing the monitor number of pixels in the horizontal direction and the vertical direction, adjacent or emphasize a portion of the pixel signals of the pixel number changing process and the input image to increase the number of pixels without changing the spatial frequency bandwidth Any one or a combination of two or more spatial frequency characteristic changing processes that reduce the difference in pixel value between matching pixels, and the calculation of the subject distance of the subject in the input image is performed on a plurality of screens of the input image. When performing for each minute region divided into two, the calculation processing is performed on the input image using all the image processing set as the image processing type, and the image processing corresponding to all the image processing types is performed. A processed image of a minute area is obtained, the section to which the subject distance calculated for each minute area belongs is specified, and the image associated with the specified section is further specified. Specify the type of processing, acquire the processed image of the minute area corresponding to the specified type of image processing, configure and output the screen of the processed image, and for the section with a small subject distance Te, associates a combination of the resolution converting process or emphasizing the resolution transformation processing and edge the spatial frequency characteristic change processing higher than the resolution of the processed image resolution of the input image, is large the object distance The interval is associated with a combination of the pixel number changing process or the pixel number changing process and the spatial frequency characteristic changing process that narrows the spatial frequency bandwidth as the resolution is lowered .

  According to the present invention, it is possible to achieve both high resolution of an image and effective action of stereoscopic effect and perspective.

<Relationship between high resolution and blur in super-resolution processing>
First, it will be described with reference to FIG. 14 that blurring included in an original image is reduced when the resolution of the image is increased by super-resolution processing. FIG. 14A is a diagram showing blur included in the original image, FIG. 14B is a diagram showing blur included in the processed image having a higher resolution, and FIG. 14C is the original image and the conventional method image. It is a figure which shows the depth perceived in.

An original image 1001 illustrated in FIG. 14A is an image in which a person is photographed against a background of a mountain. The person is located on the near side, is within the depth of field of the imaging system, and can be regarded as being in focus. Accordingly, the amount of blurring (blurring amount) 1102 representing the blurring of the minute area 1101 in the background is larger than the blurring amount 1104 of the minute area 1103 (including the person) of the person. Here, the blur amount is an amount indicating how much a point on the subject spreads on the display image (point spread).
In other words, the blur amount 1104 of the minute area 1103 of the person has a small point spread. On the other hand, in the distant view of the mountain as the background, the focus is greatly deviated because the depth of field of the imaging system is finite. Therefore, the amount of blur 1102 in the distant micro-area 1101 has a large point spread.

  FIG. 14B shows the blur amount of the processed image after the super-resolution processing. The blur amount 1204 of the human minute region 1203 and the blur amount 1202 of the background minute region 1201 are reduced as compared with the case of the original image 1001 by increasing the resolution by super-resolution processing. Therefore, each blur amount 1204 and 1202 has a reduced point spread as compared with the original image 1001.

In FIG. 14C, the horizontal axis represents the arrangement of minute regions including the same pattern, the vertical axis (left side) represents the blur amount, and the vertical axis (right side) represents the perceived depth. In other words, the horizontal axis shows the minute regions corresponding to the same pattern in order to compare the blur amount 1102 of the minute region of the original image 1001 with the blur amount 1202 of the minute region of the processed image 1002. The vertical axis indicates both the amount of blur and the stereoscopic effect and perspective (perceived depth) because the amount of blur depends on the distance to the subject.
The solid line 1301 shown in FIG. 14C is the characteristic of the original image 1001, and the broken line 1302 is the characteristic of the conventional system image (characteristic of the processed image 1002). As shown in FIG. 14C, the blur amount indicated by the broken line 1302 is smaller than the blur amount indicated by the solid line 1301 by the super-resolution processing. Therefore, when the blur amount is compared for a minute region including the same pattern in the original image 1001 and the processed image 1002, the G point 1303 and the H point 1304 are obtained, and the depth perceived by the processed image 1002 is the original image 1001. Becomes narrower than the perceived depth.

  Next, the best mode for carrying out the present invention (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings as appropriate.

<< First Embodiment >>
First, an overview of a processing function for controlling the perceived depth (depth feeling) will be described with reference to FIG. FIG. 1A is a diagram showing an overview of conventional processing functions, and FIG. 1B is a diagram showing an overview of processing functions of the present invention.
As shown in FIG. 1A, in the conventional processing, the image processing 102 is performed on the image input 101, and the result is displayed in the super-resolution processing method 104 (the above-described super-resolution) in the processing system that performs the image display 103. Image processing) was used.

Next, in the processing of the present invention, as shown in FIG. 1B, the image processing 102a includes a plurality of methods (pixel number enlargement processing method 121 (pixel number) including the super-resolution processing method 104 (resolution conversion processing). Change processing), edge enhancement processing method 122 (spatial frequency characteristic change processing), smoothing processing method 123 (spatial frequency characteristic change processing), and the like. For the image input 101, a depth information analysis 124 for analyzing the depth information of the image is provided. The “method designation” 125 output from the depth information analysis 124 is input to the image processing 102 a. The image processing 102a performs image processing by appropriately selecting and combining the methods provided in the image processing 102a in accordance with the method designation 125. The image processing result is displayed on the image display 103.
The parameter setting for the analysis process in the depth information analysis 124 can be changed by the image quality setting 126 and the process setting 127.
The image quality setting 126 has a function of setting how much depth is emphasized. That is, the image quality setting 126 has a function of setting the degree of edge enhancement and the degree of smoothing.
The process setting 127 has a function of setting process parameters. Details of the process setting 127 will be described later.

Here, the pixel number enlargement processing method 121, the edge enhancement processing method 122, and the smoothing processing method 123 used in the image processing 102a will be described below.
The pixel number enlargement processing method 121 is a method with almost no change in resolution. For example, the number of pixels is increased by interpolation such as linear interpolation or non-linear interpolation between pixels. Therefore, when the pixel number enlargement processing method 121 is used, the spatial frequency bandwidth of the processed image 1002 (see FIG. 14B) hardly changes with respect to the spatial frequency bandwidth of the original image 1001.
The edge enhancement processing method 122 is a method for enhancing a difference in pixel values (for example, (luminance, color difference) or RGB) between adjacent pixels. For example, the edge enhancement processing method 122 can enhance the contour.
The smoothing processing method 123 is a method for reducing a difference in pixel values between adjacent pixels. For example, the smoothing processing method 123 can blur the outline. That is, the smoothing processing method 123 can make the spatial frequency bandwidth of the processed image 1002 narrower than the spatial frequency bandwidth of the original image 1001.

Next, the relationship between the resolution (processing) and the sense of depth, that is, the depth distance (estimation) in the depth information analysis 124 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of the relationship between resolution (processing) and depth distance (estimation).
According to human perceptual characteristics, the relationship between the resolution (processing) and the depth distance (estimation) is such that, as the depth distance (estimation) increases, the resolution (processing) decreases as the depth distance (estimation) increases. It tends to be good. This is because, as shown in FIG. 14C, when the blur amount is large, the perceived depth is felt far away. Then, for an image region having a somewhat large amount of blur, a method that does not improve the blur amount (for example, the pixel number enlargement processing method 121 or the smoothing processing method 123) is used instead of performing super-resolution processing. It would be better to be there.

Next, the method designation 125 that is the output of the depth information analysis 124 will be described with reference to FIGS. FIG. 3 is a diagram illustrating an example of method selection in the case of specifying the “small depth distance” method. FIG. 4 is a diagram illustrating an example of method selection in the case of designating a method of “large depth distance”.
FIG. 3 illustrates a case where the depth information analysis 124 determines that the depth distance is small for a minute region in the image input 101. From the result of the depth information analysis 124, the method designation 125a is information for selecting the super-resolution processing. Thereby, the resolution of the image input 101 is increased by the super-resolution processing 104, and the image with the increased resolution is displayed on the image display 103.

  FIG. 4 illustrates a case where the depth information analysis 124 determines that the depth distance is large for a minute region in the image input 101. From the result of the depth information analysis 124, the method designation 125 b is information for selecting processing by the smoothing processing method 123 after processing by the pixel number enlargement processing method 121. As a result, the number of pixels of the image input 101 is increased by the pixel number enlargement processing method 121, and then smoothed by the smoothing processing method, and the processed image is displayed on the image display 103.

  Next, detailed processing functions of the depth information analysis 124 will be described with reference to FIG. FIG. 5A is a diagram illustrating an example of a processing function when focus blur amount estimation is used in depth information analysis, and FIG. 5B is a process when spatial frequency analysis is used in depth information analysis. It is a figure which shows an example of a function, (c) is a figure which shows an example of a processing function at the time of using depth distance estimation in depth information analysis.

As shown in FIG. 5A, the depth information analysis 124a receives input of an image and executes processing in the order of region division 141, focal blur amount estimation 142, depth amount threshold value processing 143, and processing assignment 144. Output the specified information.
The area division 141 has a function of dividing input image information into minute areas on the screen. The micro area is an area having a width of the arrangement of adjacent pixels vertically and horizontally, and is, for example, a rectangular area of (horizontal) dx × (vertical) dy shown in FIG. It is. Note that the size of the minute area is set by the process setting 127.
Then, each process of focal blur amount estimation 142, depth amount threshold processing 143, and processing allocation 144 is performed for each minute region.

The focal blur amount estimation 142 has a function of estimating the focal blur amount of the camera lens and estimating the depth distance from the estimation result. The focal blur amount is estimated based on, for example, the correlation value including the pixel value in the minute region or the surrounding pixel value including the minute region.
The depth amount threshold value processing 143 has a function of setting a threshold value of the length of the estimated depth distance. This threshold value is used as a comparison reference when determining a method to be used for processing according to the estimated length of the depth distance. Note that this threshold value is made variable by the image quality setting 126.
The process assignment 144 has a function of comparing and determining the estimated depth distance length and the threshold set by the depth amount threshold process 143 and determining which method is used as long as it is within a range between certain thresholds. .

  In the focus blur amount estimation 142 described above, the depth distance is estimated from the focus blur amount. However, only the function of estimating the focus blur amount (focus blur amount analysis) may be used. In this case, it is assumed that the depth amount threshold processing 143 has a function of setting a threshold value of the estimated focal blur amount.

As shown in FIG. 5B, the depth information analysis 124b receives input of an image and executes processing in the order of region division 151, spatial frequency analysis 152, depth amount threshold processing 153, and processing allocation 154. Outputs specified information.
The area division 151 has a function of dividing the input image information into minute areas on the image, like the above-described area division 141. However, the size of the micro area used in the area division 141 may be different. Note that the size of the minute area is set by the process setting 127.
Then, each process of spatial frequency analysis 152, depth amount threshold processing 153, and processing allocation 154 is performed for each minute region.

The spatial frequency analysis 152 has a function of obtaining a spatial frequency bandwidth and estimating a depth distance from the result. The reason why the depth distance can be estimated from the spatial frequency bandwidth is based on the fact that the spatial frequency bandwidth of the subject image depends on the distance to the subject because the depth of field of the camera lens is finite. That is, a far object is blurred and the spatial frequency bandwidth is narrow, and a near object is less blurred and the spatial frequency bandwidth is wide. Based on this relationship, the depth distance is estimated for each minute region.
The depth amount threshold value processing 153 has a function of setting a threshold value of the length of the estimated depth distance. This threshold value is used as a comparison reference when determining a method to be used for processing according to the estimated length of the depth distance. Note that this threshold value is made variable by the image quality setting 126. The threshold value may be different from the setting by the depth amount threshold value processing 143 described above.
The process assignment 154 has a function of comparing and determining the length of the estimated depth distance and the threshold set by the depth amount threshold process 153 and determining which method is used as long as it is within a range between certain thresholds. .

  In the spatial frequency analysis 152 described above, the depth distance is estimated from the spatial frequency bandwidth. However, only the function of calculating the spatial frequency bandwidth may be used. In this case, it is assumed that the depth amount threshold processing 153 has a function of setting a threshold value of the calculated spatial frequency bandwidth.

As shown in FIG. 5C, the depth information analysis 124c receives image input and executes processing in the order of region division 161, depth distance estimation 162, depth amount threshold processing 163, and processing allocation 164. Outputs specified information.
The area division 161 has a function of dividing the input image information into minute areas on the image, similarly to the above-described area divisions 141 and 151. However, the size of the micro area used in the area divisions 141 and 151 may be different. Note that the size of the minute area is set by the process setting 127.
Then, each process of depth distance estimation 162, depth amount threshold value processing 163, and process allocation 164 is performed for each minute region.

The depth distance estimation 162 has a function of estimating the depth distance. The depth distance estimation 162 calculates the depth distance estimated by the focal blur amount estimation 142 (see FIG. 5A) and the depth distance estimated by the spatial frequency analysis 152 (see FIG. 5B), respectively. The weighted depth distance is calculated by adding weights to both sides.
The depth amount threshold processing 163 has a function of setting a length threshold of the weighted depth distance. This threshold is used as a comparison reference when determining a method to be used for processing according to the length of the weighted depth distance. Note that this threshold value is made variable by the image quality setting 126. Further, this threshold value may be different from the setting by the depth amount threshold value processing 143 and 153 described above.
The process assignment 164 has a function of comparing and determining the length of the weighted depth distance and the threshold set by the depth amount threshold process 163 and determining which method is used as long as it is within a range between certain thresholds.

  Next, the flow of processing in the first embodiment will be described with reference to FIGS. FIG. 6A is a diagram illustrating an example of a minute region, FIG. 6B is a diagram illustrating an example of a blur amount of the minute region, and FIG. 6C is a relationship between the blur amount and the depth distance estimation value. It is a figure which shows an example. FIG. 7A is a diagram illustrating an example of the relationship between the depth distance estimation value and the depth amount threshold value, and FIG. 7B is a diagram illustrating an example of a processing method allocation table that associates the depth amount threshold value with the processing method. It is.

FIG. 6A shows an original image 201 to be processed. Then, the screen of the original image 201 is divided into a plurality of minute regions 202. This process is performed by the area division 141 shown in FIG.
The size of the minute region 202 is (horizontal) dx × (vertical) dy, and is determined by the processing setting 127 (see FIG. 5). The size of the minute area 202 is determined in consideration of the number of vertical and horizontal pixels of the entire original image 201, the spatial frequency bandwidth of the original image 201, the image quality after image processing, the calculation processing speed when mounted, and the like.

FIG. 6B shows that the blur amount (focus blur amount) Brxy is calculated and estimated for each minute region 202. In FIG. 6C, a data range 211 indicates data plotted by obtaining a relationship between the blur amount Brxy and a known depth distance in advance. The prediction function 210 indicates a tendency derived by regression analysis, a least square method, or the like using data in the data range 211. When the blur amount Brxy (see FIG. 6B) is calculated, the depth distance estimated value (d) can be calculated (estimated) using the prediction function 210. Note that the prediction function 210 is represented as a straight line in FIG. 6C, but may be a higher-order curve. The prediction function 210 may be a mathematical formula or a table.
The processes shown in FIGS. 6B and 6C are performed by the focal blur amount estimation 142 shown in FIG.
In addition, when calculating the spatial frequency bandwidth instead of calculating the blur amount Brxy, the processing is performed by the spatial frequency analysis 152 illustrated in FIG. Furthermore, when calculating the weighted depth distance estimation value (d) instead of calculating the blur amount Brxy, the processing is performed by the depth distance estimation 162 shown in FIG.

  Next, FIG. 7A shows that a threshold value for dividing the depth distance estimation value (d) into a plurality of sections is set. FIG. 7A shows an example in which six threshold values (d1 to d6) are set. The processing for setting this threshold is performed by the depth amount threshold processing 143 shown in FIG. The threshold value is determined by the image quality setting 126 (see FIG. 5).

  FIG. 7B shows a processing method allocation table 220 (depth information) in which processing methods corresponding to the sections divided by the threshold values (d1 to d6) are associated with each other. For example, in the processing method allocation table 220, if the depth distance estimation value (d) is within the section from d3 to d4, image processing is executed by the processing method 4. The association between the threshold and the processing method shown in the processing method allocation table 220 is determined by the image quality setting 126 (see FIG. 5).

  Then, as indicated by the arrow in FIG. 7A, the flow of processing is such that when the blur amount Brxy is given, the estimated depth distance (d) is between the threshold values d3 and d4 using the prediction function 210. It is determined that Next, it is determined with reference to the processing method allocation table 220 that processing is performed in the processing method 4. This process is performed by the process assignment 144 shown in FIG.

The processing flow shown in FIGS. 6A and 6B and FIGS. 7A and 7B uses the processing flow in the depth information analysis 124a, that is, the focus blur amount (blur amount Brxy). Shown if there was. The processing flow in the depth information analysis 124b is shown in FIGS. 6A and 6B and FIGS. 7A and 7B by replacing the focal blur amount (blur amount Brxy) with the spatial frequency bandwidth. The process flow is the same. In addition, the processing flow in the depth information analysis 124c (see FIG. 5C) also replaces the focal blur amount (blur amount Brxy) with the weighted depth distance, and FIGS. The processing flow is the same as shown in a) and (b).
Although detailed description is omitted, a depth vector may be estimated by detecting a motion vector for each minute region and using the fact that the size of the motion vector is large near and small near. In this case, the focal blur amount is replaced with the magnitude of the motion vector, and the processing flow is the same as that shown in FIGS. 6 (a) and 6 (b) and FIGS. 7 (a) and 7 (b).

<< Specific Example 1 >>
The processing flow shown in FIG. 7 is shown by a specific example using the depth information analysis 124a shown in FIG. 5, and a specific example of the effect of improving the sense of depth perceived in the processed image is shown in FIG. This will be described with reference to FIG. FIG. 8A is a diagram showing a specific example showing the setting of the threshold value of the depth distance estimation value, and FIG. 8B shows a specific example showing the relationship between the region divided by the threshold value and the minute region. FIG. 8C is a diagram illustrating a specific example of a processing method allocation table that associates regions classified by thresholds with processing methods. FIG. 9A is a diagram showing a specific example showing the blur amount of the processed image, and FIG. 9B is a specific example showing the perceived depth in the original image, the processed image, and the conventional system image. FIG.

As shown in FIG. 8A, the depth distance estimation value (d) is divided into five regions (A, B, C, D, E) by threshold values dA, dB, dC, dD. This threshold value is set by the image quality setting 126 (see FIG. 5A).
And in each area | region (A, B, C, D, E), as shown in FIG.8 (b), the person of the original image 201 is in the area E with the shortest depth distance estimated value (d). The background is set in the region D, the region C, the region B, and the region A in order from the nearest due to the difference in the depth distance estimation value (d).
Then, as shown in FIG. 8C, the processing method allocation table 230 is set by the image quality setting 126 (see FIG. 5A). That is, the processing method (5, 4, 3, 2, 1) corresponding to each region (A, B, C, D, E) is related. In the processing method assignment table 230, a high resolution method (super-resolution processing method) is assigned to the processing of the region E of the closest person. Further, the processing of the farthest background area A is assigned with a simple pixel number enlargement method without changing the resolution so as to preserve the blur amount of the original image 201. In the processing of the regions B, C, and D that are intermediate regions, after high resolution (super-resolution processing), smoothing processing (strong, medium, weak) with different frequency bandwidth filters (strong, medium, weak) Medium, weak) is assigned. Note that the smoothing process (strong) has a function of increasing the blur amount more than the smoothing process (medium). Further, the smoothing process (medium) has a function of increasing the blur amount more than the smoothing process (weak).

  As shown in FIG. 9A, in the post-processing image 240 obtained after the processing shown in FIG. 8, the minute region 243 belonging to the region E is processed by the super-resolution processing method. Therefore, the blur amount 244 of the minute region 243 is smaller than the blur amount of the original image 201 (see FIG. 8B). Further, the micro area 241 belonging to the area A is processed by the simple pixel number enlargement method. Therefore, the blur amount 242 of the minute area 241 is not different from the blur amount of the original image 201.

The solid line 250 shown in FIG. 9B is the characteristic of the original image, the alternate long and short dash line 251 is the characteristic of the processed image processed using the processing method shown in FIG. 8C, and the broken line 252 is the conventional method image. It is a characteristic.
As is clear from FIG. 9B, the blur amount indicated by the broken line 252 is smaller than the blur amount indicated by the solid line 250 in a minute region including the same pattern. On the other hand, the blur amount of the dashed-dotted line 251 is close to the blur amount of the broken line 252 when the blur amount is small, but close to the solid line 250 when the blur amount is large. That is, in a minute region including the same pattern, the perceived depth of the point A 253 of the alternate long and short dash line 251 is farther than the point B 254 of the broken line 252.
Since the maximum blur amount of the alternate long and short dash line 251 substantially matches the solid line 250, the perceived depth for the processed image 240 is substantially the same as in the original image.

<< Specific Example 2 >>
The processing flow shown in FIG. 7 is shown in another specific example using the depth information analysis 124a shown in FIG. 5 and another specific example of the effect of improving the perceived depth perception in the processed image. This will be described with reference to FIG. FIG. 10A is a diagram showing another specific example showing the relationship between the area and the micro area divided by the threshold, and FIG. 10B is a process for associating the micro area and the processing method divided by the threshold. It is a figure which shows the other specific example of a system allocation table, (c) is a figure which shows the other specific example showing the depth perceived in an original image, a post-process image, and a conventional system image.

As shown in FIG. 10 (a), the person of the original image 201 is in the region E having the shortest depth distance estimated value (d), and the background is the region in order from the closest due to the difference in the depth distance estimated value (d). It is assumed that the image is divided into D, region C, region B, and region A.
Then, as shown in FIG. 10B, the processing method allocation table 260 is set by the image quality setting 126 (see FIG. 5A). That is, the processing methods (a, b, c, d, e) corresponding to the respective regions (A, B, C, D, E) are related. In the processing method assignment table 260, a method of performing edge processing for emphasizing the perceived depth distance closer to the processing of the closest person region E after high resolution (super-resolution processing) is assigned. . Thereby, in the area | region E, the high frequency part of a spatial frequency is emphasized and the blurring amount reduces relatively.
Further, the processing of the farthest background area A is assigned with a method in which the number of simple pixels is expanded and then smoothed by filtering to further enhance blur. Then, the processing methods shown in the processing method assignment table 260 are assigned to the processes of the regions B, C, and D that are intermediate regions.

The solid line 270 shown in FIG. 10C is the characteristic of the original image, the alternate long and short dash line 271 is the characteristic of the processed image processed using the processing method shown in FIG. 10B, and the broken line 272 is the conventional image. It is a characteristic.
As is apparent from FIG. 10C, the blur amount indicated by the broken line 272 is smaller than the blur amount indicated by the solid line 270 in a minute region including the same pattern. In the range where the blur amount is small (region E), the perceived depth (blur amount) of the alternate long and short dash line 271 can produce an effect of enhancing the stereoscopic effect by edge processing, and the perceived depth of the dashed line 272 ( Smaller than the blur amount).
In the range where blur amount is large (area A), the perceived depth (blur amount) of the alternate long and short dash line 271 is larger than the perceived depth (blur amount) of the solid line 270 due to the effect of the smoothing process. That is, the perceived depth of the C point 273 of the alternate long and short dash line 271 is farther than the D point 274 of the solid line 270 in a minute region including the same pattern. As a result, the range of the blur amount indicated by the alternate long and short dash line 271 is wider than the range of the blur amount indicated by the solid line 270, so that the perceived depth is also increased.
As described above, in the second specific example, the perceived depth is made farther than the original image, and the three-dimensional effect and the perspective effect can be applied.

<< Second Embodiment >>
Next, a configuration example of an image display device to which the above-described processing is applied in the first embodiment will be described with reference to FIG. FIG. 11 is a diagram illustrating a configuration example of an image display device.
In FIG. 11, the image display apparatus 300 includes an input unit 311, a recording / playback unit 312, a control unit 313, a user interface unit 314, a content storage unit 315, an image signal processing unit 316, an audio output unit 317, and a display unit 318. Is done.
Note that the double lines in FIG. 11 indicate the flow of image and audio signals, and the single lines indicate the flow of control signals.

The input unit 311 receives an input of a broadcast signal, a video signal, and a broadcast / video content 320 including an image and sound via a broadcast wave including a television signal, a network, or the like. The input unit 311 may include a tuner that receives a broadcast wave, a LAN connector that connects to a network, or a USB connector. The input unit 311 may be provided with a terminal for digitally inputting an image signal or an audio signal, or may be provided with an analog input terminal such as a composite terminal or a component terminal. Alternatively, the input unit 311 may be a receiving unit that transfers data wirelessly.
The recording / playback unit 312 records or plays back the content received by the input unit 311. Note that a function of passing a signal received from the input unit 311 may be provided.

The control unit 313 has a function of controlling each unit (311, 312, 314 to 318) of the image display device 300, and manages information transmission between the units (311, 312, 314 to 318). . The control unit 313 includes, for example, a CPU (Central Processing Unit) (not shown) that executes arithmetic processing and a memory (not shown) that is a main storage device used by the CPU for arithmetic processing. The memory is realized by a RAM (Random Access Memory) or a ROM (Read Only Memory). Then, the application program stored in the memory is expanded in the RAM, and the CPU implements various processes by executing it.
The user interface unit 314 has a function for a user to operate the image display apparatus 300.

The content storage unit 315 has a function of storing content via the recording / playback unit 312. The content storage unit 315 may be, for example, a hard disk drive, a flash memory, or a removable media disk drive.
The image signal processing unit 316 has a function of processing the image signal described in the first embodiment. A detailed configuration of the image signal processing unit 316 will be described later.
The audio output unit 317 has a function of outputting the audio signal reproduced by the recording / reproducing unit 312. The audio output unit 317 may be, for example, a speaker.
The display unit 318 has a function of displaying the image processed by the image signal processing unit 316. The display unit 318 may be, for example, a plasma panel module, an LCD (Liquid Crystal Display) module, or a projector device.

The image display apparatus 300 may be, for example, a plasma television, a liquid crystal television, a cathode ray tube, a projector, or an apparatus using another device. The image display apparatus 300 includes the image signal processing unit 316 that performs the processing described in the first embodiment, so that the resolution of the image signal received from the input unit 311 is increased, and the perspective and stereoscopic effect are provided. It can be displayed on the display unit 318 as a high-quality image signal.
The processing described in the first embodiment has been described as being performed by the image signal processing unit 316, but may be realized by software (program) in the control unit 313.

Next, a configuration example of the image signal processing unit 316 illustrated in FIG. 11 will be described with reference to FIG. 12 (see FIGS. 1, 5, and 11 as appropriate). FIG. 12 is a diagram illustrating a configuration example of the image signal processing unit.
The input to the image signal processing unit 316 is an image signal input 401 output from the recording / playback unit 312, an image quality setting 412 received by the user interface unit 314 and output via the control unit 313, and a control unit by the producer. This is a process setting 411 set via 313. The output of the image signal processing unit 316 is an image signal output 403, which is input to the display unit 318.
The function of the image quality setting 412 is the same as the function of the image quality setting 126 shown in FIG. The function of the process setting 411 is the same as the function of the process setting 127 shown in FIG.

The image signal processing unit 316 (image signal processing circuit) includes a depth information analysis processing unit 421, a threshold determination / processing unit 422, a super-resolution processing unit 423, a pixel number enlargement processing unit 424, a filter characteristic setting unit 425, and a filter processing unit. 426 and an output selection unit 427.
The functions of the depth information analysis processing unit 421 are the function of the area division 141 and the function of the focal blur estimation 142 shown in FIG. 5A, the function of the area division 151 and the spatial frequency analysis 152 shown in FIG. And the function of the area division 161 and the function of the depth distance estimation 162 shown in FIG. Further, the function of the depth information analysis processing unit 421 is set by the processing setting 411.
The functions of the threshold determination / processing unit 422 are the functions of the depth amount threshold processing 143 and the processing allocation 144 shown in FIG. 5A and the functions and processing allocation of the depth amount threshold processing 153 shown in FIG. 154 and any one or a combination of the depth threshold processing 163 and the processing allocation 164 shown in FIG. 5C. The function of the threshold determination / processing unit 422 is set by the image quality setting 412.
A line 430 drawn from the threshold determination / processing unit 422 to the output selection unit 427 corresponds to the “method designation 125” illustrated in FIG.

The super-resolution processing unit 423 has a function of performing processing by the above-described super-resolution processing method (resolution conversion processing).
The pixel number enlargement processing unit 424 has a function of performing processing by a pixel number enlargement processing method (pixel number change process).
The filter characteristic setting unit 425 receives the output of the image quality setting 412, determines the level of strength of the filter in the smoothing process and the level of edge enhancement, and the filter in the edge enhancement process or smoothing process in the filter processing unit 426. It has a function to control the strength of.
The filter processing unit 426 receives the output of the super-resolution processing unit 423 and the output of the pixel number enlargement processing unit 424, and performs edge enhancement processing (spatial frequency characteristic change processing) or smoothing processing (spatial frequency characteristic change processing). It has a function. However, although only one line is output from the filter processing unit 426 to the output selection unit 427 in FIG. 12, the type of image signal corresponds to the number of filters used in the edge enhancement processing or smoothing processing. Is output.
Note that the super-resolution processing unit 423, the pixel number enlargement processing unit 424, and the filter processing unit 426 always perform repetitive processing for each micro region having the same size as the micro region divided by the depth information analysis processing unit 421. It shall be.

  The output selection unit 427 receives a line 430 indicating the output of the threshold determination / processing unit 422, that is, “method designation 125 (see FIG. 1)”, and outputs an image processed by the designated processing method for each minute region. Has a function to select.

  Note that each of the process setting 411 and the image quality setting 412 includes a function for storing setting information at the time of shipment, a function for storing setting information set by the user, and the like.

  As described above, the image display apparatus 300 (see FIG. 11) according to the present embodiment performs the depth information analysis 124 (see FIG. 1B) on the image to be processed, and analyzes the depth information. Based on the result, an image processing method (super-resolution processing method 104, pixel number enlargement processing method 121, edge enhancement processing method 122, smoothing processing method 123, etc. (see FIG. 1B)) is performed for each minute region of the image. Since a function of selecting and processing appropriately is provided, it is possible to prevent the stereoscopic effect and the perspective from being impaired even when the resolution of the image is increased.

(A) is a figure which shows the outline | summary of the conventional processing function, (b) is a figure which shows the outline | summary of the processing function of this invention. It is a figure which shows an example of the relationship between resolution (process) and depth distance (estimation). It is a figure which shows an example of the system selection in the case of designation | designated of the system of "the depth distance is small." It is a figure which shows an example of the system selection in the case of designation | designated of the system of "large depth distance". (A) is a figure which shows an example of the processing function at the time of using a focus blurring amount estimation in depth information analysis, (b) is a processing function at the time of using spatial frequency analysis in depth information analysis. It is a figure which shows an example, (c) is a figure which shows an example of the processing function at the time of using depth distance estimation in depth information analysis. (A) is a figure which shows an example of a micro area | region, (b) is a figure which shows an example of the blur amount of a micro area | region, (c) is an example of the relationship between the blur amount and depth distance estimated value. FIG. (A) is a figure which shows an example showing the relationship between a depth distance estimated value and a depth amount threshold value, (b) is a figure which shows an example of the processing method allocation table which links | relates a depth amount threshold value and a processing method. . (A) is a figure which shows one specific example showing the setting of the threshold value of a depth distance estimated value, (b) is a figure which shows one specific example showing the relationship between the area | region classified by the threshold value, and a micro area | region. And (c) is a diagram showing a specific example of a processing method allocation table for associating regions classified by thresholds with processing methods. (A) is a figure which shows one specific example showing the blurring amount of a processed image, (b) shows one specific example showing the depth perceived in an original image, a processed image, and a conventional system image. FIG. (A) is a figure which shows the other specific example showing the relationship between the area | region classified by the threshold value, and a micro area | region, (b) is processing method allocation which associates the micro area | region classified by the threshold value and a processing system. It is a figure which shows the other specific example of a table, (c) is a figure which shows the other specific example showing the depth perceived in an original image, a post-process image, and a conventional system image. It is a figure which shows the structural example of an image display apparatus. It is a figure which shows the structural example of an image signal process part. (A) is a figure which shows the original image used as a process target, (b) is a figure which shows the spatial frequency spectrum of an original image, (c) is a figure which shows the image after a process after performing a super-resolution process (D) is a figure which shows the spatial frequency spectrum of the image after a process. (A) is a figure which shows the blur contained in the original image, (b) is a figure which shows the blur contained in the post-processing image which made high resolution, (c) is a perception in the original image and the conventional system image It is a figure which shows the depth made.

Explanation of symbols

124 Depth information analysis 125 Method designation 126,412 Image quality setting 127,411 Processing setting 141,151,161 Region division 142 Focus blur amount estimation 143,153,163 Depth amount threshold processing 144,154,164 Processing allocation 201 Original image 210 Prediction function 220, 230, 260 Processing method allocation table 240 Processed image 250, 270 Original image characteristic 251, 271 Processed image characteristic 252, 272 Conventional method image characteristic 300 Image display device 313 Control unit 314 User interface unit 316 Image signal processing unit 318 Display unit 421 Depth information analysis processing unit 422 Threshold determination / processing unit 423 Super-resolution processing unit 424 Pixel number enlargement processing unit 425 Filter characteristic setting unit 426 Filter processing unit 427 Output selection unit

Claims (5)

A plurality of sections divided according to the subject distance related to the depth of the subject in the image, and a storage unit that stores depth information that associates the sections with the type of image processing;
The subject distance of the subject in the input image is calculated, the section to which the calculated subject distance belongs is identified with reference to the storage unit, and the image processing associated with the identified section is further performed. Identify the type,
An image signal processing circuit that performs arithmetic processing on the input image and outputs a processed image using the specified type of image processing,
The type of the image processing is at least a composite image using a single frame or a plurality of frames of the input image, restoring the aliasing component of the spatial frequency spectrum to widen the spatial frequency bandwidth and changing the number of pixels in the horizontal and vertical directions. Resolution conversion processing that increases, pixel number change processing that increases the number of pixels without changing the spatial frequency bandwidth, and enhancement of part of the pixel signal of the input image or reduction in pixel value difference between adjacent pixels Any one or a combination of two or more of spatial frequency characteristic change processing,
When performing calculation of the subject distance of the subject in the input image for each minute region obtained by dividing the screen of the input image into a plurality of areas,
Performing an arithmetic process on the input image using all the image processing set as the type of image processing to obtain a processed image of the micro region corresponding to all the types of image processing;
Identifying the section to which the subject distance calculated for each minute region belongs, further identifying the type of image processing associated with the identified section;
Acquire a processed image of the minute region corresponding to the specified type of image processing, configure and output a screen of the processed image,
For the section where the subject distance is small, the resolution conversion process that makes the resolution of the processed image higher than the resolution of the input image or the combination of the resolution conversion process and the spatial frequency characteristic change process that emphasizes edges associate,
The pixel number changing process or the pixel number changing process and a combination of the spatial frequency characteristic changing process for narrowing the spatial frequency bandwidth as the resolution is reduced are associated with the section where the subject distance is large.
Image signal processing circuit, characterized in that. Depth information that associates the focal blur amount of the image with the subject distance is stored in the storage unit,
In calculating the subject distance of the subject in the input image, a focal blur amount analysis is performed to calculate the focal blur amount of the subject of the input image, and the depth blur is referred to and the calculated focal blur amount is used to calculate the subject blur distance. Calculating the subject distance,
The image signal processing circuit according to claim 1. Depth information associating the spatial frequency bandwidth of the image with the subject distance is stored in the storage unit,
In calculating the subject distance of the subject in the input image, a spatial frequency analysis is performed to calculate a spatial frequency bandwidth of the subject of the input image, and the depth information is referred to and the calculated spatial frequency bandwidth is used to calculate the subject distance. Calculating the subject distance,
The image signal processing circuit according to claim 1. An image display device that performs arithmetic processing on an input image and displays the processed image,
An image signal processing unit, a display unit for displaying the processed image, a control unit for outputting image quality setting information for controlling the image quality of the processed image to the image quality signal processing unit, and a subject distance related to the depth of the subject in the image A plurality of sections divided according to the storage section, and a storage unit that stores depth information that associates the sections with the type of image processing.
The image signal processor is
Accepting the image quality setting information from the control unit, setting to change one or both of the number of sections and the width of the section,
The subject distance of the subject in the input image is calculated, the section to which the calculated subject distance belongs is identified with reference to the storage unit, and the image processing associated with the identified section is further performed. Identify the type,
Using the specified type of image processing, the input image is subjected to arithmetic processing to output a processed image,
The type of the image processing is at least a composite image using a single frame or a plurality of frames of the input image, restoring the aliasing component of the spatial frequency spectrum to widen the spatial frequency bandwidth and changing the number of pixels in the horizontal and vertical directions. Resolution conversion processing that increases, pixel number change processing that increases the number of pixels without changing the spatial frequency bandwidth, and enhancement of part of the pixel signal of the input image or reduction in pixel value difference between adjacent pixels Any one or a combination of two or more of spatial frequency characteristic change processing,
When performing calculation of the subject distance of the subject in the input image for each minute region obtained by dividing the screen of the input image into a plurality of areas,
Performing an arithmetic process on the input image using all the image processing set as the type of image processing to obtain a processed image of the micro region corresponding to all the types of image processing;
Identifying the section to which the subject distance calculated for each minute region belongs, further identifying the type of image processing associated with the identified section;
Acquiring a processed image of the micro region corresponding to the specified type of the image processing, configuring a screen of the processed image, displaying the configured screen of the processed image on the display unit,
For the section where the subject distance is small, the resolution conversion process that makes the resolution of the processed image higher than the resolution of the input image or the combination of the resolution conversion process and the spatial frequency characteristic change process that emphasizes edges associate,
The pixel number changing process or the pixel number changing process and a combination of the spatial frequency characteristic changing process for narrowing the spatial frequency bandwidth as the resolution is reduced are associated with the section where the subject distance is large.
An image display device characterized by that . A plurality of sections divided according to the subject distance related to the depth of the subject in the image, and a storage unit that stores depth information that associates the sections with the type of image processing;
The subject distance of the subject in the input image is calculated, the section to which the calculated subject distance belongs is identified with reference to the storage unit, and the image processing associated with the identified section is further performed. Identify the type,
An image signal processing method used in an image signal processing circuit that performs arithmetic processing on the input image and outputs a processed image using the specified type of image processing,
The type of the image processing is at least a composite image using a single frame or a plurality of frames of the input image, restoring the aliasing component of the spatial frequency spectrum to widen the spatial frequency bandwidth and changing the number of pixels in the horizontal and vertical directions. Resolution conversion processing that increases, pixel number change processing that increases the number of pixels without changing the spatial frequency bandwidth, and enhancement of part of the pixel signal of the input image or reduction in pixel value difference between adjacent pixels Any one or a combination of two or more of spatial frequency characteristic change processing,
The image signal processing circuit includes:
When performing calculation of the subject distance of the subject in the input image for each minute region obtained by dividing the screen of the input image into a plurality of areas,
Performing an arithmetic process on the input image using all the image processing set as the type of image processing to obtain a processed image of the micro region corresponding to all the types of image processing;
Identifying the section to which the subject distance calculated for each minute region belongs, further identifying the type of image processing associated with the identified section;
Acquire a processed image of the minute region corresponding to the specified type of image processing, configure and output a screen of the processed image,
For the section where the subject distance is small, the resolution conversion process that makes the resolution of the processed image higher than the resolution of the input image or the combination of the resolution conversion process and the spatial frequency characteristic change process that emphasizes edges associate,
The pixel number changing process or the pixel number changing process and a combination of the spatial frequency characteristic changing process for narrowing the spatial frequency bandwidth as the resolution is reduced are associated with the section where the subject distance is large.
Image signal processing method, characterized in that.

Images