JP2017156821A - Trembling recognition amount estimation device and trembling recognition amount estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately estimate a trembling amount which a viewer who views a trembling video recognizes, namely a trembling recognition amount.SOLUTION: A trembling recognition amount estimation device 1 comprises: a movement vector for every area detection part 10 for detecting a movement vector; a movement area determination part 30 for determining presence/absence of movement; a movement area number application part 40 for applying a movement area number; a rotation center rotation angle calculation part 50 for acquiring a rotation center and a rotation angle in whole screen; a rotation vector calculation part 60 for acquiring a rotation vector; a rotation residual vector calculation part 70 for acquiring a rotation residual vector; a space frequency energy calculation part 100 for acquiring a corrected space frequency energy; a correlation value between movement areas calculation part 110 for acquiring a correlation value between movement areas; a correlation value between a boundary and an inner part of a movement area calculation part 120 for acquiring a correlation value between a boundary and an inner part of a movement area; and a total trembling energy calculation part 130 for calculating a total trembling energy.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a motion recognition amount estimation device and a motion recognition amount estimation program for estimating a motion recognition amount for a video with shaking.

  Discomfort induced by screen shaking (video shaking) tends to increase with the increase in the display size when viewing broadcasts, recorded videos, personal computer videos, etc. at home. There are also cases in which people suffer from health problems due to motion sickness. Most of the screen shaking is caused by the motion of the video camera at the time of shooting, that is, global motion. In order to prevent such a situation, it is almost essential to install a “camera shake correction” function in a hand-held video camera or the like. In addition, a lot of “Video Stabilization” software has been developed to reduce the global motion contained in the captured video. Furthermore, an attempt has been made to estimate an unpleasant state by searching for a relationship between global motion and physiological indices such as pulse and blood pressure (see Non-Patent Document 1).

  However, with the above-mentioned “camera shake correction”, long-period shaking cannot be corrected or is insufficient, and “video stabilization” eliminates or reduces the shaking of the video within a short period of time, or smooth screen movement. It is not based on an objective standard that it can be left only if it is shaken by what kind of property and how big by simply converting it. Moreover, even if physiological indices such as pulse and blood pressure can be used as this objective standard, it is possible to specify only the conditions for reaching health hazard and the current physiological state for those conditions. .

  On the other hand, suppliers of video content such as broadcasts and movies have not only video safety but also global motion (total screen movement) or a certain area of the screen in the video production stage. Care must be taken not to cause discomfort associated with movement. However, since the above-described conventional technology does not specifically grasp the psychological “discomfort level” of the viewer with respect to the movement of the entire screen or a certain area, the video based on the “discomfort level”. It was unsuitable for judging the quality of.

“Feasibility Study Report on Practical Use of Visual Sickness Guidelines Verification System”, Mechanical Systems Promotion Association, March 2008

  In order to solve this problem, the applicant of this application analyzes the video using the relationship between the movement of the entire screen or a certain area of the screen and the discomfort obtained in the subjective evaluation experiment without using a physiological index. An unpleasantness estimation device that estimates an uncomfortable degree with respect to screen shaking based on the physical feature values obtained in this way was developed (Japanese Patent Application No. 2015-040609, unpublished at the time of filing this application).

  However, even with the above-described discomfort level estimation apparatus that introduces various techniques for reducing the discomfort level estimation error, the square root of the square error average of the discomfort level estimated in five ranks for a general captured image. (R.m.s.e (root mean square error)) remains about 0.5 rank, and an error of ± 1 rank or more occurs in one scene in 20 scenes of an image estimated to have the same rank of discomfort. It was. If practicality is required, it is desirable that errors of ± 1 rank or more occur within 1 scene in 100 scenes. This is because the r. m. s. This corresponds to suppressing e to 0.4 rank or less, that is, further reducing the error by 20%.

  However, in interviews with subjects in the evaluation experiment of discomfort, there were many comments such as “There are images that feel shaking but not uncomfortable,” or “discomfort varies depending on the image even with shaking of the same magnitude.” It has been found that it is difficult to reduce the discomfort degree estimation error to a target value with the method of the discomfort degree estimation device that directly estimates the discomfort degree from the physical feature quantity of the video.

  In other words, in order to estimate the degree of discomfort of screen shaking with high accuracy, first, the amount of perceived shaking is estimated with high accuracy, and then the degree of discomfort is estimated using the estimated amount of perception of shaking and the physical features of the video. Need to be estimated.

  The present invention has been created in view of the above-described circumstances, and it is possible to estimate with high accuracy the amount of shaking initially recognized when a viewer views a video with shaking (smoothing video). It is an object of the present invention to provide a motion recognition amount estimation device and a motion recognition amount estimation program.

  In order to solve the above-described problem, the motion recognition amount estimation device according to the present invention is based on the motion vector in the video and the partial spatial frequency of the video, that is, the motion amount initially recognized by the viewer who viewed the video. A motion recognition amount estimation device for estimating motion recognition amount, a motion vector detection unit for each region, a first filter unit, a motion region determination unit, a motion region number assigning unit, a rotation center / rotation angle calculation unit A rotation vector calculation unit, a rotation residual vector calculation unit, a second filter unit, a spatial frequency sensitivity correction unit for each region, a spatial frequency energy calculation unit, a correlation value calculation unit between motion regions, and a motion region An internal boundary correlation value calculation unit and a fluctuation energy correction unit are provided.

  The motion recognition amount estimation device detects a motion vector between temporally adjacent images of the video for each region obtained by dividing the screen into a plurality of regions by a region-specific motion vector detection unit, and a first filter unit detects the motion vector. A motion vector is subjected to frequency sensitivity correction to obtain a corrected motion vector, and a motion region determination unit determines that there is motion in the corresponding region when the size of the corrected motion vector is equal to or greater than a threshold, A region number assigning unit assigns a motion region number that is different for each region belonging to the same motion region and reflects the context between the regions based on the difference of the corrected motion vector from the adjacent region.

  Note that the first filter unit may include a first digital filter and a second digital filter. With this configuration, the motion recognition amount estimation device performs frequency sensitivity correction on the vertical component of the motion vector using the first digital filter, and applies the second digital filter to the horizontal component of the motion vector. Frequency sensitivity correction different from that of one digital filter is performed.

  Further, the motion recognition amount estimation device may be configured to determine that there is motion when the motion region determination unit includes the corrected motion vector that is equal to or greater than the threshold value within a predetermined time immediately before.

  Further, in the motion recognition amount estimation device, the sum of the squares of the magnitudes of the motion vectors based on the rotation angle is maximized in the region obtained by integrating all the regions to which the motion region numbers are assigned by the rotation center / rotation angle calculation unit. And calculating a rotation center and a rotation angle when a sum of square errors between the motion vector based on the rotation angle and the corrected motion vector is minimized, and the rotation vector calculation unit calculates the motion vector based on the rotation angle in each region. The rotation vector is calculated as a rotation vector, and the residual of the rotation vector and the corrected motion vector in each region is calculated as a rotation residual vector by the rotation residual vector calculation unit.

  In addition, the fluctuation recognition amount estimation device obtains a corrected rotation vector by performing frequency sensitivity correction on the rotation vector by the second filter unit.

  The second filter unit may include a third digital filter. According to such a configuration, the motion recognition amount estimation device uses the third digital filter to change the frequency characteristic of the relative sensitivity with respect to the rotation vector to the other component subjected to frequency sensitivity correction by the first filter unit. Corresponding frequency relative sensitivity correction, that is, frequency relative sensitivity correction different from both the first digital filter and the second digital filter is performed.

  In addition, the motion recognition amount estimation device performs sensitivity correction according to the spatial frequency component on the luminance signal of the video for each region obtained by dividing the screen by the region-specific motion vector detection unit by the region-specific spatial frequency sensitivity correction unit. Then, after obtaining the spatial frequency sensitivity corrected video signal, the spatial frequency energy calculation unit calculates the energy of the spatial frequency sensitivity corrected video signal as the corrected spatial frequency energy.

  In addition, the motion recognition amount estimation device determines whether the foreground region and the background region at the motion region boundary are based on the region to which the motion region number is assigned and the rotation residual vector by the motion region correlation value calculation unit. The correlation value of the rotation residual vector is calculated as the correlation value between motion regions.

  In addition, the motion recognition amount estimation device may include a motion vector and a motion of the motion region boundary itself based on the region to which the motion region number is assigned and the rotation residual vector by the motion region inner boundary correlation value calculation unit. The correlation value of the rotation residual vector with the inside of the region is calculated as the correlation value between the motion region inner boundaries.

  Finally, the perturbation recognition amount estimation device uses a total motion energy calculation unit to calculate the square of the magnitude of the corrected rotation vector in the region and the rotation residual for each region to which the same motion region number is assigned. The individual motion energy obtained by calculating the sum of the squares of the vectors is sequentially corrected according to the corrected spatial frequency energy, the correlation value between the motion regions, and the correlation value between the motion region inner boundaries, and then the motion. A total sum of the individual shaking energy corrected in the region is obtained, and the square root of the obtained sum is used as the shaking energy for each motion region for each motion region number, and then the motion region of all the motion regions is classified by the motion region. The total shaking energy corresponding to the estimated value of shaking perception is obtained by nonlinearly adding shaking energy.

  With such a configuration, even when a subject to be photographed having different spatial frequency components in various places on the screen such as a general video moves differently from the background while blocking the background, the viewer perceives the fluctuation. The amount, that is, the amount of motion recognition can be estimated with high accuracy.

  The present invention can also be embodied as a motion recognition amount estimation program that causes a computer to function as the motion recognition amount estimation device described above.

  According to the present invention, even when a subject to be photographed having various spatial frequency components at various points on the screen such as a general image moves differently from the background while blocking the background, the viewer recognizes it. It is possible to estimate the amount of motion to be performed, that is, the amount of motion recognition with high accuracy.

It is a block diagram which shows the structural example of a discomfort degree estimation system provided with the fluctuation recognition amount estimation apparatus which concerns on embodiment of this invention. It is a block diagram which shows the fluctuation recognition amount estimation apparatus which concerns on embodiment of this invention. It is a graph which shows an example of the frequency characteristic of the motion recognition sensitivity for every component of a motion vector. It is a figure which shows an example of the frequency characteristic of the shake recognition relative sensitivity of the rotation direction component equivalent to the ratio of the shake recognition sensitivity of the rotation direction component with respect to the shake recognition sensitivity of the radial direction component of a motion vector. It is a figure which shows an example of the characteristic which carries out a sensitivity correction | amendment with respect to the luminance signal of an image | video according to a spatial frequency component. It is a figure which shows an example of the characteristic which correct | amends the vibration energy according to area | region according to the correlation value between motion areas. It is a figure which shows an example of the characteristic which correct | amends the vibration energy according to area | region according to the correlation value between motion area | region inner boundaries.

Hereinafter, embodiments of the present invention will be described with reference to the drawings as appropriate.
As shown in FIG. 1, a discomfort degree estimation system A according to an embodiment of the present invention includes a motion recognition amount estimation device 1 and a discomfort degree estimation device 2. The discomfort degree estimation device 2 is capable of suppressing the discomfort degree estimation error by using the motion recognition amount estimation device 1, and includes a feature amount extraction unit 3 and a discomfort degree estimation unit 4 as functional units. And comprising. However, since the configuration of the discomfort degree estimation device 2 shows an example of use of the motion recognition amount estimation device 1 according to the embodiment of the present invention, it is not an essential element in the embodiment of the present invention.

<Moving recognition amount estimation device>
As shown in FIG. 2, the motion perception amount estimation apparatus 1 according to the embodiment of the present invention estimates motion perception amount with respect to screen motion based on a motion vector in a video. Detection unit 10, first filter unit 20, motion region determination unit 30, motion region number assigning unit 40, rotation center / rotation angle calculation unit 50, rotation vector calculation unit 60, and rotation residual vector calculation Unit 70, second filter unit 80, region-specific spatial frequency sensitivity correction unit 90, spatial frequency energy calculation unit 100, motion region correlation value calculation unit 110, and motion region inner boundary correlation value calculation unit 120. And a total shaking energy calculation unit 130. Among such functional units, the motion vector detection unit 10 for each region, the first filter unit 20, the motion region determination unit 30, the rotation vector calculation unit 60, the rotation residual vector calculation unit 70, the second filter unit 80, and each region Regarding the combination of the spatial frequency sensitivity correction units 90, a plurality (N) sets are provided corresponding to a plurality (1 to N) regions obtained by dividing the screen.

<Motion vector detection unit by region>
The region-specific motion vector detection unit 10 acquires a video signal of a region corresponding to the detection unit 10, and based on the acquired video signal, a motion vector between temporally adjacent images (continuous frames) of the video (Difference between frames) is detected in time series. The region-specific motion vector detection unit 10 outputs the detection result to the first filter unit 20. In this embodiment, the area-specific motion vector detection unit 10 detects the vertical component and the horizontal component of the motion amount as the motion vector.

<First filter part>
The first filter unit 20 acquires the motion vector detected by the region-specific motion vector detection unit 10, and applies the corrected motion vector by performing different frequency sensitivity corrections in the vertical direction and the horizontal direction on the acquired motion vector. obtain. The first filter unit 20 outputs the obtained corrected motion vector to the motion region determination unit 30, the motion region number assigning unit 40, the rotation center / rotation angle calculation unit 50, and the rotation residual vector calculation unit 70. . In the present embodiment, the first filter unit 20 includes a first digital filter 21 and a second digital filter 22 in parallel.

  The first digital filter 21 performs frequency sensitivity correction on the vertical component of the motion vector, and the impulse corresponding to the sensitivity of the vertical motion perception shown in FIG. 3 with respect to the vertical component of the motion vector. By applying digital filter processing that convolves and integrates the response, the vertical component of the corrected motion vector is obtained. The first digital filter 21 uses the vertical direction component of the obtained corrected motion vector as a motion region determination unit 30, a motion region number assigning unit 40, a rotation center / rotation angle calculation unit 50, and a rotation residual vector calculation unit. Output to 70.

  The second digital filter 22 performs frequency sensitivity correction on the left-right direction component of the motion vector, and an impulse corresponding to the sensitivity of the motion perception amount in the left-right direction shown in FIG. 3 with respect to the left-right direction component of the motion vector. By applying digital filter processing that convolves and integrates the response, the left-right direction component of the corrected motion vector is obtained. In addition, the second digital filter 22 uses the motion direction determination unit 30, the motion region number assigning unit 40, the rotation center / rotation angle calculation unit 50, and the rotation residual vector calculation unit as the left and right direction components of the obtained corrected motion vector. Output to 70.

  In the first digital filter 21, the second digital filter 22, and the third digital filter 81 to be described later, discrete Fourier transform can be used instead of convolution integration. Since the estimation apparatus 1 performs frequency correction for each of a large number of areas on one screen, it is desirable to use a convolution integral with a small amount of calculation.

<Motion region determination unit>
The motion region determination unit 30 acquires the corrected motion vector output from the first filter unit 20, and indicates that “there is motion” in the corresponding region when the magnitude of the motion vector is equal to or greater than a predetermined threshold. judge. In the present embodiment, the motion region determination unit 20 squares the vertical motion amount (that is, the magnitude of the vertical component of the corrected motion vector) and the horizontal motion amount (that is, corrected) for simplification of calculation. It can be determined that “there is motion” when the sum of the square of the left and right direction component of the motion vector is equal to or greater than a threshold value (the square of the threshold value). The motion area determination unit 30 sets different values for “with motion” and “without motion”, for example, “0” for “with motion” and “−1” for “no motion”. The setting result is output to the motion region number assigning unit 40 as the determination result of the presence or absence of motion.

<Motion area number assigning unit>
The motion region number assigning unit 40 assigns a motion region number to each region belonging to the same motion region with respect to the region determined to have motion. The motion region number assigning unit 40 is a region adjacent to one region when the magnitude of a difference vector between the corrected motion vector of the one region and the corrected motion vector of the adjacent region is equal to or less than a predetermined value. Are the “same motion region”, and the same “motion region temporary number” is assigned to them. Such a predetermined value can be set by using the relationship between the movement of the entire screen or a certain area of the screen and the amount of motion perception obtained in the subjective evaluation experiment.

  Further, the motion region number assigning unit 40 determines the front-rear relationship of the regions by examining which region shields the other region between frames at the boundary between adjacent “different motion regions”. Based on the result of examining all the motion areas, after rearranging the “motion area temporary numbers” in order from the motion area at the back, “1” along the order of rearrangement in each motion area. A formal “motion area number” is assigned in order. When multiple non-adjacent motion regions are adjacent to the same motion region, the context of the multiple motion regions cannot be specified, but the context is broken between all adjacent motion regions. As long as you do not, you can change the order. However, even when two non-adjacent motion areas A and B are adjacent to the same motion area C and have the same context, another motion area D adjacent to the motion area C is still in motion area A. And B are adjacent to each other and have different contexts, the order of the contexts of the motion regions A, B, C, and D is uniquely determined. It should be noted that a known method similar to a method for detecting occlusion of a subject in a video can be employed as a method for determining the order of the order of movement regions.

  Here, the motion region numbers are assigned in order from the motion region at the back, but it is only necessary that the size relationship of the motion region numbers represents the context, so that the motion region number from the foremost motion region is in order. A motion area number may be assigned.

  Further, “−1” is assigned to the area determined not to move, as in the case of the movement area determination unit 30. This value does not need to be limited to “−1” and may be a specific value different from any of the motion region numbers assigned to the regions determined to have motion, but the motion region number is a positive value. Therefore, it is desirable to set “−1” which is the largest negative value.

  In addition, even if the region is determined to have motion, “0” is assigned to a region that has a different motion from any of the adjacent regions, similar to the motion region determination unit 30. This value need not be limited to “0”, and is preferably set to “0” that can be easily distinguished from a positive motion region number and a region number “−1” determined to have no motion. .

  The motion region number assigning unit 40 uses the motion region number assigned to each region in the above procedure as the rotation center / rotation angle calculation unit 50, the motion region correlation value calculation unit 110, and the motion region internal boundary correlation value calculation unit 120. , And the total sway energy calculation unit 130.

<Rotation center / rotation angle calculation unit>
The rotation center / rotation angle calculation unit 50 acquires the corrected motion vector output from the first filter unit 20 and the motion region number output from the motion region number assigning unit 40, and the motion region number is the above-described motion region number. In all areas in the screen that are neither “0” nor “−1”, the corrected motion vector is used, the sum of the squares of the magnitude of the motion vector due to the rotation angle is maximized, and the motion vector due to the rotation angle The rotation center and the rotation angle when the sum of the square error (mean square error) between the actual corrected motion vector and the actual corrected motion vector is minimized are calculated. Further, the rotation center / rotation angle calculation unit 50 outputs the calculated rotation center and rotation angle to the rotation residual vector calculation unit 60.

  When the center of rotation of the screen is not aligned with the center of gravity of the moving area, the viewer pays attention to the inclination of the horizontal and vertical lines, so the rotation in all moving areas is in phase. In this case, the viewer is centered on the center of rotation where the sum of the squares of the amount of rotation is greater, not the amount of rotation after subtracting the average of the in-phase motion (the value of the rotation angle multiplied by the distance to the center of gravity) The rotation amount (value obtained by multiplying the rotation angle by the distance to the rotation center) is perceived.

  If the sensitivity of the motion perception amount in the in-phase rotation direction is the same as that of the motion perception amount in other directions, either the rotation amount after subtracting the average of the motion or the rotation amount centered on the rotation center is used. Even if there is, the total shaking energy, which is the sum of the shaking energy of all the movement components, becomes the same.

  However, as shown in FIG. 3, the sensitivity of the perceived amount of motion in the in-phase rotation direction is 2 Hz or less and higher than the sensitivity of the perceived amount of motion in the other directions. The energy is larger when the rotation amount around the rotation center is used than when the rotation amount after subtracting the average of movement is used.

  As described above, the motion recognition amount estimation device 1 according to the present invention calculates the rotation center and the rotation angle without subtracting the average of the motion, so that the center of rotation of the screen is shifted from the center of gravity of the region in which the motion exists. The accuracy of estimating the amount of motion perception is improved.

<Rotation vector calculation unit>
The rotation vector calculation unit 60 acquires the rotation center and rotation angle output from the rotation center / rotation angle calculation unit 50, and based on the rotation center and rotation angle, the rotation direction at the center of the region (to the rotation center). Vector in the direction orthogonal to the direction), and a set of the vertical and horizontal components of the vector is output as a rotation vector to the rotation residual vector calculation unit 70 and the second filter unit 80.

<Rotation residual vector calculation unit>
The rotation residual vector calculation unit 70 acquires the corrected motion vector output from the first filter unit 20 and the rotation vector output from the rotation vector calculation unit 60, and calculates the corrected motion vector and the rotation vector. A difference vector is calculated, and a set of a vertical component and a horizontal component of the calculated difference vector is used as a rotation residual vector, and a correlation value calculation unit 110 between motion regions, a correlation value calculation unit 120 between motion region internal boundaries, and And output to the total shaking energy calculation unit 130.

  The rotation residual vector can be decomposed into a radial direction (direction toward the rotation center) component and a rotation direction component that is out of phase with the rotation vector. However, the frequency sensitivity correction for the radial direction component and the non-in-phase rotational direction component in FIG. 3 performs the frequency sensitivity correction of the vertical direction component of FIG. It is clear that the frequency sensitivity correction of the left-right component in FIG. 3 is the same, and the frequency sensitivity correction has already been performed by the first filter unit 20 described above. Absent.

<Second filter section>
The second filter unit 80 acquires the rotation vector output from the rotation vector calculation unit 60, obtains a corrected rotation vector by performing frequency relative sensitivity correction on the rotation vector, and obtains the corrected rotation vector thus obtained. It outputs to the total shaking energy calculation part 130. In the present embodiment, the second filter unit 80 includes a third digital filter 81.

  The third digital filter 81 performs a digital filter process that convolves and integrates the same impulse response corresponding to the frequency characteristic of the relative sensitivity of the motion perception amount shown in FIG. 4 for each of the vertical and horizontal components of the rotation vector. As a result, an up-down direction component and a left-right direction component of the corrected rotation vector are obtained and output to the total shaking energy calculation unit 130.

  Here, regarding the frequency sensitivity correction of the rotation vector, the vertical component and the horizontal component based on the relative sensitivity of the rotational component with respect to the sensitivity of the vertical component and the relative sensitivity of the rotational component with respect to the sensitivity of the horizontal component. However, the same frequency relative sensitivity correction is performed for the vertical component and the horizontal component based on the relative sensitivity of the rotational component to the sensitivity of the radial component.

  This is because when the original vector (that is, the corrected motion vector) is decomposed again into the radial component and the rotational component, a part of the vertical component of the original vector becomes the horizontal component of the rotational vector, This is because part of the horizontal direction component of the original vector becomes the vertical direction component of the rotation direction vector.

  As described in the description of the rotation residual vector calculation unit 70, the same frequency as the frequency sensitivity correction of the radial component performed by the frequency sensitivity correction of the vertical component and the frequency sensitivity correction of the horizontal component by the first filter unit 20 Since the sensitivity correction has already been applied to the rotational direction component, the correction to be newly performed by the third digital filter 81 is a frequency relative sensitivity correction based on the relative sensitivity of the rotational direction component to the sensitivity of the radial direction component. ing.

<Spatial frequency sensitivity correction unit by region>
The area-specific spatial frequency sensitivity correction unit 90 performs a screen display by the area-specific motion vector detection unit as pre-processing for calculating the energy after the spatial frequency sensitivity correction necessary for correcting the influence of the visibility of the area. For each divided region, the spatial frequency sensitivity correction shown in FIG. Further, the spatial frequency sensitivity correction unit 90 for each region reflects the luminance signal of the image subjected to the spatial frequency sensitivity correction (before reflection of the influence level of the light and dark contrast) in order to reflect the influence level of the local light and dark contrast. By dividing by the square root of the direct current component of the video signal in the region, a spatial frequency sensitivity corrected video signal is obtained. In addition, the spatial frequency sensitivity correction unit 90 for each region outputs the obtained spatial frequency sensitivity corrected video signal (after reflecting the influence level of light and dark contrast) to the spatial frequency energy calculation unit 100.

  The characteristics shown in FIG. 5 do not cause a large error even if the characteristics in the horizontal, vertical, and diagonal directions are the same (more precisely, the sensitivity in the high frequency region in the diagonal direction is 80% of the sensitivity in the horizontal and vertical directions). However, the increase in the estimation error of the motion perception amount is less than 0.1 rank), so that the two-dimensional shape obtained by rotating the characteristic of FIG. 5 by 360 ° with the vertical axis passing through the origin of the graph as the central axis Spatial frequency sensitivity correction can be performed by convolving and integrating a two-dimensional impulse response, which is a spatial frequency sensitivity characteristic, with a luminance signal of an image.

<Spatial frequency energy calculator>
The spatial frequency energy calculation unit 100 obtains the sum of the squared values of the spatial frequency sensitivity corrected video signals output from the regional spatial frequency sensitivity correction unit 90 in the region, thereby affecting the visibility of the region. The energy after the spatial frequency sensitivity correction necessary for correcting the degree is calculated. In addition, the spatial frequency energy calculation unit 100 outputs the calculated energy after the spatial frequency sensitivity correction to the total vibration energy calculation unit 130.

<Motion region correlation value calculation unit>
The inter-motion area correlation value calculation unit 110 acquires outputs from the motion area number assigning unit 40 and the rotation residual vector calculation unit 70, and calculates the inter-motion area correlation value based on these outputs. Specifically, the correlation value calculation unit 110 between the motion regions calculates the correlation value of the rotation residual vector between the foreground region and the background region at the boundary of the motion region, thereby determining the degree of influence of the shake correlation between the foreground and the background. A correlation value between motion areas necessary to correct for is calculated. Further, the inter-motion region correlation value calculation unit 110 outputs the calculated inter-motion region correlation value to the total shaking energy calculation unit 130.

In the present embodiment, the inter-motion area correlation value calculation unit 110 calculates the average motion vector of the pixel group in the area A adjacent to the boundary between the area A and the area B, that is, the average motion vector as V _A , and the area A and the area B as When the average motion vector of the pixel group in the region B adjacent to the boundary of the region, that is, the average motion vector is V _B , the inter-motion region correlation value C ₁ in a unit time (for example, 2 to 3 seconds) is calculated by the following equation: To do.
C ₁ = Σ (V _A · V _B ) / Σ (‖V _A ‖ × ‖V _B ‖)
Here, “·” represents an inner product.

<Motion region inner boundary correlation value calculation unit>
In a swaying image, even if the area is the same and the size of the swaying, the amount of perception of motion is greater when the entire region is swaying than when only the interior is swaying without moving the boundary of the region. Has the property of becoming larger. The motion region inner boundary correlation value calculation unit 120 acquires outputs from the motion region number assigning unit 40 and the rotation residual vector calculation unit 70, and calculates a correlation value between the motion region inner boundaries based on these. Specifically, the correlation value calculation unit 120 between the motion region inner boundaries calculates the correlation value of the rotation residual vector between the motion vector of the motion region boundary itself and the motion region, thereby correcting the above-described property. The correlation value between the motion region inner boundaries necessary for performing the above is obtained and output to the total shaking energy calculation unit 130.

In the present embodiment, the correlation value calculation unit 120 between the motion region inner boundaries is V _A1 , which is the average motion vector of the pixel group on the outermost periphery (boundary with another region) of the region A, that is, the outermost periphery of the region A. When the average of the motion vectors of the other pixel groups (that is, inside), that is, the average motion vector is V _A2 , the correlation value C ₂ between the motion region inner boundaries in a unit time (for example, 2 to 3 seconds) is calculated by the following equation To do.
C ₂ = Σ (V _A1 · V _A2 ) / Σ (‖V _A1 ‖ × ‖V _A2 ‖)
Here, “·” represents an inner product.

<Total shaking energy calculation part>
The total shaking energy calculation unit 130 includes the motion region number output from the motion region number assigning unit 40, the rotation residual vector output from the rotation residual vector calculation unit 70, and the second filter unit 80. Between the corrected rotation vector output, the corrected spatial frequency energy output from the spatial frequency energy calculation unit 100, the correlation value between motion regions output from the correlation value calculation unit 110 between motion regions, and the inner boundary between motion regions The motion area correlation value output from the correlation value calculation unit 120 and the same motion area to which a positive value is assigned based on the motion area number of each area output from the motion area number assignment unit 40 For each number region, the sum of the square of the magnitude of the rotation vector and the square of the magnitude of the rotation residual vector is calculated. In addition, the total shaking energy calculation unit 130 performs a process of multiplying the calculated result by a value obtained by multiplying the corrected spatial frequency energy by α as a correction for the influence of the visibility of the region (α≈0.3). Then, as a correction for the degree of influence of the correlation between the foreground and the background, a process of multiplying a correction value corresponding to the correlation value between the motion areas shown by the characteristics of FIG. 6 is performed. In addition, the total shaking energy calculation unit 130 is multiplied by the correction value corresponding to the correlation value between the motion areas, and further depends on how much the boundary of the motion area itself is shaking in synchronization with the inside of the area. As correction, the individual shake energy is obtained by applying the correction value corresponding to the correlation value between the motion area internal boundaries shown in the characteristics of FIG. 7, and then the sum of the individual shake energy is calculated for each motion area number. After that, the total motion energy corresponding to the motion perception amount estimation value is calculated by performing nonlinear addition on all motion region numbers. The total shaking energy calculation unit 130 outputs the calculated total shaking energy to the discomfort degree estimation device 2 (see FIG. 1) as an estimated value of the shaking recognition amount.

  Here, for the motion region number-specific swing energy, the square root of the sum of the individual swing energy in each motion region number is used. The reason is that, as in the example shown in FIG. 1, when the degree of discomfort is estimated using the motion recognition amount estimation device 1 according to the present invention, the degree of discomfort degree estimation accuracy is the highest as with normal energy. This is because it is clear that the value obtained by multiplying the square of the area by the area is not the total vibration energy, but the square root of the value is the total vibration energy.

The total shaking energy calculation unit 130 calculates and outputs the total shaking energy corresponding to the shaking recognition amount estimation value by nonlinearly adding the shaking energy for each moving region for all the motion region numbers. The method of nonlinear addition, for another motion area number upset all motion area number n energy a _n,
(Σa _n ^γ ) ^{1 / γ}
Calculate and do.

Here, since the vibration energy of the present invention is the square root of normal energy, when γ = 2, it corresponds to the calculation of the total vibration energy using normal energy.
When γ> 1, the larger the value of γ, the smaller the total shaking energy obtained by the nonlinear addition, and the larger the number of motion region numbers, the smaller the total shaking energy.
In particular, when γ≈6, it has been clarified that the estimation error of the perceived amount of motion when the plurality of regions individually show different motion is minimized.

<Estimated discomfort estimation device>
Next, returning to FIG. 1, the feature extraction unit 3 and the discomfort level estimation unit 4 of the discomfort level estimation device 2 will be described.

<Feature extraction unit>
The feature amount extraction unit 3 acquires a video signal, and extracts the extracted physical feature amount based on the acquired video signal into a physical feature amount of the video that causes discomfort other than the motion recognition amount. Output to the discomfort degree estimation unit 4. Examples of the physical feature amount of the video that causes the discomfort other than the motion recognition amount include a spatial frequency, a temporal frequency, a color distribution, a localization rate, an attractiveness degree, and the like.

<Discomfort level estimation unit>
The discomfort degree estimation unit 4 acquires the motion recognition amount estimated by the motion recognition amount estimation device 1 and the physical feature amount extracted by the feature amount extraction unit 3, and performs nonlinear addition, subtraction, integration, and the like. Thus, the discomfort level is calculated and output to an external device (notification unit that notifies the user of the discomfort level) such as a display and a speaker.

<Operation example>
Subsequently, an operation example of the motion recognition amount estimation device 1 according to the embodiment of the present invention will be described with reference to FIGS. 2 to 7. Since the operation example of the discomfort degree estimation apparatus 2 in FIG. 1 shows a use example of the motion recognition amount estimation apparatus 1 according to the embodiment of the present invention, it is excluded from the description here.

  First, the region-specific motion vector detection unit 10 detects a motion vector between temporally adjacent images (continuous frames) of a video using a video signal in a region corresponding to the detection unit 10, Output to the filter unit 20. Subsequently, the first filter unit 20 performs frequency sensitivity correction on the motion vector, and outputs the corrected motion vector to the motion region determination unit 30 or the like.

  Subsequently, the motion region determination unit 30 determines that there is motion when the magnitude of the corrected motion vector is equal to or greater than the threshold value, and outputs the result to the motion region number assigning unit 40 as a determination result of the presence or absence of motion. Subsequently, the rotation vector calculation unit 40 assigns a motion region number reflecting the front-rear relationship between the motion regions to each region belonging to the same motion region, and outputs the motion region number to the rotation center / rotation angle calculation unit 50 or the like.

  Subsequently, the rotation center / rotation angle calculation unit 50 calculates the rotation center and rotation angle in the entire screen and outputs them to the rotation residual vector calculation unit 60. Subsequently, the rotation vector calculation unit 60 calculates a motion vector based on the rotation angle in each region as a rotation vector using the rotation center and the rotation angle, and outputs the rotation vector to the rotation residual vector calculation unit 70 and the second filter unit 80. To do.

  Subsequently, the rotation residual vector calculation unit 70 calculates a residual between the corrected motion vector and the rotation vector in each region as a rotation residual vector, and the inter-motion region correlation value calculation unit 110, the motion region internal boundary It outputs to the correlation value calculation part 120 and the total sway energy calculation part 130. Subsequently, the second filter unit 80 performs frequency sensitivity correction corresponding to the relative sensitivity of the rotation vector with respect to the rotation residual vector, and outputs the corrected rotation vector to the total sway energy calculation unit 130.

  Meanwhile, the spatial frequency sensitivity correction unit 90 for each region performs spatial frequency sensitivity correction on the video signal in the same region as the motion vector detection unit 10 for each region, and the spatial frequency sensitivity corrected video signal is converted to the spatial frequency energy calculation unit 100. Output to. Subsequently, the spatial frequency energy calculation unit 100 calculates the energy of the spatial frequency sensitivity corrected video signal as the corrected spatial frequency energy, and outputs it to the total shaking energy calculation unit 130.

  On the other hand, the correlation value calculation unit 110 between motion regions calculates the correlation value of the rotation residual vector between the foreground region and the background region at the boundary of the motion region as the correlation value between motion regions, and the total shaking energy calculation unit 130 Output to. On the other hand, the correlation value calculation unit 120 between the motion region inner boundaries calculates the correlation value of the rotation residual vector between the motion vector of the motion region boundary itself and the inside of the motion region as the correlation value between the motion region inner boundaries. And output to the total shaking energy calculation unit 130.

  Subsequently, the total shaking energy calculation unit 130 calculates the individual shaking energy obtained by calculating the sum of the square of the magnitude of each rotation residual vector and the square of the magnitude of each corrected rotation vector for each movement area number. On the other hand, after performing correction in order according to the corrected spatial frequency energy, the correlation value between motion regions, and the correlation value between motion region inner boundaries, the sum is obtained within the motion region, and the square root is calculated for each motion region number. Based on the separate vibration energy, the movement energy of all the movement areas is nonlinearly added and output as the total vibration energy corresponding to the estimated fluctuation perception value.

As described in the above description, the motion recognition amount estimation device 1 according to the embodiment of the present invention does not have a one-to-one correspondence with the degree of discomfort with the motion video, but is obtained by the motion recognition amount estimation device 1. By using the estimated motion recognition amount and the physical feature amount of the video, it is expected that the accuracy of estimation of discomfort can be improved to a level that cannot be achieved by the conventional device for estimating the discomfort of motion video.
In addition, the estimated motion recognition amount obtained by the motion recognition amount estimation device 1 can also be used as a substitute index for discomfort when the video content producer performs video correction to reduce screen motion. is there. That is, when the motion recognition amount estimation device 1 is used by the video content creator at the production stage, the motion recognition is set as a determination target for determining the quality of the video and to what extent the screen shake included in the video should be reduced. Since the amount estimation value can be presented to the video content producer, not only can the time, labor, and cost required for production be reduced, but also the safety of the supplied video content can be improved.
In addition, when the shake recognition amount estimation device 1 is used on the viewer side, it is shaken before viewing or immediately before display during viewing with respect to a video produced and distributed without guaranteeing that screen shake is safe. Since it is possible to issue a warning by estimating the amount of recognition and outputting it to a display or speaker, it is possible to prevent health damage due to video sickness.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, A design change is possible suitably in the range which does not deviate from the summary of this invention. For example, the present invention can be embodied as a motion recognition amount estimation program that causes a computer to function as the motion recognition amount estimation device 1. Alternatively, the logarithmic value of the total motion energy may be output as the motion perception amount estimation value so that the motion perception amount estimation value linearly corresponds to the psychological evaluation value. Further, the motion recognition amount estimation device 1 can be embodied as a device integrated with the discomfort degree estimation device 2. In addition, the motion perception amount estimated by the motion perception amount estimation device 1 is not used only for estimating the degree of discomfort, but can be used as a basis for a single psychological index or another psychological index.

A discomfort level estimation system 1 motion recognition amount estimation device 2 discomfort level estimation device 3 feature amount extraction unit 4 discomfort level estimation unit 10 region-specific motion vector detection unit 20 first filter unit 21 first digital filter 22 second digital Filter 30 Motion region determination unit 40 Motion region number assigning unit 50 Center of rotation / rotation angle calculation unit 60 Rotation vector calculation unit 70 Rotation residual vector calculation unit 80 Second filter unit 81 Third digital filter 90 Spatial frequency sensitivity by region Correction unit 100 Spatial frequency energy calculation unit 110 Correlation value calculation unit between motion regions 120 Correlation value calculation unit between motion region internal boundaries 130 Total shake energy calculation unit

Claims (5)

A motion perception amount estimation device that estimates motion perception amount for screen motion based on motion vectors in a video,
For each region obtained by dividing the screen into a plurality of regions, a motion vector detection unit for each region that detects a motion vector between temporally adjacent images of the video,
A first filter unit that obtains a corrected motion vector by performing frequency sensitivity correction on the detected motion vector;
A motion region determination unit that determines that there is motion in the region corresponding to a case where the size of the corrected motion vector is equal to or greater than a threshold;
A motion region number assigning unit that assigns a motion region number reflecting the front-rear relationship between regions for each region belonging to the same motion region with respect to the region determined to have motion;
Using the corrected motion vectors of all the regions to which the motion region numbers are assigned, the sum of the squares of the magnitudes of the motion vectors according to the rotation angle is maximized, and the motion vector according to the rotation angle and the corrected motion are A rotation center / rotation angle calculation unit for calculating the rotation center and rotation angle when the sum of square errors with the vector is minimum;
A rotation vector calculation unit that calculates a rotation vector of the area using the corrected motion vector of the area, the rotation center, and the rotation angle;
A rotation residual vector calculation unit that calculates a difference between the corrected motion vector of the region and the rotation vector as a rotation residual vector;
A second filter unit that obtains a corrected rotation vector by performing frequency relative sensitivity correction on the rotation vector;
A spatial frequency sensitivity correction unit for each region that obtains a spatial frequency sensitivity corrected video signal by performing spatial frequency sensitivity correction on the video,
A spatial frequency energy calculating unit that obtains corrected spatial frequency energy by calculating energy of the spatial frequency sensitivity corrected video signal;
Based on the region to which the motion region number is assigned and the rotation residual vector, motion is calculated by calculating a correlation value of the rotation residual vector between the foreground region and the background region at the boundary of the motion region. A correlation value calculation unit between motion regions for obtaining a correlation value between regions;
Based on the region to which the motion region number is assigned and the rotation residual vector, the correlation value of the rotation residual vector between the motion vector of the boundary of the motion region itself and the same motion region is calculated. A motion region inner boundary correlation value calculating unit that obtains a correlation value between motion region inner boundaries,
For each of the regions to which the same motion region number is assigned, the correction is performed on the individual vibration energy obtained by calculating the sum of the square of the size of the corrected rotation vector and the square of the rotation residual vector in the region. After performing correction in order according to the completed spatial frequency energy, the correlation value between the motion regions, the correlation value between the motion region inner boundaries, and determining the sum of the individual shaking energy corrected within the motion region, The calculated root sum of the sum is used as the motion energy per motion region for each motion region number, and then the motion energy per motion region for all motion regions is nonlinearly added to obtain a total corresponding to the motion perception amount estimation value. A total sway energy calculator to obtain sway energy;
An apparatus for estimating the amount of perception of shaking characterized by comprising: The first filter unit is
A first digital filter that performs frequency sensitivity correction on the vertical component of the motion vector;
A second digital filter that performs a frequency sensitivity correction different from that of the first digital filter on the horizontal component of the motion vector;
The motion recognition amount estimation device according to claim 1, comprising: The motion recognition amount estimation according to claim 1 or 2, wherein the motion region determination unit determines that there is motion when there is the corrected motion vector equal to or greater than the threshold value within a predetermined time immediately before. apparatus. The second filter unit performs a third frequency relative sensitivity correction corresponding to a frequency characteristic of a relative sensitivity with respect to the rotation vector and another component subjected to the frequency sensitivity correction by the first filter unit. The motion recognition amount estimation device according to any one of claims 1 to 3, further comprising a digital filter.   A motion recognition amount estimation program for causing a computer to function as the motion recognition amount estimation device according to claim 1.

Images