JP2012165338A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent a viewer from feeling sick due to the partial movement of an image within a screen.SOLUTION: An image processing apparatus extracts an image object from an inputted image, calculates the proportion of the extracted image object in the screen, and detects the shake frequency of the extracted image object. Then, the image processing apparatus determines a threshold of the proportion in the screen with the detected shake frequency. If the calculated proportion of the image object to the screen is the determined threshold or more, the image processing apparatus corrects the image object so that the movement of the image object becomes less active.

Description

  In particular, the present invention relates to an image processing apparatus, an image processing method, and a program suitable for use in processing a video including shaking.

  Conventionally, thin and large-screen displays such as liquid crystal displays and plasma displays have become widespread. With such a display, it is possible to view beautiful images with a high sense of realism, but there are concerns about effects on the living body such as image sickness and light-sensitive seizures. For example, there are many image shakes caused by camera shake in videos taken by home video cameras by ordinary people, and video sickness is likely to be induced by such camera shake, rotation, and rapid zooming. . In addition, when viewing on a large-screen display, video sickness is more likely to be induced as the viewed video is larger.

  Therefore, various methods have been proposed as a method for preventing video sickness or photosensitive seizures with a reproducible device. As an example, there has been proposed a method for reducing and displaying a screen when it is detected that the screen on which content is displayed is shaken (see, for example, Patent Document 1).

JP 2008-299241 A JP 2000-184336 A

  However, even if the degree of shaking of the entire screen is less than or equal to a value at which the influence on the living body is a concern, partial movement may approach that value, and this cannot be corrected by the conventional technology. Therefore, in an environment displayed on a large screen such as a wall display, video sickness may be caused to a user who is sensitive to shaking.

  The present invention has been made in view of the above-described problems, and an object thereof is to prevent motion sickness due to partial movement in a screen.

  An image processing apparatus according to the present invention includes: an input unit that inputs a video; an extraction unit that extracts an object from the video input by the input unit; and a calculation that calculates a ratio of an object extracted by the extraction unit to a screen Means for detecting the shaking frequency of the object extracted by the extracting means, the ratio of the object to the screen calculated by the calculating means, and the detection result of the shaking frequency by the detecting means. And correction means for performing correction to reduce shaking.

  According to the present invention, video sickness can be prevented even when a partial area of the screen is shaking, and viewing can be performed more safely.

1 is a block diagram illustrating a characteristic functional configuration example of an image processing apparatus according to an embodiment. It is a block diagram which shows the structural example of the display display apparatus of 1st Embodiment. It is a figure which shows the example of a screen of the image | video containing the shaked image object. It is a figure which shows the example of a screen after performing video sickness prevention correction. It is a flowchart which shows an example of the process sequence of an image sickness prevention correction. It is a figure which shows the relationship between a shaking frequency and an occupation rate. It is a block diagram which shows the structural example of the video camcorder of 2nd Embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

(First embodiment)
<Configuration of image processing apparatus>
FIG. 1 is a block diagram illustrating a characteristic functional configuration example of the image processing apparatus 1 according to the present embodiment. The configuration of the image processing apparatus will be described below with reference to FIG.
In FIG. 1, a video input unit 2 inputs video data from the outside. The video output unit 8 outputs a video that has been subjected to video sickness prevention correction. In the present embodiment, the video output unit 8 includes, for example, a display device for displaying video and a control circuit thereof. The image object extraction unit 3 extracts image objects constituting the video screen.

  The occupation ratio calculation unit 4 calculates the ratio (occupancy ratio) that the size of the extracted image object occupies the entire screen. The shaking frequency detection unit 5 detects the shaking frequency of the image object. The correction control unit 6 determines whether there is a possibility of causing motion sickness due to the shaking of the image object, and determines a threshold value for the occupation ratio. The correction control unit 6 further includes a correction determination unit 31 and a determination threshold value determination unit 32. The correction determination unit 31 determines whether there is a possibility of causing motion sickness by comparing the occupation ratio of the image object with a threshold value. The determination threshold value determination unit 32 determines the threshold value of the occupation ratio of the image object according to the detection result of the shaking frequency.

  The motion sickness correction unit 7 performs motion sickness prevention correction on an image object that has been determined to cause motion sickness. In the present embodiment, processing for dulling the motion of an image object that may cause video sickness is performed using a filter in the time axis direction. Specifically, a motion vector of an image object is analyzed between a plurality of frames, and a motion component of a specific frequency in the motion vector is attenuated. It should be noted that an area partially missing by suppressing the movement of the image object is interpolated by the background image at the same location in the preceding and following frames. This missing area interpolation processing may use, for example, the method disclosed in Patent Document 2 or any other known technique. Further, interpolation may be performed from a region in the vicinity of the missing region, or blurring processing may be performed to compensate for a joint between the region to be interpolated.

FIG. 2 is a block diagram illustrating a schematic configuration example when applied to the display device 20 as an example of the image processing device 1. In addition, the same number is attached | subjected to the same structure as FIG. 1, and description is abbreviate | omitted about the structure which is common in FIG.
In FIG. 2, a video input interface 21 constituting the video input unit 2 inputs video content from the outside. The video input interface 21 includes, for example, an analog composite signal, an analog component signal, an interface of a digital video signal standard such as HDMI or DVI, and inputs various video contents. The image quality enhancement processing unit 22 performs image processing such as enlargement / reduction, contour correction, and color correction on the input video content.

  The OSD composition unit 23 superimposes various operation information such as a video port number and volume setting on the display screen. The display device control unit 24 performs display control for displaying an image on the display device 25, generates a synchronization signal at a timing according to the characteristics of the display device 25, and displays display screen data according to the synchronization signal. Read and output. The display device control unit 24 also converts the signal voltage level and creates an auxiliary signal.

  The display device 25 is configured by an arbitrary display method such as liquid crystal, plasma, projection, CRT, organic EL, or SED. The overall control unit 26 includes a CPU, peripheral circuits, a memory, and the like, and controls the entire display device 20 including control for video sickness prevention correction. Although not shown, the display device 20 includes an audio processing unit for reproducing audio data, an amplifier, a speaker, a communication interface, a user interface, a power source, and the like.

<Flow of processing to prevent motion sickness>
In the following, the processing for image sickness prevention correction is described. In the present embodiment, the occupation ratio of the image object extracted from the screen and the shaking frequency of the image object are detected, and the threshold value is determined according to the detection result of the shaking frequency. Then, when the occupation ratio of the image object exceeds the threshold value, control is performed to perform video sickness prevention correction and display.

FIG. 5 is a flowchart showing an example of a processing procedure for video sickness prevention correction by the display device 20 according to the present embodiment. Note that the processing shown in FIG. 5 is processing performed under the control of the overall control unit 26 after the image quality improvement processing of the image for one frame is performed by the image quality improvement processing unit 22.
First, in step S101 in FIG. 1, the video input unit 2 inputs an image (frame) of one frame of video content. In step S102, the image object extraction unit 3 extracts one image object from the input image.

  Next, in step S103, the occupation ratio calculation unit 4 calculates the ratio (occupancy ratio) of the size of the extracted image object to the entire screen. In step S104, the shaking frequency detection unit 5 detects the shaking frequency of the image object.

  Next, in step S <b> 105, the determination threshold value determination unit 32 determines the threshold value of the occupation ratio of the image object based on the detection result of the vibration frequency by the vibration frequency detection unit 5. A method for determining the threshold will be described later. In step S106, the correction determination unit 31 determines whether the size ratio of the image object is equal to or greater than the threshold value determined in step S105. As a result of this determination, if it is equal to or greater than the threshold, the process proceeds to step S107, and if it is less than the threshold, the process proceeds to step S108.

  Next, in step S107, the video sickness correction unit 7 performs video sickness correction on the image object. In step S108, it is determined whether or not the last image object has been processed in the input image. As a result of the determination, if there is an image object that has not yet been processed, the process returns to step S102, and if all the image objects have been processed, the process proceeds to step S109.

  In step S109, the OSD composition unit 23 performs OSD composition on the input image, and the display device control unit 24 displays an image for one frame on the display device 25, and ends the process.

<Determination method of threshold value>
In general, video sickness is caused when the screen shakes at a low frequency. The shaking frequency causing image sickness is on the order of several Hz, and is distributed around the peak frequency. The peak frequency is a value around 1.5 Hz, for example, and this peak frequency is hereinafter referred to as a dangerous frequency. In step S105 in FIG. 5, the occupation ratio threshold is determined to be lower as the difference between the preset dangerous frequency and the fluctuation frequency is smaller. Thus, when the difference is small (close to the dangerous frequency), even a smaller image object may cause video sickness, so the threshold value is lowered.

  The threshold is determined according to a function as shown in FIG. 6, for example. FIG. 6 is a diagram showing the relationship between the shaking frequency and the occupation ratio. In FIG. 6, a dotted line 61 represents a dangerous frequency that is highly likely to cause image sickness, and the dangerous frequency is set to 1.5 Hz. The broken line 62 is a line indicating a threshold value corresponding to the shaking frequency. Here, the threshold value is the minimum value (60% in FIG. 6) when the vibration frequency is near the dangerous frequency. Then, the threshold value increases as the distance from the dangerous frequency increases, and the threshold value becomes constant when the upper limit value (90% in FIG. 6) is reached.

  As described above, the threshold value is minimized when the shaking frequency of the image object is in the vicinity of the dangerous frequency. Therefore, control is performed so as to perform video sickness prevention correction even for a relatively small image object. This is because if the image object swings at a dangerous frequency, even a slightly small image object may cause video sickness to a sensitive user. On the other hand, when the vibration frequency is a frequency away from the dangerous frequency, the possibility of causing video sickness is reduced. Therefore, control is performed so that video sickness prevention correction is performed only on image objects that occupy a large proportion of the screen. As described above, video sickness can be prevented by correcting the minimum necessary portions while maintaining the movement of the original video as much as possible.

<Example screen for correction of motion sickness>
Hereinafter, an example of a specific image that has undergone image sickness prevention correction will be described with reference to FIGS. 3 and 4.
FIG. 3 is a diagram illustrating an example of a video display screen 50 including a shaken image object. The display screen 50 shown in FIG. 3 includes three image objects: a person object 51, a house object, and a ship object 53. The human object 51 is moving due to dancing or the like, and is shaking at a frequency near the dangerous frequency. For example, it is assumed that the size of the person object 51 occupies 70% of the screen and shakes at 1.2 Hz. Since the house object 52 is fixed, it is stationary. The ship object 53 is swaying at a dangerous frequency due to the influence of waves and wind. For example, it is assumed that the size of the ship object 53 occupies 10% of the screen and shakes at 1.5 Hz.

  FIG. 4 is a diagram illustrating an example of the display screen 55 after the video sickness prevention correction is performed in the present embodiment. In the video before correction, the person object 51 is shaken at a frequency (1.2 Hz) near the dangerous frequency (1.5 Hz), and occupies 70% of the screen. As shown in FIG. 6, when the shaking frequency is 1.2 Hz, since the occupation ratio threshold is exceeded, correction of video sickness prevention is performed on the person object 51.

  On the display screen 55 shown in FIG. 4, a human object 56 from which shaking has been removed by correction is displayed. On the other hand, the ship object 53 is swaying at the same vibration frequency as the dangerous frequency, but since the ratio of the screen to the screen is smaller than the threshold value, the ship object 53 is not corrected but is displayed with the vibration remaining.

  As described above, according to the present embodiment, video sickness can be prevented even when an image in which a part of the region is shaken is displayed, so that viewing can be performed more safely. In this embodiment, the motion dullness prevention process is performed as the video sickness prevention correction, but other video sickness prevention correction may be performed. For example, a process of blunting the edge of the image object may be performed, or a stationary texture may be superimposed.

  Further, the function for determining the threshold is not limited to the example shown in FIG. 6, and any function may be used. In the present embodiment, a function that increases linearly on both sides of the critical frequency is used. However, a function that increases only on one side may be used, and an arbitrary curve function such as a quadratic curve may be used.

  Further, a range of the shaking frequency may be determined according to the occupation ratio of the screen, and correction may be made when the shaking frequency is within this range. In this case, the occupation ratio calculation unit 4 is configured to input the occupation ratio information to the determination threshold value determination unit 32, and the fluctuation frequency detection unit 5 is configured to input the fluctuation frequency information to the correction determination unit 31, and the determination threshold value determination unit At 32, the range of the shaking frequency is determined. At this time, the range of the vibration frequency is determined so that the determination range becomes narrower as the occupation ratio is smaller with the dangerous frequency as the center.

  For example, in the example shown in FIG. 6, when the occupation ratio is 80%, the range of the swing frequency is determined as 0.8 to 2.2 Hz, and when the occupation ratio is 70%, the range of the swing frequency is determined. It is determined as 1.1 to 1.9 Hz. As a result, when the vibration frequency is close to the dangerous frequency, even a small image object is corrected. As described above, this configuration can also achieve the same effect as the configuration described in the present embodiment.

(Second Embodiment)
In the first embodiment described above, the motion sickness prevention correction is performed before the video is displayed. However, as shown in the present embodiment, the motion sickness prevention correction may be performed in advance after shooting the video. In this embodiment, an example of application to a video camcorder will be described. Note that a characteristic functional configuration example of the image processing apparatus in the present embodiment is the same as that in FIG.

<Configuration of image processing apparatus>
FIG. 7 is a block diagram illustrating a schematic configuration example when applied to a video camcorder 40 as an example of the image processing apparatus 1. In addition, the same number is attached | subjected to the same structure as FIG. 1, and description is abbreviate | omitted about the structure which is common in FIG.
In FIG. 7, the image pickup unit 41 constituting the video input unit 2 includes a lens, an image pickup device, and an image pickup control unit. An arbitrary sensor such as a CCD or a CMOS sensor is used as the image sensor. The image quality enhancement processing unit 42 performs image processing such as color correction and contour correction on the video captured by the imaging unit 41.

  The video output interface 43 configuring the video output unit 8 outputs video data being shot and video data stored in the storage 45 to the outside. The video output interface 43 includes, for example, an analog composite signal, an analog component signal, and an interface of a digital video standard such as HDMI or DVI. The codec 44 is JPEG, MPEG, H.264, or the like. The video data is compressed or expanded by a compression algorithm such as H.264 or DV.

  The storage 45 is a recording medium that stores compressed video data, and includes an HDD, a DVD, an SD card, or the like. Video data captured by the image sensor is compressed by the codec 44 and stored in the storage 45. The video data stored in the storage 45 is expanded by the codec 44 and output from the video output interface 43. The overall control unit 46 includes a CPU, peripheral circuits, a memory, and the like, and performs overall control of the video camcorder 40 such as shooting by an image sensor including control for image sickness prevention correction. Although not shown, the video camcorder 40 includes a finder mechanism for confirming a video to be shot, a microphone for inputting audio data, an amplifier, an audio processing unit, a user interface, a power source, and the like. .

  The flow of video sickness correction processing in the present embodiment is almost the same as that in the first embodiment described above, and a description thereof will be omitted. In the present embodiment, the data is output to the codec 44 or the video output interface 43 in step S109.

  As described above, according to the present embodiment, video sickness prevention correction can be performed on the video camcorder side. When viewing this video on a display, the image object that swings at a frequency around the dangerous frequency has already been removed from the video, so that it is possible to prevent video sickness.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

2 Video input unit 3 Image object extraction unit 4 Occupancy ratio calculation unit 5 Shake frequency detection unit 6 Correction control unit 7 Video motion sickness correction unit

Claims (9)

An input means for inputting video;
Extraction means for extracting an object from the video input by the input means;
Calculating means for calculating a ratio of the object extracted by the extracting means to the screen;
Detecting means for detecting a shaking frequency of the object extracted by the extracting means;
An image processing apparatus comprising: a correction unit that corrects the shaking of the object according to a ratio of the screen calculated by the calculating unit and a detection result of a shaking frequency by the detecting unit.   The correction means determines a threshold value of a ratio of the object to the screen according to a detection result of the vibration frequency of the object, and corrects the shaking of the object when the ratio of the object to the screen exceeds the determined threshold value. The image processing apparatus according to claim 1, wherein:   The image processing apparatus according to claim 2, wherein the correction unit determines the threshold value to be lower as a difference between a preset frequency and the shaking frequency of the object is smaller.   The correction means determines a range of a shaking frequency according to a ratio of the object to the screen, and corrects the shaking of the object when the shaking frequency of the object is within the determined range. The image processing apparatus according to claim 1.   The correction means determines a range of a swing frequency so as to include a preset frequency, and determines that the range of the swing frequency becomes narrower as a ratio of the object to the screen is smaller. Item 5. The image processing apparatus according to Item 4.   The image processing apparatus according to claim 1, wherein the input unit includes an imaging unit including an imaging element.   The image processing apparatus according to claim 1, further comprising a display control unit that displays on the display unit a video in which the object is corrected by the correction unit. An extraction process for extracting objects from the input video;
A calculation step of calculating a ratio of the object extracted in the extraction step to the screen;
A detection step of detecting a shaking frequency of the object extracted in the extraction step;
An image processing method comprising: a correction step of correcting the shaking of the object according to a ratio of the screen calculated in the calculation step and a detection result of a shaking frequency in the detection step. An extraction process for extracting objects from the input video;
A calculation step of calculating a ratio of the object extracted in the extraction step to the screen;
A detection step of detecting a shaking frequency of the object extracted in the extraction step;
A program that causes a computer to execute a correction step of correcting the shaking of the object in accordance with the ratio of the screen calculated in the calculating step and the detection result of the shaking frequency in the detecting step.

Images