WO2015016691A1

WO2015016691A1 - Method for converting and reproducing image for providing moderate three-dimensional effect between two-dimensional image and three-dimensional image

Info

Publication number: WO2015016691A1
Application number: PCT/KR2014/007190
Authority: WO
Inventors: 최재혁
Original assignee: 고포디 테크놀로지 엘엘씨
Priority date: 2013-08-02
Filing date: 2014-08-04
Publication date: 2015-02-05
Also published as: KR101433082B1

Abstract

Disclosed is a method for reproducing a source image by converting the source image into an image having a three-dimensional effect which is moderate between two-dimensional and three-dimensional images by monocular cues. The method, which is executed by a computer, comprises: firstly, sequentially converting respective source image frames of source image data into corresponding target image frames so as to generate converted image data, wherein the conversion into the corresponding target image frames of the respective source image frames is obtained by scaling the source image frames in a predetermined ratio so as to form two converted image frames which are substantially the same as each other, and by combining the two converted image frames so as to form the target image frames; reproducing the target image frames of the converted image data on a screen; and respectively displaying images of the same two converted image frames, which form the respective target image frames, on a left half side and a right half side of a display screen such that the same two converted image frames, which form the respective target image frames, are respectively and separately seen by a left eye and a right eye of a consumer viewing image content with three-dimensional glasses having two lenses.

Description

Image conversion and playback method giving a three-dimensional effect between 2D and 3D images

The present invention relates to image processing, and more particularly, by providing an intermediate three-dimensional effect between a two-dimensional image and a three-dimensional image, image signal conversion and reproduction that can reduce visual fatigue or psychological anxiety caused by viewing a complete three-dimensional image. It is about a method.

Since the release of the three-dimensional film Avatar, people's interest in stereoscopic images has exploded. The market for stereoscopic images also grew significantly. Various TVs, monitors, etc., which can watch 3D images (3D images), have also been released, and related 3D image technologies have been further developed.

When one looks at an object with two eyes, one can feel the three-dimensional effect of convergence, which is a phenomenon in which the eyes of both eyes converge toward the object and the parallax of both eyes of a person having a distance of about 65 mm. This is due to two factors (binocular vision) of binocular disparity, which is a phenomenon in which a difference occurs in the retina. Due to these two factors, the left and right eyes see different two-dimensional images, each with a predetermined deviation, and when these two images are transmitted through the retina to the brain, the brain fusions exactly with each other and the perception of the depth of the original stereoscopic image. of depth, presence, and perception of volume. In order to display a stereoscopic image, two different images having binocular disparity are required. Stereoscopic image content for 3D display can be produced through a binocular lens shooting method that simultaneously shoots with two cameras that are about 65mm apart to have binocular parallax. There is also a method of converting image contents of a single eye view taken by a single-eye lens camera through hardware and software calculations to achieve the same effect as a two-eye lens shooting method.

By the way, the stereoscopic image due to the binocular vision has gained much attention immediately after the success of the movie avatar, but recently, consumers' desire to purchase stereoscopic images has been stalled for a while. The reason why the popularity of stereoscopic images has become weaker is that the cost of viewing devices is high, while stereoscopic contents are largely lacking. But the more fundamental reason is that concerns about watching stereoscopic images can be uncomfortable and even adversely affect people's health.

People who watched stereoscopic images due to binocular vision are known to have visual fatigue and psychological instability. Visual fatigue and instability associated with viewing 3D images include eyestrain, eyes feel heavy, double vision, dry eyes, heavy head, headache Shoulder stiffness, shoulder pain, back pain, nausea, giddiness, and headache. Statistics show that about 12% of people are unable to see stereoscopic images properly due to various medical conditions, and about 30% of people have very weak stereoscopic vision, making them poorly aware of depth. Such people have little or no significant effect on the stereoscopic image. Because of these points, there are surprisingly many consumers who have a rejection of viewing stereoscopic images in three dimensions.

Many people who have watched stereoscopic images due to binocular vision have experienced eye fatigue and dizziness, and there is also a strong suspicion that the continuous viewing of stereoscopic images does not adversely affect the human body. In order to activate the industry of stereoscopic images, it is very important to study the problem and to establish stability standards for 3D display and contents. The factors causing visual psychological fatigue in stereoscopic images are not clearly identified. However, it is generally acceptable that the discrepancy between the focal length and convergence angle of the two eyes, the unnatural image, the disparity of the motion parallax, and the optical characteristics of the left and right images can be the cause of visual psychological fatigue. Although these factors are related to the error of stereoscopic image reproducing system including wrong shooting of stereoscopic images and wrong display, it is understood that these factors are closely related to the original attributes of stereoscopic images themselves. In stereoscopic images caused by binocular vision, excessive protrusion and speed of moving objects are felt, and if the image contains a lot of colorful colors and intricate patterns, the part adds a three-dimensional feeling to leave afterimages of patterns and colors. Watching these videos for a long time causes visual fatigue and psychological anxiety, such as dizziness.

Various techniques for minimizing dizziness when viewing 3D images have been proposed. For example, a technology for correcting an error of a stereoscopic image caused by a difference in capability of an individual photographer when producing a stereoscopic image (Korean Patent Publication No. 10-2012-0064560. Name of the Invention: A stereoscopic image production system supporting minimizing visual fatigue And the method) are known. The prior art reports that there are various factors such as geometric errors between left and right images, color errors, afterimages of images, and objects visible only on one image as errors of stereoscopic images that may occur in the shooting stage. We present stereoscopic correction technology that can be minimized. However, this technique is focused on error correction at the shooting stage, and thus is not a fundamental solution to the problem.

After all, making images look stereoscopic is not good for everyone. The fundamental solution to various health problems caused by viewing stereoscopic images may be to minimize or eliminate stereoscopic effects.

Meanwhile, even in the same image, the impression of the viewer varies according to the size of the screen on which the image is displayed. It is hard to feel the inspiration of this movie through the big screen of a movie theater on a small screen such as a computer monitor or a smartphone. This is because there is a difference in the size of the screen that the user sees. If not as much as the screen of a movie theater, it can be said that if the screen of the small size can be enlarged and the image can be seen to be much larger than the actual size of the screen, it can meet the needs of users. As the viewing of video contents through computer monitors has become widespread, in recent years, with the explosive increase in the population of smartphones, the viewing of video contents and TVs using smartphones has become commonplace. Such demands can be said to be stronger when viewing images on a smartphone or computer monitor with a relatively small screen size.

There is a need to find a solution to view an image larger than the actual screen. However, the solution needs to be a third method, not a method of using a stereoscopic image that causes discomfort such as dizziness.

In addition, the user may have normal gaze direction characteristics, but may have abnormal gaze characteristics such as strabismus. Technical considerations are also required to ensure that the characteristics of strabismus allow the user to comfortably view the images.

The present invention provides a three-dimensional to depth feeling and a sense of realism and a sense of presence by monocular factors, which are differentiated from 2D images that do not provide any stereoscopic feeling, but also cause visual fatigue or psychological anxiety caused by three-dimensional feeling depending on binocular factors. It is an object of the present invention to provide a video conversion and reproduction method that does not.

Another object of the present invention is to provide an image converting and reproducing method for viewing an image generated by the above method much larger than an actual screen.

In another aspect, the present invention, by adjusting the position of the image displayed in the side-by-side form on the display screen of the video player according to the characteristics of the user's gaze direction, the image conversion that allows a user with strabismus to enjoy a comfortable image And to provide a regeneration method.

According to an aspect of the present invention for achieving the above object, in the image conversion unit, by converting each source image frame (Fi) of the source image data to a corresponding target image frame (Fi ') in sequence to the target image frame A first step of generating the converted image data consisting of (Fi '), wherein the conversion of each source image frame (Fi) to the corresponding target image frame (Fi') is a ratio of the source image frame (Fi ') By scaling to create two substantially identical transformed video frames, and combining the two transformed video frames to form the target image frame Fi '; A second step of providing the converted image data to the image reproducing unit; And reproducing the target image frames Fi 'of the converted image data provided by the image reproducing unit on a screen, and displaying the images of the two substantially identical converted image frames constituting each target image frame Fi'. The left eye and the right half of the display screen are respectively displayed so that the same two converted image frames constituting each target image frame (Fi ') are viewed by a stereoscopic lens including two lenses. There is provided a video data conversion and playback method comprising a third step of reproducing so that it can be seen separately from each other.

In the above method, the ratio applied to the scaling is such that the two substantially identical transformed video frames have an image size that can be displayed in the left half and right half regions of the display screen of the image reproducing apparatus, respectively. It is preferable to be determined.

According to one example of the method, the two substantially identical converted video frame, to reduce the size of the source video frame (Fi) in the horizontal direction and the vertical direction in half each reduced image frame and then the two reduced image frame It is two reduced image frames obtained by duplication.

According to another example of the method, the two substantially identical transformed image frames are (i) even with a first transformed image frame constructed by taking only pixel data in odd-numbered rows and columns of the source image frame Fi. It consists of a second converted image frame composed of only the pixel data in the first row and column.

The image data converting and reproducing method may further include: displaying a target for correcting an image display area in a left half region and a right half region divided by left and right sides of the display screen; And receiving image display area correction coordinate information of any one of the left half region and the right half region through a user input means, and changing the image display region coordinates according to the image display region correction coordinate information. have.

In the image data converting and reproducing method, the image display in the third step is performed in a portable image reproducing apparatus including a smartphone or a tablet computer, and the image content consumer of the stereoscopic mirror provides a viewing angle of 40 to 70 °. It is preferable that the two convex lenses simultaneously view the two substantially identical converted image frames through the stereoscopic mirror with the convex lens separated by 30 to 80 mm from the display screen.

In the video data conversion and playback method, the first step, the second step, and the third step may be performed in an image reproducing device of a consumer of an image content including both the image converting unit and the image reproducing unit. have.

In the video data converting and reproducing method, the first and second steps are performed by an image providing system on the side of an image content provider including the image converting unit, and the third step comprises the image reproducing unit. The image content may be performed by an image reproducing device of a consumer. In this case, when the image conversion unit performs the second step, the image conversion unit transmits the converted image data to the image reproduction unit in a streaming method or a broadcast method.

In the second step, when the image providing system provides the converted image data to the image reproducing apparatus, the source image data is also provided. In this case, the third step is performed when the video content consumer instructs playback by selecting the latter of the source video data and the converted video data.

According to the image converting and reproducing method of the present invention, since the two images visible to the left and right eyes of the user are completely identical images (that is, no offset between each other), the 3D image effect using binocular disparity is It can't be felt, but it's not like the feeling of a simple 'two-dimensional' image when the left eye and the right eye are in common. Since the left and right eyes view the 'two' identical images separately, you can only feel the so-called 'three-dimensional effect of monocular factors' (described later) through each of the left and right eyes. The viewing feeling of the reproduced video is halfway between the full 3D and simple two-dimensional. Therefore, almost no visual fatigue or psychological anxiety, which is felt in a complete 3D stereoscopic image, is rather comfortable, but a sense of reality and presence is alive and refreshed. Therefore, side effects (visual fatigue and psychological anxiety) caused by viewing 3D images can be prevented.

In addition, when wearing a stereoscopic mirror with a magnification while playing in stereoscope method, you can enjoy the effect of enlarging the playback image considerably. That is, the image can be enlarged and seen according to the lens magnification of the stereoscopic mirror to be used. For example, when viewing a playback image of a stereoscopic mirror composed of two lenses having a focal length of 6 cm, the user can obtain the same effect as when viewing a 125-inch screen magnified 25 times in front of 1.5 meters. This has the effect of offsetting the shortcomings of a video player such as a smartphone with a small screen size.

In addition, according to the method of the present invention, in displaying two converted image frames in a side by side format on the left half region and the right half region of the display, the display position of the image frames can be adjusted according to the characteristics of the user's gaze direction. have. Therefore, users with strabismus can adjust the position of the video display according to their own gaze direction, thereby making it easier to view the video.

1 is a view conceptually illustrating an image display and a viewing method according to the present invention;

2 illustrates a state in which the left eye (including the left lens) and the right eye (including the right lens) of the user view the left half and the right half of the display screen, respectively, aligned at their centers.

FIG. 3 is a diagram schematically showing a converted image frame in a side-by-side manner in which an original frame is reduced in half, and then duplicated two reduced image frames.

4 briefly shows a means for performing image conversion processing according to the present invention,

FIG. 5 schematically shows the configuration of an image content provider system (necessary for processing an image transformation in which the original image is left and right juxtaposed with two reduced identical images;

6 is a flowchart illustrating a processing procedure of an application program in which an image conversion algorithm according to the present invention is implemented.

7 schematically illustrates a configuration of a video player 500 of a content consumer,

FIG. 8 is a flowchart illustrating a method of reproducing an image according to the present invention at an image content consumer when providing the image into two types, an original image and an image converted according to the present invention.

9 is a flowchart schematically illustrating a processing procedure of an application program for processing video conversion when video data is provided in the form of a live broadcast signal.

10 is a view for explaining the effect that the screen is enlarged according to the lens focal length of the stereoscopic when viewing the converted image in accordance with the stereoscopic mirror,

11 (a) and 11 (b) illustrate a case where a user views two identical converted images reproduced in a side by side form on a smartphone using a simple stereoscopic head and a stereoscopic stereoscopic head mounted mirror according to an embodiment of the present invention. Respectively,

12 illustrates a case in which the eyes of the user's eyes are normally positioned at the center of the left and right half-half regions of the display screen (in case of (a)) and when they are displaced due to strabismus (in case of (b) to (d)). By way of example,

13 is an explanatory diagram of an embodiment of a target for adjusting the position of an image display area displayed on a display screen and a corrected image display area.

Hereinafter, with reference to the accompanying drawings will be described a preferred embodiment of the present invention.

The present inventors have found the following points in the process of exploring techniques for solving side effects (visual fatigue and psychological anxiety) caused by stereoscopic images caused by binocular factors. That is, when the same two images are viewed simultaneously through the left eye and the right eye in substantially the same visual direction (ie, the two directions of the two images facing each other in both eyes are parallel to each other), The effect of double recognition of the three-dimensional effect occurs. In addition, when the two images are greatly enlarged through the convex lens, the stereoscopic effect by the monocular factors can be recognized more clearly, and the realism and presence of the image are combined together so that the stereoscopic or simple 2D image due to the existing binocular factor is combined. You can enjoy the video with a completely different feeling from the audience's view. The present invention is based on this concept.

In the present invention, 'the same two images' include two completely identical image frames, such as the original and a duplicate, when the original image frame is duplicated, and of course, they are not completely identical but can be regarded as 'substantially identical'. The concept includes up to two image frames having the same identity. For example, two image frames (eg, a first image frame configured by extracting only even fields and a second image frame extracted only odd-numbered fields) may be configured by extracting specific image data that are similar enough to be recognized from one original image frame. Or a first image frame formed by extracting only odd pixels in a row and a column direction, and a second image frame formed by extracting only even pixels, or an original image frame, for example, interpolation (eg, adjacent pixel interpolation, Linear image interpolation, cubic interpolation, B-spline interpolation, etc.) may be a representative example of 'two image frames having the same' as two image frames obtained by reducing or expanding at an appropriate ratio. no. The 'substantially identical two images' are derived from the same original image frame, and widely cover those two image frames having no wide-angle difference such as binocular disparity between the two image frames and having only a slight difference in pixel data values.

The 'monocular factor' that is the basis of the three-dimensional effect provided by the present invention is mainly an empirical depth clue, for example, as follows. Recognizing depth when the same object appears large when it's close and small when it's far away (relative size of the object), when two parallel lines actually converge in one point in three-dimensional space (line perspective) Recognizing through experience that obscured objects are behind when overlapping (overlapping); perception of depth in experience when distant objects appear blurred by atmospheric scattering effects (atmospheric scattering); Perception of depth when recognizing texture changes in an object (texture gradient); guessing the three-dimensional shape of an object through light and shadow (light and shadow); Perception (relative velocity) and perception of depth due to a change in focus on an object (focus) are typical monocular factors.

The stereoscopic effect caused by such monocular factors is inherently different from the stereoscopic effect of a normal 3D image. The stereoscopic effect of the image due to conventional binocular factors may be obtained by photographing the same object at different angles or by processing the image to have such a difference by a wide-angle difference corresponding to binocular disparity. It is a three-dimensional feeling when the left and right eyes view the images in different directions by binocular disparity. The stereoscopic effect according to the monocular factor provided by the present invention hardly causes side effects (visual fatigue and psychological anxiety) exhibited by such stereoscopic images due to such a binocular factor. It is also different from the normal 2D image, which has no sense of three-dimensionality, which is perceived when the 'one image' is viewed in 'left and right eyes' in different directions.

The present invention is a method for allowing the user to separately view the same two images through the left eye and the right eye, respectively. The display screen 20 of the image device 10 used by the user is divided into left and right as illustrated in FIG. 1. And displaying the same two images in the left image display area 22L and the right image display area 22R in the left screen area 20L and the right screen area 20R, respectively. As shown in FIG. 2, the user views two images of the left image display area 22L and the right image display area 22R into the left and right eyes through two

convex lenses

30L and 30R having a predetermined magnification, respectively. see. At this time, the left eye and the right eye see the same left image and the right image in substantially the same visual direction. For normal eyes other than strabismus, the direction VD1 from the left lens 30L to the center point CP1 of the left screen region 20L and from the right lens 30R to the center point CP2 of the right screen region 20R. Direction VD2 is substantially parallel.

In order to be able to display such an image, the original image needs to be properly processed. An exemplary image processing method proposed by the present invention is to convert an original image into a new frame for each frame. The converted frame is a frame in which two images that are substantially identical to the left half and the right half of the frame are arranged in a side-by-side (SBS) form. The resolution of the converted frame may be the same as or different from the resolution of the original frame. The image data composed of the converted image frames is directly made by the system of the image provider (e.g., broadcaster, YouTube, etc.) and provided to the user's image device, or consumes the image content. May be made using original image data provided from an image provider.

First, a case of generating converted image data at the image provider side will be described. According to an example, each frame of the original image is reduced in half, and then the reduced image is duplicated in two and juxtaposed to the left and right to produce a converted frame corresponding to the frame. 3 schematically shows one example of converting image data in this manner, and FIG. 4 shows a configuration of means for performing such image conversion processing.

Assume that the original image frame 100 to be converted is a digital image having a and b pixels, respectively. According to an exemplary image processing method, first, for image conversion, the image conversion unit 200 receives an original image and performs image conversion processing to output the converted image. The image conversion process first reads the original image in units of frames to produce a reduced image frame 110 in which the number of horizontal pixels and the number of vertical pixels of each image frame 100 are reduced by half. Then, the reduced image frame 110 is copied twice to create a new SBS-type image frame 120 in which the two reduced image frames 110 are juxtaposed on the left and right sides. The image frame 120 is the converted image frame 120 according to the image processing method of the present invention.

Instead of this method, as mentioned above, two reduced image frames 110 may be made from the original image frame 100 by using various known interpolation methods or extracting field data.

The content provider side system provides the processed video to the user's video playback device in a streaming transmission method or a broadcasting transmission method through a wired or wireless communication network. The user's image reproducing apparatus only needs to decode the received image data and reproduce each converted image frame 120 as it is. Since the converted image frame 120 itself is composed of two identical reduced frames arranged side by side, the same two images are simultaneously displayed on the left half and right half regions of the screen on the display screen of the image reproducing apparatus. The user sees the displayed image using, for example, a stereoscopic mirror in which two convex lenses correspond to the two eyes. At this time, the left eye sees the image of the left half of the converted image frame 120 and the right eye sees the right half of the image 110 by the distance between the display screen and the two convex lenses and the refractive indices of the two convex lenses. See). Since the user sees the same image of the left eye and the right eye at the same time, the user can feel the stereoscopic effect due to the monocular factor double.

This is different from general 3D stereoscopic images. In a typical 3D stereoscopic image, the two images seen by the left and right eyes are not the same, and the non-identical images (photographed) viewed from two directions with a deviation corresponding to binocular disparity (one image is distorted compared to the other image) Video). The human brain perceives the depth (stereometry) of the image by the difference between the images received by both eyes. In order to feel the three-dimensional feeling of the image, it is necessary to show an image having a difference between both eyes, and also an image having a difference according to the distance of an object. In the case of the present invention, since the two images entering the left and right eyes are the same, a stereoscopic feeling like a general 3D stereoscopic image does not appear. Instead, if the same two images are seen separately through the left and right eyes, each of the two images is a sense of coolness with a sense of depth or three-dimensional by the monocular factors included in the image.

In addition, by utilizing the techniques used to view stereoscopic images, the converted images according to the present invention can be viewed with higher quality. In particular, when the two images are magnified through two convex lenses of stereoscopic mirrors, the sense of presence and reality are added, and a new feeling image which is not experienced in a general 2D or 3D image can be viewed. Details thereof will be described later.

FIG. 5 schematically illustrates a configuration of an image content provider system 300 necessary to process an image transformation in which the same original image is reduced and left and right juxtaposed as described above. The system 300 may include a data store 310 or a camera 320 as a content source. A CPU 330, a memory 340, and an input / output device serving as a user interface 350 are also included.

Apart from these hardware resources, an application program (hereinafter, referred to as an image conversion program) and a user interface program implementing the image conversion algorithm described below are also required. 6 is a flowchart schematically showing an execution procedure of this video conversion program.

As an example, consider a case in which the source image to be converted is stored in the data storage 310 in the form of a file. The CPU 330 executes an image conversion program to generate a converted image frame (also referred to as a target image frame) by scaling each source image frame of the original image by a predetermined ratio. The ratio applied to the scaling is preferably set to a value such that the two target video image frames have an image size that can be displayed in the left half region and the right half region of the display screen of the video reproducing apparatus, respectively. The ratio of reduction or enlargement of the source image frame may be preferably determined in consideration of the correlation between the resolution of the source image data and the resolution of the display of the image device. It is necessary to perform conversion in a range in which the resolution of the target image frame converted from the source image frame does not exceed the resolution of the display. For example, if the resolution of the source image data is the same as that of the display of the imaging device, as mentioned above, the two targets of which the resolution of the source image frame is reduced in the ratio of half or less in the horizontal and vertical directions respectively are juxtaposed on the left and right sides. You can configure the frame. If the resolution of the source image data is, for example, about 1/4 of the resolution of the display of the imaging apparatus, the resolution of the source image data may be increased up to twice.

For scaling, first, the number of pixels of the target image frame, that is, the resolution, is determined (step S10). Examples of resolutions set as standards may include various types such as 1080p (1920x1080 pixels), 720p (1280x720 pixels), 480p (640x480 pixels), and the like. If the resolution of the source image is, for example, 1080p (1920x1080 pixels), the resolution of the target image frame may also be set to 1920x1080. Of course, it is possible to set the value higher or lower than this resolution. If the target image frame has a higher resolution than the source image frame, the frame size is larger. On the contrary, if the target image frame has a lower resolution, the frame size is smaller. The CPU 330 secures sufficient buffer memory space in consideration of the resolution of the target image frame.

The CPU 330 reads the source image file from the data storage 310 (step S12). The image files may be files compressed according to a predetermined compression standard such as MP4, avi, wmv, mkv, and the like. If it is such a compressed video file, the video file is decompressed using the decoding tool of the corresponding standard (step S14). Decode based on eg 30 frames per second.

The video conversion process according to the present invention is now performed with the decoded video data. First, each video frame Fi 100 of the decoded video data is scaled by a predetermined ratio (step S16). The scaling ratio is determined according to how much to enlarge or reduce the original video frame Fi 100. Hereinafter, an example in which each image frame Fi 100 is reduced to half the size will be described. One exemplary method of reducing each image frame Fi 100 by half is to reduce the horizontal and vertical pixel counts of each image frame Fi 100 by half, respectively. That is, in the horizontal and vertical directions of each image frame Fi 100, for example, all the even-numbered pixels are removed and the image is newly constructed with only the odd-numbered pixels, or all the odd-numbered pixels are removed and only the even-numbered pixels are removed in the horizontal and vertical directions. To reconstruct the video. The reduced image frame Fi / 2 (110) is a screen in which the number of pixels is reduced in half and the area is reduced to one quarter in comparison with the original image in the horizontal and vertical directions, respectively. According to the above example, the number of pixels of the reduced image frame Fi / 2 110 is 960x540.

When the image Fi / 2 110, which is reduced in half for each image frame, is prepared, the reduced image Fi / 2 110 is left and right in the converted image region of the memory 340 as shown in FIG. By copying twice in parallel to form a new image frame Fi '(120) (S18). The new video frame Fi 120 configured by combining two reduced frames of the original video frame is the converted video frame or the target video frame that the present invention intends to obtain through the conversion.

The converted target image frames Fi '120 are encoded in a desired file format (MP4, avi, wmv, mkv, etc.) in consideration of storage efficiency or transmission efficiency and then stored in the data storage 310 (S20). step).

Then, when necessary or in response to the request of the content consumer, the converted image file stored in the data storage 310 is provided to the content playback device of the content consumer (step S22). The method of providing the converted video file to the content consumer may be, for example, a wired or wireless transmission method such as a streaming method, a broadcasting method, or a download method, but is not necessarily limited thereto.

On the other hand, as an example of a broadcasting transmission method, a video broadcasted by a television station in real time also performs the above-described conversion processing prior to radio transmission, and then converts the converted video into a broadcast signal such as over-the-air broadcast, cable broadcast, or DMB broadcast. You can also send. In this case, the source of the conversion target image may be the camera device 320 instead of the data storage 310. That is, an image frame generated by the camera device 320 is provided to the CPU 330 in real time, and the CPU 330 scales each image frame of the image data at a predetermined ratio in the same manner as described above. To be broadcasted by the broadcasting method. In detail, the CPU 330 processes an image frame provided by the camera device 320 to the image, which is scaled by a predetermined ratio for each frame through the processing of steps S16 and S18 described above, for example, an image reduced to 1/2. After making it, it is converted to a combined video frame by juxtaposing it. Then, the converted video frames are collected and subjected to compression encoding processing in accordance with the transmission format of the broadcast signal, and transmitted by the broadcasting method.

According to the method of pre-converting the source video frame into the target video frame in the content provider system and providing the same to the video playback device of the user who is the content consumer, the video playback device of the user receives the converted video signal and receives the normal video. This can be done according to the signal reproduction method. Then, the image displayed on the display screen of the image reproducing apparatus will display an image in which the same two images are arranged side by side.

In providing the actual image content to which the present invention is applied, the image content provider side may provide the same image in two forms to the content consumer side. That is, one is the image data of the target image frame Fi 'obtained by converting two identical image frames reduced from the source image frame to the left and right as described above, and the other is an image composed of the original source image frame which is not converted. Data. Content consumers may have the choice of video in both of them in consuming video content.

When the content consumer receives and reproduces the converted image data including the converted target image frames (Fi ′), the left half image and the right half image constituting each image frame of the converted image data are the left eye of the user. Play the video so that you can see each one separately. As an exemplary method for this, any one of the known 3D stereoscopic image display methods may be combined with the reproduction of the converted image data. 7 schematically illustrates a configuration of an exemplary video playback device 500 of a content consumer that can do so.

The image reproducing apparatus 500 includes a communication unit 510, a CPU 530, a memory 540, a storage unit 520, a user interface 550, a video driver 530, a display 570, and the like. The communication unit 510 receives video data provided by an external video providing system (for example, a broadcasting system or a streaming server system) and delivers demodulated video data to the CPU 430. The CPU 530 utilizes a data processing space provided by the memory 540 to perform overall control of the image reproducing apparatus 500 and to reproduce image data transmitted through the communication unit 510 in a desired stereoscopic image reproducing method. Perform various data processing necessary to The storage unit 520 functions as an application program, image data before and after processing, and a data storage space (for example, a hard disk or a flash memory) for storing a program or data necessary for performing the present invention. The user interface 550 provides a function that allows the user to give a command and check the processing result, including specifying one of the original image and the converted image. The video driver 560 converts the image data provided by the CPU 530 into a display driving signal and provides the converted image data to the display 570. The display 570 displays an image on the screen according to the display driving signal.

8 is a flowchart illustrating a method of reproducing an image according to the present invention at the image content consumer side, when the image content provider provides two types of the original image and the converted image according to the present invention.

The content consumer may select one of the original source image and the converted image through the user interface 550. The image selection information is provided to the CPU 530 (step S30). Only when the converted video is selected, the processing described below is performed. If the original video is selected, the processing of Fig. 8 described below is not performed, and the original video is reproduced by a normal playback method.

The CPU 530, which has received the image selection information from the user interface 550, requests the content providing system to provide the image while providing information on the desired converted image to the content providing system through the receiver 510 (S32). However, this step may be omitted depending on the transmission method of the video content. For example, when the content provider transmits the original source video and the converted video at the same time by broadcasting, the user's request procedure may be omitted, and the user may selectively take the converted video from the received video. On the other hand, if the content provider side provides the video content in response to the demand of the content consumer, such a request will be necessary.

When the content provider system transmits the converted image data, the communication unit 510 receives the converted image data and transmits the converted image data to the CPU 530. When the video data is allowed to be stored, the CPU 530 may store the video data in a file form in the storage unit 520 and perform the playback process described below when necessary. If the transmitted video is a live broadcast video, the communication unit 510 may demodulate and then provide the demodulation process to the CPU 530. When the video data is a video provided by a streaming method or a live broadcast video, the CPU 530 performs the following processing for reproducing the video data in real time while receiving the video data stream.

First, the CPU 530 temporarily stores the converted image data stream provided through the communication unit 510 in the buffer memory (step S34). The converted video data is read and decoded by predetermined units (step S36). The CPU 530 also performs additional processing in accordance with the specifications of the stereoscopic image reproduction method (viewing method) and the display 570 for each decoded screen frame Fi (step S38). Then, the processed image data is provided to the video driver 560 (step S40). The video driver 560 generates a driving signal for driving the display 570 based on the image data provided by the CPU 530, and drives the display 570 with the driving signal. As a result, an image is displayed on the display 570 according to a desired stereoscopic image reproduction method.

The content consumer sees the image played on the display 570. The playback image should match the stereoscopic image viewing method used by the consumer. Known three-dimensional image viewing method (playing method) can be largely divided into glasses and glasses-free method. As an eyeglass viewing method (playback method), a stereoscope method, a head mounted display (HMD), a red-blue eyeglass method (Anaglyph), a polarizer method, a time division to a shutter glass method Etc. are known. The glasses-free viewing method (playing method) includes a parallax barrier method and a lenticular method. The premise that these spectacle viewing methods and the glasses-free viewing methods are common premise is that different left-eye images and right-eye images are artificially assigned to the optical angle in order to make the playback image feel three-dimensional. Is the point.

In contrast, the present invention is fundamentally different from the conventional stereoscopic image reproducing method in that the left eye image and the right eye image are the same. That is, in the present invention, the same left image and right image constituting each frame Fi 'of the converted image are used as the left eye image and the right eye image. The present invention differs from the conventional stereoscopic image reproducing method in that an image frame Fi 'converted by the above-described conversion method is used as a source image for reproduction. In addition, the left and right eyes are located at the center of each of the left and right images, and the stereoscopic image reproduction is performed in that the direction connecting the centers of the left and left images is substantially parallel to the direction connecting the centers of the right and right images. And the viewing method is different.

The stereoscopic method, known as one of the 3D imaging techniques, juxtaposes two images having wide-angle deviations as much as binocular disparity to the left and right, and forms the two images on both eyes separately through two convex lenses juxtaposed to the left and right. It is a way to make the image feel in three dimensions. This stereoscopic method can be applied to the present invention. In this case, the left image and the right image of the converted image frame Fi 'are displayed on the left half and the right half of the display 570 or vice versa. In addition, the content consumer wears a stereoscopic mirror in which two convex lenses are juxtaposed to the left eye and the right eye, and the left half of the display 570 is viewed as the left eye, and the right half is viewed as the right eye. In other words, the left eye and the right eye view the same two images with no wide angle difference.

In the case of the present invention, since two images with no wide-angle difference are viewed, the three-dimensional feeling as much as when viewed with the conventional stereoscopic image viewing method is not felt. However, it is not the same as the feeling of seeing a 'one' image in which the left and right eyes are common. In the present invention, since the left eye and the right eye separately view the two 'same' images, the viewing feeling is about a three-dimensional feeling between the three-dimensional effect and the simple two-dimensional feeling. It is so comfortable that almost no visual fatigue or psychological anxiety can be felt in a full stereoscopic image, but it looks like a living and refreshing feeling to some extent.

In particular, when wearing two-dimensional mirror juxtaposed to the left and right of the two lens facing the display of the image reproducing apparatus and looking at the two images displayed on the display 570, the effect of the image is greatly enlarged according to the magnification of the lens to obtain Can be. This is particularly effective when watching an image with a smartphone having a small size of the display 570. If the user wears a stereoscopic mirror and views the two images with the left and right eyes, respectively, while the two images are reproduced in the left and right half of the screen, respectively, the size of the image felt by the user is greatly enlarged according to the lens focal length.

11 shows the image displayed on the left and right two regions of the display screen 20 of the smartphone 10 through the

stereoscopic mirrors

50 and 60, using the smartphone 10 as the user's video playback device. An example is shown. In the simplified stereoscopic mirror 50 of FIG. 11 (a), two

convex lenses

30L and 30R are mounted on a frame 52 having an eyeglass shape, and the leg ends 40L and 40R of the frame 52 are smart. The left and right outer portions of the display screen 20 of the phone 10 are provided to be picked up. In the head mounted stereoscopic mirror 50 of FIG. 11 (b), two

convex lenses

30L and 30R are mounted to a position of two eyes in a frame 62 that is shaped to be used on a user's head. The front of the frame 62 is provided with a detachable smartphone 10, the speaker 64 is provided on both ears of the user.

As the

lenses

30L and 30R of the

stereoscopic mirrors

50 and 60, it is preferable to use a simple plano covex lens, or an aspherical lens or a double-sided aspheric lens for minimizing chromatic aberration and refraction. The diameter of the lens is preferably 20 to 50mm. Since the viewing angle that a person can see comfortably is about 40 to 70 degrees, it is preferable to use a lens having characteristics that can provide such a viewing angle. The focal length of the lens is suitably between about 30 and 80 mm. That is, when viewing images displayed on the image reproducing apparatus 10 such as a smart phone using the stereoscopic mirrors 50 and 60, the display screen of the image reproducing apparatus is separated from the two lenses by an interval of about 30 to 80 mm. It is preferable. If the lens is too close to the display screen, the viewing angle is too wide, which can cause eye fatigue. If the viewing angle is too narrow, there is a limit to increasing the viewing magnification of the screen, and in order to enlarge the screen with a large magnification, the lens and the display screen must be separated from each other, causing various inconveniences. In addition, since the distance between the eyes of the user varies from person to person, it is preferable that each user adjusts the distance between the centers of the two lenses to approximately 45 to 70 mm according to their own eye distance.

FIG. 10 is a stereoscopic lens when a user's video player is a smartphone (Samsung Galaxy S4 model) having a screen size of 11cm x 6.5cm in width and length, and a virtual screen is positioned 150cm in front of a stereoscopic lens. In the case of four types of focal lengths of 6 cm, 12 cm, 18 cm and 24 cm, the size of the screen seen through a stereoscopic mirror is shown schematically. According to FIG. 10, in the case of a lens having a focal length of 6, an image having a length of about 277.24 cm is displayed on a virtual screen 1.5 m away. In other words, it has the same effect as the user visually (without wearing a stereoscopic body) a TV having a horizontal size of 277cmm at a distance of 1.5m. However, in the case of the present invention, the two images are arranged in a side-by-side form and are half-divided so that they are enlarged to 138.62 cm. Half of the actual width of the smartphone is 5.5cm, so it is approximately 25 times larger. In other words, the user can play the stereoscopic image of the converted image according to the present invention on the smartphone of the size (5-inch screen diagonally), and the user wears a stereoscopic mirror composed of two lenses having a focal length of 6 cm to view the reproduced image. The same effect can be seen with the naked eye seeing a 125-inch screen 25 times magnified 1.5 meters forward.

If the lens focal length of the stereoscopic mirror becomes longer, the effect of screen magnification is halved. For example, if the focal length of the lens is 24cm, it is like viewing a screen with a width of 72.66cm on the virtual screen 1.5m forward. In this case, the screen magnification is approximately 6.5 times (72/11), so that the user perceives a 36-inch television about the same size as the naked eye looking at 1.5 meters away.

The difference between the playback image according to the present invention described above, a conventional stereoscopic image, and a simple 2D image is summarized in the following table.

Table 1

division	Conventional stereoscopic image	Image of the present invention	Simple two-dimensional image
Source footage	Two images for left and right eyes with a wide angle difference (non-identical images)	Two images for left eye and right eye with no difference in wide angle (same image)	One image for the left eye and the right eye
How to watch	View left eye and right eye images in the left and right eyes, respectively	View left eye and right eye images in the left and right eyes, respectively	Both the left and right eyes view one video
Spectating feeling	Three-dimensional	About halfway between three-dimensional feeling and two-dimensional feeling	Two-dimensional
Other		If you wear a magnified lens, you can zoom in on the small display image.

Alternatively, the process of converting the source video frame into the target video frame can be performed by the video reproducing apparatus of the user consuming the content. Even in this case, the user's video reproducing apparatus scales (reduces or enlarges the video frame) the original video signal composed of the source video frame received from the content provider system to produce a target video signal. When performing the scaling process, it is desirable to consider the correlation between the resolution of the original source video frame and the resolution of the display screen.

Next, a case in which the operation of creating the converted image frame Fi 'is performed by the user who consumes the image will be described. As a means for converting the image, for example, the image reproducing apparatus 500 having a hardware resource as shown in FIG. 7 is sufficient. There is also a need for an application program configured to execute an image conversion processing algorithm. For example, the application program can be converted in a smartphone, tablet PC, general PC, and the like to convert an image. It may be desirable to implement this application program in a form of inclusion as some functions of a video player.

The video data to be converted may be stored in a file format in the storage unit 520, video data streamed in real time from an external content providing system, or video data carried on a broadcast signal transmitted by an external broadcast system. In either case, the content provider does not differ much from generating the converted image. That is, the original source image frame is scaled to a desired ratio and converted into a target image frame, and the converted target image frame is duplicated in two to be displayed in the left half and the right half of the display screen, respectively. The basic concept of allowing the left and right eyes to see the image displayed in the left half of the display screen and the image displayed in the right half through the lens is the same.

First, consider a case where the image data is provided in the form of a live broadcast signal. 9 is a flowchart schematically showing an execution procedure of an application program for processing video conversion in this case.

The image provided by the external broadcasting system is a general digital image. The content consumer may choose whether to watch the unconverted original image or the converted image according to the present invention. This option may be exercised through the user interface 550 (step S50). If it is selected to watch the converted video, the CPU 530 performs the video conversion processing described below.

In detail, the image data received in the form of a broadcast signal will be provided to the CPU 530 through the communication unit 510. Since this video data will normally be encoded according to a specific compression standard, the CPU 530 decodes the received video data (step S52). Each screen frame Fi of the decoded video data is scaled by a desired ratio (step S54). The scaling ratio is preferably determined in consideration of the resolution of the image data and the resolution of the display 570. For example, if the resolution of the display 570 is 720p (1280x720 pixels), consider splitting the screen of the display 570 to the left and right to display the same two target image frames on the split left and right screens. . In this case, the conversion is performed to reduce each source image frame by half. As a method of reducing the source image frame by half, the above-described method, that is, removing all the even pixels in the horizontal and vertical directions of the image frame and using the remaining odd pixels as the converted image frame can be used. . The number of pixels in the converted frame is 640x360. If the image is displayed on the display in a half-contracted state without any other conversion, the size of the image displayed in the side-by-side form is half the display size. That is, with respect to the reference coordinate of the display (when the coordinate of the upper left is set to (0, 0), for example), the coordinate of the upper left (starting position) of the left half region (left eye display area) of the display 570 screen is (0). 179, and the coordinates of the lower right (end position) are (639, 539). In addition, the coordinates of the upper left (start position) of the right half area (right eye display area) of the display 570 screen are (640, 179), and the coordinates of the lower right (end position) are (1279, 539). . If the resolution of the source image is different from that of the display 570, the image frame may be converted using an appropriate scaling method according to the position and size of the display area of the display.

Each screen frame (Fi / 2) obtained by the scaling process is duplicated in two and arranged to be parallel to the left and right to form a new image frame (Fi ') (step S56).

The CPU 530 sequentially generates the converted image frame Fi 'and performs necessary processing according to the stereoscopic image reproducing method applying the converted screen frames on the other hand (step S58).

The processed screen frames are collected and provided to the video driver 560. That is, the CPU 530 divides two duplicate image frames Fi 'into two predetermined memory regions of the video driver 560-two memory regions corresponding to the left half region and the right half region of the display 570. Store each in-. Then, the video driver 560 reads out the two duplicated image frames Fi 'from the two memory regions and processes them to be displayed on the left half and right half regions of the display 570 (step S60).

If allowed, the CPU 530 may collect the new image frames Fi ′ obtained in step S46 and store them in a file form in the storage unit 520, and later read and play the stored files when necessary.

Next, conventional image files stored in the storage unit 520 may be converted into a converted image file according to the present invention. That is, the CPU 530 reads the source image file to be converted from the storage unit 520 and sequentially executes a combination of decoding in step 42, image reduction in step 44, and juxtaposed image in step 46. The screen frame (Fi) is processed into a converted video frame (Fi '). Subsequently, steps S48 and S50 are performed to perform playback in parallel with the conversion, or the converted frames Fi 'are collected and stored in a file format in the storage unit 520 and played back when necessary.

On the other hand, if the user has an abnormal muscle of the eyes of the eyes, such as strabismus, it is desirable to be able to adjust the display position of the two images of the side-by-side type displayed on the display to match the eyes of the two eyes. Do. Explain how to do this.

A user with normal binocular rather than strabismus has a visual direction of both eyes as shown in (a) of FIG. 12. The center (CP1) and the right half region (the center of the left half region 20L of the display 20, 570) 20R) will be towards the center (CP2). However, the eyes of the eyes of the strabismus are different. For example, the visual direction of the two eyes viewed through the two

lenses

30L and 30R of the

stereoscopic mirrors

50 and 60 shows a slight deviation in the left eye and a right deviation in the right eye as shown in FIG. As shown in (c), the left eye is slightly upward while being narrowed in the left and right directions, or as in (d), the left eye has a slight deviation from the center (CP1) to the upper right and the right eye has a large deviation upward. The visual direction of at least one of the two eyes does not coincide with the two centers CP1 and CP2. It is necessary to correct the center of the left image and the center of the right image displayed in the side-by-side form on the display screen 20 so as to match the visual directions of the eyes of the users themselves.

To this end, first, targets C1 and C2 for image display area correction are respectively displayed on the left image display area 22L and the right image display area 22R of the

displays

20 and 570, as shown in FIG. 13A. ). The targets C1 and C2 may use a circle in which a centerline is displayed. However, the targets C1 and C2 may be various shapes such as a rectangle and a hexagon. FIG. 13 (b) shows a state where the user who is out of focus with the pupil is recognized when using the stereoscopic mirrors 50 and 60 to view the side-by-side image displayed so as to be horizontal, as shown in FIG. 13 (a). One case. When a user sees the side-by-side image as shown in FIG. 13 (b), the second target C2 is moved by -dx and -dy to match. That is, the

image reproducing apparatuses

10 and 500 adjust the coordinates CS2 of the right image display area 22R through an input device (for example, a remote control or joystick connected by Bluetooth) or a touch screen sensor. The

display

20 and 570 receive an input of a position adjustment signal which is commanded to move by (-dx, -dy). When the user of the

image reproducing apparatus

10 or 500 sees that the target coincides when viewing the left and right juxtaposed images displayed on the

displays

20 and 570, the adjustment end signal is input by the user at that time, and the image reproducing apparatus is input. The CPU 530 of (10, 500) receives the adjustment end signal and records the coordinate positions of the adjusted left image display area 22L and / or right image display area 22R in the storage unit 520. do. 13 (c) shows the left image display area 22L and / or the right image display area 22R adjusted to be suitable for the user. As a result, the left image and the right image of the converted target image frame are displayed in the left image display area 22L and the right image display area 22R, respectively, which are positioned to match the eyes of the user's eyes.

As described above, the conversion of an image frame may be performed in advance in the content provider system to provide the converted image content to the content consumer, or the content consumer may perform the conversion process of the image frame. In either case, the pre- and post-processing of the transformation resulting from differences in the transformation environment or conditions may be slightly different, but the core algorithms of the image transformation are substantially the same. Alternatively, it may be possible to embed the image conversion algorithm into an operating system program. In other words, it is possible to provide a user with a right to divide a screen, and to split the screen into two left and right sides according to a user's selection, and to simultaneously display both images to be played on the divided two screens.

Claims

In the image conversion unit, first converting each source image frame (Fi) of the source image data to a corresponding target image frame (Fi ') to generate the converted image data consisting of the target image frame (Fi') Step, wherein the conversion of each source video frame (Fi) to a corresponding target video frame (Fi ') to scale the source video frame (Fi) by a predetermined ratio to create two substantially identical converted video frame, Achieved by combining the two converted image frames to form the target image frame Fi ';

A second step of providing the converted image data to the image reproducing unit; And

The image reproducing unit reproduces the received target image frames Fi 'of the converted image data on a screen, and displays the images of the two substantially identical converted image frames constituting each target image frame Fi'. By displaying the left half and the right half of the screen, respectively, the left eye and the left eye of the image content consumer viewing the same two converted image frames constituting each target image frame (Fi ') with a stereoscope including two lenses. And a third step of reproducing the right eye so that the right eye can be seen separately.
The method of claim 1, wherein the ratio applied to the scaling is such that the two substantially identical converted image frames have an image size that can be displayed in the left half region and the right half region of the display screen of the image reproducing apparatus, respectively. Video data conversion and playback method characterized in that.
The method of claim 1, wherein the two substantially identical converted video frames are made to reduce the size of the source video frame (Fi) in half in the horizontal and vertical directions, respectively, and then reduce the reduced video frames into two. Video data conversion and reproduction method characterized in that the two reduced image frames obtained by copying.
2. The method of claim 1, wherein the two substantially identical transformed image frames comprise (i) an even numbered first and second transformed image frames formed by taking only pixel data in odd-numbered rows and columns of the source image frame Fi. 3. And a second converted video frame composed of only pixel data in rows and columns.
The method of claim 1, further comprising: displaying a target for correcting an image display area in the left half and right half divided by the left and right of the display screen;

And receiving the image display area correction coordinate information of any one of the left half region and the right half region through a user input means, and changing the image display region coordinates according to the image display region correction coordinate information. Video data conversion and playback method.
The stereoscopic two-convex lens of claim 3, wherein the image display in the third step is performed in a portable image playback device including a smartphone or a tablet computer, and the image content consumer provides a viewing angle of 40 to 70 degrees. And simultaneously view the two substantially identical converted image frames through the stereoscopic mirror in a state spaced 30 to 80 mm from the display screen.
The image reproducing apparatus as claimed in claim 1, wherein the first, second and third steps are performed in an image reproducing apparatus of the image content consumer, which includes both the image converting unit and the image reproducing unit. How to convert and play data.
The image content consumer side according to claim 1, wherein the first step and the second step are performed by an image providing system of an image content provider having the image converting unit, and the third step is provided with the image reproducing unit. Image data conversion and playback method characterized in that performed by the video playback device.
The image data conversion method according to claim 8, wherein when the image conversion unit performs the second step, the converted image data is transmitted to the image reproduction unit by a streaming method or a broadcast method. How to play.
10. The method of claim 9, wherein in the second step, the image providing system provides the source image data together with providing the converted image data to the image reproducing apparatus.
12. The method of claim 10, wherein the third step is performed when the video content consumer instructs playback by selecting the latter of the source video data and the converted video data in the video playback device. How to play.