KR20160135660A - Method and apparatus for providing 3-dimension image to head mount display - Google Patents

Abstract

A method and apparatus for providing a stereoscopic image are provided. An electronic device for providing a stereoscopic image selectively generates the image of a virtual viewpoint according to a users position. An image synthesizing part generates the image of a virtual viewpoint corresponding to the position state of the user by using the image provided to the electronic device. A display part outputs an image corresponding to the position state. The image corresponding to the position state may include the image of the virtual point and may include the image provided to the electronic device.

Description

METHOD AND APPARATUS FOR PROVIDING 3-DIMENSION IMAGE TO HEAD MOUNT DISPLAY FIELD OF THE INVENTION [0001]

The following embodiments relate to a method and apparatus for image processing, and more particularly to a method for providing a stereoscopic image for a head mounted display.

Recently, as the technology related to the head-mount display (HMD) is developed, HMDs using the latest technology are introduced.

In particular, a recent HMD can detect the motion of the user's head in real time when the user wears the HMD, and provides a view in that direction regardless of which direction the head is moving.

Further, the HMD has a left lens and a right lens. Each of the left lens and the right lens provides a concave curved panoramic display image. Such a panoramic display image provides a wide viewing angle, and the user can see the display of the virtual reality by moving his / her pupil.

For example, recent HMDs use a tracking technique to recognize the movement of the user's head and produce stereoscopic images provided to both eyes of the user. By providing the binocular image corresponding to the view point of the user, the HMD can allow the user to perceive that he is in the virtual reality.

As a method for providing a multi-view image by the HMD as described above, a method of generating two biometric images by attaching images taken by a multi-view camera to each other is generally used .

In such a method, the following problems may occur. 1) First, the correlation between neighboring images must be used in order to generate a connected image. 2) It is also required to attach images to each other for every frame of the image. 3) In addition, since a connected image must be generated based on a fixed resolution, an image quality deteriorates in which the resolution of the image provided is further reduced compared to the resolution of the original image.

One embodiment of the present invention can provide an apparatus and method for selectively generating an image at a virtual viewpoint according to a user's viewpoint.

A method for providing stereoscopic images to a user on an electronic device, the method comprising: generating an image of a virtual viewpoint corresponding to a positional state of the user using an image provided to the electronic device; And outputting an image corresponding to the position state, wherein the image corresponding to the position state includes the image at the virtual time point.

An image corresponding to the position state may include an image provided to the electronic device.

The image corresponding to the position state may include a left image provided in the left eye of the user and a right image provided in the right eye of the user.

The time of the left image and the time of the right image may be different from each other.

The time of the left image and the time of the right image may be determined based on the interval between the eyes of the user.

The location status may be information for tracking the location of the user based on the viewpoint of the image initially provided to the user.

The image providing method may further include generating the position state by recognizing movement of the head of the user.

The positional state may include information indicating a displacement of movement of the user's head to the left or right.

The location state may include information indicating a rotation of the user's head to the left or right.

If the virtual viewpoint is the same as the viewpoint of one of the one or more images of the provided image, the same viewpoint as the virtual viewpoint of the one or more images can be used as the virtual viewpoint image.

The image corresponding to the position state may be one or more images of one or more viewpoints.

The provided image may be one or more images of one or more views.

Wherein the step of generating an image at the virtual time point is performed when at least one of the views of one or more images corresponding to the position state is not the same as any one of the one or more viewpoints of the one or more provided images .

The image providing method may include: determining whether the position state has changed; And updating the camera parameters according to the change of the position state.

Updating the camera parameters may include updating the camera parameters in accordance with movement of the user's head to the left or right.

Updating the camera parameters may include updating the camera parameters in accordance with rotation of the user's head to the left or right.

The change of the positional state may indicate the displacement of the positional movement according to the movement of the user to the left or right with reference to the reference time.

The change of the positional state may indicate a rotation angle of the user with respect to the left or right rotation based on the reference time point.

The camera parameters may be a matrix including a focal length in the x-axis and a focal length in the y-axis.

The provided image may be a multi-view image of High Efficiency Video Coding (HEVC).

The provided image may be a stitched image of High Efficiency Video Coding (HEVC).

The image of the virtual viewpoint may be generated using a depth map corresponding to the provided image.

The image at the virtual viewpoint may be generated using a disparity map corresponding to the provided image.

In another aspect, in an electronic device, an image synthesizing unit generates an image of a virtual viewpoint corresponding to a position state of a user using an image provided to the electronic device; And a display unit for outputting an image corresponding to the position state, wherein the image corresponding to the position state includes the image at the virtual time point.

The head-mounted display according to claim 1, further comprising: an image synthesizer for generating an image at a virtual viewpoint corresponding to a user's positional state using an image provided to the head-mounted display; And a display unit for outputting an image corresponding to the position state, wherein the image corresponding to the position state includes the image at the virtual time point.

There is provided an apparatus and method for selectively generating an image at a virtual viewpoint according to a user's viewpoint.

1 is a structural diagram of an electronic device for providing a stereoscopic image according to an exemplary embodiment of the present invention.
Figure 2 illustrates an electronic device using an HMD according to an example.
Figure 3 illustrates an electronic device implemented as an HMD according to an example.
FIG. 4 illustrates an image provided to an electronic device according to an example.
5 is a flow diagram of a method of providing an image to an electronic device user in accordance with an embodiment.
6 is a flowchart of a method for determining whether an image at a virtual viewpoint is required according to an example.
FIG. 7 illustrates a structure of a 3D image system according to an embodiment.
Figure 8 illustrates a multi-view display with intermediate view composition according to an example.
9 shows a stitched image according to the movement of the user's head.
10 illustrates a computer system for implementing an electronic device according to an embodiment.

The following detailed description of exemplary embodiments refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the location or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the embodiments. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the exemplary embodiments is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained.

In the drawings, like reference numerals refer to the same or similar functions throughout the several views. The shape and size of the elements in the figures may be exaggerated for clarity.

The terms used in the examples are intended to illustrate the embodiments and are not intended to limit the invention. In the examples, the singular includes the plural unless otherwise stated in the specification. It is noted that the terms "comprises" and / or "comprising" used in the specification are intended to be inclusive in a manner similar to the components, steps, operations, and / And that additional configurations may be encompassed within the scope of the embodiments of the exemplary embodiments or the technical ideas of the exemplary embodiments. When it is mentioned that a component is "connected" or "connected" to another component, the two components may be directly connected or connected to each other, It is to be understood that other components may be present in the middle of the components.

The terms first and second, etc. may be used to describe various components, but the components should not be limited by the terms above. The above terms are used to distinguish one component from another. For example, without departing from the scope of the right, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

In addition, the components shown in the embodiments are shown independently to represent different characteristic functions, which does not mean that each component is composed of separate hardware or one software constituent unit. That is, each component is listed as each component for convenience of explanation. For example, at least two of the components may be combined into a single component. Also, one component can be divided into a plurality of components. The integrated embodiments and the separate embodiments of each of these components are also included in the scope of the right without departing from the essence.

Also, some components are not essential components to perform essential functions, but may be optional components only to improve performance. Embodiments may be implemented only with components that are essential to implementing the essentials of the embodiments, and structures within which the optional components are excluded, such as, for example, components used only for performance enhancement, are also included in the scope of the right.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to facilitate embodiments of the present invention by those skilled in the art. In the following description of the embodiments, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear.

In the embodiments below, an image can mean visual information of a moment. The images processed in the steps in the following embodiments may be visual information of a moment. For example, the image may be a frame at time t. A video may include one or more images. One or more images of the video may have a time series order.

The steps of the following embodiments can be repeatedly performed over time. As the steps are repeatedly performed, a series of images can be processed sequentially.

In the following embodiments, "depth" and "disparity" may have a reciprocal relationship with each other. That is to say, when a pixel represents a particular object, the "depth" of the pixel can be the inverse of the "displacement" of the pixel. In embodiments, "depth" and "disparity" may be used interchangeably.

In the following embodiments, the "depth map" can be used with the same meaning as "depth image ".

The "depth map" may represent the depth value of the pixel for the corresponding image. For example, for pixels having the same coordinate values of "depth map" and "image", the pixel value of the pixel of "image" may represent the color of the object that the pixel represents, The pixel value of the pixel may indicate the distance from the viewpoint of the object represented by the pixel.

In the following embodiments, the "displacement map" can be used in the same sense as the "displacement image ".

The "displacement map" can represent the displacement value of the pixel with respect to the corresponding image. For example, in pixels having the same coordinate values of "displacement map" and "image", the pixel value of the pixel of "image" may represent the color of the object that the pixel represents, The pixel value of the pixel may represent the displacement of the object represented by the pixel. The displacement of the object can represent the reciprocal of the distance from the viewpoint of the object.

1 is a structural diagram of an electronic device for providing a stereoscopic image according to an exemplary embodiment of the present invention.

The electronic device 100 may include an image receiving unit 110, an image decoding unit 115, a tracking unit 120, an image synthesizing unit 130, and a display unit 140.

The display unit 140 may include a left display unit 141 and a right display unit 142.

The image receiving unit 110 may receive a multi-view image and a depth map provided from the outside. The image receiving unit 110 may transmit the provided multi-view image and the depth map to the image combining unit 130.

The image provided to the electronic device 100 may be a multi-view image of High Efficiency Video Coding (HEVC). In addition, the image provided to the electronic device 100 may be a stitched image of the HEVC.

An image provided from the outside may be an encoded image. For example, the image receiving unit 110 may receive a bitstream from the outside, and the externally provided multi-view image and depth map may be encoded in the bitstream, entropy encoded Data and compressed data.

The image decoding unit 115 may generate a decoded image by performing decoding on the encoded provided image. Hereinafter, the provided image used in the image combining unit 130 may be a decoded image generated by the image decoding unit 115. [

For example, the image decoding unit 115 may perform decoding on the encoded multi-view image and the encoded depth map to generate a decoded multi-view image and a decoded depth map. Hereinafter, the multi-view image and the depth map used by the image synthesis unit 130 may be a decoded multi-view image generated by the image decoding unit 115 and a decoded depth-map.

For example, the image decoding unit 115 may generate a decoded multi-view image and a decoded depth map by performing HEVC-based decoding on the multi-view image and the depth map of the HEVC. The decoded multi-view image and the decoded depth-map may be suitable for compositing images of the virtual viewpoint described below. The image decoding unit 115 may generate a decoded image by performing decoding according to the HEVC on the stitched image of the HEVC. The decoded image may be suitable for the synthesis of the image of the virtual point of view described below.

The tracking unit 120 may generate a signal of a position state. The signal of the position state may include information indicating the position state.

The time point required can be determined according to the position state. For example, depending on the user's position and / or posture, it may be determined at what point the electronic device 100 should provide the user with an image.

The location status may be information that tracks the location of the user's head. For example, the position state can track the state of the left and right movement position and the rotation position. Alternatively, for example, such as location status may indicate the user of the position of the head (position) and / or orientation of the electronic device (100) (pose). The position and / or posture of the user's head may refer to the position and / or posture of the electronic device 100 or the tracking portion 120 to which the user's head is attached. In the following, the position and / or posture of the user's head may be replaced by the position and / or posture of the electronic device 100 or the position and / or posture of the tracking portion 120. [

The location status may be information that tracks the location of the user based on the viewpoint of the image initially displayed by the electronic device 100. [ Alternatively, the location status may be information that tracks the location of the user based on the viewpoint of the image initially provided to the user. For example, the positional state may track the state of the user's left and right movement position and / or the rotation position based on the time of the image initially displayed by the electronic device 100 or the time of the image initially provided to the user.

The image combining unit 130 can generate a video image at a desired timing. The required point of time may be the time point of the stereoscopic image output from the electronic device 100. The viewpoint of the stereoscopic image output from the electronic device 100 can be determined according to the position of the head of the user or the user. That is to say, the required time point may be a time point depending on the position and / or posture of the user or the user's head. Alternatively, the time point required may be a time point depending on the position and / or posture of the electronic device 100.

The stereoscopic image may include images of a plurality of viewpoints. The image combining unit 120 may generate an image at each viewpoint of a plurality of desired viewpoints.

If the requested point in time is the same as the point of view of one of the one or more images of the multi-view image from the external to the electronic device 100, the image synthesizer 130 determines that the required one or more images of the multi- It is possible to use the image at the same time point as the time point as the image at the required time point.

If the required point of view is different from the viewpoint of the multi-view image provided from the outside, the image synthesizer 130 may generate an image at a virtual viewpoint using the multi-view image and the depth map. Here, the virtual viewpoint may indicate a time point different from the viewpoint of the multi-view image among the required viewpoints.

The image synthesizer 130 may generate an image at a virtual viewpoint using an image neighboring the virtual image among one or more images of the multi-view image. The image synthesis unit 130 may generate an image of a virtual viewpoint using an image at a point closest to the virtual viewpoint among the one or more images of the multi-view image. Alternatively, the image synthesis unit 130 may generate an image of a virtual viewpoint using the image at the closest point to the left or right of the virtual viewpoint, among the one or more images of the multi-view image. Alternatively, the image synthesis unit 130 generates an image of a virtual viewpoint by using an image at a closest point to the left of the virtual viewpoint and an image at a closest point to the right of the virtual viewpoint, among the one or more images of the multi- can do.

A user of the electronic device 100 may recognize a stereoscopic image through his or her eyes. That is to say, the time required to provide the stereoscopic image to the user may be two. The requested point of time may include a left point of view of the user's left eye and a right point of view of the user's right eye.

When the left viewpoint is different from the viewpoint of the multi-view image, the image composition unit 130 can generate an image at the left viewpoint. In addition, when the right viewpoint is different from the viewpoint of the multi-view image, the image composition unit 130 can generate an image at the right viewpoint.

The display unit 140 can output a video image at a desired timing. The left display unit 141 can output the left view image. And the right side display unit 142 can output the image at the right side view.

Configuration of multi-view image

The multi-view image may include one or more images. In the above description, the required time point is the same as the time point of the multi-view image, which means that the required time point is the same as one of the one or more time points of the multi-view image. Alternatively, the requested time point is the same as the time point of the multi-view image, which means that the requested time point includes the image at the time point when the multi-view image is requested. The fact that the requested point of view is different from the point of view of the multi-view image may mean that the requested point-in-time is not the same as the point in time at which one or more points of the multi-view image are required. Alternatively, the requested time point is different from the time point of the multi-view image, which means that the required time does not include the image at the same time point as the multi-view image is requested.

Each image of one or more images may be an image at a specific point in time. The views of one or more images may be different from each other. That is to say, one or more images of the multi-view image may be images generated by one or more cameras photographing the object and the background at different locations. The photographed object and background can be represented as pixels in the image.

In conjunction with the multi-view image, a depth map corresponding to the multi-view image may be provided to the electronic device 100. The pixels of the multi-view image and the pixels of the depth map may correspond to each other. In the pixels of the multi-view image and the depth map corresponding to each other, the pixel value of the pixel of the multi-view image can represent the color of the photographed point, and the pixel value of the pixel of the depth map is the camera of the photographed point And a depth value indicating a distance from the reference point. Alternatively, the pixel value of the pixel of the depth map may be a disparity value that indicates the inverse of the distance of the photographed point from the camera.

Each image of one or more images may be a stereoscopic image. When each image is a stereoscopic image, each image may include a color image and a depth map.

In a color image and a depth map, one pixel of the color image and one pixel of the depth map may correspond to each other. For example, in a color image and a depth map, the pixels of the color image having the same coordinates and the pixels of the depth map may correspond to each other. In the pixels of the color image and the depth map corresponding to each other, the pixel value of the pixel of the color image may indicate the color of the photographed point, and the pixel value of the pixel of the depth map may indicate the distance of the photographed point from the camera May be a depth value indicating a depth value.

Alternatively, the pixel value of the pixel of the depth map may be a disparity value that indicates the inverse of the distance of the photographed point from the camera.

Operations and functions of the image receiving unit 110, the image decoding unit 115, the tracking unit 120, the image synthesizing unit 130, and the display unit 140 will be described in detail below.

Selective decoding

It may require an excessive amount of computation for the image decoding unit 115 to perform decoding on all of one or more images of the encoded multi-view image. In the decoding of the encoded image by the image decoding unit 115, the image decoding unit 115 may selectively decode only the image corresponding to the predetermined condition. For example, the image decoding unit 115 may selectively decode only an image required to provide a user with an image at a desired point in time, from among one or more images of the encoded multi-view image.

For example, the image decoding unit 115 may include one or more of the following: 1) an image used to generate an image at a virtual viewpoint, and 2) an image used at the same point in time as the requested viewpoint Only decoding can be performed.

In order to selectively decode one or more images of the encoded multi-view image by the image decoding unit 115, the image synthesizing unit 130 outputs information for selecting the image to the image decoding unit 115 Lt; / RTI > The information for selection of the image may represent the image (s) of which one or more images of the encoded multi-view image should be decoded by the image decoding unit 115. Alternatively, the information for selecting the image may represent the image (s) selected by the image synthesizer 130 among the one or more images of the encoded multi-view image.

The image synthesis unit 130 may derive which one of the one or more images of the multi-view image is required to provide an image at a desired time point, and may determine which one of the one or more images of the multi- It is possible to generate information for selection of the image according to whether it is required to provide the image at the required time.

Figure 2 illustrates an electronic device using an HMD according to an example.

At least some of the image receiving unit 110, the image decoding unit 115, the tracking unit 120, the image synthesizing unit 130, and the display unit 140 may be separate devices.

The image synthesis 130 may be connected to the image receiving unit 110, the image decoding unit 115, the tracking unit 120, and the display unit 140, respectively.

An HMD can be used to provide a stereoscopic image to the user. For example, the display unit 140 may be an HMD. The left display 141 may be the left lens of the HMD. The right display 142 may be the right lens of the HMD.

Figure 3 illustrates an electronic device implemented as an HMD according to an example.

The electronic device 100 may be an HMD. For example, the image receiving unit 110, the encoding decoding unit 115, the tracking unit 120, the image synthesizing unit 130, and the display unit 140 may be components in the HMD.

FIG. 4 illustrates an image provided to an electronic device according to an example.

An image may be provided from the outside to the electronic device 100. [ The image provided to the electronic device 100 may be a multi-view image. In addition, the image provided to the electronic device 100 may be a multi-view and depth map.

The image provided to the electronic device 100 may be transmitted to the electronic device 100 via the network 410. [

For example, the image provided to the electronic device 100 may be a multi-view image generated by a multi-view camera. In FIG. 4, as a multi-view camera, a first camera 420 and a second camera 430 are shown. The multi-view camera may be a camera array.

Alternatively, for example, the image provided to the electronic device 100 may be a computer graphic (CG) and / or a multi-view image of a real image. The multi-view image of CG and / or real world can be generated by other electronic devices.

The image provided to the electronic device 100 may be a compressed image.

5 is a flow diagram of a method of providing an image to an electronic device user in accordance with an embodiment.

In step 510, the image receiving unit 110 may receive an image provided to the electronic device 100. [

One or more images may be provided to the electronic device 100. One or more views of one or more images provided to the electronic device 100 may be different from one another. That is, the image provided to the electronic device 100 may be a multi-view image.

The image provided to the electronic device 100 may be one or more images of one or more views. For example, the image provided to the electronic device 100 may be a multi-view image. In addition, the image provided to the electronic device 100 may include a depth map.

At least some of the plurality of images provided to the electronic device 100 may be images at a real time point. The image at the actual time point may be an image taken by a camera disposed at a position corresponding to a specific point in time.

In step 515, the image decoding unit 115 may generate a decoded image by performing decoding on the encoded image that has been encoded.

Hereinafter, the image provided to the electronic device 100 may represent an image decoded by the image decoding unit 115. [

In step 520, the tracking unit 120 may obtain information on the positional state of the user. The tracking unit 120 can generate the position state by recognizing the movement of the user's head.

For example, the position state may include information indicating the position and posture of the user. The user's position and posture can indicate the position and posture of the user's head. In the following, "user's position and posture" and "position and posture of the user's head"

For example, the position state may indicate a change in the position state of the user. The positional state may be such that the tracking unit 120 can detect a change in the positional state and generate positional state information indicating a change in the positional state or the positional state. The positional state may indicate movement and rotation of the user's head. The positional state may include information indicating a displacement of movement of the user's head to the left or right. The positional state may include information indicating a rotation of the user's head to the left or right.

The image combining unit 130 may obtain the information of the position state from the tracking unit 120. [ The image synthesis unit 130 may calculate a required time based on the obtained position state. The requested point in time may indicate the point of view of the image to be provided to the user according to the position state of the user.

In operation 530, the image synthesis unit 130 may determine whether an image of a virtual viewpoint corresponding to the user's position state is required.

If an image at a virtual viewpoint is required, step 540 may be performed. If an image at the virtual time point is not required, step 550 may be performed.

For example, when there are one or more images corresponding to the positional state and one or more images provided to the electronic device 100, a step 540 of generating an image of a virtual viewpoint may include: At least one of the plurality of electronic devices 100 is not identical to any of the one or more views of the images provided to the one or more electronic devices 100. [ In other words, the step 540 of generating an image at the virtual time point may include, for one image provided to the electronic device 100, an image corresponding to the position state of the user when the image corresponding to the user's position state is not provided And may be performed to generate an image at a virtual viewpoint.

In step 530, described in more detail below with reference to FIG.

In operation 540, the image synthesizer 130 may generate an image at a virtual viewpoint corresponding to the user's position using the image provided to the electronic device 100. [

The image combining unit 130 may use a depth map in generating an image at a virtual viewpoint. The image synthesizer 130 may generate an image of a virtual viewpoint corresponding to the position state of the user using the image provided to the electronic device 100 and the depth map corresponding to the provided image. Alternatively, the image combining unit 130 may generate an image of a virtual viewpoint corresponding to the user's position state using camera parameters corresponding to the position of the head of the user.

 The image at the virtual time point may be a new time point different from the time point of the image provided to the electronic device 100. [ In other words, the image synthesis unit 130 may synthesize an image at a virtual viewpoint using the image provided to the electronic device 100. [

If step 540 is performed, then step 550 may be performed to provide an image of the virtual time point generated in step 540.

In step 550, the display unit 140 may output an image corresponding to the position state of the user.

In outputting the image, the display unit 140 may determine which of the image provided to the electronic device 100 and the image at the virtual viewpoint is to be output. The display unit 140 may determine an image to be output together with an image of a virtual viewpoint among the one or more images provided to the electronic device 100. [

If step 540 is performed, the image corresponding to the position state may include the image at the virtual time point.

If step 550 is not performed, the image corresponding to the position state may be the image provided to the electronic device 100 at step 510.

Also, an image corresponding to the position state may be one or more images of one or more viewpoints. For example, two images may be provided in the left and right eyes of the user, respectively, depending on the positional state of the user. The image corresponding to the position state may include a left image provided in the left eye of the user and a right image provided in the right eye of the user.

The plurality of views of the plurality of images corresponding to the position state may be different from each other. For example, the viewpoints of the left image and the right image may be different from each other. The image synthesis unit 130 may determine a difference between a viewpoint of a left image and a viewpoint of a right image based on an interval between the eyes of the user.

When there are a plurality of images corresponding to the positional state, at least a part of the images may be images at a virtual time point. In addition, the remaining images of the plurality of images, other than the images at the virtual viewpoint, may be images provided to the electronic device 100. [ That is, the image corresponding to the position state may include 1) an image at a virtual viewpoint, and 2) an image provided to the electronic device 100.

The output of the image may mean outputting the data of the image to the outside. Alternatively, the output of the image may mean outputting the image in a form recognizable through the user's view.

6 is a flowchart of a method for determining whether an image at a virtual viewpoint is required according to an example.

Step 530 described above with reference to FIG. 5 may include steps 610, 620, 630, 640 650, 660, and 670 described below.

In step 610, the image synthesis unit 130 may determine whether the user's position state has changed.

The change in the position state can be determined based on the viewpoint of the image output by the electronic device 100. [ The viewpoint of the image output by the electronic device 100 may be a reference time point. The change of the positional state may indicate the displacement of the positional movement according to the movement of the user to the left or right side based on the reference point. The change of the positional state may indicate a rotation angle of the user with respect to the left or right rotation based on the reference point. Alternatively, a change in the positional state may indicate a displacement and / or a rotational angle of the positional movement.

When the position state is changed, processing according to the changed position is required.

If it is determined that the user's location status has changed, step 620 may be performed. If it is determined that the user's position status has not changed, step 670 may be performed.

In step 620, the image synthesis unit 130 may update the camera parameters according to the change of the position state.

Step 620 may include steps 630, 640, 650, and 660. As at least a portion of step 620, steps 630 and 650 may be performed, respectively.

In step 630, the image synthesis unit 130 may determine whether the position state indicates movement to the left or right of the user's head.

If the position state indicates movement to the left or right of the user's head, step 640 may be performed. If the position state does not indicate movement to the left or right of the user's head, step 670 may be performed.

In step 640, the image synthesis unit 130 may update the camera parameters according to the movement of the user's head to the left or right. The updated camera parameters may reflect movement of the user's head to the left or right side.

If step 640 is performed, then step 670 may be performed.

In step 650, the image synthesis unit 130 may determine whether the position state indicates rotation to the left or right of the user's head.

If the position state indicates a rotation to the left or right of the user's head, step 660 may be performed. If the position state does not indicate a rotation to the left or right of the user's head, step 670 may be performed.

In step 660, the image synthesis unit 130 may update the camera parameters according to the rotation of the user's head to the left or right. The updated camera parameters may reflect a rotation to the left or right of the user's head.

If step 640 is performed, then step 670 may be performed.

The above-described steps 630, 640, 650 and 660 may be exemplary only. For example, steps 630 and 640 may be performed before or after steps 650 and 660. Alternatively, for example, steps 630 and 650 may be performed, and then steps 640 and 660 may be performed at once or separately. When steps 640 and 650 are performed at one time, the image synthesizer 130 may update the camera parameters according to the movement and / or rotation of the user's head to the left or right.

In step 670, the image synthesis unit 130 may determine whether the time indicated by the position state is the same as the time of the image provided to the electronic device 100. [ Alternatively, the image synthesis unit 130 may determine whether there is a time point equal to a time point indicated by the position state among one or more time points of the multi-view image provided to the electronic device 100. [

If the same point in time exists, step 540 described above with reference to FIG. 5 may be performed. In operation 540, the image synthesis unit 130 may generate an image of a virtual viewpoint corresponding to the user's position state using the image and camera parameters provided to the electronic device 100. [

If the same time does not exist, step 550 described above with reference to FIG. 5 may be performed.

Hereinafter, a method of synthesizing an image at a virtual viewpoint according to an example will be described.

The user can wear his or her electronic device 100 or the HMD and turn his or her head to the left or right or move it in the longitudinal direction. The positional state sensed by the tracking unit 120 may include R _hmd indicating the rotation of the user to the left or right and T _hmd indicating the movement of the user in the longitudinal direction. Further, the positional state may include a viewing angle F indicating the direction in which the electronic device 100 or the user wearing the HMD is looking. That is to say, the electronic device 100 or the HMD can be regarded as looking at an object as a virtual camera.

The camera parameters at a particular point in time v can be expressed as Equations 1, 2 and 3 below.

Matrix K may represent an internal camera parameter.

Here, f _x can indicate the focal length on the x axis. f _y can represent the focal length in the y- axis. C _x can represent the x coordinate of the principle point of the image plane. C _y can represent the y coordinate of the principal point of the image plane.

In Equation (2), the matrix R may represent the rotation matrix of the camera.

In Equation (3), the matrix T may represent the position of the camera. The location of the camera represented by matrix T may be a location on the world coordinate.

The image synthesis unit 130 may project the coordinate values ( x , y , z ) on the world coordinate to an arbitrary point at a specific point in time v .

The projection of the coordinate value ( x , y , z ) on the world coordinate to an arbitrary point of the virtual point of view v can be expressed by the following equation (4).

Coordinates (x _v, y _v) may represent the point in virtual time v.

According to Equation (4), if the depth value at the virtual viewpoint v is known, the image composing unit 130 obtains the coordinate value ( x ( x )) on the world coordinate through re- projection using the depth value at the virtual viewpoint v , , y , z ).

Image combining unit 130, the coordinate value (x, y, z) inside the parameters of the virtual viewpoint v K 'and external parameters (R' with respect to | coordinate value in "vapor viewpoint v by applying T) '(x' , y ').

로 표현될 수 있다. 이동 방향 부호

는 좌측 또는 우측으로의 이동에 따른 부호를 나타낼 수 있다.The values T ', which are changed by tracking the movement of the user or electronic device 100, are defined by the movement distance T _hmd and the movement direction sign

. ≪ / RTI > Direction code

May indicate a sign according to movement to the left or right.

The moving distance T _hmd can be expressed by the following equation (5).

의 값은 -1 또는 1일 수 있다.

May be either -1 or 1.

The values T ', which are changed by tracking the movement of the user or electronic device 100, can be expressed as: < EMI ID = 6.0 >

만큼 회전하는 것으로 간주될 수 있다. 상기의 간주에 따라 R _hmd 는

로 변경될 수 있다. 또한, R _hmd 는 아래의 수학식 7과 같이 표현될 수 있다. R _hmd is centered on the x-axis

As shown in FIG. According to the above consideration, R _hmd is

. ≪ / RTI > Also, R _hmd can be expressed by the following Equation (7).

The image synthesis unit 130 may generate a virtual viewpoint v 'generated by tracking according to the following equation (8).

The above-described virtual viewpoint generation method is an example. The image synthesis unit 130 may generate a virtual viewpoint using the input position state and various interpolation methods.

FIG. 7 illustrates a structure of a 3D image system according to an embodiment.

A three dimensional (D) imaging system may include a transmitter and a receiver.

The receiver may include a 3D video encoder.

The receiver may include a 3D receiver, a stereo receiver, and a 2D receiver. The 3D receiver may include a 3D video decoder. The stereo receiver may include a stereo video decoder. The 2D receiver may include a 2D video decoder.

The transmitter can acquire the contents of the multi-view image using a stereo camera, a depth camera, a multi-view camera, and a conversion of a 2D image into a 3D image. The number of viewpoints of the images of the multi-view image may be M. M may be an integer of 2 or more.

The multi-view image may include images of M viewpoints. In Figure 7, the images of the M views are shown as video pictures. A video picture can represent an image at a specific point in the video.

In addition, the multi-view image may include depth maps of images of M views. In addition, the multi-view image may include additional information related to the camera.

The multi-view image can be compressed by a multi-view video encoding method. The bit-stream of the compressed multi-view image may be transmitted from the transmitter to the receiver over the network.

The receiver may perform decoding for the transmitted bit-stream using a multi-view video decoding method. The images of M start points can be restored by decoding.

The receiver can output images of M views using a display suitable for the receiver.

The 3D receiver can generate an image at a point in time when the multi-view image is not included using the intermediate view synthesis using the multi-view image. Intermediate view synthesis is described in detail below with reference to FIG.

Figure 8 illustrates a multi-view display with intermediate view composition according to an example.

In FIG. 8, decoded multi-view-depth (MVD) data is shown. The image provided to the electronic device 100 may include three images and three depth maps. In FIG. 8, the images and depth maps provided in the electronic device 100 may include V1, D1, V5, D5, V9, and D9.

The image synthesis unit 130 applies depth-based image based rendering (DIBR) to the images of M view points restored by decoding the decoded MVD data, thereby generating images V2, V3, V4, V6 , V7, and V8.

The DIBR may be a method of generating an image at a virtual viewpoint using 3D warping using a multi-view image and a depth map. One commonly used method of the DIBR is to move the coordinate values to a three-dimensional coordinate system using camera parameters and depth maps corresponding to a specific point in time, to project the coordinate values shifted to the three- can do.

The two stages of the DIBR can be as follows. 1) In the first step, a point of an image at a specific point in time is projected in the world coordinate system. In the above projection, the depth value of the corresponding point of the deep map can be used. 2) In the second step, the fold projected in the world coordinate system can be projected back to the virtual point on the image plane of the virtual camera at the virtual point of view.

The image synthesizer 130 may provide images of N viewpoints including images provided to the electronic device 100 and images at virtual viewpoints. N may be an integer greater than M.

The image synthesizer 130 may provide a desired viewpoint image according to the location of the user among the N viewpoint images. In Fig. 8, it is shown that for each of the first position, the second position, and the third position, the image required for each of the user's right eye and left eye is provided.

Images of the N viewpoints can be played back in accordance with various multi-view 3D display devices. A stereoscopic image can be provided to the user through playback.

As the number of viewpoints provided by the multi-view display increases, the number of images at a virtual viewpoint requiring synthesis may also increase. This increase may require a large amount of computation. In the above-described embodiment, the electronic device 100 can selectively reduce the amount of calculation for providing an image at a virtual viewpoint by selectively performing synthesis of an image at a virtual viewpoint.

The image synthesis unit 130 may provide M images of N view points using N images included in the multi-view image and N depth maps corresponding to the images. The image combining unit 130 can generate only the image at the desired time point using the position state acquired by the tracking unit 120 without generating the image at all possible virtual viewpoints. This generation scheme can reduce the amount of computation required for a spectacles multi-view display to synthesize virtual images of all views. In addition, the image combining unit 130 may selectively reduce the resolution of the image compared to the original image or the image distortion by generating the image of the virtual viewpoint.

9 shows a stitched image according to the movement of the user's head.

As described above, according to the movement of the user, the user's head, or the electronic device 100, the viewpoint of the image to be provided to the user can be changed. By compositing, synthesized views can be provided to the user.

For example, when the head of the user moves to the left, the image synthesizing unit 130 can generate a left stitched image. Alternatively, if the user's head moves to the right, the image combining unit 130 can generate a right-stitched image.

10 illustrates a computer system for implementing an electronic device according to an embodiment.

The electronic device 100 may be implemented as the computer system 1000 shown in Fig.

10, the computer system 1000 may include at least some of the processing unit 1010, the communication unit 1020, the memory 1030, the storage 1040, and the bus 1090. [ Components of computer system 1000 such as processing unit 1010, communication unit 1020, memory 1030 and storage 1040 may communicate with each other via bus 1090.

The processing unit 1010 may be a semiconductor device that executes the processing instructions stored in the memory 1030 or the storage 1040. [ For example, the processing unit 1010 may be at least one processor.

The processing unit 1010 can process a job required for the operation of the computer system 1000. [ The processing unit 1010 may execute the code of the operation or step of the processing unit 1010 described in the embodiments.

The processing unit 1010 may perform generation, storage, and output of information, and may perform operations of steps performed in the other computer system 1000.

The communication unit 1020 can be connected to the network 1099. May receive data or information required for operation of computer system 1000 and may transmit data or information required for operation of computer system 1000. [ The communication unit 1020 can transmit data to another device via the network 1099 and can receive data from another device. For example, the communication unit 1020 may be a network chip or a port.

Memory 1030 and storage 1040 can be various types of volatile or non-volatile storage media. For example, the memory 1030 may include at least one of a ROM (ROM) 1031 and a RAM (RAM) The storage 1040 may include internal storage media such as RAM, flash memory and a hard disk, and may include removable storage media such as memory cards and the like.

The function or operation of the computer system 1000 may be performed as the processing unit 1010 executes at least one program module. Memory 1030 and / or storage 1040 may store at least one program module. The at least one program module may be configured to be executed by the processing unit 1010. [

The computer system 1000 may further include a user interface (UI) input device 1050 and a UI output device 1060. UI input device 1050 may receive input of a user required for operation of computer system 1000. The UI output device 1060 can output information or data according to the operation of the computer system 1000.

The computer system 1000 may further include a sensor 1070.

The image receiving unit 110 may correspond to the communication unit 1020. The tracking unit 120 may correspond to the sensor 1070. The image decoding unit 115 and the image synthesizing unit 130 may correspond to the processing unit 1010. The display unit 140 may correspond to the UI output device 1060.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

100: Electronic device
110:
120:
130:
140:

Claims (20)

A method of providing a stereoscopic image to a user of an electronic device,
Generating an image of a virtual viewpoint corresponding to the positional state of the user using the image provided to the electronic device; And
Outputting an image corresponding to the position state
Lt; / RTI >
And the image corresponding to the position state includes the image at the virtual time point. The method according to claim 1,
And an image corresponding to the position state includes an image provided to the electronic device. The method according to claim 1,
An image corresponding to the position state includes a left image provided in the left eye of the user and a right image provided in the right eye of the user,
Wherein the viewpoint of the left image and the viewpoint of the right image are different from each other. The method of claim 3,
Wherein the viewpoint of the left image and the viewpoint of the right image are determined based on an interval between the eyes of the user. The method according to claim 1,
Wherein the location information is information for tracking the location of the user based on a viewpoint of an image initially provided to the user. The method according to claim 1,
Generating the position state by recognizing movement of the head of the user;
Further comprising the steps of: The method according to claim 1,
Wherein the position state includes information indicating a displacement of movement of the user's head to the left or right side. The method according to claim 1,
Wherein the location state includes information indicating a rotation of the user's head to the left or right. The method according to claim 1,
Wherein the virtual point of time is the same as the point of time of one of the one or more images of the provided image, the image of the same point in time as the virtual point of time is used as the image of the virtual point of time. The method according to claim 1,
Wherein the image corresponding to the position state is one or more images of one or more views,
Wherein the provided image is one or more images of one or more views,
Wherein the step of generating an image at the virtual time point is performed when at least one of the views of one or more images corresponding to the position state is not the same as any one of the one or more viewpoints of the one or more provided images Image providing method. The method according to claim 1,
Determining whether the position state has changed; And
Updating the camera parameters according to the change of the position state
Further comprising the steps of: 12. The method of claim 11,
Wherein updating the camera parameters comprises:
Updating the camera parameters in accordance with movement of the user's head to the left or right
The method comprising the steps of: 12. The method of claim 11,
Wherein updating the camera parameters comprises:
Updating the camera parameters in accordance with rotation of the user's head to the left or right
The method comprising the steps of: 12. The method of claim 11,
Wherein the change of the position state indicates a displacement of the position movement according to the movement of the user to the left or right based on the reference time. 12. The method of claim 11,
Wherein the change of the position state represents a rotation angle of the user with respect to a left or right rotation based on a reference time point. 12. The method of claim 11,
Wherein the camera parameter is a matrix including a focal length in the x-axis and a focal length in the y-axis. The method according to claim 1,
Wherein the provided image is a multi-view image of High Efficiency Video Coding (HEVC). The method according to claim 1,
Wherein the image of the virtual viewpoint is generated using a depth map corresponding to the provided image. In an electronic device,
An image synthesizer for generating an image of a virtual viewpoint corresponding to a position state of the user using the image provided to the electronic device; And
A display unit for outputting an image corresponding to the position state,
Lt; / RTI >
And the image corresponding to the position state includes the image at the virtual time point. In a head mount display,
An image synthesizer for generating an image of a virtual viewpoint corresponding to a positional state of the user using the image provided to the head mount display; And
A display unit for outputting an image corresponding to the position state,
Lt; / RTI >
Wherein the image corresponding to the position state includes the image at the virtual time point.

Images