WO2016204323A1

WO2016204323A1 - User view point-linked image processing device and method therefor

Info

Publication number: WO2016204323A1
Application number: PCT/KR2015/006113
Authority: WO
Inventors: 최광진; 박영준; 나경건
Original assignee: (주)에프엑스기어
Priority date: 2015-06-17
Filing date: 2015-06-17
Publication date: 2016-12-22

Abstract

Disclosed are a user view point-linked image processing device and a using method therefor, the device comprising: a sensor unit for generating motion data by sensing a change in the view point of a user; a communication interface unit for receiving an image package including image data from a media server; a scene formation unit for arranging a virtual camera in a virtual space and forming a scene with respect to the image data, by using the image package; and a virtual camera positioning change unit for changing the positioning of the arranged virtual camera by using the motion data, such that the positioning corresponds to the change in the view point of the user.

Description

User viewpoint linked image processing device and method

The present invention relates to an image processing apparatus and method, and more particularly, to detect a movement of a viewpoint of a user using various sensors and to change a part of an image provided to the user so as to correspond to the movement of the viewpoint of the user. It relates to an image processing apparatus and a method.

In general, when taking an image using an optical device such as a camera and displaying the captured image, when viewing an image that does not enter every scene in the user's field of view, such as a wide viewing angle image, the image does not enter the field of view of the current user. You need to move the part of the image that is displayed to see the screen.

Conventionally, in this case, the user changes a portion of the image displayed through a keyboard or a touch operation, but such a change is inconvenient in that the user has to perform a separate input action.

Prior Art: Korean Patent Publication 10-2008-0022544

In order to solve such a problem, there is a need for a method in which a separate operation is not required in changing a part of an image displayed in a direction or a position intended by a user.

In accordance with one embodiment of the present invention, an apparatus for synchronizing user views may include: a sensor unit configured to generate a motion data by detecting a change in a viewpoint of a user, a communication interface unit receiving an image package including image data from a media server; By using the image package, using a scene configuration unit for arranging a virtual camera in the virtual space and constituting a scene for the image data and the motion data, of the arranged virtual camera to correspond to a change in the viewpoint of the user And a virtual camera posture converting unit configured to convert postures.

In addition, in the user-view linked image processing apparatus, the posture of the virtual camera may include a direction or a position of the virtual camera.

The sensor unit for detecting a change in the viewpoint of the user may include at least one of a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU) sensor, a global positioning system (GPS) sensor, and a vision sensor. It may be characterized by including.

In addition, in the user-view linked image processing apparatus, the image package further includes a camera parameter for posture conversion of a virtual camera, and the camera parameter is a parameter for an actual camera that captures an image of the image data. It can be characterized.

The virtual camera posture converting unit may convert the posture of the arranged camera by using the motion data and the camera parameter.

In addition, in the user-view linked image processing apparatus, the image data may include at least one of a wide viewing angle streaming image and a stereoscopic image.

In addition, the user-view linked image processing apparatus further includes a projection geometry generation unit that generates projection geometry for outputting a wide viewing angle image by using camera parameters of an actual camera that captures the image of the image data. The generation unit may generate the projection geometry by correcting the vertex or texture coordinates of the reference geometry by using the camera parameter and transforming the corrected reference geometry into a curved shape.

In the user-view linked image processing apparatus, the camera parameter may include at least one of a camera lens parameter, a camera stabilization parameter, and a stereo camera rig misalignment correction parameter. .

In the user-view linked image processing apparatus, the scene configuration unit may adjust the scene by dynamically adjusting the position of the virtual camera using the camera stabilization parameter.

The user-view linked image processing apparatus may further include an image rendering unit which renders at least a part of the scene viewed by the virtual camera in real time in the converted camera posture and displays it on a display unit.

In accordance with another aspect of the present invention, a user view linked image providing media server comprises: an image package storage unit having an image package including image data and camera parameters for posture conversion of a virtual camera, and another image package terminal; It includes a communication interface for transmitting to the, wherein the camera parameter may be characterized in that the parameter for the actual camera that captured the image of the image data.

In addition, in the user-view linked image providing media server, the posture of the virtual camera may include a direction or a position of the virtual camera.

The image data may include at least one of a wide viewing angle streaming image and a stereoscopic 3D in the user view linked image providing media server.

The camera parameter may include at least one of a camera lens parameter, a camera stabilization parameter, and a stereo camera rig misalignment correction parameter in the user viewpoint linked image providing media server. have.

In accordance with another aspect of the present invention, there is provided a method of synchronizing user views with a user, by generating a motion data by detecting a change in a user's viewpoint by a sensor, and transmitting image data from a media server by a communication interface unit. Receiving an image package comprising a step of placing a scene and a virtual camera for the image data in a virtual space using the image package, and using the motion data, to correspond to a change in the viewpoint of the user And converting the pose of the arranged virtual camera.

In addition, in the user-view linked image processing method, the posture of the virtual camera may include a direction or a position of the virtual camera.

In addition, in the user-view linked image processing method, the image package further includes a camera parameter for posture conversion of the virtual camera, wherein the camera parameter is a parameter for an actual camera that captured the image of the image data. It can be characterized.

Also, in the user-view linked image processing method, the converting the posture of the arranged virtual camera may include changing the posture of the virtual camera using the motion data and the camera parameter. Can be.

In the user view-linked image processing method, the image data may include at least one of a wide viewing angle streaming image and a stereoscopic 3D image.

The method may further include displaying, on a display unit, at least a part of the scene viewed by the virtual camera in real time in the converted virtual camera posture.

According to an embodiment of the present invention, as the user changes his viewpoint such as turning his / her head, the part of the image displayed to correspond to the changed viewpoint may be changed. Accordingly, a separate input action using a hand is not required, and since a portion of an image displayed according to movement of a viewpoint is different, an intuitive user experience can be provided as compared with the prior art.

1 is a block diagram of a user view (VIEW POINT) linked image processing apparatus according to an embodiment of the present invention.

2 is a view for explaining the operation of the sensor unit 100 according to an embodiment.

3 is a conceptual diagram illustrating a configuration of an image package 10 according to an embodiment of the present invention.

4 is a view for explaining the operation of the scene and the virtual camera in a virtual space according to an embodiment of the present invention.

5A to 5E are diagrams for describing projection geometry projection and texturing according to a vertex transformation method of projection geometry, according to an exemplary embodiment.

6A through 6E are diagrams for describing projection geometry projection and texturing according to a texture coordinate transformation method of projection geometry, according to an exemplary embodiment.

7 is a diagram illustrating a network environment including a viewpoint-linked video providing media server according to an embodiment of the present invention.

8 is a flowchart of a user view linked image processing method according to an embodiment of the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms "comprise" or "having" are intended to indicate that there is a listed feature, number, step, operation, component, part, or combination thereof, one or more other features or numbers, It is to be understood that the present invention does not exclude the possibility or addition of steps, actions, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal sense unless clearly defined herein. . Like reference numerals in the drawings denote like elements. However, in describing the embodiments, when it is determined that detailed descriptions of related known functions or configurations may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted. In addition, the size of each component in the drawings may be exaggerated for description, it does not mean the size that is actually applied.

Embodiments described herein may have aspects that are wholly hardware, partly hardware and partly software, or wholly software. As used herein, "unit", "module", "device" or "system" and the like refer to hardware, a combination of hardware and software, or a computer related entity such as software. For example, parts, modules, devices, or systems herein refer to running processes, processors, objects, executables, threads of execution, programs, and / or computers. computer, but is not limited thereto. For example, both an application running on a computer and a computer may correspond to a part, module, device, system, or the like herein.

Embodiments have been described with reference to the flowchart presented in the drawings. Although the method is shown and described in a series of blocks for the sake of simplicity, the invention is not limited to the order of the blocks, and some blocks may occur in different order or simultaneously with other blocks than those shown and described herein. Various other branches, flow paths, and blocks may be implemented in order to achieve the same or similar results. In addition, not all illustrated blocks may be required for the implementation of the methods described herein. Furthermore, the method according to an embodiment of the present invention may be implemented in the form of a computer program for performing a series of processes, which may be recorded on a computer-readable recording medium.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a user view (VIEW POINT) linked image processing apparatus according to an embodiment of the present invention. Referring to FIG. 1, in an exemplary embodiment, the user-view linked image processing apparatus 1000 may include the sensor unit 100, the communication interface unit 200, the scene composition unit 300, and the virtual camera posture converting unit 400. Include. In another embodiment, the user-view linked image processing apparatus 1000 may further include at least one of the projection geometry generator 500, the image renderer 600, and the display 700.

The sensor unit 100 generates motion data by detecting a change in viewpoint of a user. In detail, the sensor unit 100 may detect a direction in which the user's eyes are directed or a current location of the user, and track the changed direction or location. In one embodiment, the sensor unit 100 may include a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU) sensor, a global positioning system (GPS) sensor, and a vision sensor, and mix at least one of the aforementioned sensors. It may be included. For example, the sensor unit 100 may detect a change in a user's viewpoint by using the vision sensor and the IMU sensor together. In this case, the sensor unit may use a method of unifying the coordinate system by calibrating the information obtained from the two sensors. As the above-described vision sensor, various sensors such as a charged coupled device (CCD) sensor, a pinned photo diode (PPD), a charge and injection device (CID), an active pixel sensor (APS), and an active column sensor (ACS) may be used. Optical motion tracking can be used.

In one embodiment, the motion data includes orientation information in which the viewpoint of the user is changed in the real space. In another embodiment, the motion data may further include location information of the user's viewpoint. The motion data is data which is the basis for changing the direction or the position to which the virtual camera existing in the virtual space faces.

2 is a view for explaining the operation of the sensor unit 100 according to an embodiment. Referring to FIG. 2, it is assumed that the current user's viewing direction is the -z axis. In addition, when the user turns his head or moves the position to change the viewing direction, the sensor unit 100 may measure the amount of the viewing direction (that is, the viewpoint).

For this measurement, as shown in FIG. 2, a sensor may be attached to the user's head, and in another embodiment, the user's viewpoint may be detected using image data without being attached to the user's body. In another exemplary embodiment, the sensor unit 100 may include a pupil tracking sensor to locate the pupil tracking sensor in the vicinity of the user's eyeball, and detect the user's viewpoint change by capturing the pupil of the user.

The communication interface 200 receives a video package containing video data from the media server. The media server is a server that provides an image package to the user-view linked image processing apparatus 1000 and may have various image contents. For example, the media server may be a server that YOUTUBE®, GOOGLE®, ITUNES®, etc. use to provide multimedia contents to the user. The communication interface 200 may receive an image package from a media server through a network. The network uses standard communication technologies / protocols. Thus, the network 3000 may be configured for Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 2G / 3G / 4G, digital subscriber line (DSL), asynchronous transfer mode (ATM). And links using technologies such as infiniband, PCI Express Advanced Switching, and the like. Similarly, the networking protocol used on the network 3000 may include multiprotocol label switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), user datagrams. User datagram protocol (UDP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (FTP), and the like. The data exchanged on the network may be in binary form (e.g., Portable Network Graphics (PNG)), hypertext markup language (HTML), extensible markup. or a description and / or format including image data within a language (XML) or the like. Additionally, some or all of the links may be secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec). It can be encrypted using conventional encryption techniques such as.

3 is a conceptual diagram illustrating a configuration of an image package 10 according to an embodiment of the present invention. Referring to FIG. 3, the image package 10 may include image data 11 and camera parameters 12. For example, the image package may be one electronic file.

The image data refers to media content content itself for providing to the user. For example, the image data may include at least one of a streaming image and a stereoscopic image (S3D), and the image may be a wide viewing angle image. The image data may be rendered and used to visually present the image to the user. In one embodiment, the image package may further include a camera parameter for posture conversion of the virtual camera. The camera parameter may be a parameter for an actual camera photographing the image data, and may include at least one of a camera lens parameter lens, a camera stabilization parameter, and a stereo camera rig misalignment correction parameter. . For example, the stereo camera rig error correction parameter may be included in a camera parameter when the image data is a stereo image (S3D).

In one example, the camera parameter 12 may function as a kind of parameter for executing a transformation command of the pose of the virtual camera in response to a change in the viewpoint of the user.

4 is a view for explaining the operation of the scene and the virtual camera in a virtual space according to an embodiment of the present invention. As shown in FIG. 4, the scene configuration unit 300 may place a virtual camera in a virtual space and configure a scene for the image data using the image package.

That is, the scene composition unit 300 may texture the image data on the projection geometry 22 projected on the virtual space R. FIG. The image data 11 is encoded when received by the communication interface 200, but may be decoded and textured by the scene composition unit. The textured image is generally provided to the user through the display device. That is, according to the prior art, the position of the virtual camera is fixed but does not change according to the user's motion, and the image is displayed at the fixed time point.

The virtual camera posture converting unit 400 according to an embodiment of the present invention may convert the posture of the arranged virtual camera to correspond to the change of viewpoint of the user by using the motion data and the camera parameter. Since the camera parameter 12 includes a camera lens parameter or a camera stabilization parameter for an actual camera, the virtual camera pose converting unit 400 uses the camera parameter 12 and the motion data to determine the virtual camera in the virtual space. Position and direction can be controlled.

In one example, as shown in FIG. 4, a portion of the image input to the virtual camera for the textured image may be region 23. For example, when the image data 11 includes a wide viewing angle image or depending on the position of the virtual camera, the entire image may not be input to the virtual camera.

In this state, for example, when the user's viewpoint moves by moving his or her head down to the right, the sensor unit 100 detects the user's movement of the viewpoint and the virtual camera pose converting unit 400 performs the motion data and the camera parameters. By using, the posture of the virtual camera in the virtual space may be converted (move by vector M in FIG. 4) in response to the movement of the user. In FIG. 4, the position of the virtual camera is shifted. However, in another embodiment, the position of the virtual camera does not move, and only the direction of the virtual camera may be changed. In another example, the position and orientation may change together.

Referring to FIG. 4, the portion of the image inputted to the virtual camera in the posture 21 becomes the area 30, and the virtual camera converter 400 determines that the virtual camera corresponds to the change of the user's field of view. By changing the posture, the virtual camera moves to the posture 21 ', and one area input to the virtual camera is moved to the area 24. The area input to the virtual camera changes according to the conversion of the camera posture, and the user Moves his viewpoint to track and view the desired part of the image.

In another embodiment, the user-view linked image processing apparatus 1000 may include a projection geometry generator configured to generate projection geometry for outputting a wide viewing angle image using camera parameters of an actual camera that captures an image of the image data. 500) may be further included. The projection geometry generation unit 500 may generate the projection geometry by correcting the vertex or texture coordinates of the reference geometry by using the camera parameter and transforming the corrected reference geometry into a curved surface. As described above, the camera parameter may include at least one of a lens parameter, a camera stabilization parameter, and a stereo camera rig error correction parameter. For example, the stereo camera rig error correction parameter may be included in a camera parameter when the image data is a stereo image (S3D).

5A and 5E are diagrams for describing projection geometry projection and texturing according to a vertex transformation method of projection geometry, according to an exemplary embodiment. In one embodiment, the projection geometry generator 500 may generate the corrected projection geometry by converting the vertex of the reference projection geometry by using the camera lens parameter. 5A, reference projection geometry 31 is shown together for comparison with the generated projection geometry 32. Referring to FIG. 6A, the projection geometry 32 in which the vertices are converted has a shape where the edge is curved and is compressed based on the center. When texturing is performed on the generated projection geometry 32, the textured image appears as shown in FIG. 5B without distortion. For example, straight lines of the subject appear as straight lines in the textured image.

5C and 5D, the scene configuration unit 300 arranges the projection geometry 32 in the virtual space. In this case, by projecting the projection geometry to any spherical surface of the virtual space, the concave curved projection geometry shown in FIG. 5B may be unfolded as shown in FIGS. 5C and 5D. FIG. 65 shows a state of texturing an image captured on the projected projection geometry, and FIG. 5E shows a state of displaying the textured image from the front at the same angle of view as the camera's angle of view when the image is captured.

6A through 6E are diagrams for describing projection and texturing of projection geometry according to a texture coordinate transformation method of projection geometry, according to an exemplary embodiment. The texture coordinates are coordinates in which the image is textured and are independent of the shape of the projection geometry. As shown in FIG. 6A, the projection geometry 42 does not change the shape of the reference projection geometry. However, when the image is textured on the projection geometry 42, as shown in FIG. 6B, the image textured on the projection geometry shows a shape inwardly distorted rather than rectangular.

Referring to FIG. 6C, when the projection geometry 42 having the texture coordinates corrected is disposed on a spherical surface in the virtual space, and the image is rendered to the projection geometry 42 to render, the image is output as shown in FIG. 6D. 6E shows a state in which the textured image is displayed from the front at the same angle of view as the camera angle when the image is captured. Through this process, the wide viewing angle image can be represented on the spherical surface of the virtual space without distortion.

In another embodiment, the scene configuration unit 300 dynamically adjusts the position of the virtual camera in the virtual space by using a camera parameter (eg, a camera lens parameter or a camera stabilization parameter) to render a corrected image. Can be corrected. In one embodiment, the scene configuration unit 300 may correct the scene by converting the projection geometry while the virtual camera is fixed. For example, the projection geometry can be moved in the up, down, left, right, or yaw, pitch, and roll directions.

Alternatively, the scene configuration unit 300 may correct the scene by dynamically adjusting the position of the virtual camera using the camera stabilization parameter. In this case, the adjustment of the position of the virtual camera may occur on a smaller scale than the movement of the viewpoint of the user, and the distortion correction is also aimed at adjusting the virtual camera. There is a difference

In an embodiment, the user-view linked image processing apparatus 1000 further includes an image rendering unit 700 which renders at least a part of the scene viewed by the virtual camera in the converted camera posture in real time and displays it on a display unit. It may include. The image rendering unit 700 serves to allow a user to actually view an image through the display unit.

In one example, the user-view linked image processing apparatus 1000 may further include a display unit 800, and the display unit 800 may include any display device such as an LCD, an OLED, a PDP, and a touch screen. May be

7 is a diagram illustrating a network environment including a viewpoint-linked video providing media server according to an embodiment of the present invention. The viewpoint-linked image providing media server 2000 may provide an image package to another device through a network. As shown in FIG. 8, the other device may include various terminals such as a smart phone 2001, a laptop 2002, a head mounted display (HMD) device 2003, a smart glass, and the like. In another embodiment, the other device may be included in the user-view linked image processing apparatus 1000 described above. The posture of the virtual camera may include a direction or a position of the virtual camera, and the image data may be at least one of a wide viewing angle streaming image and a stereo image S3D.

That is, the viewpoint-linked image providing media server includes an image package storage unit having an image package including image data and camera parameters for posture conversion of a virtual camera, and a communication interface unit transmitting the image package to another terminal. can do. The camera parameter is a parameter for an actual camera that captures an image of the image data, and the detailed description thereof is as described above.

8 is a flowchart of a user view linked image processing method according to an embodiment of the present invention. Referring to FIG. 8, in the user-interview-linked image processing method, a sensor is configured to detect motion of a user's viewpoint and generate motion data (S10). The image includes image data from a media server by a communication interface unit. Receiving a package (S20), using the image package, disposing a scene and a virtual camera for the image data in a virtual space (S30) and using the motion data, the user's viewpoint detected And converting the posture of the arranged virtual camera to correspond to the change (S40). The posture of the virtual camera may include a direction or a position of the virtual camera. That is, in response to the user's viewpoint change, the virtual camera may move or move closer to the textured scene. In addition, the direction of the virtual camera may change in response to a change in the viewpoint of the user.

In an embodiment, the image package may further include a camera parameter for posture conversion of the virtual camera, and the camera parameter may be a parameter for an actual camera that captures an image of the image data. In addition, the image data may include a wide viewing angle streaming image.

In one embodiment, the converting the pose of the arranged virtual camera may include: changing the pose of the virtual camera based on using the motion data and the camera parameter. Can be. The step may be executed by the virtual camera pose converting unit.

The method may further include displaying, on a display unit, at least a part of the scene viewed by the virtual camera in real time in the transformed virtual camera posture.

Although the present invention described above has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and variations may be made therefrom. However, such modifications should be considered to be within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

Claims

A sensor unit configured to generate a motion data by detecting a change in viewpoint of a user;

A communication interface unit for receiving an image package including image data from a media server;

A scene constitution unit for arranging a scene for the image data and arranging a virtual camera in a virtual space using the image package; And

And a virtual camera posture converting unit configured to convert a posture of the arranged virtual camera to correspond to a change in the viewpoint of the user by using the motion data.
The method of claim 1,

And a posture of the virtual camera includes a direction or a position of the virtual camera.
The method of claim 1,

The sensor unit for detecting a change in the viewpoint of the user,

And a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU) sensor, a global positioning system (GPS) sensor, and a vision sensor.
The method of claim 1,

The video package,

And a camera parameter for converting a posture of a virtual camera, wherein the camera parameter is a parameter for an actual camera which captured the image of the image data.
The method of claim 4, wherein

The virtual camera attitude conversion unit,

And a posture of the arranged camera using the motion data and the camera parameter.
The method of claim 1,

The image data includes at least one of a wide viewing angle streaming image and a stereoscopic image (Stereoscopic 3D).
The method of claim 1,

And a projection geometry generator configured to generate projection geometry for outputting a wide viewing angle image using camera parameters of an actual camera photographing the image data.

The projection geometry generating unit corrects the vertex or texture coordinates of the reference geometry by using the camera parameter, and generates the projection geometry by modifying the corrected reference geometry into a curved shape. .
The method according to claim 4 or 7,

The camera parameter is,

And at least one of a camera lens parameter, a camera stabilization parameter, and a stereo camera rig misalignment correction parameter.
The method of claim 1,

The scene configuration unit,

And the camera stabilization parameter to dynamically adjust the position of the virtual camera to correct the scene.
The method of claim 1,

And a video rendering unit which renders at least a part of the scene viewed by the virtual camera in real time in the converted camera posture and displays it on a display unit.
An image package storage unit which holds an image package including image data and camera parameters for posture conversion of a virtual camera; And

And a communication interface configured to transmit the image package to another terminal, wherein the camera parameter is a parameter for an actual camera which captured the image of the image data.
The method of claim 12,

And a posture of the virtual camera includes a direction or a location of the virtual camera.
The method of claim 11,

And the image data comprises at least one of a wide viewing angle streaming image and a stereoscopic 3D image server.
The method of claim 11,

The camera parameter is,

And at least one of a camera lens parameter, a camera stabilization parameter, and a stereo camera rig misalignment correction parameter.
Generating, by the sensor, motion data by detecting a viewpoint change of a user;

Receiving, by the communication interface unit, an image package including image data from the media server;

Arranging a scene and a virtual camera for the image data in a virtual space using the image package; And

And converting a posture of the arranged virtual camera to correspond to a change in the viewpoint of the user by using the motion data.
The method of claim 15,

The posture of the virtual camera includes a direction or a position of the virtual camera.
The method of claim 15,

The video package,

And a camera parameter for converting a posture of a virtual camera, wherein the camera parameter is a parameter for an actual camera which captured the image of the image data.
The method of claim 17,

The converting the posture of the arranged virtual camera may include changing the posture of the virtual camera by using the motion data and the camera parameter.
The method of claim 15,

The image data includes at least one of a wide viewing angle streaming image and a stereoscopic image (Stereoscopic 3D).
The method of claim 15,

And real-time rendering at least a portion of the scene viewed by the virtual camera in the converted virtual camera posture and displaying the same on a display unit.