KR102094772B1 - Method and server for controlling video - Google Patents

Abstract

The present invention is an image control method, comprising: (a) obtaining location information of a user located in a real space; (b) matching the location of the user to a virtual space created to correspond to the real space based on the location information; (c) generating a virtual image for the virtual space by displaying a virtual object associated with each user at the location of the user matched with the virtual space; And (d) providing the generated virtual image to the image output device.

Description

METHOD AND SERVER FOR CONTROLLING VIDEO

The present invention relates to an image control technology, and relates to an image control method and a server that synchronizes the images output between different types of virtual reality devices and reflects the location of the real space in which the user wearing the virtual reality device is actually located. will be.

In the case of devices that provide virtual reality (VR) images, it is very difficult to share the same virtual space between different models, and devices that provide virtual reality images such as a head mounted display (HMD) There is a problem in that the location of the users wearing in the real space and the location in the virtual reality image are not synchronized, and thus a dangerous situation such as collision between users occurs. Therefore, when a plurality of people experience virtual reality content in the same space, it is necessary to synchronize the virtual reality images viewed by a plurality of people and to ensure that the experiences in the virtual reality images are actually reflected.

Korean Patent Publication No. 10-2012-0013021

The object of the present invention is to acquire the location of each user based on data obtained from a laser, microphone, or image capture device disposed in a real space, reflect it in the virtual image of the virtual space, and wear the corresponding virtual image by the user It is to provide an image control method and a server to provide a head mounted display or a projector installed in a real space, so that both a user wearing a head mounted display and a user not wearing the head can experience the same virtual image.

A first aspect of the present invention for achieving the above object is an image control method comprising: (a) obtaining location information of a user located in a real space; (b) matching the location of the user to a virtual space created to correspond to the real space based on the location information; (c) generating a virtual image for the virtual space by displaying a virtual object associated with each user at the location of the user matched with the virtual space; And (d) providing the generated virtual image to the image output device.

Preferably, the step (a) may include obtaining information about the location of the user from a laser device, a microphone device, or an imaging device disposed in the real space, and calculating position coordinates for each user. You can.

Preferably, the step (b) may include determining a location on each user's virtual space by matching the location coordinates for each user with the location coordinates of the virtual space.

Preferably, in step (d), if the video output device corresponds to a head mounted display worn by at least one of the users, depending on the location and viewpoint of the user wearing the head mounted display, And, it may further include the step of adjusting the viewpoint of the virtual image to be output through the head-mounted display.

Preferably, in step (d), when the image output device corresponds to a projector for outputting a virtual image to the real space, the viewpoint of the virtual image to be output through the projector is adjusted according to a preset viewpoint direction. The method may further include, but the projector may output a virtual image on the front, bottom, left, and right sides of the real space.

A second aspect of the present invention for achieving the above object is an image control server, a location information acquisition unit for acquiring location information of a user located in a real space; A location matching unit matching the user's location to a virtual space generated to correspond to the real space based on the location information; An image generation unit that displays a virtual object associated with each user at a location of the user matched with the virtual space to generate a virtual image for the virtual space; And an image providing unit that provides the generated virtual image to the image output device.

Preferably, the location information acquiring unit may obtain information about the location of the user from a laser device, a microphone device, or an imaging device disposed in the real space, and calculate position coordinates for each user.

Preferably, the location matching unit may determine a location on each user's virtual space by matching the location coordinates for each user with the location coordinates of the virtual space.

Preferably, it may further include an image viewpoint adjustment unit for adjusting the viewpoint of the virtual image to be provided to the image output device.

Preferably, when the image viewing unit adjusts the video output device to a head mounted display worn by at least one of the users, according to the position and viewpoint of the user wearing the head mounted display, The viewpoint of the virtual image to be output may be adjusted through the head mounted display.

Preferably, the image viewpoint adjusting unit adjusts the viewpoint of the virtual image to be output through the projector according to a preset viewpoint direction when the image output device corresponds to a projector for outputting a virtual image to the real space. , The projector may output virtual images on the front, bottom, left, and right sides of the real space.

As described above, according to the present invention, since the same virtual image can be shared regardless of the type of the device outputting the virtual image, and the user's position in the virtual space is synchronized with the real space, multiple users in the same space When they perform an experience using virtual images, there is an effect capable of interaction between users and interaction with a real space.

1 is a block diagram of an image control system according to a preferred embodiment of the present invention.
2 is a block diagram of an image control server according to an embodiment.
3 is a flowchart illustrating an image control method according to an embodiment.

Hereinafter, advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and the ordinary knowledge in the technical field to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification. “And / or” includes each and every combination of one or more of the items mentioned.

Although the first, second, etc. are used to describe various elements, components and / or sections, it goes without saying that these elements, components and / or sections are not limited by these terms. These terms are only used to distinguish one element, component or section from another element, component or section. Accordingly, it goes without saying that the first element, the first component or the first section mentioned below may be the second element, the second component or the second section within the technical spirit of the present invention.

In addition, in each step, an identification code (for example, a, b, c, etc.) is used for convenience of explanation, and the identification code does not describe the order of each step, and each step is clearly specified in context. Unless the order is specified, it may occur differently from the order specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, “comprises” and / or “comprising” refers to the components, steps, operations and / or elements mentioned above, the presence of one or more other components, steps, operations and / or elements. Or do not exclude additions.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings commonly understood by those skilled in the art to which the present invention pertains. In addition, terms defined in the commonly used dictionary are not ideally or excessively interpreted unless specifically defined.

In addition, in describing embodiments of the present invention, when it is determined that a detailed description of known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to a user's or operator's intention or practice. Therefore, the definition should be made based on the contents throughout this specification.

1 is a block diagram of an image control system according to a preferred embodiment of the present invention.

Referring to FIG. 1, the image control system 100 includes an image output device 110, an image control server 120, a laser device 130, a microphone device 140, and an image photographing device 150.

The image output device 110 is a device that outputs a virtual image that is a virtual reality image for a virtual space. Preferably, the image output device 110 may be a device directly worn by a user, for example, a head mounted display (HMD), a device capable of outputting a virtual image, such as a projector, a PC, It may be a mobile terminal. Here, the image output device 110 is located in a limited specific real space, and there is no limit to the number.

The image control server 120 is a device that generates and provides a virtual image. Preferably, the image control server 120 is connected to each image output device 110, a laser device 130, a microphone device 140, and an image photographing device 150 to locate users located in the same real space. The reflected virtual image may be generated and the generated virtual image may be provided to each image output device 110.

Each of the laser device 130, the microphone device 140, and the image photographing device 150 is installed in the real space and acquires information for determining the location of each user located in the real space, thereby controlling the image control server 120. To provide. Preferably, at least one of the laser device 130, the microphone device 140, and the image photographing device 150 may be disposed in the real space, and the number of devices disposed in the real space is limited. none.

2 is a block diagram of an image control server according to an embodiment.

Referring to FIG. 2, the image control server 120 includes a location information acquisition unit 210, a location matching unit 220, an image generation unit 230, an image providing unit 240, and a control unit 250. . The control unit 250 controls the operation and data flow of the location information acquisition unit 210, the location matching unit 220, the image generation unit 230, and the image providing unit 240. Hereinafter, an image control method performed in the image control server 120 will be described in more detail with reference to FIG. 3.

In the image control method according to the present invention, a laser device 130, a microphone device 140, or an image photographing device 150 is installed in a specific real space that is a limited space, and a virtual image is projected on a wall surface of the real space. An image output device (for example, a projector) for installation is installed, and a user who wears an image output device (for example, an HMD) for outputting lyrics and a user who does not wear is located in the corresponding real space. It is performed in the environment.

First, the image control server 120 constructs a virtual space corresponding to a specific real space, generates a virtual image for the virtual space, and provides it to the image output device 110.

Referring to FIG. 3, the location information acquisition unit 210 acquires location information of a user located in a real space (step S310). Preferably, the location information acquisition unit 210 is information about the distance and azimuth from the laser device 130 or the microphone device 140 disposed in the real space to each user, or the depth value from the image taking device 150, And acquiring information about computer vision data in an image to determine the number and location of users located in the real space.

More specifically, when the signal generated from the signal source of each of the laser device 130, the microphone device 140, or the image photographing device 150 is reflected from the user corresponding to the target, the location information acquisition unit 210 Based on the phase change value of the reflected signal, the distance between the laser device 130, the microphone device 140, or the image photographing device 150 and the user generating the signal is measured. Here, a TOF (Time of Flight) method may be used. Next, the location information acquisition unit 210 is a triangulation based on the azimuth angle at the position where the laser device 130, the microphone device 140, or the image photographing device 150 where the measured distance and signal are generated ( triangulation) to determine the user's location.

Alternatively, the location information acquiring unit 210 acquires a depth value from images acquired from a plurality of image photographing devices 150 respectively located in the XYZ plane of the real space, and acquires RGB values of still images. . Here, the image photographing apparatus 150 may correspond to a depth camera. Next, the location information acquisition unit 210 may determine the user's location based on the depth value, the RGB value, the reference point in the preset screen, and the angle of view of the image photographing apparatus 150. That is, the location where the laser device 130, the microphone device 140, or the image photographing device 150 is installed is stored in advance, and the location information obtaining unit 210 includes the laser device 130, the microphone device ( 140) Alternatively, the location of the image photographing device 150 and the distance from each device to the user can be used to determine the location of the user in a specific real space.

Here, it has been described that the location information obtaining unit 210 determines the user's location by receiving information obtained from the laser device 130, the microphone device 140, or the image photographing device 150. 130, the user's location may be determined directly from the microphone device 140, or the image photographing device 150, and in this case, the location information acquisition unit 210 may include a laser device 130, a microphone device 140, or The determined user's location may be provided from the image photographing device 150. In addition, the user's location is determined based on the information obtained from each of the laser device 130, the microphone device 140, or the imaging device 150, or the information obtained from each of them is combined or combined to obtain the user's The location can be determined, and the method of determining the user's location is not limited to this.

Preferably, the location information acquisition unit 210 may finally determine the user's location and number of people by filtering information about the user's location by distance based on a specific wall surface of the real space based on the user's location information. .

The location matching unit 220 matches the user's location to the virtual space generated to correspond to the real space based on the location information determined through the location information obtaining unit 210 (step S320). Preferably, the location matching unit 220 may determine the location on each user's virtual space by matching the location coordinates for each user with the location coordinates of the virtual space.

In one embodiment, the image control server 120 may include a motion information acquisition unit (not shown in the drawing), and the motion information acquisition unit acquires information about a user's motion located in a real space. Preferably, the head mounted display corresponding to the image output device 110 that the user wears may include a motion recognition device that identifies a wearer's motion, and the motion information acquisition unit wears the head mounted display from the motion recognition device It is possible to obtain information about the operation of the user. In addition, the motion information acquisition unit estimates the motion of a user interacting with the wearer (a user wearing an HMD and viewing an image output through the projector) based on the position and azimuth of the user wearing the head mounted display. You can. In addition, when the image photographing apparatus 150 is a 3D camera, since 3D scanning of a specific real space through the image photographing apparatus 150 is possible, the operation information acquisition unit receives 3D scanning information photographed from the image photographing apparatus 150. On the basis of this, it is possible to acquire motion information of a user located in a real space, and when the laser device 130 is installed in more than a certain number in the real space, scanning of an area unit through the laser device 130 is possible, thereby obtaining motion information The unit may acquire motion information of a user located in a specific real space based on the scan information of the area unit obtained from the laser device 130.

The image generation unit 230 generates a virtual image for the virtual space by displaying the virtual objects associated with each user at the location of the user matched with the virtual space (step S330). Preferably, the image generator 230 synchronizes and displays the location of each user in the real space and the location in the virtual image, and synchronizes the motion of each user and the motion of the virtual object acquired from the motion information acquisition unit Can be displayed.

The image providing unit 240 provides the generated virtual image to the image output device (step S340). Preferably, before providing the virtual image through the image providing unit 240, the image viewpoint adjusting unit (not shown in the drawing) may adjust the viewpoint of the virtual image to be provided to the image output device 110. That is, since the virtual image generated based on the location coordinates of each user is provided after the viewpoint is adjusted according to the image output device 110 to be provided, the virtual image and the state of the real space can be more accurately synchronized.

In one embodiment, when the image output device 110 corresponds to a head mounted display (HMD) worn by at least one of the users located in the real space, the image viewpoint adjusting unit is a user wearing the HMD According to the position and the direction of the view (direction that the user is looking at), it is possible to adjust the viewpoint of the virtual image to be output through the HMD. Preferably, the HMD may include an inertial measurement device (IMU), and the image viewpoint adjustment unit may determine the viewpoint direction by receiving information about the direction, speed, and acceleration of the HMD from the inertial measurement device. In another embodiment, when the image output device 110 corresponds to a projector installed in a real space, the image viewpoint adjusting unit may adjust a viewpoint of a virtual image to be output through the projector according to a preset position and a viewpoint direction. Here, the virtual image output through the projector can be projected on the front, bottom, left and right walls of the real space.

Preferably, steps S310 to S340 described above may be repeatedly performed. That is, the location information acquisition unit 210 continuously maintains the number and location of users located in the real space according to the information continuously acquired through the laser device 130, the microphone device 140, or the image photographing device 150. , And the location matching unit 220 matches the updated person and location on the virtual space, and the image generation unit 230 displays the virtual object on the virtual image according to the number and location of the user matched on the virtual space. A virtual image is generated by adjusting the number and location, and the image providing unit 240 is capable of providing the virtual image whose viewpoint is adjusted through the image viewpoint adjusting unit to each image output device 110.

In addition, data and images obtained according to the repetitive performance of steps S310 to S340 may be stored, and according to the present invention, as the stored data and images are provided, new users are reflected in the past virtual reality to experience a similar experience. I can do it. Here, the method described above can be applied to the method in which a new user is reflected in the past virtual reality. For example, in the case of history class content, students and teachers can participate and experience in the space and data that have been built in advance, and in the case of content such as concert video, new users can participate in the saved concert video to experience the scene. Will be.

Therefore, according to the present invention, the location of each user in the real space is reflected in the virtual image, and the user viewing the virtual image output from the HMD and the user viewing the virtual image output from the projector are all users located in the corresponding real space. Since the location can be confirmed within the virtual image, multiple users can interact with each other and experience various experiences while viewing the same virtual image in a specific real space.

Meanwhile, the image control method according to an embodiment of the present invention may also be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

For example, computer-readable recording media include ROM, RAM, CD-ROM, magnetic tapes, hard disks, floppy disks, removable storage devices, and non-volatile memory (Flash Memory). , Optical data storage, and the like.

In addition, the computer-readable recording medium may be distributed over computer systems connected by a computer communication network, and stored and executed as code readable in a distributed manner.

Although the preferred embodiments of the image control method and the server according to the present invention described above have been described, the present invention is not limited to this, and various modifications are made within the scope of the claims and detailed description of the invention and the accompanying drawings. Possible and this also belongs to the present invention.

100: video control system
110: video output device
120: video control server
130: laser device
140: microphone device
150: video recording apparatus
210: location information acquisition unit
220: position matching unit
230: image generation unit
240: video providing unit

Claims (12)

(a) obtaining location information of at least one user wearing a head mounted display (HMD) located in the real space and a user without the HMD for each distance based on a specific wall surface of the real space;
(b) determining a location on each user's virtual space by matching the location coordinates for each user with the location coordinates of the virtual space;
(c) generating a virtual image for the virtual space by displaying a virtual object associated with each user at a location of the user matched with the virtual space, and adjusting a viewpoint of the virtual image to be provided to the image output device; And
(d) providing the generated virtual image to the HMD and a projector for outputting the virtual image to the real space, wherein the image control method comprises:
Step (c) is,
In the HMD, a viewpoint of a virtual image to be output through the HMD is adjusted according to a position and a viewpoint direction of a user wearing the HMD,
In the projector, the location information acquiring unit is a step of adjusting a viewpoint of a virtual image to be displayed on the front, bottom, left, and right sides of the real space at a specific wall position in the real space for acquiring the user's location. Characterized image control method.
The method of claim 1, wherein step (a) is
And obtaining information on the location of the user from a laser device, a microphone device, or an image capturing device disposed in the real space, and calculating position coordinates for each user.
delete delete delete A location information acquiring unit for acquiring location information of at least one user wearing a head mounted display (HMD) located in a real space and a user not wearing an HMD for each distance based on a specific wall surface of the real space;
A location matching unit that matches a location coordinate for each user with a location coordinate in a virtual space to determine a location on each user's virtual space;
An image generation unit that displays a virtual object associated with each user at a location of the user matched with the virtual space to generate a virtual image for the virtual space;
An image viewpoint adjusting unit that adjusts a viewpoint of a virtual image to be provided to the image output device; And
An image control server including an image providing unit that provides the generated virtual image to the HMD and a projector for outputting the virtual image to the real space,
The video viewpoint adjustment unit,
In the HMD, a viewpoint of a virtual image to be output through the HMD is adjusted according to a position and a viewpoint direction of a user wearing the HMD,
In the projector, the location information acquisition unit adjusts a viewpoint of a virtual image to be displayed on the front, bottom, left, and right sides of the real space at a specific wall position in the real space for acquiring the user's location. Video control server.
The method of claim 6, wherein the location information acquisition unit
Image control server, characterized in that for obtaining the information about the location of the user from a laser device, a microphone device, or an image capture device disposed in the real space, to calculate the position coordinates for each user.
delete delete delete delete A computer-readable recording medium recording a program capable of executing the method of claim 1 or 2 with a computer.

Images