KR101961841B1 - Image apparatus and operating method of image apparatus - Google Patents

Abstract

The present invention relates to an image apparatus capable of realizing wireless synchronization playback in various environments, and an operation method thereof. According to the present invention, the method for operating an image apparatus comprises: a step of determining whether a display apparatus is connected; a step of acquiring a group key corresponding to an apparatus group including the image apparatus and one or more other image apparatuses when it is determined that the display apparatus is connected; a step of providing address information for playing content acquired when connecting to a server with the group key to the other image apparatuses included in the apparatus group; a step of transmitting a playback command for the content to the other image apparatuses based on whether each of the other image apparatuses completes the content based on the address information; and a step of generating and providing image information for the display apparatus based on calibration information for playing the content or playing at least a part of the content in an area corresponding to the display apparatus from the entire area for playing the content.

Description

IMAGE APPARATUS AND OPERATING METHOD OF IMAGE APPARATUS [0001]

The following embodiments relate to a method of operating a video device and a video device.

It is possible to provide a large screen image using a plurality of display devices such as multi-screen devices or multi-projectors in a movie theater, an exhibition hall or a museum. When a plurality of display devices are used to provide a single large-screen image, it is not easy to arrange the screen by each of the plurality of display devices, determine the content to be reproduced, and manage the reproduction time of the content. In addition, it is also possible to perform calibration for setting an area of contents to be reproduced by each of the plurality of display devices, operation for causing a larger-sized image provided by the plurality of display devices to be displayed more naturally, A complicated operation for synchronizing images reproduced by the user is required.

A method of operating a video device according to one side comprises: determining whether the video device is connected to a display device; Obtaining a group key corresponding to a device group including the video device and one or more other video devices according to a determination that the video device is connected to the display device; Acquiring address information for content reproduction by accessing a server using the group key; Providing the address information to the other imaging devices included in the device group; Transmitting a playback command for the content to the other video apparatuses based on whether or not loading of content based on the address information is completed in each of the other video apparatuses; Generating image information for the display device based on calibration information for reproducing at least a part of the content in an area corresponding to the display device among the entire area for reproducing the content and the content step; And providing the image information to the display device.

The address information may include a playlist including information on contents to be reproduced in each of the plurality of video apparatuses included in the device group, and a content reproduction order in each of the plurality of video apparatuses.

Whether the loading of the content has been completed may be determined based on whether the first remaining time for completing the loading of the content is shorter than the second remaining time for completing the reproduction of the content.

The method of claim 1, further comprising: determining whether initialization is required when power is supplied from the display device; Operating as a hot spot host to obtain information for connection from the mobile device according to the determination that the initialization is necessary; Accessing the access point using the information for the connection and receiving the group key; And storing the group key.

The method of operating the imaging device may further include determining whether the imaging device is a master imaging device determined by the mobile device.

The method comprising: determining whether the video device stores the group key; And switching the operation mode of the video apparatus to the master mode according to the determination that the group key is stored.

Wherein the step of generating image information for the display device includes synchronizing the plurality of image devices, which are determined by inputting a time offset between a plurality of image devices included in the device group into a Random Sample Consensus (RANSAC) And generating image information for the display device based on the synchronization information.

Wherein the display device corresponding to the video apparatus among the plurality of video apparatuses included in the device group includes the calibration information for setting at least a partial area to be reproduced from the entire area for reproduction of the content And a step of deciding whether or not to perform a search.

The content may include at least one of multimedia content, multi-page content, moving image content, game content, and text.

According to one aspect, the imaging apparatus includes a sensing unit that senses whether or not the imaging device is connected to the display device; Acquiring a group key corresponding to a device group including the video apparatus and one or more other video apparatuses according to a determination that the video apparatus is connected to the display apparatus, Providing address information for the other video devices to the other video devices included in the device group based on whether or not the loading of content based on the address information has been completed in each of the other video devices, Based on the content and the calibration information for reproducing at least a part of the content in an area corresponding to the display device among the entire area for reproduction of the content, A processor for generating information; And a communication interface for providing the image information to the display device.

Wherein the processor determines whether initialization is required as power is supplied from the display device and operates as a hot spot host to acquire information for connection from the mobile device according to the determination that the initialization is necessary, And can receive and store the group key.

Wherein the process determines whether the video apparatus is the master video apparatus determined by the mobile device based on whether the group key is stored or not, . ≪ / RTI >

The processor may generate image information for the display device based on synchronization information for synchronizing the plurality of imaging devices, which is determined by inputting a time offset between a plurality of imaging devices included in the device group to the RANSAC model have.

The processor may determine the calibration information for setting at least some areas of a plurality of video apparatuses included in the apparatus group that the display apparatus corresponding to the video apparatus should reproduce from the entire area for playing the content.

The video apparatus may further include a memory for storing the loaded content based on the address information.

The video device according to one side includes a communication interface for receiving address information for content reproduction from the master video device of the device group; And a control unit for loading the content for the device group based on the address information, determining whether the loading of the content is completed, determining whether the loading of the content is completed, Based on calibration information for reproducing at least a part of the content in a region corresponding to the content and a display device connected to the video apparatus among all regions for reproduction of the content according to a reproduction command for the content, Lt; RTI ID = 0.0 > a < / RTI >

The communication interface may provide the image information to the display device.

The processor determines whether the video apparatus is connected to the display apparatus, checks whether the video apparatus is connected to the display apparatus, determines whether the group key is stored or not, , The operation mode of the video apparatus can be switched to the slave mode.

Wherein the processor generates synchronization information for synchronizing the master imaging device and the imaging device by inputting a time offset between a plurality of imaging devices included in the device group to the RANSAC model, It is possible to generate image information for the device.

The processor may determine the calibration information for setting at least some areas of a plurality of video apparatuses included in the apparatus group that the display apparatus corresponding to the video apparatus should reproduce from the entire area for playing the content.

1 is a diagram for explaining a configuration of a system in which an operation method of a video apparatus according to an embodiment is performed;
2 is a flowchart illustrating an operation method of a video apparatus according to an exemplary embodiment;
3 is a flowchart illustrating a method of initializing a video device according to an exemplary embodiment.
4 is a flowchart illustrating an operation method of a video apparatus according to another embodiment;
5 is a flow diagram illustrating a method of operation of a mobile device in accordance with one embodiment.
FIG. 6 is a diagram for explaining a video reproducing method of a video apparatus according to an embodiment; FIG.
7 is a flowchart illustrating a method of reproducing an image of a video apparatus according to an exemplary embodiment.
8 is a view for explaining a method of generating a mask according to an embodiment;
9-10 illustrate a method for determining a reference texture in accordance with one embodiment.
11 is a flowchart illustrating a method of synchronizing a video apparatus according to an exemplary embodiment.
12 is a view for explaining time information exchanged between a slave image device and a master image device according to an exemplary embodiment;
13 is a flow diagram illustrating a method for generating synchronization information in accordance with one embodiment.
14 is a flow diagram illustrating a method for an outlier to obtain filtered sampling data in accordance with one embodiment;
15 is a flow diagram illustrating a method for determining synchronization information in accordance with one embodiment.
FIG. 16 is a diagram illustrating a pseudo code illustrating a process of applying time information obtained through a Simple Network Time Protocol (SNTP) packet to a RANSAC model according to an exemplary embodiment; FIG.
17 is a block diagram of a video device according to one embodiment.

It is to be understood that the specific structural or functional descriptions disclosed herein may be implemented by way of example only, and that the embodiments may be embodied in various other forms and are not limited to the embodiments described herein It does not.

The terms first or second may be used to describe various components, but such terms should be understood only for the purpose of distinguishing one component from another. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Expressions that describe the relationship between components, such as "between" and "between" or "neighboring to" and "directly adjacent to" should be interpreted as well.

The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises ", or" having ", and the like, are used to specify one or more of the features, numbers, steps, operations, elements, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

FIG. 1 is a diagram for explaining a configuration of a system in which an operation method of a video apparatus according to an embodiment is performed. 1, a system 100 according to one embodiment includes a server 110, a mobile device 130, display devices 151, 153, 155, 157, and video devices 171, 173, 175 , 177).

The server 110 registers the video devices 171, 173, 175, and 177 and can manage and / or control the video devices 171, 173, 175, and 177 through the web. The server 110 can manage information such as which user's video devices 171, 173, 175, and 177 belong to, which group belongs, what content and when and where to play it.

The server 110 may be a web server. Or the server 110 may be, for example, a single server computer or similar system, or a plurality of servers arranged in one or more server banks or other arrangements. The server 110 may be located in a single facility, or it may be a server "cloud " distributed across many different geographic locations.

The mobile device 130 may control the video devices 171, 173, 175, and 177 in cooperation with the video devices 171, 173, 175, and 177. For example, the mobile device 130 may change the settings of at least one of the imaging devices 171, 173, 175, and 177, or may control the operation.

The mobile device 130 may display the time and / or the time to play content and / or content to be played on the display devices 151, 153, 155, and 157 via the video devices 171, 135, respectively. The content may include, for example, multimedia content, multi-page content, video content, streaming content, game content, and text.

The mobile device 130 controls each of the display devices 151, 153, 155, and 157 through the video devices 171, 173, 175, and 177 so that the display devices 151, 153, 155, ) Can be turned on or off. The display devices 151, 153, 155, 157 may be, for example, multi-screen devices or multi-projectors.

The imaging devices 171, 173, 175, and 177 may be implemented in the form of a small stick device, for example. The video devices 171, 173, 175 and 177 are connected to an HDMI (High Definition Multimedia Interface) port or a USB (Universal Serial Bus) port of each of the display devices 151, 153, 155 and 157, . Each of the video apparatuses 171, 173, 175, and 177 can be operated by receiving power from each of the display apparatuses 151, 153, 155, and 157.

The video devices 171, 173, 175, and 177 may communicate with each other through a wireless network, such as Wi-Fi, IEEE 802.11 wireless local area network (WLAN)

The video apparatuses 171, 173, 175, and 177 synchronize not only the video file but also a video stream and a game content through a streaming service such as YouTube to display apparatuses 151, 153, 155, and 157 Can be reproduced through.

The user 135 who wants to use the video devices 171, 173, 175 and 177 creates his or her account on the web provided by the server 110, 173, 175, and 177 may be connected to (or coupled to) the display devices 151, 153, 155, and 157 to proceed with the device registration procedure.

The user 135 may create a device group for playing back one large screen by selecting the video devices 171, 173, 175, and 177, for example, after logging into the mobile app.

171, 173, 175, and 177 are connected to the display devices 151, 153, 155, and 157 to supply power to the video devices (s) 171, 173, (E.g., access point (AP) information) from the mobile device 130, if the initialization is necessary.

Each of the video device (s) 171, 173, 175, and 177 may be assigned a device identifier. The device identifier may be broadcast into a device group via UDP (User Datagram Protocol). For example, the HostAP module included in each of the video devices 171, 173, 175, and 177 automatically operates to provide a Wi-Fi SSID (Service Set Identifier) service unique to the video devices 171, 173, Set identifier). The SSID is a unique identifier that exists in the header of every packet transmitted through the wireless LAN.

The mobile device 130 may connect to each imaging device (s) 171, 173, 175, 177 hosted with a unique SSID. The mobile device 130 may communicate the address of the access point (AP) to access the imaging device (s) 171, 173, 175, The video device (s) 171, 173, 175, 177 connect to the access point and the mobile device 130 can also connect to the same access point. The user 135 can view the list of video devices connected to the common access point through the mobile device 130 and can select the video devices included in the device group. The mobile device 130 can automatically complete the registration procedure by sending the information about the set device group to the server 110 automatically.

In addition, the user 135 can manage content through a web or mobile application, and can configure and / or manipulate playback conditions of the content. The user 135 is connected to the video devices 171, 173, 175, 177 and / or the display devices 151, 153, 155, 157 via the web page or mobile device 130 provided through the server 110. [ Can be controlled. The user 135 can turn off the power of the video apparatuses 171, 173, 175, 177 and / or the display apparatuses 151, 153, 155, 157 when the reproduction of the content is completed. 173, 175, and 177 and / or display devices 151, 153, 155, and 157 when the user 135 wants to reproduce the content again.

The mobile device 130 may set any of the imaging device (s) 171, 173, 175, and 177 to a master imaging device (e.g., imaging device 171). The mobile device 130 may generate a group key corresponding to the device group and then transmit the group key to the master video device 171. [ The mobile device 130 may transmit to the server 110 information including video devices included in the device group, information on the master video device 171, and group key. At this time, the remaining video devices other than the master video device 171 among the video devices included in the device group may correspond to the slave video devices 173, 175, and 177.

When each of the master video apparatus 171 and the slave video apparatuses 173, 175 and 177 is connected to the display apparatus, it is checked whether initialization (for example, device group setting, etc.) Or to operate in the slave mode. For example, each of the video apparatuses can confirm whether or not the group key is stored. If the group key is stored, the video apparatuses can operate in the master mode, and if the group key is not stored, to operate in the slave mode.

The master imaging device 171 may try to log into the server 110 using the group key and the server 110 may validate the group key. When the verification of the group key is completed, the server 110 can control the master imaging device 171 through a process of log-in to the master imaging device 171. The server 110 may provide the master video device 171 with address information for content reproduction.

The master video apparatus 171 acquiring address information for content reproduction from the server 110 may provide address information for content reproduction to the slave video apparatuses 173, 175, and 177. The slave video devices 173, 175, and 177 receiving the address information for content reproduction from the server 110 may load the content based on the address information. When the loading of the content is completed, the slave video devices 173, 175, and 177 can transmit a completion signal to the master video device 171. [ The master video apparatus 171 can transmit a content reproduction command to the slave video apparatuses 173, 175, and 177 upon receiving the completion signal from the slave video apparatuses 173, 175, and 177.

The operation of the master imaging device, the slave imaging device, and the mobile device according to one embodiment will be described in more detail with reference to FIGS. 2 to 5 below.

When the device registration procedure described above is completed for all the video devices 171, 173, 175, and 177 included in the device group, each of the video devices 171, 173, 175, and 177 accesses the entire large screen ) Of the video device to be reproduced can be set. Thereafter, when the reproduction command is transmitted from the master video apparatus 171, the slave video apparatuses 173, 175, and 177 can reproduce a partial area of the content according to their respective calibration information.

The imaging device according to one embodiment can provide an easy calibration method with respect to the configuration of an arbitrary display device. The video device is interlocked with the user interface (UI) of the mobile device 130 by Wi-Fi, for example, so that the user 135 can directly view the screen of the mobile device 130, , 153, 155, and 157 can set at least some areas to be reproduced.

When the calibration is completed in all the video devices 171, 173, 175, and 177 included in the device group, each of the video devices 171, 173, 175, and 177 can store its own calibration information. Upon receiving the content reproduction command, the video apparatuses 171, 173, 175, and 177 generate video information using the stored calibration information and transmit the generated video information to the connected display apparatus.

In accordance with an embodiment, each calibration information may be transmitted to the server 110. In this case, the server 110 manages the geometry information of each of the display devices 151, 153, 155, and 157 combined with the respective video devices 171, 173, 175, and 177 to provide a single large screen , And the display devices 151, 153, 155, and 157, respectively.

The video apparatuses 171, 173, 175 and 177 receive the entire contents through the display apparatuses 151, 153, 155 and 157 corresponding to the video apparatuses 171, 173, 175 and 177 on the basis of the calibration information. Area (e.g., a large screen) can be played. At this time, each of the display devices 151, 153, 155, and 157 may reproduce at least a part of the entire area of the content.

Once the calibration information for each of the imaging devices 171, 173, 175, and 177 is determined, the user 135 may need to repeat the setting or adjustment process for each of the imaging devices 171, 173, 175, The user can perform operations for reproducing the content through the mobile device 130 or the web page.

The user 135 can upload a desired content through a web page provided through the server 110 or a mobile application provided through the mobile device 130, and can construct a play list to reproduce a large screen image. At this time, the uploaded contents can be managed by the contents management system (CMS) of the server 110 or the mobile device 130.

For example, if the user 135 creates a play schedule on the web and sets up a desired play list, the play list can be stored in the server 110. When the playback command is transmitted from the server 110 in accordance with the start time of a screening place such as an auditorium or an exhibition hall, each of the video devices 171, 173, 175, and 177 communicates with the server 110, 153, 155, and 157 according to the calibration information for each of the video devices 171, 173, 175, and 177. In addition, the display device 151, 153, 155, The method by which the imaging apparatus according to one embodiment performs the calibration will be described in detail with reference to Figs. 6 to 10 below.

In addition, the video apparatus according to an exemplary embodiment can support detailed time synchronization between the video apparatuses 171, 173, 175, and 177 for playing back content using a simple protocol.

For example, after the user 135 connects multiple video devices 171, 173, 175, and 177 on the wireless network, the video and audio streams are synchronized in the video devices 171, 173, 175, . In one embodiment, video devices may be synchronized using a Simple Network Time Protocol (SNTP) protocol that includes a time stamp. At this time, the time synchronization of each of the video apparatuses 171, 173, 175, and 177 is required for the time stamp to be valid in each video apparatus.

In general, NTP (Network Time Protocol) can be used to synchronize devices connected on the network. NTP is a robust protocol that can be applied in a variety of environments, but it is difficult to use in certain situations. Especially, it is difficult to utilize the whole NTP function because of factors such as development environment and OS (Operating System) in IoT (Internet of Things) device or mobile device which are widely used recently.

SNTP is simpler in operation than NTP and can be easily implemented in any environment where it can be programmed. However, if you are connected to a wireless network and you depend on SNTP for synchronization, the delay and time error can change significantly every hour. Therefore, when SNTP is synchronized in a wireless network environment, the fluctuation of the synchronized result may be larger than when the wired LAN is connected. Such a large fluctuation can be a big problem when reproducing contents such as video synchronously between devices. For example, since it takes several tens of milliseconds (ms) per frame, the time synchronization error between the devices must be less than several milliseconds (ms) so that the content can be properly reproduced without synchronization errors.

In one embodiment, synchronization errors can be minimized by inputting a time offset obtained through SNTP packet transmission in a wireless network to a Random Sample Consensus (RANSAC) model. The RANSAC model may correspond to a model that performs regression analysis on a data set (e.g., an inlier) that belongs to a normal distribution.

A method of synchronizing a video apparatus according to an exemplary embodiment will be described in detail with reference to FIGS. 11 to 16 below.

2 is a flowchart illustrating an operation method of a video apparatus according to an exemplary embodiment of the present invention. Referring to FIG. 2, a video device (e.g., a master video device) according to an embodiment determines whether it is possible to acquire a group key corresponding to a device group (210). For example, the imaging apparatus can check whether a group key is stored in a storage space such as a memory or confirm flag information indicating that a group key is stored.

If it is determined in step 210 that acquisition of the group key is not possible, the imaging apparatus can operate in the initialization mode or the slave mode. For example, the imaging device may operate in accordance with the flowchart of FIG. 3 or FIG.

If it is determined in step 210 that acquisition of the group key is possible, the imaging device acquires a group key corresponding to the imaging device and the device group including one or more other imaging devices (220). For example, the video apparatus can check whether a group key is stored in a cache or memory of the video apparatus, and obtain a group key. At this time, the video device storing the group key may correspond to the master video device of the device group. The video device can determine whether the video device is the master video device determined by the mobile device by whether the group key is acquired.

The imaging device may attempt to log in to the server using the group key based on whether or not the group key is stored at boot time.

According to the embodiment, the video apparatus can determine whether or not the group key is stored in the memory, and can switch the operation mode of the video apparatus itself to the master mode according to the determination that the group key is stored. Alternatively, if it is determined that the group key is not stored, the video apparatus can switch its operation mode to the slave mode.

The video apparatus acquires address information for content reproduction by accessing the server using the group key (230). The address information may be, for example, a content URL (Uniform Resource Locator). Or address information may include, for example, information about content to be played in a plurality of video apparatuses included in a device group, a playlist including a content playback order in a plurality of video apparatuses, and the like . The playlist may be a set of content URLs in accordance with the playback order of the content.

The video device provides the address information to the other video devices included in the device group (240). Other imaging devices may be, for example, slave imaging devices.

The video device transmits a playback command for the content to other video devices based on whether or not the loading of the content based on the address information has been completed in each of the other video devices (250). At this time, whether or not the loading of the content has been completed can be determined based on, for example, whether or not the first remaining time for completing the loading of the content is shorter than the second remaining time for completing the reproduction of the content. In other words, if the remaining time until the loading of the content is completed is 15 seconds and the remaining time until the reproduction of the content is completed is 20 seconds, the video apparatus can determine that the loading of the content has been completed. On the other hand, if the time remaining until the reproduction of the currently playing content is completed is 10 seconds, the video apparatus can determine that the loading of the content is not completed.

The imaging device can acquire pre-stored calibration information. The calibration information may include information for setting at least a region in which a display device corresponding to a video device among a plurality of video devices included in the device group should reproduce the entire area for reproduction of the content. The imaging device generates image information for the display device based on the content and the calibration information (260). The video device provides video information to the display device (270).

3 is a flowchart illustrating a method of initializing a video apparatus according to an exemplary embodiment of the present invention. Referring to FIG. 3, a video apparatus (master video apparatus) according to an exemplary embodiment may determine whether initialization is required (320) as power is supplied 310 from a display apparatus.

If it is determined in step 320 that initialization is not required, the imaging apparatus may terminate the initialization operation. At this time, whether or not the initialization is necessary can be determined based on whether or not the video apparatus includes information for connection (for example, AP address, etc.). For example, if it includes information for connection, the imaging device may determine that initialization is not required. Alternatively, separate flag information indicating whether initialization is necessary or initialization is completed may be used.

If it is determined in step 320 that initialization is required, the imaging device may act as a hot spot host 330 to obtain information for connection from the mobile device 340.

The imaging device may access the access point using the information for the connection obtained in step 340 and receive the group key (350).

The imaging device may store the group key in a cache or memory (360).

4 is a flowchart illustrating an operation method of a video apparatus according to another embodiment. Referring to FIG. 4, an imaging device (e.g., a slave imaging device) according to an exemplary embodiment receives address information for reproducing content from a master imaging device of a device group (410).

The imaging device loads the content for the device group based on the address information (420).

The video device determines whether the content has been loaded (430), and the video device transmits the determination result (completion of loading) to the master video device (440).

The video device receives a playback command for the content from the master video device (450).

The video apparatus generates video information for the display apparatus based on the content and the calibration information for reproducing at least a part of the content in a region corresponding to the display apparatus connected to the video apparatus in the entire region for reproducing the content ).

The imaging device provides imaging information to the display device (470).

According to the embodiment, the video apparatus can determine whether or not the video apparatus is connected to the display apparatus, and can check whether or not the group key is stored according to the determination that the video apparatus is connected to the display apparatus. If it is checked that the group key is not stored, the video apparatus can switch the operation mode of the video apparatus to the slave mode.

5 is a flow diagram illustrating a method of operation of a mobile device according to one embodiment. Referring to FIG. 5, the mobile device may transmit information for connection to imaging devices (510). The mobile device may, for example, connect to a video device operating as a hotspot host and transmit the AP address to the video device.

The mobile device may determine 520 the device group that includes at least some of the imaging devices using the information for the connection transmitted in step 510. [ The mobile device may, for example, connect to the AP and expose a list of imaging devices connected to the AP to the user. The mobile device may determine a group of devices comprising the selected video devices in response to a user's input selecting at least some of the video devices.

The mobile device may generate a group key corresponding to the device group (530). The group key may be composed of a 128-bit random number, for example, a Universal Unique Identifier (UUID).

The mobile device may determine 540 the master imaging device among the imaging devices included in the device group, and may transmit the generated group key to the master imaging device in step 530 (550). The mobile device may determine the master imaging device, for example, by randomly selecting any of the imaging devices included in the device group. The method of determining the master video device can be variously modified according to predetermined conditions such as determining the video device having the best communication environment as the master.

The mobile device may send group information including the group key to the server (560).

Calibration method according to one embodiment

FIG. 6 is a diagram for explaining a video reproducing method of a video apparatus according to an embodiment. Referring to FIG. 6A, a screen is shown in which contents are reproduced by four video apparatuses on a projection surface including an edge.

The video apparatus according to an exemplary embodiment can reproduce contents for various types of space in which planar and curved surfaces are mixed as well as planar and curved surfaces. For example, it is assumed that the contents describing the handwriting process are reproduced in the corner spaces 610, 620, 630, and 640 of the exhibition hall, and the other contents are reproduced through the monitor 605 in the middle of the corresponding space. At this time, it is assumed that four video apparatuses are required to reproduce content in the corner space of the exhibition hall.

The video apparatuses according to an exemplary embodiment may perform a calibration in which a user interacts with a user interface (UI) of a mobile device and Wi-Fi to set at least a partial region of the entire large screen (content area) .

When the reproduction command for the content is transmitted from the master imaging apparatus, the slave imaging apparatus can cause the corresponding display apparatus to reproduce a part of the content together with the master imaging apparatus according to the respective calibration information.

The user can create a device group that plays back one large screen, for example, by selecting four video devices after logging into the mobile app. At this time, the first display device corresponding to the first image device among the four image devices displays a partial image corresponding to the first area 610 of the projection surface of the corner space of the exhibition hall, The display device displays a partial image corresponding to the second area 620, the third display device corresponding to the third video device displays a partial image corresponding to the third area 630, and the fourth display device corresponding to the fourth video device The display device may be calibrated to reproduce a portion of the image corresponding to the fourth area 640. [ At this time, it can be seen that the first region 610 to the fourth region 640 are different from each other in their shapes.

For example, each display device can be arranged to project different areas of the corner space of the exhibition hall. The areas projected by the display devices may overlap each other in a predetermined area. The imaging device controlling each display device may display a plurality of points (e.g., four points) in the projection area of the display device it controls for calibration. According to an embodiment, a plurality of points displayed by each imaging device may be displayed (e.g., in different colors or different shapes, etc.) to be distinct from a plurality of points displayed by different imaging devices.

When a plurality of points (for example, four points) are displayed on the projection surface of the corner space of the exhibition hall through the control of each imaging device, the user can use the user interface of the mobile device to display a plurality of points on the projection surface And it is possible to set some areas to be reproduced by the video apparatuses (corresponding to the display apparatuses) according to the shape of each area of the projection surface. Depending on the embodiment, at least one point may be added or removed.

When the setting from the first area 610 to the fourth area 640 is completed, each image device can generate a mask corresponding to the remaining area except the area to be reproduced, and prevent the image from being reproduced in the remaining area.

The calibration process for setting at least some areas to be reproduced by the imaging device according to the embodiment may be performed automatically by the imaging device. As will be described in detail below, the information about the projection space is acquired using a 360-degree camera, and by controlling each of the display devices to project images of different patterns, images are displayed on the display devices ) May be automatically set.

Referring to FIG. 6 (b), a screen is shown in which the content is reproduced by the three video devices on the projection surface of the curved surface. The projection surface shown in FIG. 6 (b) has the same cylindrical shape as the projection surface of FIG. 6 (a).

For example, suppose you want to reproduce the contents of a marine image in a cylindrical exhibition hall.

The video apparatus according to an exemplary embodiment can detect a space (a cylindrical space of 180 degrees) in which marine contents can be reproduced by using an omnidirectional camera capable of shooting 360 degrees. The imaging device may acquire a structured pattern projected from each of the display devices corresponding to the plurality of imaging devices included in the device group. For example, the imaging device (first imaging device) projects a vertical structured pattern, the second imaging device projects a horizontal structured pattern, and the third imaging device projects a zigzag structured pattern lets do it.

Based on the detection result of the space (cylindrical space of 180 degrees) and the structuring pattern of each image device, the image device displays an area to be reproduced by each of the display devices corresponding to the image device among the contents of the marine image reproduced in the cylindrical space of 180 degrees You can decide.

For example, if the display device corresponding to the corresponding video device (first video device) among the 180-degree cylindrical spaces displays content (for example, 1/3 of the entire content of the marine image) for the first area 650 The display device corresponding to the second video device reproduces the content for the second area 660 and the display device corresponding to the third video device reproduces the content for the third area 670 Can be determined to play. Thus, the marine contents reproduced in the exhibition hall constructed in a cylindrical shape can be reproduced on the projection plane by the display devices (corresponding to the display devices included in the device group).

7 is a flowchart illustrating a method of reproducing an image of a video apparatus according to an exemplary embodiment of the present invention. Referring to FIG. 7, a video apparatus according to an embodiment obtains information on a space for reproducing contents (710). The video apparatus can acquire information on the space for reproducing the content by receiving information on the space for reproducing the content from the user. In this case, the information on the space for reproducing the content may include, for example, a space type such as a plane shape, a curved surface shape, a cube shape, a cylinder shape, and the like, . The video apparatus may also receive a selection of the projection plane from which the content is to be reproduced from the user based on the information on the space for reproducing the content. Here, '(reproduction of contents)' can be understood to mean projection of contents using a projector or screening (display) of contents using a screen device.

The imaging device generates a mask for setting at least a partial area to be reproduced by the display device corresponding to the imaging device on the projection surface determined based on the information on the acquired space in step 710 (720). For example, the projection plane may be divided and allocated to a plurality of display devices, and each display device may project an image to an area assigned to itself. In this case, each display device may project an image only in a part of the entire area that can be projected by itself, and information about a part of the area may be stored as a mask.

The entire area of the content may be reproduced on the projection surface by a group of devices comprising the imaging device and one or more other imaging devices. At this time, the mask may be, for example, an alpha map mask.

Depending on the embodiment, the display device may be multi-screen devices or multi-projectors. For example, if the display device is a multi-projector, the imaging device according to one embodiment can determine at least some areas to be reproduced by the display device through the following automatic method.

The video apparatus can sense a space for playing the content by using an omnidirectional camera capable of shooting 360 degrees. The imaging device may acquire a structured pattern projected from each of the display devices corresponding to the plurality of imaging devices included in the device group. The imaging device can determine at least some areas of each of the entire areas of the contents to be reproduced by each of the display devices based on the detection result of the space and the structured pattern.

Depending on the embodiment, the imaging device may determine at least some areas of the display device to be reproduced by the following manual method. The imaging device can display a plurality of points on the projection surface using a display device. The imaging device may move at least one of the plurality of points based on user input. The imaging device may determine a portion of the area based on the plurality of points including the moved point. For example, some areas may be determined as polygons with vertices at each point.

The imaging device may generate masks corresponding to the remaining areas except at least some areas on the projection surface using points determined by automatic determination or user input. The imaging device can adjust pixel values corresponding to the remaining area using a mask. For example, by adjusting the masking value corresponding to the remaining area to '0', the image device can prevent the image from being output to the area corresponding to the mask. A more detailed description of how to generate a mask will be given later with reference to FIG.

The imaging device determines (730) a reference texture that includes reference coordinates of pixels of an area that is actually projected onto the projection surface in response to coordinates of pixels of at least some of the areas during content reproduction of the display device. The reference coordinates may be, for example, a UV coordinate for a planar image or a surface image, and a 360 degree image may be a latitude and longitude coordinate. The reference texture may also be a UV texture corresponding to a set of UV coordinates or a PHI map corresponding to a set of latitude and longitude coordinates.

The imaging device may determine the reference texture using, for example, a homography matrix that corresponds to the relationship between the content and the projected image actually projected on the projection surface. At this time, the relationship between the content (P _x ) and the projected image Px 'actually projected on the projection surface can be established as shown in the following equation (1).

Here, H may correspond to a homography matrix.

In the case of a UV texture, the coordinates of the pixels in a specified area can be expressed in relative coordinates. For example, referring to FIG. 9, each pixel in the entire content may have UV coordinates within (0, 0), (1, 0), (0, 1), (1, 1) A part of the area determined in correspondence with each image device corresponds to an area on the projected image Px ', and a content pixel matching each pixel in a certain area can be determined using Equation (1). In one example, a homography matrix may be used to match a pixel on the projected image Px 'to a pixel on the content Px to determine the UV coordinates of the pixel on the projected image Px'. The UV coordinate of the pixel on the projection image Px 'may be set to the UV coordinate of the matched pixel on the content Px.

The imaging device reproduces at least a portion of the area using the mask and reference texture (740). The imaging device may store at least one of the mask and the reference texture in association with the imaging device, or may store and match the display device corresponding to the imaging device. When playing the content, the video apparatus generates video information using the stored mask and reference texture, and transmits the generated video information to the display device. For example, the imaging device can use the stored mask to determine the area in the projectable area of the display device associated therewith that should actually project the image, and determine the value of each pixel in the determined area of the seal projection area using the reference texture You can decide. As described above, the reference texture includes information on which pixel value in the content should be referred to in order to determine the value of each pixel in the seal projection area.

8 is a diagram for explaining a method of generating a mask according to an embodiment. 8, a portion 810 in which the first display device corresponding to the first video device can reproduce the content in the entire projection surface 805, and a second display device corresponding to the second video device reproduce the content A portion 850 is shown. At this time, the first video device and the second video device may be video devices belonging to the same device group.

For example, when a playback command for content is transmitted from the mobile device, the first display device reproduces a region 830 corresponding to a part of the entire area of the content in the portion 810, 850, it is possible to reproduce the area 860 corresponding to a part of the whole area of the contents.

In an embodiment, the user can set an area to be reproduced by the display device through a user interface (UI) of a mobile device interlocked with the image device.

For example, the imaging device may display a plurality of points (e. G., Any four points within portion 810) on the projection surface 805 using a display device. Depending on the embodiment, the imaging device may determine where the plurality of points are initially displayed based on the basic properties of the projection surface 805 (e.g., the type of projection space, the size of the projection space, etc.).

The user may move the four points to the edge of the area 830 using the user interface of the mobile device. The imaging device may set at least some of the areas 830 to be played by the first display device by the four points moved to the edges of the area 830. [

Similarly, the imaging device displays a plurality of points on the projection surface 805, and the user moves the four points corresponding to the position of each corner of the portion 850 through the user interface of the mobile device, ) To the edge of the frame. The imaging device may set at least some of the areas 860 to be reproduced by the second display device by the four points moved to the edge of the area 860. [

The imaging device may generate mask information for the first display device. For example, the masking value corresponding to the partial area 830 may be adjusted to '1' and the masking value corresponding to the remaining area 820 may be adjusted to '0'.

Likewise, the imaging device may generate mask information for the second display device. The imaging apparatus may adjust the masking value corresponding to the partial area 860 to '1' and adjust the masking value corresponding to the remaining area 870 to '0'.

The first video apparatus may be configured such that a signal transmitted by the mobile device or the server causes the area 830 reproduced by the first display apparatus corresponding to the first video apparatus to be reproduced by the second display apparatus corresponding to the second video apparatus It can be seen that an overlapped area 840 occurs with the area 860. In this case, the first imaging device may adjust the masking value for the overlapping area 840 to, for example, 0.5. As with the second imaging device and the first imaging device, the masking value for the overlapping area 840 can be adjusted to, for example, 0.5. Accordingly, the brightness of the area overlappingly projected by the plurality of display devices can be equal to the brightness of the area projected by the single display device.

In addition, the first or second imaging device may mask the boundary regions 835 and 865 of the partial regions included in the overlap region 840 and the masking values of the boundary region including the regions adjacent to the boundary lines 835 and 865 Smoothing can be performed. Thus, the visibility of the boundary between the regions divided and projected by the plurality of display devices can be eliminated.

The imaging device can calculate the boundary area using various edge extraction algorithms such as first order differential method, sobel edge extraction method, prewitt dege extraction method, and the like. The imaging device may smooth the boundary region using various smoothing algorithms such as, for example, a Gaussian algorithm or a histogram equalization.

9 is a diagram for explaining a method of determining a reference texture according to an embodiment. Referring to FIG. 9, there is shown a UV texture 910, which is an example of a reference texture.

The UV coordinate system may correspond to a two-dimensional coordinate system that is a conversion reference for texture mapping to texture image (UV texture 910) in three-dimensional space.

The UV coordinates can have a value from 0 to 1 at a minimum. The UV coordinates can be expressed as a value from 0 to 1, regardless of the resolution of the reference texture (UV texture 910).

Figure 9 (a) shows two triangles that do not yet have a reference texture (UV texture 910). At this time, vertices v0, v1, and v2 form one triangle, and vertices v0, v2, and v3 form another triangle.

9B shows a case where the range of UV coordinates of v0, v1, v2 and v3 for rendering using the reference texture is (0,0) to (1,1), and FIG. Indicates the case where the range of the UV coordinates for rendering is (0,0) to (0.5,1). According to FIG. 9 (c), only the left half of the reference texture (UV texture 910) is visible.

9D shows a case where the range of UV coordinates for rendering is (0,0) to (0.5, 0.5), in which only the upper half of the left half of the reference texture (UV texture 910) Can be.

FIG. 9E shows a case where the range of UV coordinates for rendering is (0,0) to (1,2). In this case, the reference texture (UV texture 910) may be repeated two times up and down. If the range of the UV coordinates for rendering is (0,0) to (2,2) as shown in FIG. 9 (f), the reference texture (UV texture 910) is twice vertically and twice horizontally Can be expressed repeatedly. In addition, if the range of the UV coordinates for rendering is (1,0) to (0,1), the left and right sides of the reference texture (UV texture 910) can be inverted.

The imaging device may use a reference texture that includes reference coordinates of pixels of an area actually projected onto the projection surface corresponding to the coordinates of pixels of at least some of the areas in the reproduction of the content of the display device in the manner described above.

FIG. 10 is a view for explaining a method of determining a reference texture according to another embodiment. Referring to FIG. 10, reference coordinates representing a 360-degree image in latitude and longitude coordinates are shown. The reference texture may be a PHI map corresponding to a set of latitude and longitude coordinates. The latitude and longitude coordinates in the pi map can function in the same manner as the UV coordinates described with reference to Fig.

A synchronization method according to an embodiment

11 is a flowchart illustrating a method of synchronizing a video apparatus according to an exemplary embodiment. Referring to FIG. 11, a video device (e.g., a slave video device) according to an embodiment exchanges time information with a master video device included in a device group in a wireless network (1110). The time information may be exchanged using SNTP packets, for example. At this time, the time information includes a first time stamp on the side of the video apparatus for transmitting the first packet to the master video apparatus at the first time, a time stamp on the side of the master video apparatus for receiving the packet transmitted at the first time from the video apparatus at the second time And a third time stamp on the master imaging device side that transmits the second packet to the imaging device at a third time after a predetermined delay from the second time. The first packet may comprise a first timestamp and the second packet may comprise a first timestamp, a second timestamp, and a third timestamp. A specific method by which the video apparatus according to an embodiment exchanges time information with the master video apparatus will be described in detail with reference to FIG.

The imaging device calculates the time offsets between the master imaging device and the imaging device based on the time information (1120). The video apparatus receives the first through third time stamps included in the time information exchanged with the master video apparatus at step 1110 and the video packet that receives the second packet transmitted at the third time from the master video apparatus at the fourth time And calculate time offsets based on a fourth time stamp on the device side. According to an embodiment, the imaging device can check the validity of the time information and calculate the time offsets based on the time information that passed the validation.

The imaging device generates synchronization information based on sampling parameters for extracting sampling data from the time offsets and filtering parameters for determining whether the sampling data is outliers (1130). At this time, the sampling parameter may be determined based on the probability that the outlier extracts the sampling data from the filtered sampling data and the ratio of the sampling data whose outliers are filtered out of the time offsets. The filtering parameters may be determined based on the standard deviation of the filtered sampling data by the outlier. The method by which the video autonomous entity generates synchronization information will be described in detail with reference to FIG.

The imaging device synchronizes to the master imaging device using the synchronization information (1140).

12 is a view for explaining time information exchanged between a slave image device and a master image device according to an embodiment.

Computer hardware technology continues to be developed around the keywords of miniaturization and high performance, and various products are now on the market. In particular, many small computers have appeared with products such as Razbai Pie. Recently, there have been a lot of products capable of high-level graphic operation. These devices support full-HD or higher video decoding, network modules and various operating systems, enabling many creative services. When such a high performance small computer is used in an image system using a plurality of displays, one small computer is connected to one display, and each one of the images can be played by taking charge of each one of the images.

Then, time synchronization is performed through the wireless network and each image is reproduced. What is important here is stable synchronization. The conditions of the wireless synchronization reproduction module for general commercial system can be summarized as follows.

1. Time precision within milliseconds for frame synchronization

2. Quick to achieve target time accuracy after starting time synchronization

3. Possible to prepare for unstable network situation

Most systems use the international standard NTP (Network Time Protocol) for time synchronization. NTP is theoretically synchronized with UTC (Coordinated Universal Time), which is the international standard time, in milliseconds. It forms a distributed network from atomic clock to hierarchical structure and exchanges packets with many clocks. The information can then be analyzed statistically through postprocessing algorithms and used to fit the clock. The complexity of this NTP is that it is aimed at more precisely aligning rather than quickly aligning clocks, making it difficult to quickly achieve target time accuracy. Therefore, in one embodiment, in order to satisfy the above condition 2, SNTP is used as a wireless synchronization reproduction module.

SNTP does not require complicated post-processing algorithms in the existing NTP standard, so its implementation is relatively simple and easy to customize. However, when time synchronization is to be performed between devices using a wireless network, the SNTP has a problem in that the filtering process for the packets to be exchanged is omitted and the stability is degraded. This is because, in an unstable wireless network compared to a wired connection, only one abnormal packet may degrade the quality of synchronization reproduction.

In one embodiment, the time samples obtained in the driving process of the SNTP are processed using the RANSAC algorithm to maintain the synchronization, thereby providing a time synchronization method on the wireless network having the fastness and accuracy based on the SNTP. It can be easily implemented in a limited computing environment, and can select reliable samples during NTP data processing. According to the experimental results, it can be seen that the synchronization method according to the embodiment maintains the time accuracy within 5 milliseconds within 5 seconds and uses a small amount of resources so that it can be stably driven in any environment.

In one embodiment, synchronization can be performed by analyzing information while exchanging packets containing time information between server-peer-clients on the network. At this time, the time information included in the packet can be referred to as a time stamp. By recording both the packet sent to the time stamp and the received packet, it is possible to securely synchronize regardless of packet loss and / or redundant transmission even if packets are exchanged with various devices at the same time.

Referring to FIG. 12, time information for exchanging packets between the slave video apparatus A 1210 and the master video apparatus B 1230 from the first time T ₁ to the fourth time T ₄ is shown. In one embodiment, there are four time stamps from the first time (T ₁ ) to the fourth time (T ₄ ) and three state variables, org, rec, and xmt. org indicates the time stamp on the side of the transmission (slave video apparatus A 1210), rec indicates the time stamp on the side of the reception (master video apparatus B 1230), xmt indicates the time on the transmission side (slave video apparatus A 1210) Stamps can be represented.

The four timestamps include a first time stamp on the side of the imaging device that transmits the first packet to the master imaging device at the first time T ₁ , a second time stamp on the side of the imaging device that transmits the packet transmitted at the first time T ₁ from the imaging device, (T ₂₎ the master video device side of the second time stamp, and a master video device for sending a second packet to an imaging device of claim the third time after a predetermined delay from 2 hours (T ₂₎ (T ₃₎ for receiving the side it is possible to include a fourth time stamp of the third time stamp, and the master third time from the video device (T ₃₎ the imaging device side for receiving the transmitted second packet to a fourth time (T ₄₎ of the.

The first packet may be transmitted from slave imaging device A 1210.

Slave imaging apparatus A 1210 transmits a transmission time stamp T ₁ through a first packet, and master imaging apparatus B 1230 can copy time stamp T ₁ to xmt.

The master video apparatus B 1230 receives the first packet at the second time T ₂ , copies the received content T ₁ to its org, and copies T ₂ to rec. Then, at a third time (T ₃ ) after a while, the master imaging apparatus B 1230 can transmit all of T _1, T _{2 and} T ₃ to the slave imaging apparatus A 1210 in the second packet. Master image device B 1230 may also copy information about T ₃ , the time at which it has transmitted the second packet, to xmt.

The slave imaging apparatus A 1210 can receive the second packet transmitted by the master imaging apparatus B 1230 at the fourth time T ₄ and perform the validity check on the second packet.

The slave video apparatus A 1210 compares whether the T ₁ included in the second packet received at the fourth time T ₄ is equal to the information stored in the xmt and checks whether the T ₃ value is not 0, Lt; / RTI > is a valid packet.

If the second packet is determined to be a valid packet, the slave video apparatus A 1210 uses the stored information T _1, T _{2, and} T ₃ to calculate a time offset (?) And a round-trip delay delay (delta).

The time offset? Represents a time difference between the slave video apparatus A 1210 and the master video apparatus B 1230, and can be obtained, for example, by the following equation (2).

The round-trip delay (delta) means the time taken to move purely until the sent packet returns, for example, the difference between the fourth time and the first time and the difference between the third time and the second time Value. ≪ / RTI >

Among the obtained time offset (?) And round trip delay (?) Information, noise due to various causes such as a network failure may be included. In one embodiment, the noise included in the time offset can be removed by the method of FIG. 13 below, and the synchronization can be finally performed using the best time offset value.

13 is a flow diagram illustrating a method for generating synchronization information in accordance with one embodiment. Referring to FIG. 13, an imaging apparatus according to an exemplary embodiment inputs the time offsets to a RANSAC model so that an outlier among the time offsets acquires filtered sampling data (1310). Here, the outlier can be understood as a value corresponding to a noise which largely deviates from the range of the normal time offset value, for example, when the time offset value jumps over 200 ms. Further, the inliers can be understood as values existing within a range of a typical time offset value.

RANSAC is a method for obtaining a mathematical model that can represent them in any data, and can select a model that receives the most support for randomly sampled data. Basically, it works well when the proportion of outliers in the sampled data has little effect on the actual value of the model that you want to guess.

When using the RANSAC model, what is needed is a reference number (t) that determines the number of iterations (k) of how many iterations of the sampling process are repeated and whether to include the time offset in the inlier list. The number of repetition times (k) may be called a sampling parameter. Further, the reference value t may be called a filtering parameter.

In order for the RANSAC model to be successful, sampling data should be drawn only from the inliers at least once during the attempt to extract the sampling data. This probability increases as the number of repetition times k increases, but the amount of computation increases as the number of repetition times k increases. Therefore, the number of repetition times k can be determined using a probability calculation formula as shown in the following Equation (3).

Here, p represents the probability of sampling data being extracted from among the inliers, and a represents the ratio of the inliers among the input data (time offsets), that is, the ratio of the sampling data outliers filtered out of the time offsets . n represents the number of sampled data sampled at one time, and k represents the number of sampling sampled data.

If the reference value t for determining whether to insert the time offset into the inlier list is too large, the discrimination power between the models is lost. If the reference value t is too small, the probability of the RANSAC algorithm itself failing may increase. In one embodiment, when the standard deviation of the inliers is 2σ, the reference value t can be determined to be 2σ or 3σ. More accurate results can be obtained by performing RANSAC after determining the reference value t by obtaining the variance of the time offsets according to the embodiment.

A method by which a video apparatus according to an embodiment obtains filtered sampling data by an outlier will be described in detail with reference to FIG.

The imaging device determines synchronization information based on the acquired sampling data (1320). The method by which the imaging apparatus determines the synchronization information based on the obtained sampling data will be described in detail with reference to Fig.

14 is a flow diagram illustrating a method for an outlier to obtain filtered sampling data in accordance with one embodiment. Referring to FIG. 14, a video apparatus according to an exemplary embodiment may extract sampling data from time offsets (1410). For example, if there are 1000 time offsets, the imaging device can extract 100 or 50 out of 1000 time offsets as sampling data.

The imaging device may determine a criteria model based on the statistical values of the sampling data (1420). The reference model may be, for example, the mean value of the time offset value, i. E., A constant model. The statistical value of the sampling data may be, for example, an average, or a variance value.

The imaging device may remove the outlier by comparing the filtering parameters and the difference between the sampling data and the reference model (1430). For example, let us say that the filtering parameter is 0.0005, and the difference between the sampling data and the reference model is 0.0003. Since the difference (0.0003) between the sampling data and the reference model is smaller than the filtering parameter (0.0005), the imaging apparatus can include the sampling data in the inlier list. On the other hand, if the difference between the sampling data and the reference model is 0.0007, since the difference (0.0007) between the sampling data and the reference model is larger than the filtering parameter (0.0005), the video apparatus can determine the corresponding sampling data as outliers and remove it.

15 is a flow diagram illustrating a method for determining synchronization information in accordance with one embodiment. Referring to FIG. 15, a video apparatus according to an embodiment may acquire a plurality of sampling data from which an outlier has been removed, corresponding to a sampling parameter (1510).

The imaging device may select one of a plurality of sampling data (1520).

The imaging device may determine 1530 whether the percentage of outliers removed from the selected sampling data is below a threshold. If it is determined in step 1530 that the percentage of outliers removed from the selected sampling data is greater than or equal to the threshold, the imaging device may terminate the operation.

If it is determined in step 1530 that the percentage of outliers removed from the selected sampling data is less than the threshold, the imaging device may calculate the standard deviation of the sampling data (1540).

The imaging device may select 1550 the sampling data having the lowest standard deviation of the standard deviation of the sampling data calculated in step 1540.

The imaging device may calculate the average of the sampling data selected in step 1550 (1560).

The imaging device may set the synchronization information to an average of the selected sampling data (1570).

FIG. 16 is a diagram illustrating a pseudo code illustrating a process of applying time information obtained through a Simple Network Time Protocol (SNTP) packet to a RANSAC model according to an embodiment.

The numerical code shown in FIG. 16 shows a process of applying the RANSAC model to the time information obtained through the SNTP. The reason why the time information obtained through SNTP in the wireless network environment is appropriate as the input of RANSAC is that abnormal packets occasionally but steadily occur.

In the algorithm shown in Fig. 16, the number of repetition times (k) is 1000, and the reference value t for determining the inlier is 0.0005. Also, the model uses the average value of the time offset values, that is, the constant model.

In one embodiment, the model uses a model containing more than 50 percent of all lyers among the whole data. To find the model with the highest reliability, each standard deviation is calculated so that the list of the inliers having the lowest standard deviation The synchronization information can be determined using the average value. With this synchronization information, according to an embodiment, it is possible to maintain a time offset of 5 ms or less within 5 seconds after the start of synchronization.

According to an exemplary embodiment, synchronization is performed using the SNTP packet and the RANSAC model, thereby achieving high-quality time synchronization even in a simple implementation and a small resource in a wireless network environment. In addition, according to an exemplary embodiment, wireless synchronization reproduction can be performed in various environments such as an embedded system and an IoT device in which NTP is not provided or is difficult to use, thereby helping to develop and apply related technologies.

Device configuration according to one embodiment

17 is a block diagram of a video device according to one embodiment. 17, a video device 1700 according to one embodiment may include a sensing unit 1710, a communication interface 1720, a processor 1730, and a memory 1740. [ The sensing unit 1710, the communication interface 1720, the processor 1730 and the memory 1740 can communicate with each other via the communication bus 1705. [

The sensing unit 1710 senses whether the imaging device 1700 is connected to the display device. The sensing portion 1710 may sense whether the imaging device 1700 is connected to the display device based on, for example, whether the imaging device 1700 is coupled to the HDMI port of the display device, or the USB port, or whether power is supplied.

The processor 1730 can determine whether initialization is required as power is supplied from the display device. The processor 1730 may act as a hotspot host to obtain information for the connection from the mobile device, as determined by the need for initialization. The processor 1730 may access the access point using the information for the connection and receive the group key. Processor 1730 may store the group key in memory 1740.

The processor 1730 can determine the calibration information that sets at least some areas of the plurality of video devices included in the device group that the display device corresponding to the video device should reproduce from among the entire area for playback of the content.

In the video device 1700, the communication interface 1720 acquires information on the space for reproducing the content. The processor 1730 generates a mask for setting at least a partial area to be reproduced by the display device corresponding to the imaging device on the projection surface determined based on the information about the space. The processor 1730 determines a reference texture including reference coordinates of pixels of an area that is actually projected on the projection surface in correspondence with the coordinates of pixels of at least some areas upon reproduction of the content of the display device.

The processor 1730 may use a forward camera to sense the space to reproduce the content. The processor 1730 may obtain a structured pattern projected from each of the display devices corresponding to the plurality of video devices included in the device group. The processor 1730 may determine at least some areas of each of the entire areas of the content to be played by each of the display devices, based on the detection result of the space and the structured pattern.

The processor 1730 may display a plurality of points on the projection surface using the display device. Processor 1730 may move at least one of the plurality of points based on user input and may determine a portion of the area based on the plurality of points including the moved point.

The processor 1730 may use the determined points from the user to create a mask corresponding to the remaining area except at least some areas on the projection surface. The processor 1730 may use a mask to adjust pixel values corresponding to the remaining regions.

The processor 1730 can adjust pixel values for an area overlapping some area to be reproduced by a display device corresponding to another video device belonging to the device group among at least some device areas. The processor 1730 may smooth the pixel values of the boundary region including boundary regions of at least some regions included in the overlap region and regions adjacent to the boundary lines.

The processor 1730 may determine the reference texture using a homography matrix corresponding to the relationship between the content and the projected image actually projected on the projection surface. Memory 1740 may store at least one of a mask and a reference texture.

The processor 1730 obtains the group key corresponding to the device group including the video device and one or more other video devices, in accordance with the determination that the detection unit 1710 is connected to the display device. For example, the processor 1730 may obtain the group key from the memory 1740.

Process 1730 determines whether the imaging device is the master imaging device determined by the mobile device based on whether or not the group key is stored. The processor 1730 may convert the operation mode of the video apparatus into the master mode according to the determination that the video apparatus is the master video apparatus.

The processor 1730 acquires address information for content reproduction by connecting to the server using the group key, and provides address information for content reproduction to other video apparatuses included in the device group. The processor 1730 transmits a playback command for the content to the other video apparatuses based on whether or not the loading of the content based on the address information has been completed in each of the other video apparatuses. The processor 1730 generates the video information for the display device based on the content and the calibration information for reproducing at least a part of the content in the area corresponding to the display device among the entire area for reproduction of the content.

Processor 1730 may generate image information for the display device based on synchronization information for synchronizing a plurality of imaging devices. At this time, the synchronization information may be determined, for example, by inputting a time offset between a plurality of video apparatuses included in the device group into the RANSAC model.

The memory 1740 may store image information generated by the processor 1730. Also, the memory 1740 can store the loaded content based on the address information for reproducing the content. The memory 1740 may be a high speed random access memory such as, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM) And may include a high-speed random access memory. Memory 1740 may include volatile memory and non-volatile memory. Memory 1740 may optionally include at least one storage device remotely located from processor 1730.

The communication interface 1720 provides the display device with the image information generated by the processor 1730.

If the processor 1730 determines that the group key is not stored, the processor 1730 can switch the operation mode of the video apparatus to the slave mode. In the video device 1700, the communication interface 1720 receives address information for content reproduction from the master video device of the device group. The processor 1730 loads the content for the device group based on the address information. The processor 1730 determines whether or not the loading of the content has been completed. The processor 1730 receives the playback command for the content from the master video apparatus that has received the determination result that the loading of the content has been completed. The processor 1730 generates a video for the display device based on the content information and the calibration information for reproducing at least a part of the content in an area corresponding to the display device connected to the video device, Information.

The communication interface 1720 of the imaging device 1700 exchanges time information with the master imaging device included in the device group in the wireless network.

The processor 1730 calculates time offsets between the master imaging device and the imaging device based on the time information. The processor 1730 generates synchronization information based on the sampling parameters and the filtering parameters, and synchronizes the synchronization information with the master imaging device.

Processor 1730 may generate synchronization information for synchronizing the master imaging device and the imaging device by inputting time offsets between the plurality of imaging devices included in the device group into the RANSAC model.

Processor 1730 can input the time offsets into the RANSAC model so that outliers of the time offsets can obtain filtered sampled data. The processor 1730 may determine the synchronization information based on the acquired sampling data.

The processor 1730 may calculate time offsets based on time information and a fourth time stamp on the side of the imaging device receiving the second packet transmitted at the third time from the master imaging device at the fourth time.

The processor 1730 may extract the sampling data from the time offsets and determine a reference model based on the statistical values of the sampling data. The processor 1730 may remove the outliers by comparing the filtering parameters with the difference between the sampling data and the reference model.

Processor 1730 may obtain a plurality of sampled data from which the outliers have been removed, corresponding to the sampling parameters. The processor 1730 may calculate the standard deviation of the sampling data based on whether the ratio of the outliers removed from the selected one of the plurality of sampling data is lower than the threshold value. The processor 1730 may select sampling data having the lowest standard deviation and set the average of the selected sampling data as synchronization information.

In addition, processor 1730 may perform an algorithm corresponding to at least one method or at least one method described above with respect to Figures 1-16. The processor 1730 may execute the program and control the imaging device 1700. [ The program code executed by the processor 1730 may be stored in the memory 1740.

The above-described operations correspond to exemplary embodiments, and the embodiments can be variously modified from the above-described operations.

The embodiments described above may be implemented in hardware components, software components, and / or a combination of hardware components and software components. For example, the apparatuses, methods, and components described in the embodiments may be implemented by, for example, a processor, a controller, a central processing unit (CPU), a graphics processing unit (GPU) arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, Application Specific Integrated Circuits (ASICS) may be implemented using one or more general purpose or special purpose computers, such as any other device capable of executing and responding to instructions.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the preferred embodiments of the present invention have been disclosed for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

A method of operating a video device,
Determining whether the display device is connected to the display device;
Obtaining a group key corresponding to a device group including the video device and one or more other video devices according to a determination that the video device is connected to the display device;
Acquiring address information for content reproduction by accessing a server using the group key;
Providing the address information to the other imaging devices included in the device group;
Transmitting a playback command for the content to the other video apparatuses based on whether or not loading of content based on the address information is completed in each of the other video apparatuses;
Generating image information for the display device based on calibration information for reproducing at least a part of the content in an area corresponding to the display device among the entire area for reproducing the content and the content step; And
Providing the image information to the display device
The method comprising the steps of: The method according to claim 1,
The address information
Information about contents to be reproduced in each of a plurality of video apparatuses included in the apparatus group, and a play list including a content reproduction order in each of the plurality of video apparatuses. The method according to claim 1,
Whether or not the loading of the content has been completed
Wherein the first time remaining for completion of loading of the content is determined based on whether the remaining time is less than a second time remaining for completion of playback of the content. The method according to claim 1,
Determining whether initialization of the video apparatus has been completed as power is supplied from the display apparatus;
Operating as a hot spot host to obtain information for connection from the mobile device according to the determination that the initialization is completed;
Accessing the access point using the information for the connection and receiving the group key; And
Storing the group key
≪ / RTI > The method according to claim 1,
Determining whether the imaging device is a master imaging device determined by the mobile device
≪ / RTI > The method according to claim 1,
Determining whether the video apparatus stores the group key; And
Switching the operation mode of the video apparatus to the master mode according to the determination that the group key is stored
≪ / RTI > The method according to claim 1,
The step of generating image information for the display device
Based on synchronization information for synchronizing the plurality of video devices, the time offset being determined by inputting a time offset between a plurality of video devices included in the device group to a Random Sample Consensus (RANSAC) The step of generating the image information
The method comprising the steps of: The method according to claim 1,
Determining, by the display device corresponding to the video apparatus among the plurality of video apparatuses included in the device group, the calibration information for setting at least a partial area to be reproduced from the entire area for reproduction of the content
≪ / RTI > The method according to claim 1,
The content
Multimedia content, multi-view content, multimedia content, game content, and text. A computer program stored in a computer-readable medium for executing the method of any one of claims 1 to 9 in combination with hardware. In the imaging apparatus,
A sensing unit that senses whether or not the display device is connected to the display device;
Acquiring a group key corresponding to a device group including the video apparatus and one or more other video apparatuses according to a determination that the video apparatus is connected to the display apparatus, Providing address information for the other video devices to the other video devices included in the device group based on whether or not the loading of content based on the address information has been completed in each of the other video devices, Based on the content and the calibration information for reproducing at least a part of the content in an area corresponding to the display device among the entire area for reproduction of the content, A processor for generating information; And
A communication interface for providing the image information to the display device;
/ RTI > 12. The method of claim 11,
The processor
Determining whether initialization of the video apparatus has been completed as power is supplied from the display apparatus, acquiring information for connection from the mobile device as a hot spot host according to the determination that the initialization is completed, And accessing the access point using information for connection to receive and store the group key. 12. The method of claim 11,
The processor
Determining whether the video apparatus is the master video apparatus determined by the mobile device based on whether the group key is stored or not, and switching the operation mode of the video apparatus to the master mode according to the determination that the video apparatus is the master video apparatus, Imaging device. 12. The method of claim 11,
The processor
And generates image information for the display device based on synchronization information for synchronizing the plurality of image devices, which is determined by inputting a time offset between a plurality of image devices included in the device group to a RANSAC model. 12. The method of claim 11,
The processor
Wherein the display device corresponding to the video apparatus among the plurality of video apparatuses included in the apparatus group determines the calibration information to set at least a part of the entire region for reproduction of the content to be reproduced. 13. The method of claim 12,
A memory for storing contents loaded on the basis of the address information,
Further comprising: In the imaging apparatus,
A communication interface for receiving address information for content reproduction from the master video device of the device group;
And a controller for controlling the loading of the content based on the address information, loading the content for the device group, determining whether loading of the content is completed, On the basis of the content and the calibration information for reproducing at least a part of the content in an area corresponding to the display device connected to the video device among the entire area for reproducing the content, Lt; RTI ID = 0.0 >
Lt; / RTI >
The communication interface
And provides the image information to the display device. 18. The method of claim 17,
The processor
The control unit determines whether or not the video apparatus is connected to the display apparatus and checks whether or not the group key is stored according to the determination that the video apparatus is connected to the display apparatus. To the slave mode. 18. The method of claim 17,
The processor
Generating synchronization information for synchronizing the master video apparatus and the video apparatus by inputting a time offset between a plurality of video apparatuses included in the apparatus group to the RANSAC model, A video device for generating video information. 18. The method of claim 17,
The processor
Wherein the display device corresponding to the video apparatus among the plurality of video apparatuses included in the apparatus group determines the calibration information to set at least a part of the entire region for reproduction of the content to be reproduced.

Images