KR20140100627A - Contents synchronization server and method thereof using motion matching - Google Patents

Contents synchronization server and method thereof using motion matching Download PDF

Info

Publication number
KR20140100627A
KR20140100627A KR1020130013170A KR20130013170A KR20140100627A KR 20140100627 A KR20140100627 A KR 20140100627A KR 1020130013170 A KR1020130013170 A KR 1020130013170A KR 20130013170 A KR20130013170 A KR 20130013170A KR 20140100627 A KR20140100627 A KR 20140100627A
Authority
KR
South Korea
Prior art keywords
motion
content
measurement data
mobile terminal
motion measurement
Prior art date
Application number
KR1020130013170A
Other languages
Korean (ko)
Other versions
KR101560673B1 (en
Inventor
윤일지
손기영
공경록
Original Assignee
삼성에스디에스 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 삼성에스디에스 주식회사 filed Critical 삼성에스디에스 주식회사
Priority to KR1020130013170A priority Critical patent/KR101560673B1/en
Priority to PCT/KR2013/010860 priority patent/WO2014123299A1/en
Publication of KR20140100627A publication Critical patent/KR20140100627A/en
Application granted granted Critical
Publication of KR101560673B1 publication Critical patent/KR101560673B1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10Services

Landscapes

  • Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Health & Medical Sciences (AREA)
  • Economics (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Strategic Management (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

There is provided a motion-based content synchronization server and method for ensuring high accuracy in matching a mobile terminal to receive content and a display device in which the content is reproduced. The content synchronization server according to the present invention receives first motion measurement data according to a motion of the user and identification information of a content being reproduced by the display device from a display device having a sensor for sensing motion of a user, A motion matching unit for comparing the first motion measurement data with the second motion measurement data to determine whether or not the motion is coincident with the motion of the user, And a content providing unit transmitting the identification information of the content to the mobile terminal or requesting the content server to transmit the content to the mobile terminal when it is determined that the motions coincide with each other.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a content synchronization server and a content synchronization method using motion matching,

The present invention relates to a content synchronization server and a content synchronization method using motion matching. And more particularly, to a content synchronization technology that enables a user of a mobile terminal to play content that is being played back on the display device through the mobile terminal by simply moving the mobile terminal and performing predetermined motion in front of the display device.

Various techniques for transmitting contents provided in a display device such as a kiosk to a mobile terminal are provided. What is important in these technologies is a method of matching a mobile terminal to receive contents and a display device in which contents are displayed.

For example, in the case of a technique of requesting content synchronization by triggering a specific motion such as shaking a mobile terminal, a kiosk terminal located nearby is searched using location information at the time when the trigger occurs, The kiosk in which the content synchronization event occurs is matched with the mobile terminal in which the trigger occurs. This technique is described in Korean Patent Publication No. 2012-0001561.

However, the above-described technique is not applicable to an environment in which a plurality of kiosk terminals are installed in adjacent areas or where users requesting content synchronization are located in adjacent areas. The location information of the mobile terminal, which the mobile terminal can provide to the content synchronization server, has a large error.

Accordingly, it is possible to provide a content synchronization technique that enables a user to easily synchronize content reproduced on a display device to his / her mobile terminal while ensuring high accuracy in matching a mobile terminal to receive content and a display device in which content is being reproduced Is requested.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a content synchronization server and method based on motion recognition that assures high accuracy in matching a mobile terminal to receive content and a display device in which the content is played back.

It is another object of the present invention to provide a motion-based content synchronization server and method in which a user's motion serving as a trigger signal of a content synchronization request is simultaneously used as a trigger signal and as a matching reference.

The technical objects of the present invention are not limited to the above-mentioned technical problems, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a content synchronization server, comprising: a display device having a sensor for sensing a motion of a user, the first motion measurement data corresponding to the motion of the user, A network interface for receiving the identification information of the content and receiving second motion measurement data according to the motion of the user from the user's mobile terminal, a second network interface for comparing the first motion measurement data with the second motion measurement data, To the mobile terminal, identification information of the content or identification information of the content associated with the content, or transmits the content to the mobile terminal if the determination result motion is determined to match A content providing unit that requests the content server .

According to an embodiment, the first motion measurement data includes data extracted from the image of the user taking the motion, and the second motion measurement data includes an acceleration generated as the mobile terminal moves according to the motion, And angular velocity.

According to an embodiment, the apparatus further includes a motion data storage unit for storing first motion measurement data feature information and second motion measurement data feature information for one or more predetermined motion, The motion of the first and second motion measurement data is determined from the second motion measurement data, and the motion indicated by the second motion measurement data is determined from the second motion measurement data, It can be determined whether or not they match.

According to an embodiment, the network interface further receives a time stamp of a time at which the motion is sensed and position information of the display device from the display device, and receives a time stamp of the time at which the motion is sensed from the mobile terminal And the motion matching unit may compare the first motion measurement data and the second motion measurement data in which the difference between the time stamp and the position information is within a predetermined threshold. That is, according to the present embodiment, the time and position at which the motion is recognized can be further compared to improve the accuracy of content synchronization.

According to an embodiment, the first motion measurement data may be identification information of a motion determined by the display device. The second motion measurement data may be identification information of a motion determined by the mobile terminal.

According to another embodiment of the present invention, the network interface additionally receives information on a reproduction time point of the content being reproduced by the display device from the display device, and the content provider, It is possible to request the content server to transmit data necessary for jumping to the playback point received from the device and playing back. That is, according to the present embodiment, when the content is reproduced through the mobile terminal, the content reproduction time point of the content synchronization point is directly interlocked.

According to another aspect of the present invention, there is provided a contents providing server including a contents storing unit for storing contents and a sensor for detecting a user's motion, and downloading and reproducing contents stored in the contents storing unit A network interface for receiving first motion measurement data from the display device in accordance with the motion of the user and receiving second motion measurement data from the user's mobile terminal in accordance with the motion of the user, A motion matching unit for comparing the two motion measurement data to determine whether or not the motion matches with each other, and a content providing unit for transmitting the data of the content reproduced by the display device to the mobile terminal when it is determined that the motion results match do.

According to another aspect of the present invention, there is provided a content synchronization method including: a display device having a sensor for sensing a motion of a user, the method comprising: recognizing the motion of the user and generating first motion measurement data Transmitting the first motion measurement data and the information indicating the content being reproduced by the display device to the content synchronization server, and transmitting the first motion measurement data to the content synchronization server, Generating second motion measurement data according to the motion and transmitting the second motion measurement data to the content synchronization server, the content synchronization server comparing the first motion measurement data and the second motion measurement data to determine whether the motion is coincident; If it is determined that the motions match, The recreational synchronization server and a step wherein the mobile terminal transmits data of the content to the mobile terminal according to the information indicating the content comprises the step of reproducing the content by receiving data of the contents from the contents synchronization server.

According to another aspect of the present invention, there is provided a method for synchronizing a content to be synchronized, the method comprising: receiving, by a mobile terminal, It is possible to reproduce through the mobile terminal.

In addition, synchronization of content is performed only when both the display device and the mobile terminal recognize the motion of the user, and the recognition can be performed only when the display device is close to the user. Therefore, Matching can be performed with high accuracy.

1 is a configuration diagram of a content synchronization system according to an embodiment of the present invention.
2 is a configuration diagram of a content synchronization system according to another embodiment of the present invention.
3A is a conceptual diagram of motion recognition on the display device side according to the present invention.
3B is a conceptual diagram of motion recognition on the mobile terminal according to the present invention.
4 is a block diagram of a contents synchronization server according to an embodiment of the present invention.
5 is a block diagram of a content providing server supporting a content synchronization function according to an embodiment of the present invention.
6 is a flowchart of a content synchronization method according to an exemplary embodiment of the present invention.
7 to 8 are a signal flow diagram of a content synchronization method performed by a content synchronization server according to an embodiment of the present invention.
9 is a signal flow diagram of a content synchronization method performed by a content providing server according to an embodiment of the present invention.
10 is a signal flow diagram of a content synchronization method according to an embodiment of the present invention, in which a display device and a mobile terminal perform motion identification.
11 is a signal flow diagram of a content synchronization method supporting a playback-time interworking function according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

First, the configuration and general operation of a content synchronization system according to an embodiment of the present invention will be described with reference to FIG.

1 is a configuration diagram of a content synchronization system according to an embodiment of the present invention. As shown in FIG. 1, the content synchronization system according to the present embodiment may include a content synchronization server 20, a content server 25, a display device 30, and a mobile terminal 40. The contents synchronization system according to the present embodiment is configured such that when the user 10 picks up the mobile terminal 40 and takes a specific motion in front of the display device 30, Is reproduced. According to one embodiment, after content synchronization is performed, playback on the display device 30 may be stopped. Whether or not the playback is stopped in the display device 30 after the content synchronization is performed may vary depending on the environment setting for the display device 30. [

The mobile terminal 40 may be a mobile terminal such as an UMPC (Ultra Mobile PC), a netbook, a PDA (Personal Digital Assistants), a portable computer, a web tablet, a wireless phone, a mobile phone, a smart phone, an e-book, a portable multimedia player (PMP), a portable game machine, a digital camera, a digital picture player, a digital video recorder, a digital video player, and the like.

The mobile terminal 40 is provided with a motion detection sensor capable of detecting movement of the mobile terminal 40. The motion detection sensor may include at least one of an acceleration sensor, a gyro sensor, and a gravity sensor, for example.

In the present embodiment, it is assumed that content to be played back on the display device 30 is also stored in the content server 25. [

The display device 30 is provided with a sensor device 31 for recognizing the user's motion so as to face the front of the display device 30. The sensor device 31 may include an image sensor for capturing an image of the user 10 and an infrared sensor for measuring the infrared light reflected by the user 10.

Hereinafter, the operation of the content synchronization system according to the present embodiment will be outlined.

The user 10 located in the sensing area of the sensor device 31 provided in the display device 30 is allowed to move to the mobile terminal 10 The sensor device 31 recognizes the motion of the user 10 and generates the first motion measurement data according to the motion. At the same time, the mobile terminal 40, which is in the hands of the user 10, measures motion by the motion of the user to generate second motion measurement data.

According to one embodiment, the first motion measurement data may include data extracted from the image of the user who took the motion. That is, the first motion measurement data may be data converted from the image of the motion of the user 10. For example, the first motion measurement data may be the human body skeleton information of the user 10 generated from the image of the user 10. The method described in U.S. Patent Publication No. 2010-0199228 can be applied to a method of obtaining human skeleton information from a user 10 captured image.

However, since the motion required in the present embodiment is not limited to the motion of the whole body but to the motion related to the hand, the first motion measurement data may be a motion between the joints related to the hand, Only the data for the skeletal edge of the frame may be included. That is, the motion means movement of a user's hand holding the mobile terminal, and the first motion measurement data may be data of at least some of the human body skeleton data of the user. By doing so, it is possible to alleviate the inclusion of unnecessary data in the first motion measurement data.

In addition, the second motion measurement data may include at least one of acceleration and angular velocity generated as the mobile terminal moves according to the motion.

Next, the display device 30 transmits the first motion measurement data, and the mobile terminal 40 transmits the second motion measurement data to the content synchronization server 20, respectively. This means that the content synchronization server 20 requests content synchronization. However, when the content synchronization server 20 receives the first and second motion measurement data separately, and when it is determined that the data for the same motion is determined as a result of the comparison of the received motion measurement data, .

However, the content synchronization server 20 can not know what contents should be synchronized with the mobile terminal 40. Therefore, the display device 30 transmits not only the first motion measurement data but also information indicating the content being reproduced in the display device 30 to the content synchronization server 20. [

The "information indicating the content being played back on the display device 30" may be, for example, identification information of the content. The identification information may be an identifier that can be recognized by the content server 25 or a URL (Uniform Resource Locator) that can access the content stored in the content server 25.

Meanwhile, since the display device 30 and the mobile terminal 40 recognize one motion at the same time, the display times of the display device 30 and the mobile terminal 40 are substantially the same. Therefore, by using this point, the content synchronization server 20 can remove the noise by excluding the measurement data for the same motion occurring at different points of time in the motion matching.

Accordingly, the mobile terminal 40 can generate the time stamp at the time of recognizing the motion, transmit the time stamp to the content synchronization server 20 together with the second motion measurement data, It is also possible to generate a time stamp at a time point at which the motion is recognized and transmit the time stamp to the content synchronization server 20 together with the first motion measurement data.

Not only time but also location coincidence can be used for noise reduction. Since the mobile terminal 40 is in the same position as the user 10 and the area in which the display device 30 can recognize the motion is not so wide, the mobile terminal 40 and the display device 30 are substantially in the same place It is expected to be. Therefore, in the content synchronization server 20, even if the motion measurement data indicating the same motion is received from the mobile terminal and the display device at the same time, if the positions of the mobile terminal and the display device are different from each other, will be.

The mobile terminal 40 can transmit the position information of the time point at which the motion is recognized to the content synchronization server 20 together with the second motion measurement data in order to utilize the coincidence of the place in the noise removal, The controller 30 may also transmit the position information at the time of recognizing the motion to the content synchronization server 20 together with the first motion measurement data. Unlike the mobile terminal 40, the display device 30 generally has no positional shift. Using this point, the display device 30 may transmit the identification information of the display device to the content synchronization server 20 instead of the location information. If the location information of each display device is stored in advance in the content synchronization server 20, the content synchronization server 20 stores the location information received from the mobile terminal 40 in the location of the pre-stored display device 30 It will be comparable.

The location information of the mobile terminal 40 may include at least one of GPS information, location information using a Wifi Positioning System (WPS), and mobile communication network cell identification information. In the case of the display device 30, since the position is fixed, unlike the mobile terminal 40, the preset position information will be transmitted.

The content synchronization server 20 compares the information received from the display device 30 with the information received from the mobile terminal 40 to determine whether the information received from both sides is a result of recognizing the same motion at the same time and place . A predefined error threshold can be used to determine the identity of time and place.

However, since the first motion measurement data is image side recognition data of motion and the second motion measurement data is motion side recognition data of motion, it is difficult to determine whether the measurement data is for the same motion by simple comparison. Thus, according to one embodiment, after converting the second motion measurement data estimate from the first motion measurement data or the first motion measurement data estimate from the second motion measurement data, Can be compared.

For example, analyzing the first motion measurement data, which includes image side recognition data of motion, e.g., data representing the motion of the human skeleton of the user 10, And the speed can be obtained. An expected value of the second motion measurement data including at least one of acceleration or angular velocity can be obtained by using the trajectory of the fingertip and its velocity. The content synchronization server 20 may compare the second motion measurement data with the second motion measurement data and determine that the motion matches if the difference is less than the threshold value.

Hereinafter, this will be described in more detail with reference to FIGS. 3A and 3B. FIG. 3A is a conceptual diagram of motion recognition on the side of the display device according to the present invention, and FIG. 3B is a conceptual diagram of motion recognition on the mobile terminal according to the present invention.

FIG. 3A is human body skeleton data measured by the sensor device 31 of the display device 30 when the user 10 performs a motion to move the hand downward from top to bottom. The human body skeleton data consists of a joint representing a joint and a skeletal edge between the joint. In the present invention, since only one arm motion is effective, only human body skeleton data of one arm is shown in FIG. And is included in the motion measurement data.

3A, the neck 11, the shoulder 12, the elbow 13, the wrist 14, and the fingertip 15 are joints. In the first motion of the motion, the elbow 13, the wrist 14 and the fingertip 15 are located almost horizontally with respect to the shoulder 12 in the second operation, In the third operation, the elbow 13, the wrist 14, and the fingertip 15 are lower than the shoulder 12. The image side recognition data identification of motion according to the present invention can be performed by identifying a pattern in which the relative position between joints changes.

3A is recognized through the motion sensor of the mobile terminal 40 as shown in FIG. 3B. For example, when the mobile terminal 40 is equipped with a sensor capable of obtaining x, y, and z axis acceleration, only the acceleration in one axial direction will be measured in the motion described with reference to FIG. 3A. This is because the user 10 will not change the gripping posture of the mobile terminal while the hand is descending from the top to the bottom. For example, when the user 10 holds the mobile terminal 40 in the direction shown in FIG. 3B, only the acceleration ( x ) in the x-axis direction has a significant value in the middle of the motion, The acceleration (? Y ,? Z ) will be recorded as almost zero.

That is, referring to FIG. 3A, it can be seen from the analysis of the first motion measurement data that the user 10 has performed a motion in which the user moves his hand from top to bottom, And the acceleration in the remaining two axial directions is expected to have almost zero.

Conversely, referring to FIG. 3B, it is expected that the user 10 has performed a motion in which the user moves his hand in one direction from the second motion measurement data.

In summary, the content synchronization server 20 calculates the second motion measurement data estimate from the first motion measurement data, compares the second motion measurement data estimate with the second motion measurement data, Determining that the motion matches if the first motion measurement data is equal to or less than the threshold value or calculating a movement pattern of the hand joint among the user's body skeleton data from the second motion measurement data, comparing the movement pattern of the hand joint with the first motion measurement data It can be determined that the motions are coincident when the movement patterns of the hand joints coincide with each other.

If it is determined that the motion is coincident with the first motion measurement data as described above or an estimate of the first motion measurement data is obtained from the second motion measurement data, It is not necessary to limit the motion for the motion detection to the predefined motion.

In addition, according to one embodiment, the content synchronization server 20 is provided with first motion measurement data feature information and second motion measurement data feature information, which are reference information for determining which motion the first and second motion measurement data represent, And the content synchronization server 20 can identify the motion indicated by the received first and second motion measurement data with reference to the first and second motion measurement data feature information.

In summary, the content synchronization server 20 stores first motion measurement data feature information and second motion measurement data feature information for one or more pre-assigned motion, and from the first motion measurement data feature information, Determine the motion indicated by the data, and determine the motion indicated by the second motion measurement data from the second motion measurement data feature information. In this case, the content synchronization is performed when the user performs the predetermined motion.

On the other hand, according to one embodiment, the first motion measurement data feature information for one or more predetermined motion is stored in the display device 30, and the display device 30 uses the first motion measurement data feature information Motion can be identified. In this case, the first motion measurement data may be the identification information of the motion determined by the display device 30. Also, the second motion measurement data feature information for one or more predefined motions may be provided to the mobile terminal 40 And the mobile terminal 40 may identify the motion itself by using the second motion measurement data feature information. In this case, the second motion measurement data may be the identification information of the motion determined by the mobile terminal 40.

If both the first motion measurement data and the second motion measurement data include the identification information of the motion identified by the display device 30 and the mobile terminal 40, the content synchronization server 20 generates two pieces of identification information It is possible to easily determine whether or not the motion is matched.

According to one embodiment, the first motion measurement data feature information may be data representing a sequential change in relative position between at least two of the hand joint, the wrist joint, the elbow joint and the shoulder joint.

According to one embodiment, the second motion measurement data feature information includes a graph of a change in acceleration magnitude for each axis along time, a graph of change in acceleration sum value for each axis according to time, A graph of variation of dispersion value, and the like.

1, the content synchronization server 20 determines whether the first motion measurement data and the second motion measurement data indicate the same motion as described above. If it is determined that the first motion measurement data and the second motion measurement data indicate the same motion, The identification information of the content received from the device 30 and the address information of the mobile terminal 40 to the content server 25 so that the content being reproduced in the display device 30 is transmitted to the mobile terminal 40 can do.

According to an embodiment, the content synchronization server 20 may transmit the content related to the content being reproduced in the display device 30 to the mobile terminal 40. For example, when the advertisement content is being reproduced on the display device 30, the discounted coupon content, which is related content of the advertisement content, may be transmitted to the mobile terminal 40. To this end, the content synchronization server 20 may store data for associated content matched for each content.

In addition, according to one embodiment, when it is determined that the first motion measurement data and the second motion measurement data indicate the same motion, the content synchronization server 20 can perform different operations according to the motion . For example, in the case of the first motion, the content being played back on the display device 30 can be transmitted to the mobile terminal 30 as it is. In the case of the second motion, the content related to the content being played back on the display device 30 is moved To the terminal 30. For example, when the user holds the mobile terminal 30 in his or her hand and takes a motion of lowering the arm from top to bottom, the content synchronization server 20 transmits the content being played back on the display device 30 to the mobile terminal 30 as it is If the user takes the motion of holding the mobile terminal 30 in the hand and raising the arm up from below, the content synchronization server 20 transmits the related content of the content being played back on the display device 30 to the mobile terminal 30, As shown in Fig.

The mobile terminal 40 can directly reproduce the content received from the content server 25 in a streaming manner. According to one embodiment, the mobile terminal 40 can play the content received from the content server 25 without any operation of the user.

The configuration of the content synchronization server 20 shown in FIG. 1 will be described later in detail with reference to FIG.

Next, a content synchronization system which is partially different from that shown in Fig. 1 will be described with reference to Fig.

The content synchronization system shown in FIG. 2 includes a content providing server 22, a display device 30, and a mobile terminal 40. The content providing server 22 may be an IPTV service server that stores content and provides a streaming service to the display device 30 or the mobile terminal 40.

1, the content providing server 22 knows the identification information of the content currently being played by the display device 30. [ Accordingly, when the motion of the user 10 is recognized, the display device 30 can transmit only the first motion measurement data without identification information of the currently playing content. The content providing server 22 can receive the first motion measurement data and identify the display device 30 from the sender address of the first motion measurement data and can display the content currently being streamed B) is also known.

The mobile terminal 40 of FIG. 2 recognizes the motion of the user 10 and transmits the second motion measurement data to the content providing server 22. The content providing server 22 compares the received first and second motion measurement data and transmits data for the content B to the mobile terminal 40 when it is determined to indicate the same motion.

Receiving the time stamp and position information from each of the mobile terminal 40 and the display device 30 means enhancing noise filtering, but is not an essential constitution of the present invention. 2 shows that the content providing server 22 does not receive the time stamp and the position information from the mobile terminal 40 and the display device 30, respectively.

The configuration of the content providing server 22 shown in FIG. 2 will be described later in detail with reference to FIG.

Next, a configuration of a content synchronization server according to an embodiment of the present invention will be described with reference to FIG. The content synchronization server 20 according to the present embodiment may further include a motion data storage unit 208 including a motion matching unit 204, a content providing unit 206 and a network interface 200. [ The network interface 200 may be connected to the display device 30, the mobile terminal 40, the content server 25, and the like through a network.

The network interface 200 receives the first motion measurement data according to the motion of the user and the identification information of the content being reproduced by the display device from the display device 30 having the sensor for sensing the motion of the user, And receives second motion measurement data from the user's mobile terminal 40 according to the user's motion.

The motion matching unit 204 compares the first motion measurement data with the second motion measurement data to determine whether or not the motion is matched. The motion matching unit 204 may determine whether or not the motion is matched in such a manner that an estimate of the second motion measurement data is obtained from the first motion measurement data or an estimate of the first motion measurement data is obtained from the second motion measurement data have.

The content providing unit 206 transmits the identification information of the content to the mobile terminal 40 or transmits the content to the mobile terminal 40 when the determination result motion is determined to match. When the identification information of the content is transmitted to the mobile terminal 40, the mobile terminal 40 can request the streaming of the content to the content server 25 to reproduce the content.

The motion matching unit 204 does not determine whether or not the motion is matched in such a manner that an estimate of the second motion measurement data is obtained from the first motion measurement data or an estimate of the first motion measurement data is obtained from the second motion measurement data The content synchronization server 20 may further include a motion data storage unit 208 for storing first motion measurement data feature information and second motion measurement data feature information for one or more predetermined motion. At this time, the motion matching unit 204 determines the motion indicated by the first motion measurement data from the first motion measurement data feature information, and extracts, from the second motion measurement data feature information, the motion indicated by the second motion measurement data And determines whether or not the motions indicated by the first and second motion measurement data coincide with each other.

FIG. 5 is a block diagram of the content providing server 22 shown in FIG. 2 of the present invention. 5, the content providing server 22 further includes a content storage unit 210 for storing content in addition to the motion data storage unit 208, the motion matching unit 204, and the content providing unit 206 do.

The network interface 200 of the content providing server 22 includes a sensor for sensing the motion of the user and receives the motion of the user from the display device 30 that downloads and reproduces the content stored in the content storage unit 210. [ And receives the second motion measurement data in accordance with the motion of the user from the user's mobile terminal 40. [

The motion matching unit 204 of the content providing server 22 compares the first motion measurement data and the second motion measurement data to determine whether or not the motion is coincident. The content providing server 206 of the content providing server 22 transmits the data of the content reproduced by the display device to the mobile terminal 40 when it is determined that the motion of the determination result matches. In addition, the motion data storage unit 208 stores the first motion measurement data feature information and the second motion measurement data feature information for one or more predetermined motion.

Up to this point, each of the components in FIGS. 4 and 5 may refer to software or hardware such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). However, the components are not limited to software or hardware, and may be configured to be in an addressable storage medium and configured to execute one or more processors. The functions provided in the components may be implemented by a more detailed component or may be implemented by a single component that performs a specific function by combining a plurality of components.

Hereinafter, a content synchronization method according to an embodiment of the present invention will be described with reference to FIG.

The content synchronization method according to the present embodiment is started by the display device playing the content (S10), and the user carrying out the motion with the mobile terminal (S12).

The motion is recognized by a display device by a visible ray photographing or an infrared ray reflection recognition method (S14), and is recognized by a mobile terminal by a method of acceleration, angular velocity, gravity direction measurement, etc. according to movement of the mobile terminal (S20) .

The display device transmits the first motion measurement data measured in accordance with the motion recognition, the currently reproduced content identification information, the position information, and the time stamp to the content synchronization server (S16). The mobile terminal also transmits the second motion Measurement data, location information of the mobile terminal, and time stamp to the content synchronization server (S22).

Next, the content synchronization server compares the first and second motion measurement data, the position information, and the time stamp to determine whether or not the motion is matched (S30). The content synchronization server determines whether the mobile terminal and the display device are in the same motion The mobile terminal transmits content data to the mobile terminal or transmits content identification information (e.g., a URL of the content) (S32) so that the mobile terminal can reproduce the content reproduced on the display device ( S34).

Hereinafter, various embodiments of the content synchronization server 20, the content providing server 22, or the content synchronization method according to the present invention will be described with reference to the signal flow diagrams shown in FIGS.

First, a content synchronization method performed by a content synchronization server according to an embodiment of the present invention will be described with reference to FIGS.

7, when the display device 30 starts playing the content (S100) and the user performs the motion in the state where the mobile terminal 40 executes the synchronization app app (S102) S104), content synchronization starts. The synchronization app is software for controlling the mobile terminal 40 to perform an operation of the mobile terminal 40 for the content synchronization method according to the present invention.

The user's motion is recognized by the display device 30 and the mobile terminal 40 at steps S106 and S110 and the display device 30 transmits the first motion measurement data and the content identification information to the content synchronization server 20 (S108), and the mobile terminal transmits the second motion measurement data or the like to the content synchronization server 20 (S112).

The content synchronization server 20 compares the first motion measurement data and the second motion measurement data according to the above-described method and determines whether or not the motion is consistent (S116). The content synchronization server 20 determines whether the data transmitted from the display device 30 or the mobile terminal 40 matches the motion, The mobile terminal 40 transmits the content identification information and the address information of the mobile terminal 40 to the mobile terminal 40 to request the mobile terminal 40 to transmit the content data to be reproduced in the display device 30 in step S120. As a result, the mobile terminal 40 receives the content data from the content server 25 (S121) and reproduces the content (S122).

8, when it is determined that the motion, the time, and the position of the data transmitted by the display device 30 and the mobile terminal 40 are identical to each other, the content synchronization server 20 transmits the data to the mobile terminal 40 And transmits the URL of the content. The URL of the content is received from the display device 30 (S108). The mobile terminal 40 accesses the URL of the content, requests the content stream (S126), receives the content streaming data (S127), and can reproduce the content (S128).

9 is a signal flow diagram of a content synchronization method performed by the content providing server 22 according to an embodiment of the present invention. As described above, the content providing server 22 transmits the content to the display device 30 (S98), and the display device 30 displays the content from the content providing server 22, for example, in a streaming manner The content providing server 22 can know what content the display device 30 is reproducing. Therefore, unlike the one shown in FIGS. 7 to 8, the display device 30 recognizing the motion of the user does not separately transmit the information about the content reproduced by the user to the content providing server 22.

9, when it is determined that the motion, the time, and the position of the data transmitted by the display device 30 and the mobile terminal 40 are coincident with each other, the content providing server 22 transmits the content To the mobile terminal 40 (S130).

10 is a signal flow diagram of a content synchronization method according to an embodiment of the present invention, in which a display device and a mobile terminal perform motion identification. 10, the first motion measurement data feature information for one or more predefined motions is stored in the display device 30, and the display device 30 uses the first motion measurement data feature information to identify itself The motion can be identified (S107). In this case, the first motion measurement data may be the identification information of the motion determined by the display device 30 (S114). In addition, the second motion measurement data feature information for one or more predefined motions is stored in the mobile terminal 40, and the mobile terminal 40 may identify the motion itself using the second motion measurement data feature information (S111). In this case, the second motion measurement data may be the identification information of the motion determined by the mobile terminal 40 (S115).

At this time, the content providing server 22 may compare the motion recognized by the display device 30 with the motion recognized by the mobile terminal 40 by comparing the first motion identification information with the second motion identification information .

11 is a signal flow diagram of a content synchronization method supporting a playback-time interworking function according to an embodiment of the present invention. The display device 30 according to the present embodiment transmits data for the content reproduction time point of the recognition time (for example, 25 minutes 30 seconds from the start of the content reproduction) to the content synchronization server 20 at the time of motion recognition (S106) (S109).

The content synchronization server 20 determines whether the content of the content transmitted from the content server 25 is identical to the content of the content transmitted to the mobile terminal 40 (S140). In step S140, the content server 100 transmits the streaming data to the display device 30, and transmits the streaming data. As a result, the mobile terminal 40 can reproduce the content by directly receiving the content reproduction point in the display device 30 (S142). According to another embodiment, when the content synchronization server 20 transmits the content URL to the mobile terminal 40, the content synchronization server 20 may transmit the received content playback point in addition to the content URL.

According to an embodiment, when the content is synchronized from the display device 30 to the mobile terminal 40 in the playback-time synchronized manner, the content playback of the display device 30 is performed at the time of playback of the content by the mobile terminal 40 Lt; / RTI > That is, the content synchronization server 20 may receive a playback start signal indicating that playback of the content is started from the mobile terminal 40, and may transmit a playback stop control signal for the content to the display device 30. [

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

Content synchronization server 20
Content server 25
Display device 30
The mobile terminal 40

Claims (15)

Receiving first motion measurement data according to a motion of the user and identification information of a content being reproduced by the display device from a display device having a sensor for sensing a motion of the user, A network interface for receiving second motion measurement data according to the first motion measurement data;
A motion matching unit for comparing the first motion measurement data and the second motion measurement data to determine whether or not the motion is matched; And
When it is determined that the motions match, transmitting identification information of the content or related content of the content to the mobile terminal, or requesting the content server to transmit the content or the related content of the content to the mobile terminal And a content providing unit for receiving the content from the server.
The method according to claim 1,
Wherein the first motion measurement data includes data extracted from the motion image of the user taking the motion,
Wherein the second motion measurement data includes at least one of acceleration and angular velocity generated as the mobile terminal moves according to the motion.
3. The method of claim 2,
The motion means movement of a user's hand holding the mobile terminal,
Wherein the first motion measurement data is at least a portion of human body skeleton data of the user.
The method of claim 3,
The motion matching unit may calculate the second motion measurement data estimate from the first motion measurement data, compare the second motion measurement data estimate with the second motion measurement data, and if the difference is less than a predetermined threshold, The content synchronization server determines that the content is to be downloaded.
The method of claim 3,
Wherein the motion matching unit calculates a movement pattern of the hand joint in the user's body skeleton data from the second motion measurement data, compares the movement pattern of the hand joint with the first motion measurement data, And determines that the motion matches.
The method according to claim 1,
Further comprising: a motion data storage for storing first motion measurement data feature information and second motion measurement data feature information for one or more predefined motion,
Wherein the motion matching unit determines a motion indicated by the first motion measurement data from the first motion measurement data feature information and determines a motion indicated by the second motion measurement data from the second motion measurement data feature information, 1 < / RTI > and 2 < th > motion measurement data.
The method according to claim 6,
Wherein the first motion measurement data is human body skeleton data of the user,
Wherein the first motion measurement data feature information is data representative of a sequential change in relative position between at least two of the hand joint, the wrist joint, the elbow joint, and the shoulder joint.
The method according to claim 1,
Wherein the network interface comprises:
Further receiving a time stamp of a time at which the motion is sensed from the display device and position information of the display device and further receiving a time stamp of a time at which the motion is sensed and position information of the mobile terminal from the mobile terminal,
The motion-
And compares the first motion measurement data and the second motion measurement data, the difference between the time stamp and the position information being within a predefined limit.
The method according to claim 1,
Wherein the first motion measurement data is identification information of a motion determined by the display device.
The method according to claim 1,
Wherein the second motion measurement data is identification information of a motion determined by the mobile terminal.
The method according to claim 1,
The content providing apparatus,
Requesting the content server to transmit the content to the mobile terminal or transmit the content to the mobile terminal when the determined motion is the first motion, and when the determined motion is the second motion, And transmits the content to the mobile terminal or requests the content server to transmit the content associated with the content to the mobile terminal.
The method according to claim 1,
Wherein the network interface further receives information on the playback time of the content being played back by the display device from the display device,
Wherein the content providing unit requests the content server to transmit, from the data of the content according to the identification information, data necessary for jumping to the reproduction time point received from the display device and reproducing the data.
13. The method of claim 12,
Wherein the network interface comprises:
And receives a playback start signal indicating that playback of the content has started from the mobile terminal and transmits a playback stop control signal for the content to the display device.
A content storage unit for storing content;
A method for receiving first motion measurement data in response to a motion of a user from a display device having a sensor for detecting motion of a user and downloading and reproducing contents stored in the content storage unit, A network interface for receiving second motion measurement data in accordance with the motion of the first network;
A motion matching unit for comparing the first motion measurement data and the second motion measurement data to determine whether or not the motion is matched; And
And a content providing unit that transmits data of the content reproduced by the display device to the mobile terminal when it is determined that the motions match.
A display device having a sensor for detecting a motion of a user recognizes the motion of the user and generates first motion measurement data according to the motion of the user, and the first motion measurement data and the content To the content synchronization server;
Generating second motion measurement data according to the user's motion and transmitting the second motion measurement data to the content synchronization server, the user's mobile terminal having a sensor for detecting motion of the mobile terminal;
Comparing the first motion measurement data with the second motion measurement data to determine whether the content is synchronized with the motion;
If the result of the determination indicates that the motions match, the content synchronization server transmits data of content according to information indicating the content to the mobile terminal; And
And receiving the content data from the content synchronization server and playing the content.
KR1020130013170A 2013-02-06 2013-02-06 Contents synchronization server and method thereof using motion matching KR101560673B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020130013170A KR101560673B1 (en) 2013-02-06 2013-02-06 Contents synchronization server and method thereof using motion matching
PCT/KR2013/010860 WO2014123299A1 (en) 2013-02-06 2013-11-27 Content synchronization server and content synchronization method using motion matching

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020130013170A KR101560673B1 (en) 2013-02-06 2013-02-06 Contents synchronization server and method thereof using motion matching

Publications (2)

Publication Number Publication Date
KR20140100627A true KR20140100627A (en) 2014-08-18
KR101560673B1 KR101560673B1 (en) 2015-10-16

Family

ID=51299870

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020130013170A KR101560673B1 (en) 2013-02-06 2013-02-06 Contents synchronization server and method thereof using motion matching

Country Status (2)

Country Link
KR (1) KR101560673B1 (en)
WO (1) WO2014123299A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101660503B1 (en) 2015-05-19 2016-10-10 주식회사 호서텔넷 Data processing device performing application according to position

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101937040B1 (en) 2017-07-20 2019-01-09 윤흥식 Online method of estimating a user's quantity of motion

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4875652B2 (en) * 2008-03-31 2012-02-15 三菱重工業株式会社 Action recognition system and method
KR20100062548A (en) * 2008-12-02 2010-06-10 에스케이 텔레콤주식회사 Game service system for providing online game of sensory type and method thereof
KR101089642B1 (en) * 2010-02-26 2011-12-06 (주)모션블루 Method and Recording Media for Providing 3-Screen Service
KR101212893B1 (en) * 2010-06-04 2012-12-14 삼성에스디에스 주식회사 Advertisement system, and method for displaying advertisement
KR20120070638A (en) * 2010-12-22 2012-07-02 (주)로이비즈 Smart phone application service system by motion control type for controlling pc game and game method providing by system thereof

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101660503B1 (en) 2015-05-19 2016-10-10 주식회사 호서텔넷 Data processing device performing application according to position

Also Published As

Publication number Publication date
WO2014123299A1 (en) 2014-08-14
KR101560673B1 (en) 2015-10-16

Similar Documents

Publication Publication Date Title
CN110199321B (en) Positioning determination for mixed reality systems
CN104054039B (en) Augmented Reality With Sound And Geometric Analysis
US20120198353A1 (en) Transferring data using a physical gesture
US20140028539A1 (en) Anatomical gestures detection system using radio signals
CN103677259B (en) For guiding the method for controller, multimedia device and its target tracker
US7936255B2 (en) Group analysis system and group analysis equipment
US20110298710A1 (en) Hand-held pointing device, software cursor control system and method for controlling a movement of a software cursor
CN102542247A (en) Information processing device, information processing method, and program
CN108600612B (en) Photographing method and mobile terminal
CN105808542B (en) Information processing method and information processing apparatus
KR20120017293A (en) Apparatus and method for providing augmented reality
CN103945109A (en) Image pickup apparatus, remote control apparatus, and methods of controlling image pickup apparatus and remote control apparatus
US9292963B2 (en) Three-dimensional object model determination using a beacon
KR101560673B1 (en) Contents synchronization server and method thereof using motion matching
CN112188105A (en) Tracking shooting method and device, intelligent device and computer readable storage medium
US11875080B2 (en) Object sharing method and apparatus
US9667946B2 (en) Method for authenticating an image capture of a three-dimensional entity
CN106370883B (en) Speed measurement method and terminal
US9715628B2 (en) Method for estimating a distance from a first communication device to a second communication device, and corresponding communication devices, server and system
KR101542477B1 (en) Method, apparatus, and terminal device for generating and processing information
JP4515510B2 (en) Recording management system, recording management method, and recording management program
KR101273634B1 (en) Tracking Method of Multiple Objects using Mobile Device in Augumented Reality Environment and System Using the same
US20220207585A1 (en) System and method for providing three-dimensional, visual search
KR101480300B1 (en) Interactive contents provision system and method using relative distance between the main screen and the second screen
KR102010782B1 (en) Method for correcting position of controller into recognizable range of multimedia device and the multimedia device therefor and target tracking device

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
AMND Amendment
E601 Decision to refuse application
AMND Amendment
X701 Decision to grant (after re-examination)
GRNT Written decision to grant
FPAY Annual fee payment

Payment date: 20180927

Year of fee payment: 4

FPAY Annual fee payment

Payment date: 20191001

Year of fee payment: 5