KR102187661B1 - Method and System for Playing Augmented Reality Photograph - Google Patents

Abstract

Disclosed are a method and system for reproducing an AR photo. An embodiment of the present invention provides the AR photo reproducing method and a device, which edits a photographed video and outputs an AR photo including a marker corresponding to a representative image selected in the video, wherein when the AR photo is photographed by a camera, the video corresponding to the marker is reproduced in an AR photo area.

Description

AR photo playback method and system {Method and System for Playing Augmented Reality Photograph}

An embodiment of the present invention relates to a method and system for reproducing AR photos.

The contents described below merely provide background information related to the present embodiment and do not constitute the prior art.

As the performance of smartphones and tablets improves and spreads, video shooting functions are widely used. With the development of information and communication, various types of information are obtained and processed using a computer, and data is provided using a public communication network from a computer connected to a LAN or the Internet.

In general, image information stored in memory cards of electronic devices is moved to a personal computer (PC) and then stored in order to store images taken with a smartphone or tablet for a long time.

In order to view the video moved to the personal computer (PC), the user must find the file stored in the personal computer (PC) and play it manually. In order to find and play a video stored in a personal computer (PC), a file name or directory must be memorized. If you don't remember the video file name or directory, there is a hassle of having to check and play many videos one by one.

Data can be exchanged between different computers connected via a network, and users can exchange various types of information including text, images, and videos.Videos are difficult to receive because of their large capacity, and There is a problem that it is difficult for a general user to transmit through the device.

This embodiment edits the captured video and outputs an AR photo including a marker corresponding to a representative image selected in the video. When an AR photo is taken with a camera, the video corresponding to the marker is played back in the AR photo area. It is an object of the present invention to provide a method and system for reproducing AR photos.

According to an aspect of the present embodiment, a marker for identifying a selected representative frame among a plurality of frames included in image data is generated, a representative image is generated by combining the representative frame and the marker, and the representative A video authoring terminal requesting to print an image as an output; A print for outputting the representative image as an Augmented Reality Photograph; When the AR photo is illuminated using the provided camera module, a code corresponding to the marker is extracted after recognizing the marker included in the AR photo, and when the code and the AR photo image illuminating the AR photo are authenticated And a video reproducing terminal configured to receive image data corresponding to the code from a server and to play the image data corresponding to the code overlaid on an area in which the AR photo exists in an output screen; And an AR photo playback server that checks whether the code received from the video playback terminal exists, and if the code exists as a result of the verification, extracts the video data matching the code and transmits the extracted video data to the video playback terminal. It provides an AR photo reproduction system, characterized in that.

As described above, according to the present embodiment, the captured video is edited, and an AR photo including a marker corresponding to a representative image selected in the video is output. When an AR photo is taken with a camera, the AR photo area There is an effect that the video corresponding to the marker is played.

According to this embodiment, when reproducing an AR photo, since a video corresponding to the marker is played only when a marker exists and a pre-stored image corresponding to the marker exists, there is an effect of enabling video distribution requiring security. have.

1 is a block diagram schematically illustrating an AR photo reproducing system according to the present embodiment.
2 is a diagram illustrating a process of selecting a captured video according to the present embodiment.
3A and 3B are diagrams illustrating a process of editing a captured video and determining a representative image according to the present embodiment.
4 is a diagram illustrating a process of printing a representative image of a selected video as an AR photo according to the present embodiment.
5 is a diagram illustrating a process of outputting a representative image as an AR photo according to the present embodiment.
6 is a diagram illustrating a process of reproducing an image in a corresponding region when an AR photo according to the present embodiment is photographed with a camera.
7 is a diagram showing a public AR photo and a private AR photo according to the present embodiment.
8 is a diagram illustrating a pin function for fixing to a screen when playing back an image corresponding to an AR photo according to the present embodiment.
9 is a diagram showing charge according to an image length corresponding to an AR photo according to the present embodiment.
10 is a diagram showing a report function for video data according to the present embodiment.
11 is a conceptual diagram showing a method of generating a marker according to the present embodiment.
12 is a diagram showing a method of matching an identification value without a marker according to the present embodiment.

Hereinafter, this embodiment will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically illustrating an AR photo reproducing system according to the present embodiment.

The AR photo reproducing system according to this embodiment includes an AR photo reproducing application 110, a terminal 120, an AR photo reproducing server 130, a printer 140, and an AR photo 150. Components included in the AR photo reproduction system are not necessarily limited thereto.

The AR photo playback application 110 refers to a program mounted and driven in the memory of the terminal 120. The AR photo playback application 110 may be mounted in an embedded form on the terminal 120 or installed in an operating system (OS).

The AR photo playback application 110 is preferably mounted on the terminal 120 and operated using various hardware provided in the terminal 120, but is not limited thereto, and may be implemented and operated as a separate device. . The AR photo playback application 110 may operate in conjunction with applications previously installed in the terminal 120.

The AR photo playback application 110 according to the present embodiment is mounted on the terminal 120 to generate a marker for identifying a representative frame for image data, and to generate a representative image by combining the marker to the representative frame. It can be used for video authoring. The AR photo reproduction application 110 requests the printer 140 to print a representative image as an output.

The AR photo playback application 110 is mounted on the terminal 120, and when the AR photo 150 is illuminated with a camera module provided in the terminal 120 to play a video corresponding to the AR photo 150, the AR photo Recognize the marker included in (150). The AR photo reproduction application 110 extracts a code corresponding to the marker and then transmits the code corresponding to the marker and the AR photo image taken of the AR photo 150 to the AR photo reproduction server 130. The AR photo playback application 110 receives image data corresponding to the AR photo image from the AR photo playback server 130, and then displays the image data corresponding to the AR photo image in the area where the AR photo 150 exists in the output screen. Overlay and play.

The terminal 120 refers to an electronic device that performs voice or data communication via a network according to a user's key manipulation. The terminal 120 includes a memory for storing a program or protocol for communicating with the AR photo reproducing server 130 via a network, and a microprocessor for calculating and controlling by executing the program.

The terminal 120 may be an electronic device such as a smart phone, a tablet, a laptop, a personal computer, and smart glasses.

The terminal 120 includes (i) a communication device such as a communication modem for performing communication with various devices or wired/wireless networks, (ii) a memory for storing various programs and data, and (iii) calculating and controlling by executing a program. It is a variety of devices including a microprocessor for. According to at least one embodiment, the memory is a computer such as a random access memory (RAM), a read only memory (ROM), a flash memory, an optical disk, a magnetic disk, or a solid state disk (SSD). It may be a readable recording/storing medium. According to at least one embodiment, the microprocessor may be programmed to selectively perform one or more of the operations and functions described in the specification. According to at least one embodiment, the microprocessor may be implemented entirely or partially as hardware such as an Application Specific Integrated Circuit (ASIC) having a specific configuration. The AR photo playback application 110 is mounted on the memory of the terminal 120.

The terminal 120 according to the present embodiment is a video authoring terminal, which loads a video captured by using the AR photo playback application 110 and edits the video. The terminal 120 selects a specific frame from the edited video using the AR photo playback application 110 and outputs it to the printer 140.

In other words, the terminal 120 generates a marker for identifying a representative frame for image data using the AR photo playback application 110. The terminal 120 generates a representative image by combining a marker with the representative frame, and requests to print the representative image as an output.

The terminal 120 is a video playback terminal, and when the AR photo 150 output from the printer 140 is projected by the camera using the AR photo playback application 110, the AR photo ( 150) is played and output.

The terminal 120 recognizes a marker included in the AR photo 150 when the AR photo 150 is illuminated with a camera module provided using the AR photo playback application 110. The terminal 120 extracts the code corresponding to the marker, and when both the code and the AR photo image are authenticated, the AR photo reproducing server 130 receives the image data corresponding to the AR photo 150 and then displays the AR in the output screen. The image data is overlaid on the area where the photo 150 is present and reproduced.

Here, the marker is a kind of 2D barcode. When the AR photo 150 is captured, the terminal 120 recognizes a marker included in one side of the AR photo 150 (eg, below the AR photo 150) and reads a code corresponding to the marker.

The terminal 120 recognizes a marker included in the AR photo when the AR photo is illuminated using an equipped camera module. The terminal 120 extracts a code corresponding to the marker and transmits the code to the AR photo reproduction server 130. The terminal 120 downloads the representative image matched with the code from the AR photo reproducing server 130.

The terminal 120 checks whether the AR photo image illuminated by the AR photo 150 and the representative image matched with the code match. When it is determined that the AR photo image and the representative image match, the terminal 120 receives image data corresponding to the code from the AR photo reproducing server 130. In other words, the terminal 120 extracts the feature points from the AR photo image, extracts the feature points from the representative image matched with the code, compares them with each other, and then the AR photo reproducing server 130 only when the feature points match by more than a preset threshold. Video data corresponding to the code is received from The terminal 120 allows the image data corresponding to the code to be overlaid on the area in which the AR photo 150 exists in the output screen to be reproduced.

When the video reproducing terminal 120 receives image data corresponding to the AR photo 150 from the AR photo reproducing server 130, the video data is overlaid on an area corresponding to the AR photo 150 and reproduced. When there is an input for a preset time on one side of the screen that outputs the image data corresponding to the AR photo 150, the terminal 120 uses the pin function to display image data that was only output to the area where the AR photo 150 exists. Is fixed to a preset area of the screen and played.

When there is a preset input (twice consecutive tap) for the output screen, the terminal 120 switches to output the image data corresponding to the AR photo 150 in full screen.

After recognizing the marker from the AR photo 150, the terminal 120 extracts the coordinates of the marker on the output screen, and then determines the area where the AR photo 150 exists as an image reproduction area based on the coordinates of the marker. The terminal 120 overlays the image data in the image reproduction area based on the coordinates of the marker while tracking the coordinates of the marker on the output screen.

The AR photo reproducing server 130 includes a hardware module identical to a typical web server or network server in terms of hardware. The AR photo reproducing server 130 may be implemented in the form of a web server or a network server. The AR photo reproducing server 130 generally communicates with a plurality of unspecified clients or other servers through an open computer network such as the Internet.

The AR photo reproducing server 130 refers to a computer system and computer software (web server program) that derives and provides a work result corresponding to a request for performing work from a client or another web server. The AR photo reproducing server 130 includes a series of application programs operating on a web server or various databases built in the device. The AR photo reproduction server 130 provides a payment service in connection with an authentication system or a financial institution payment system.

The AR photo reproducing server 130 according to the present embodiment plays a video using an augmented reality (AR) that retrieves digital information for a real object matched to the AR photo 150 at the request of the terminal 120.

The AR photo reproducing server 130 basically allows the video data matched with the personal user account and the AR photo 150 to be played for individual consumers, and when the AR photo 150 is applied to advertisements disclosed in newspapers, magazines, magazines, etc. Video data matched with the advertisement product is played.

The AR photo reproducing server 130 may basically share an individual user account and digital information matched with the AR photo 150 online for individual consumers. The AR photo reproducing server 130 matches and stores a marker and image data with the representative image received from the terminal 120.

When the marker received from the terminal 120 is a marker sold for business, the AR photo reproducing server 130 extracts the code corresponding to the marker and then assigns the code to the user account that reads the marker first, so that only the buyer responds to the marker. Control to play the video to be played.

The AR photo reproducing server 130 receives a code corresponding to a marker and an AR photo image from the terminal 120 that plays the video. The AR photo reproducing server 130 checks whether a code corresponding to a marker received from the terminal 120 for reproducing a video exists. If the code exists as a result of the verification, the AR photo reproduction server 130 extracts a representative image matched with the code. When the AR photo reproducing server 130 matches the AR photo image and the representative image, the AR photo reproducing server 130 extracts image data matched with the representative image and transmits the extracted image data to the video reproducing terminal 120.

The AR photo reproduction server 130 extracts feature points from an AR photo image, extracts feature points from a representative image, and compares them with each other.

As a result of the comparison, only when the feature points of the AR photo image and the feature points of the representative image match more than a preset threshold, the AR photo reproducing server 130 extracts image data corresponding to the representative image and transmits it to the video playback terminal 120. .

When a new upload request is received from the video authoring terminal 120, the AR photo reproducing server 130 checks whether the representative image newly requested to be uploaded and the same image already registered exist. As a result of the check, if the representative image newly requested to be uploaded and the same image already registered exist, the AR photo playback server 130 checks whether the code matched to the same image and the code matched to the newly requested representative image are the same. Confirm. As a result of the confirmation, when the code matched to the representative image newly requested to be uploaded and the code matched to the same image are different from each other, the AR photo reproducing server 130 matches and registers a different code to the representative image newly requested to be uploaded.

The AR photo reproducing server 130, when the AR photo image and the representative image do not match, when the AR photo image matches any one frame among a plurality of frames included in the image data matched with the code, the image matched with the code Data may be transmitted to the video playback terminal 120.

When the code received from the terminal 120 is a code representing a public marker, the AR photo reproducing server 130 transmits image data corresponding to the public code to the terminal 120 in a non-login state or a login state. do.

When the code received from the terminal 120 is a code representing a private marker, the AR photo reproducing server 130 transmits image data corresponding to the private code to the terminal 120 in the logged-in state.

The AR photo reproducing server 130 checks the user account that played the video data corresponding to the private code by using the login information to determine the delivery path of the private AR photo including the private code.

The AR photo reproducing server 130 grants replay permission to each user account in order to replay the image data corresponding to the private code. When the terminal 120 having a user account granted permission to play video data corresponding to the private code illuminates the private AR photo including the private code, the AR photo reproducing server 130 is a user account to which the play permission is granted. Requests an additional code to the terminal 120 having a.

The AR photo reproducing server 130 plays the video data corresponding to the private code only when the additional code input from the terminal 120 having the user account to which the reproduction permission is granted is authenticated, the terminal having the user account ( 120). When the image data in the album is deleted, the AR photo reproducing server 130 deletes all user account information to which the reproduction permission has been granted.

The terminal 120 may transmit information on identification information (user account) of the terminal together when requesting to play image data corresponding to the private marker. The AR photo reproduction server 130 may additionally check the image reproduction permission for the user account and transmit image data corresponding to the private marker to the terminal 120.

When the AR photo reproducing server 130 matches and uploads the image data to the marker, the free image and the paid image are classified and charged according to the length of the image data. The AR photo reproducing server 130 charges a fee according to the video length corresponding to the AR photo 150.

When the length (running time) of the image data requested to be uploaded is within a preset first threshold time (eg, 15 seconds), the AR photo reproduction server 130 divides the image data into free images and uploads them. The AR photo reproducing server 130 enables video data to be uploaded for free when viewing advertisements on the terminal 120.

The AR photo reproducing server 130 transmits a free video to the terminal 120 when viewing an advertisement on the terminal 120.

When the length (running time) of the image data requested to be uploaded exceeds a preset second threshold time (eg, 1 minute), the AR photo playback server 130 divides the image data into a paid image and uploads the image. . The AR photo reproducing server 130 may upload an image by charging a paid coin on the image data classified as a paid image in a preset time unit (eg, 30 seconds). When receiving and playing a paid image from the terminal 120, the AR photo reproducing server 130 consumes paid coins (eg, 1 coin per 30 seconds) held in a preset threshold unit (eg, in units of 30 seconds).

The printer 140 receives data corresponding to the AR photo 150 including a marker from the terminal 120. The printer 140 outputs data corresponding to the AR photo 150 as an output object.

2 is a diagram illustrating a process of selecting a captured video according to the present embodiment.

The terminal 120 stores an image captured by using a camera module provided, or generates a 3D object image using a 3D object using a separate module. As shown in FIG. 3, the terminal 120 loads a captured video or a generated 3D object image using the AR photo playback application 110.

The terminal 120 outputs a video captured using the AR photo playback application 110 or a generated 3D object image as a video list on one screen. The terminal 120 selects and plays one of the video lists using the AR photo playback application 110. A video selected from the video list is played on one side of the screen of the terminal 120, and a video frame is displayed on the other side of the screen.

3A and 3B are diagrams illustrating a process of editing a captured video and determining a representative image according to the present embodiment.

The terminal 120 generates image data by editing the length of the captured image or 3D object image using the AR photo playback application 110. When the terminal 120 selects any one of a plurality of frames included in the image data as the representative frame, it generates a marker for identifying the representative frame. In other words, the terminal 120 generates a marker in the form of a two-dimensional barcode including an identifier for the representative frame.

The terminal 120 generates a representative image by combining the representative frame and the marker. In other words, the terminal 120 generates a representative image including a marker in the representative frame, matches the representative image with the image data, and uploads it to the AR photo reproduction server 130.

The AR photo reproducing server 130 matches and stores a representative image corresponding to a code in an individual account, and matches and stores image data with the representative image. The AR photo reproduction server 130 assigns an identifier according to the length of the image data and stores it together with a code for identifying the marker.

As shown in FIG. 3A, the terminal 120 edits a video captured by using the AR photo playback application 110 or a 3D object image generated by adjusting the length, and then assigns an identifier to the corresponding part.

The terminal 120 uploads the edited video using the AR photo playback application 110 to the AR photo playback server 130. The AR photo playback server 130 matches and uploads the edited video to a personal account.

As shown in FIG. 3B, when the terminal 120 uploads the edited video to the AR photo reproducing server 130, the terminal 120 selects a representative frame from among the edited video. The terminal 120 generates a marker for identifying the representative frame using the AR photo playback application 110. The terminal 120 generates a representative image in which the marker is combined with the representative frame using the AR photo playback application 110. When the terminal 120 selects a representative frame among a plurality of frames included in the image data, a basic image (the first screen or a screen having a preset playback time) may be automatically selected.

When the terminal 120 generates a representative image of the edited video and then registers it with the AR photo playback server 130, the terminal 120 displays a marker even if a duplicate image (ie, the same image) already exists in the AR photo playback server 130. Registration is possible when different codes for identification are matched.

4 is a diagram illustrating a process of printing a representative image of a selected video as an AR photo according to the present embodiment.

As shown in FIG. 4, the terminal 120 outputs a recorded video or a list of edited videos obtained by editing the generated 3D object image. The edited video list allows the representative image selected from the edited video to be output in the form of a thumbnail.

When any one is selected from the list of edited videos output in the form of thumbnails using the AR photo playback application 110, the terminal 120 outputs a representative image corresponding to the selected edited video, and outputs a print menu for the representative image. do. If it is desired to change the representative image, the terminal 120 may check the frame of the edited video using the AR photo playback application 110 and change the representative image.

5 is a diagram illustrating a process of outputting a representative image as an AR photo according to the present embodiment.

The terminal 120 outputs the representative image corresponding to the selected edited video using the AR photo playback application 110, and when there is an input to the print menu for the representative image, the printer 140 displays the representative image. Request output for In this case, as the marker is included on the representative image, the terminal 120 requests the printer 140 to output the representative image including the marker.

The printer 140 outputs an AR photo 150 printed on paper or photographic paper as an output object of a representative image including a marker. When the terminal 120 wants to share the representative image including the marker with the other party, the terminal 120 may transmit the representative image including the marker to the other party by using a message menu or a mail menu. The counterpart terminal that has received the representative image including the marker may also print the representative image and output the AR photo 150.

6 is a diagram illustrating a process of reproducing an image in a corresponding region when an AR photo is captured by a camera according to the present embodiment.

The terminal 120 recognizes a marker included in the AR photo displayed on the screen when the AR photo 150 is illuminated by a camera module provided using the AR photo playback application 110.

The terminal 120 downloads an image corresponding to the marker from the AR photo reproducing server 130, and if the previously stored image matches the image corresponding to the marker, the terminal 120 receives the image data from the AR photo reproducing server 130 and The image data is overlaid on the area corresponding to the AR photo that is output to and reproduced.

The terminal 120 extracts feature points from an image corresponding to the marker. The terminal 120 extracts feature points from a pre-stored image, compares them with each other, and receives image data corresponding to the marker from the AR photo reproducing server 130 only when the feature points match by more than a preset threshold.

When the terminal 120 illuminates the AR photo 150 including the marker while the camera module is running, the marker included in the AR photo 150 is recognized. The terminal 120 extracts a code value (a kind of identification code) corresponding to the marker.

The terminal 120 checks whether an image corresponding to the code value exists from the AR photo reproducing server 130. As a result of checking, if an image corresponding to the code value exists, the terminal 120 downloads the image corresponding to the code value from the AR photo reproducing server 130.

The terminal 120 authenticates the image based on the downloaded image. In other words, the terminal 120 verifies whether the code of the downloaded image and the previously stored image match. As a result of the verification, if the code of the downloaded image and the pre-stored image match, the terminal 120 loads the image or 3D object matched to the image from the AR photo reproducing server 130. The terminal 120 outputs an image or 3D object by overlaying it on an area corresponding to the AR photo 150.

The terminal 120 uses the AR photo reproducing application 110 to authenticate the image matching with the marker, not only the marker in the AR photo 150. Since the representative image is one frame in the image and a plurality of frames are included in the image, the AR photo reproduction server 130 checks whether or not the representative image exists in the plurality of frames in the image based on the marker.

The AR photo reproducing server 130 loads a file corresponding to the marker. The AR photo reproducing server 130 loads the tracking image in the corresponding file and transmits it to the terminal 120.

The terminal 120 checks whether the tracking image downloaded from the AR photo reproducing server 130 and the image pre-stored in the terminal 120 match.

As a result of the comparison, when the previously stored image and the tracking image match, the terminal 120 causes the video corresponding to the AR photo 150 to be played. As a result of the comparison, if the previously stored image and the tracking image are different, the terminal 120 controls the video corresponding to the AR photo 150 to not be played.

If the AR photo reproducing server 130 receives the code value but there is no image corresponding to the code value, no image is played. The AR photo reproducing server 130 transmits an image corresponding to the code value to the terminal 120. When the terminal 120 finds a scene matching the image received from the AR photo reproducing server 130, the terminal 120 plays the image corresponding to the AR photo 150.

In another embodiment, when the terminal 120 illuminates the AR photo 150 including the marker while the camera module is running, the marker included in the AR photo 150 is recognized. The terminal 120 extracts a code (a kind of identification code) corresponding to the marker. The terminal 120 transmits a code (a kind of identification code) and an AR photo image corresponding to the marker to the AR photo reproduction server 130.

The AR photo reproducing server 130 primarily checks whether the code corresponding to the marker matches a previously stored code. As a result of checking, if the code corresponding to the marker matches the previously stored code, the AR photo reproducing server 130 secondaryly extracts a pre-stored representative image corresponding to the marker.

The AR photo reproducing server 130 compares the AR photo image received from the terminal 120 with a pre-stored representative image to determine whether they match. When the AR photo image and the representative image match, the AR photo reproducing server 130 extracts pre-stored image data corresponding to the marker.

The AR photo reproducing server 130 performs secondary authentication based on the AR photo image received from the terminal 120. In other words, the AR photo reproducing server 130 verifies whether the AR photo image received from the terminal 120 matches the previously stored representative image. As a result of the verification, if the AR photo image and the previously stored representative image match, the AR photo reproducing server 130 loads the image or 3D object matched with the representative image. The AR photo reproduction server 130 transmits an image or 3D object matched to the representative image to the terminal 120.

The terminal 120 outputs an image or 3D object by overlaying it on an area corresponding to the AR photo 150. As a result of the comparison, if the AR photo image and the previously stored representative image do not match, the AR photo reproducing server 130 controls the video corresponding to the AR photo 150 to not be played. That is, even if the code corresponding to the marker is copied and leaked, if the same image as the representative image stored in the AR photo playback server 130 is not leaked, the risk of hacking is low because the video cannot be played.

If a real photo is lost or a pre-stored image is leaked and the video itself is to be deleted, if the video is deleted from the album of the AR photo playback server 130, the video itself is deleted.

When the AR photo reproduction server 130 receives the code corresponding to the marker from the terminal 120, it extracts a representative image corresponding to the code.

When the representative image corresponding to the code does not exist but the image data corresponding to the code exists, the AR photo reproducing server 130 includes a plurality of AR photo images received from the terminal 120 included in the image data corresponding to the code. If it matches any one of the frames of, the image data corresponding to the code is transmitted to the terminal 120.

7 is a diagram showing a public AR photo and a private AR photo according to the present embodiment.

When a counterpart terminal registers a user account in the AR photo reproducing server 130, the user can set the video to be played only to the selected account user.

When the AR photo reproducing server 130 sells a celebrity video card for business, only the card purchaser can play the video. That is, only the card buyer can play the celebrity video by linking with the purchase account. The code is attributed to the user account that reads the code first. For example, it is possible to pass a picture to another person and identify the route of picture delivery.

The AR photo 150 can be classified into a public AR photo 810 or a private AR photo 820. The public AR photo 810 includes a public marker, and the private AR photo 820 includes a private marker.

When receiving a code value corresponding to a public marker from the terminal 120, the AR photo reproduction server 130 extracts a public AR photo image corresponding to the code value corresponding to the public marker. When receiving the code value corresponding to the private marker from the terminal 120, the AR photo reproduction server 130 extracts a private AR photo image corresponding to the code value corresponding to the private marker.

The terminal 120 recognizes the public marker included in the public AR photo 810 when the public AR photo 810 including the public marker in the non-login state or the logged in state is illuminated with the camera module running, and the public marker The public code (a kind of identification code) corresponding to is extracted.

The AR photo reproduction server 130 receives a public AR photo image and a public code from the terminal 120 in a non-login state. The AR photo reproducing server 130 extracts the representative image matched with the public code if there is a public code, and only when the representative image and the public AR photo image match, the terminal in a non-login state. Send to 120. The terminal 120 overlays and outputs image data corresponding to the public AR photo 810 on an area in which the public AR photo 810 exists in a non-login state or a logged in state.

When the terminal 120 displays the private AR photo 820 including the private marker in the logged-in state with the camera module running, the terminal 120 recognizes the private marker included in the private AR photo 820, and a code corresponding to the private marker (A kind of identification code) is extracted.

The AR photo reproduction server 130 receives a private AR photo image and a private code from the terminal 120 in the logged-in state. If there is a private code, the AR photo reproducing server 130 extracts the representative image matched with the private code, and sends the video data matched to the representative image only when the representative image and the private AR photo image match. 120). The terminal 120 overlays and outputs image data corresponding to the private AR photo 820 on the area where the private AR photo 820 exists in the logged-in state.

The AR photo reproducing server 130 may check a user who played an image based on the private AR photo 820 by using login information. That is, in order to play the video corresponding to the private AR photo 820, a play permission granted for each user account must exist.

When a user who has a user account granted permission to play the private AR photo 820 uses the terminal 120 to illuminate the private AR photo 820 while logged in, the AR photo playback server 130 After recognizing a code corresponding to 820, an additional code is requested, and when an additional code is input, a corresponding image may be played.

When the image data in the album is deleted, the AR photo reproducing server 130 deletes all user account information to which the reproduction permission has been granted.

8 is a diagram illustrating a pin function for fixing to a screen when playing an image corresponding to an AR photo according to the present embodiment.

When the terminal 120 overlays and outputs the image data of the area where the AR photo 150 exists, the terminal 120 outputs the image data to the area where the AR photo 150 exists regardless of the screen of the terminal 120. Accordingly, as shown in FIG. 8, image data output only in an area in which the AR photo 150 on the screen exists can be fixed to one side of the screen of the terminal 120 by using a pin function.

For example, if one side of the output screen of the terminal 120 is pressed, the image data that was output only in the area where the AR photo 150 is present is displayed as a pin function and the playback area is adjusted to fit the screen of the terminal 120. It is fixed to the location (eg, the center of the screen) and output. When the output screen of the terminal 120 is tapped twice in succession, the image data that has been output only in the area where the AR photo 150 exists is adjusted to be output to the entire screen of the terminal 120.

9 is a diagram showing charge according to an image length corresponding to an AR photo according to the present embodiment.

When the image data matching the AR photo 150 is uploaded or transmitted, the AR photo reproducing server 130 may divide and charge a free image and a paid image.

When the AR photo playback server 130 uploads image data matching the AR photo 150, when viewing an advertisement in a terminal that uploads the image data, the image within a preset first threshold time (eg, 15 seconds) To be able to upload for free.

When the running time of the image data matched with the AR photo 150 exceeds a preset second threshold time (eg, 1 minute), the AR photo playback server 130 uses a preset time unit (eg, 30 seconds). Charged coins to allow uploading of videos. In other words, the AR photo reproducing server 130 allows one paid coin (eg, 100 won) to be consumed every 30 seconds elapsed in a preset threshold time unit.

When viewing an advertisement even from the standpoint of a terminal playing free video data matching the AR photo 150, the free video can be viewed within a preset first threshold time (eg, 15 seconds).

A terminal that plays paid data matching the AR photo 150 may watch paid videos while consuming paid coins (eg, 1 coin per 30 seconds) held in a preset threshold unit (eg, 30 seconds).

10 is a diagram showing a report function for video data according to the present embodiment.

In this embodiment, the AR photo reproducing server 130 can play the video only when the real AR photo 150 exists. When video data matched to the AR photo 150 provided by the AR photo reproducing server 130 is provided, a report function is output from the terminal viewing the video. When the camera is pointed at the AR photo 150 in the terminal, a report button for the video is output, and after the user selects the report button for the video, the user selects one of illegal video, sexual video, and advertisement video among the reasons for reporting. can do.

11 is a conceptual diagram showing a method of generating a marker according to the present embodiment.

When the terminal 120 uses the AR photo playback application 110 to generate a marker for identifying a selected representative frame among a plurality of frames included in the image data, the terminal 120 recognizes the color of the representative frame to determine a color contrasting with the complementary color. Create a marker with.

In other words, the terminal 120 checks the RGB value of the area where the marker is inserted among the representative frame. For example, if the area in which the marker is inserted among the representative frame is identified as white, the terminal 120 may generate the marker in black so as to have a complementary color contrasting with the white color, and then combine with the representative frame to generate a representative image.

When the terminal 120 uses the AR photo playback application 110 to generate a marker having a complementary color that contrasts with the color of the area into which the marker is inserted, it is possible to generate a different complementary color for each area after dividing the area of the marker. Do.

In other words, the terminal 120 checks the RGB value of the area where the marker is inserted among the representative frames, and when the area where the marker is inserted is made of a plurality of colors, the terminal 120 determines the area of the marker to have various complementary colors contrasting with the plurality of colors. After dividing (eg, dividing into quarters), each region can be created to have a different complementary color and then inserted into a representative frame.

In addition, the terminal 120 may analyze pixel values in the representative frame, determine an area that is most contrasting when a marker of a preset color is inserted, and then insert the marker.

Also, as for the color of the marker, the background color of the marker is set to the same color by taking the RGB value of the pixel of the adjacent area (eg, upper left) at the position where the marker will be placed, and the brightness value of the corresponding color is half (128). The color of the code (data area) can be set to black or white.

12 is a diagram showing a method of matching an identification value without a marker according to the present embodiment.

For video authoring, the terminal 120 selects a representative frame from among a plurality of frames included in the image data and transmits the representative frame to the AR photo reproduction server 130 instead of generating a marker for identifying the representative frame.

The AR photo reproducing server 130 performs object recognition as a characteristic within a representative frame. The AR photo reproducing server 130 selects any one area (eg, a swimmer) among a plurality of object areas recognized in the representative frame. The AR photo reproducing server 130 divides the selected object area (for example, a swimmer) into preset units, calculates a pixel value (RGB value) for each divided area, and adds the accumulated pixel value (RGB value). ). The AR photo reproducing server 130 matches and stores an accumulated pixel value (RGB value) instead of a marker in a representative frame.

The AR photo reproducing server 130 transmits the object area (eg, a swimming person) and an accumulated pixel value (RGB value) to the AR photo reproducing application 110 in the terminal 120. The AR photo playback application 110 in the terminal 120 matches and stores an accumulated pixel value (RGB value) in the object area. When the AR photo 150 is illuminated using a camera module equipped with the terminal 120, the terminal 120 performs object recognition that is characteristic within the AR photo 150 using the AR photo playback application 110. do.

The terminal 120 uses the AR photo playback application 110 to check whether an area corresponding to a previously stored object area (eg, a swimmer) exists among the recognized object areas in the AR photo 150. . If there is an area that matches the previously stored object area (eg, a swimmer) among the recognized object areas in the AR photo 150, the terminal 120 is the accumulated pixel value (RGB value) matched to the matching object area. After extracting the AR photo image and transmits it to the AR photo reproducing server 130. The AR photo reproducing server 130 checks whether the accumulated pixel value (RGB value) received from the terminal 120 exists, and if the same pixel value exists as a result of checking, extracts a representative frame matched to the pixel value, and When the image and the representative frame match, the image data matched with the representative frame is extracted and transmitted to the terminal 120.

For video authoring, the terminal 120 selects a representative frame from among a plurality of frames included in the image data and transmits the representative frame to the AR photo reproduction server 130 instead of generating a marker for identifying the representative frame. When registering a representative frame in the AR photo playback server 130, if a duplicate image (i.e., the same image) already exists, a plurality of recognized objects within the representative frame after performing object recognition that is characteristic within the representative frame Select an area (eg, wave) other than the already designated area among the object areas of.

The AR photo reproducing server 130 divides the newly selected object area (e.g., wave) into a preset unit, calculates a pixel value (RGB value) for each divided area, and adds the accumulated pixel value (RGB value) Create The AR photo reproducing server 130 matches and stores an accumulated pixel value (RGB value) instead of a marker in a representative frame.

The AR photo reproducing server 130 transmits the object area (eg, wave) and the accumulated pixel value (RGB value) to the AR photo reproducing application 110 in the terminal 120. The AR photo playback application 110 in the terminal 120 matches and stores an accumulated pixel value (RGB value) in the object area. When the AR photo 150 is illuminated using a camera module equipped with the terminal 120, the terminal 120 performs object recognition that is characteristic within the AR photo 150 using the AR photo playback application 110. do.

The terminal 120 uses the AR photo playback application 110 to check whether an area corresponding to a previously stored object area (eg, a wave) exists among the recognized object areas in the AR photo 150. If there is an area matching the previously stored object area (eg, wave) among the recognized object areas in the AR photo 150, the terminal 120 extracts the accumulated pixel value (RGB value) matched to the matching object area. Then, the AR photo image is transmitted to the AR photo reproducing server 130 together with the AR photo image. The AR photo reproducing server 130 checks whether the accumulated pixel value (RGB value) received from the terminal 120 exists, and if the same pixel value exists as a result of checking, extracts a representative frame matched to the pixel value, and When the image and the representative frame match, the image data matched with the representative frame is extracted and transmitted to the terminal 120.

The above description is merely illustrative of the technical idea of the present embodiment, and those of ordinary skill in the technical field to which the present embodiment belongs will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present exemplary embodiments are not intended to limit the technical idea of the present exemplary embodiment, but are illustrative, and the scope of the technical idea of the present exemplary embodiment is not limited by these exemplary embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

110: AR photo playback application
120: terminal 130: AR photo playback device
140: printer 150: AR photo
810: Public AR Photo 820: Private AR Photo

Claims (5)

Generates a marker for identifying a selected representative frame among a plurality of frames included in image data, creates a representative image by combining the representative frame and the marker, and requests to print the representative image as an output A video authoring terminal;
A print for outputting the representative image as an Augmented Reality Photograph;
When the AR photo is illuminated using the provided camera module, when a public marker included in the AR photo is recognized, a public code corresponding to the public marker is extracted, and a private marker included in the AR photo ( Private Marker) is recognized, a private code corresponding to the private marker is extracted, and when the public code, the private code, and the AR photo image illuminating the AR photo are authenticated, a server corresponding to the public code and the private code A video reproducing terminal configured to receive video data and display video data corresponding to each of the public code and the private code overlaid on an area in which the AR photo exists in an output screen; And
It checks whether the public code and the private code received from the video playback terminal exist, and if the private code exists as a result of the verification, a playback permission is granted for each user account in order to play the video data corresponding to the private code, Requesting an additional code to a terminal having a user account granted with the playback permission, and extracting the video data matched with the private code only when the additional code input from the terminal having the user account granted with the playing permission is authenticated It is transmitted to a terminal having a user account to which the playback permission is granted, and if the public code exists as a result of the verification, the video data matching the public code is extracted and transmitted to the video playback terminal, and the length of the video data is If it is within a set first threshold time, the video data is classified as a free video, and when the length of the video data exceeds a preset second threshold time, the video data is classified as a paid video, and An AR photo reproducing server that controls to consume paid coins held in a predetermined threshold unit when reproducing the divided image data;
AR photo reproduction system comprising a. The method of claim 1,
The image data,
An AR photo playback system comprising an image captured using a camera module provided in the video authoring terminal or a 3D object image generated using a separate authoring module. The method of claim 2,
The video playback terminal,
Downloading a representative image matching the code from the AR photo reproduction server, and when the representative image matching the code matches the AR photo image, receiving image data corresponding to the code from the AR photo reproduction server AR photo reproduction system, characterized in that to output. The method of claim 3,
The video playback terminal,
Image data corresponding to the code from the AR photo reproducing server only when the feature points are extracted from the AR photo image, the feature points are extracted from the representative image matched with the code, and compared with each other, and the feature points match more than a preset threshold. AR photo reproduction system, characterized in that receiving a. The method of claim 4,
The AR photo reproduction server,
If the code received from the video playback terminal exists, a representative image matching the code is extracted and transmitted to the video playback terminal, and the representative image matched with the code from the video playback terminal is matched with the AR photo image. And transmitting the image data matched to the representative image to the video playback terminal only when it is confirmed whether or not.

Images