KR102580250B1 - Mobile terminal and system - Google Patents

Abstract

We present a portable terminal and system that allows one video to be randomly played among a plurality of videos stored in association with an image. The presented mobile terminal extracts a target image from a captured image, extracts a target image corresponding to the target image from stored candidate images, and selects a target image from among a plurality of videos related to the extracted target image according to a set rule to be sent to the server. The video received from is played within the area of the target image on the display.

Description

Mobile terminal and system {Mobile terminal and system}

The present invention relates to a portable terminal and system, and more specifically, to a portable terminal and system capable of playing a video associated with an image.

As the spread of smartphone terminals increases, the performance of smartphone terminals such as CPU, camera image quality, and memory storage space is gradually improving, and the speed of the wireless communication network is becoming faster.

Accordingly, there is a trend to move away from mobile communication services centered on voice calls and toward providing various forms of Internet services and multimedia services.

In particular, services that go beyond viewing various images on the Internet and use multimedia content such as videos or advertisements are increasing significantly. Moreover, with the recent rapid spread of smartphones, the number of cases where images and videos are provided via the Internet and mobile communication networks through smartphones is also increasing significantly.

In this regard, there is a need for a method that can automatically and in real time provide video based on the image or provide various types of additional services through a client terminal for images distributed through networks such as the Internet or mobile communication networks. there is.

The matters described in the above background technology are intended to aid understanding of the background of the invention and may include matters that are not disclosed prior art.

Prior Art 1: Republic of Korea Patent Publication No. 10-2020-0100274 (2020.08.26) Prior Art 2: Republic of Korea Patent No. 10-1303166 (2013.08.28) Prior Art 3: Republic of Korea Patent No. 10-1189609 (2012.10.04) Prior Art 4: Republic of Korea Patent No. 10-1404596 (2014.05.30)

The present invention was proposed in consideration of the above-described conventional circumstances, and its purpose is to provide a portable terminal and system that allows one video to be randomly played among a plurality of videos stored in association with an image.

In order to achieve the above object, a portable terminal according to a preferred embodiment of the present invention includes a camera; display; a memory for storing candidate images; and extracting a target image from a captured image captured by the camera, extracting a target image corresponding to the target image from the candidate images, and selecting a target image from among a plurality of videos associated with the extracted target image according to a set rule to be sent from the server. and a processor that plays the received video within the area of the target image on the display.

The determined rule may be the selection probability of each of a plurality of videos related to the target image.

According to a preferred embodiment of the present invention, a video received for the same target image may be different from a video previously received for the same target image.

The processor may play a video selected from among a plurality of videos associated with the target image according to the current location information of the mobile terminal and received from a server within the area of the target image on the display.

The received video may include instructions for the user.

The received video may include coupon content.

Meanwhile, a control method of a mobile terminal according to a preferred embodiment of the present invention is a control method of a mobile terminal including a camera, a display, and a memory, comprising: storing candidate images in the memory; extracting a target image from a captured image captured by the camera; extracting a target image corresponding to the target image from the candidate images; And a step of selecting a video received from a server among a plurality of videos related to the extracted target image according to a set rule and playing it within the area of the target image on the display.

Meanwhile, a system according to a preferred embodiment of the present invention includes: a first server storing candidate images; and a second server that stores a plurality of videos associated with each of the candidate images, wherein the second server selects a video according to a set rule among a plurality of videos associated with a target image extracted from the candidate images. , the selected video is transmitted to the mobile terminal so that the selected video can be played within the area of the target image on the display of the mobile terminal.

The first server may receive a target image from a mobile terminal and extract a target image corresponding to the target image from the candidate images.

The second server may select a video selected for the same target image differently from a video previously selected for the same target image.

The second server may record the transmission of the video transmitted to the mobile terminal by associating it with the account of the user of the mobile terminal.

The second server may transmit a video collection completion event signal to the mobile terminal when all of the plurality of videos related to the target image are associated with the mobile terminal user's account and recorded for transmission.

The second server has a limit value set for the number of transmissions of the video selected for the same target image, and if the limit value for the number of transmissions is exceeded for the user, the video selected for the target image may not be transmitted to the mobile terminal. there is.

The transmission number limit value may be a transmission number limit value within a predetermined period.

The second server may set a transmission time range for the video selected for the target image, and if the current time does not fall within the transmission time range, the second server may not transmit the video selected for the target image to the mobile terminal.

The second server may select a video from among a plurality of videos related to the target image according to the current location information of the mobile terminal.

The number of videos selected and transmitted in relation to the target image may be limited in relation to location.

According to the present invention with this configuration, one image can be randomly selected and played back from among a plurality of images (eg, videos) stored in relation to an image.

By allowing multiple images (e.g., videos) to be played randomly (continuously) for the same image, it is possible to arouse the user's enjoyment.

By linking location, different images (e.g., videos) can be collected from various locations.

By imposing restrictions on the number, time, location, etc. of video (eg, video) transmission or playback, the difficulty of video collection can be increased and the reward effect for video collection can be doubled.

In addition, it is effective in that it can be used for various lottery events such as drinking game service, random coupon service, etc.

1 and 2 are diagrams for explaining the basic concept of the present invention.
Figure 3 is a diagram for explaining the schematic operation of a system according to an embodiment of the present invention.
Figure 4 is a diagram for explaining the schematic operation of a system according to another embodiment of the present invention.
Figure 5 is a configuration diagram of the entire system according to an embodiment of the present invention.
FIG. 6 is an internal configuration diagram of the mobile terminal shown in FIG. 5.
FIG. 7 is a diagram used to explain that videos for the same target image received from a mobile terminal over time are different.
FIG. 8 is a diagram used to explain a case where a mobile terminal receives video from multiple locations by linking locations.
Figure 9 is a diagram used to explain a case where a mobile terminal receives instructions from a server.
FIG. 10 is a diagram used to explain a case where the mobile terminal shown in FIG. 5 receives an extraction command from a user.
Figure 11 is a diagram used to explain target image extraction when a captured image has a border pattern.
Figures 12a and 12b are diagrams showing another example of a border pattern.
Figure 13 is a diagram used to explain the case where the outer trajectory of the target image is not a square.
Figure 14 is a diagram used to explain a case where a target image is attached to a curved object.
Figure 15 is a diagram used to explain video playback on a mobile terminal based on an advertisement for a bus stop shelter.
FIG. 16 is an internal configuration diagram of the image server shown in FIG. 5.
FIG. 17 is a diagram used to explain a case in which candidate images are stored in association with a user account in the memory shown in FIG. 16.
FIG. 18 is a diagram used to explain a case in which candidate images are stored in association with location information in the memory shown in FIG. 16.
Figure 19 is an internal configuration diagram of the video server shown in Figure 5.
Figure 20 is a diagram used to explain a case where a video collection completion event signal is transmitted from the server to the mobile terminal.
Figure 21 is a flowchart for explaining a control method in a mobile terminal according to an embodiment of the present invention.
Figure 22 is a flowchart for explaining a control method in a server according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present application. No.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate overall understanding when describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

1 and 2 are diagrams for explaining the basic concept of the present invention.

As shown in FIG. 1, the user first operates the mobile terminal 10 to run the app and then photographs the photo frame 1 on the table and its surroundings, for example. The photo frame 1 contains, for example, a still image of a Halloween pumpkin 2.

The display 12 of the mobile terminal 10 displays a photo frame 1 positioned on a table.

Then, the mobile terminal 10 transmits the captured image to the server. Here, the captured image is an image displayed as a whole on the front of the display 12 of the mobile terminal 10, and in the case of FIG. 1, it may include a part of the table and the photo frame image 3.

Then, the server selects an object to be recognized from the received captured image. The server will extract the photo frame image (3) from the captured image and select it as the recognition target.

Then, the server extracts a candidate image corresponding to the photo frame image 3 from among the candidate images stored internally.

Next, the server transmits a video (for example, a video of a person moving with a Halloween pumpkin on his head) related to the extracted candidate image to the mobile terminal 10.

Accordingly, a video 1a showing a person moving with a Halloween pumpkin on his head is played on the display 12 of the portable terminal 10. At this time, the video 1a may be played for a short period of time. Then, the video 1a is played within the area of the photo frame image 3 displayed on the display 12. The video 1a can be displayed in augmented reality.

Even if the user holds the photo frame with his hand and tilts it to the left by a predetermined amount as shown in FIG. 2, the video 1a can be played within the area of the photo frame image 3 displayed on the display 12.

Figure 3 is a diagram for explaining the schematic operation of a system according to an embodiment of the present invention.

First, the mobile terminal 10 extracts the target image 9 from the captured image. Here, the captured image may mean an image displayed on the entire front of the display 12. The target image 9 may refer to an image of an area in a captured image where an object to be recognized exists. For example, the target image 9 may refer to an image of an area that can be called a region of interest within a captured image. At this time, the area of the target image 9 may include a predetermined image (A; still image). Anyone working in the same industry can fully understand the method of extracting a target image from a captured image using well-known techniques.

In the system of FIG. 3, the mobile terminal 10 may have a plurality of candidate images 5, and the mobile terminal 10 selects a target image 9 from among the plurality of pre-stored candidate images 5. Extract the target image.

In this way, if the target image 9 is extracted from the captured image and the target image corresponding to the target image 9 is extracted, it can be said that the captured image has been recognized.

Then, the mobile terminal 10 transmits the extracted target image to the server 50. Preferably, the mobile terminal 10 may transmit the identifier of the extracted target image to the server 50.

The server 50 pre-stores a plurality of candidate images 6 and a plurality of videos 7 associated with each candidate image in the form of files. Here, the plurality of candidate images 6 of the server 50 include the plurality of candidate images 5 of the mobile terminal 10. In other words, the candidate image 5 stored in the mobile terminal 10 can be understood as being uploaded to the server 50.

Accordingly, the server 50 can easily find a candidate image corresponding to the received target image among the plurality of candidate images 6. This is because, among the plurality of candidate images 6 of the server 50, there is a candidate image having the same identifier as the identifier of the received target image.

Then, the server 50 selects a video Aa according to a set rule or randomly from among a plurality of videos associated with the found candidate image (i.e., target image) to the target image 9 on the display 12 of the mobile terminal 10. ) is transmitted to the mobile terminal 10 so that it can be played within the area.

Here, the determined rule may be the selection probability of each of a plurality of videos related to the target image. For example, if there are 5 videos related to the target image, the selection probability of the first video is set to 40%, the selection probability of the second video is set to 30%, and the selection probability of the third video is set to 20%. The set rule may be the selection probability for each video associated with the target image, such as the selection probability of the fourth video being set at 7% and the selection probability of the fifth video being set at 3%.

Of course, the above set rules may mean selecting items one by one in order each time. In other words, for the same target image (i.e., the same captured image), the set rules are selected one by one each time, such as selecting the first video the first time, the second video the second time, and the third video the third time. It may mean doing.

Randomly described above may mean selecting randomly regardless of the established rules described above.

Because of this, the mobile terminal 10 reproduces the received video Aa within the area of the target image 9 on the display 12. At this time, the video (Aa) may be displayed in augmented reality.

According to the above-described FIG. 3, when the mobile terminal 10 recognizes a captured image, the server 50 does not need to recognize the image.

Figure 4 is a diagram for explaining the schematic operation of a system according to another embodiment of the present invention.

In the case of FIG. 4, the server 50 pre-stores a plurality of candidate images 6 to be compared with the target image and a plurality of videos 7 associated with each candidate image in the form of files.

First, the mobile terminal 10 extracts the target image 9 from the captured image (that is, the image displayed on the entire front of the display 12). In the case of FIG. 4 , as previously described, the target image 9 may refer to an image of an area in a captured image where an object to be recognized exists. For example, the target image 9 may refer to an image of an area that can be called a region of interest within a captured image.

Then, the mobile terminal 10 transmits the extracted target image 9 to the server 50.

The server 50 extracts a target image corresponding to the target image 9 from a plurality of pre-stored candidate images 6. The target image (9) can be seen as having a reduced recognition range compared to the captured image. That is, since the server 50 extracts the target image based on the target image 9 rather than the captured image, the target image can be extracted more quickly than extracting the target image from the captured image.

Since a plurality of videos are associated with the extracted target image (i.e., candidate image), the server 50 selects one video from among the plurality of videos associated with the target image among the candidate images according to a set rule, and the selected video is It is transmitted to the mobile terminal 10 so that it can be reproduced within the area of the target image 9 on the display 12 of the mobile terminal 10.

Because of this, the mobile terminal 10 reproduces the received video Aa within the area of the target image 9 on the display 12. The video (Aa) can be displayed in augmented reality.

According to FIG. 4 described above, the mobile terminal 10 provides the target image in the captured image to the server 50, and the server 50 extracts the target image based on the target image, so the target image is extracted from the captured image. The target image can be extracted more quickly than extracting .

Figure 5 is a configuration diagram of the entire system according to an embodiment of the present invention.

The system of FIG. 5 may include a mobile terminal 10, a network 20, and a server 50.

The mobile terminal 10 may or may not store a plurality of candidate images.

The portable terminal 10 can capture a video and automatically or manually select a representative frame from among the frames of the captured video. The mobile terminal 10 can use the representative frame as a candidate image (i.e., representative image), associate it with the corresponding video, and upload it to the server 50.

When associating a candidate image (i.e., a representative image) and a video, the mobile terminal 10 may associate a plurality of videos with one candidate image.

The mobile terminal 10 may upload related candidate images (i.e., representative images) and a plurality of videos to the server 50, and may also store the generated candidate images (i.e., representative images) in its internal memory. . In other words, if there is a need to store candidate images, save them.

If necessary, the representative image may not be selected from the captured video, but an image different from the video may be selected as the representative image. For example, a menu board image may be selected as a representative image. If there are three food menus and a video of each food menu is filmed, their representative image should be the menu image. So, you can also associate a video about a food menu with a menu board image.

Meanwhile, the portable terminal 10 may use a stored video or a provided video without shooting a video.

Additionally, the portable terminal 10 may print the stored candidate image (i.e., representative image) through a printer.

Although only one mobile terminal 10 is shown in FIG. 5 described above, it should be understood that in reality, a plurality of mobile terminals 10 exist. Accordingly, each mobile terminal 10 can upload the candidate images (ie, representative images) and videos associated with each other to the server 50 .

First, a case where a plurality of candidate images are stored in the mobile terminal 10 will be described. In this case, it is assumed that the server 50 stores candidate images from each mobile terminal 10 and a plurality of videos associated with each candidate image.

In this case, the mobile terminal 10 may extract a target image from a captured image captured by a camera and extract a target image corresponding to the target image from candidate images. The mobile terminal 10 transmits the extracted target image (preferably, the identifier of the target image) to the server 50, and then selects the extracted target image from the server 50 according to a set rule or at random. You can receive the selected video. That is, the server 50 transmits a video selected according to a set rule or randomly from among a plurality of videos related to the target image to the mobile terminal 10 so that it can be played within the area of the target image on the display of the mobile terminal 10. You can. Accordingly, the mobile terminal 10 can play the received video within the area of the target image on the display. At this time, the video may be displayed in augmented reality.

If necessary, when the mobile terminal 10 transmits the target image to the server 50, the mobile terminal user's account may also be transmitted. This may be necessary for managing transmission records for each user account on the server 50 side.

This time, a case where a candidate image is not stored in the mobile terminal 10 will be described. In this case, it is assumed that the server 50 stores candidate images from each mobile terminal 10 and a plurality of videos associated with each candidate image.

In this case, the portable terminal 10 extracts a target image from a captured image captured by a camera. The mobile terminal 10 may transmit the extracted target image to the server 50. Accordingly, the server 50 extracts a target image corresponding to the target image from among the candidate images, and displays a video selected according to a set rule or at random among a plurality of videos associated with the extracted target image on the mobile terminal 10. It can be transmitted to the mobile terminal 10 so that it can be reproduced within the area of the target image. Accordingly, the mobile terminal 10 can play the received video within the area of the target image on the display. At this time, the video may be displayed in augmented reality.

When transmitting the target image to the server 50, the mobile terminal 10 may also transmit the account of the corresponding mobile terminal user. This may be necessary for managing transmission records for each user account on the server 50 side. In addition, since the server 50 only needs to compare the candidate images of the user account of the mobile terminal with the target image, the target image corresponding to the target image can be extracted from the candidate images more quickly than when comparing with all candidate images. .

Meanwhile, in FIG. 5 , the mobile terminal 10 may be implemented as an electronic device capable of accessing the server 50 through the network 20 . For example, the mobile terminal 10 is a wireless communication device that guarantees portability and mobility, and includes Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), and Personal Handyphone System (PHS). , PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone ( It may include all types of handheld-based wireless communication devices such as smartphones, smartpads, tablet PCs, etc.

The network 20 may refer to a connection structure that allows information exchange between nodes such as the mobile terminal 10 and the server 50.

For example, the wireless communication network 20 includes wireless LAN (WLAN), Wireless Broadband (Wibro), WCDMA (Wideband CDMA), IEEE 802.16, and Long Term Evolution (LTE). )), LTE-A (Long Term Evolution-Advanced), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, adjacent magnetic field It may include, but is not limited to, Near Field Communication (NFC), ultrasonic communication (Ultra Sound Communication (USC)), Visible Light Communication (VLC), and Wi-Fi. no.

Meanwhile, the wired communication network 20 includes wired LAN (Local Area Network), wired WAN (Wide Area Network), Power Line Communication (PLC), USB communication, Ethernet, and serial communication. communication), optical/coaxial cable communication, etc., but is not limited thereto.

In Figure 5, the server 50 may include an image server 30 and a video server 40.

Here, the image server 30 can store and manage a plurality of candidate images. As needed, a plurality of candidate images in the image server 30 can be updated.

The image server 30 may receive a target image extracted from a captured image from the mobile terminal 10 and extract a target image corresponding to the target image from a plurality of candidate images.

When transmitting the target image from the mobile terminal 10 to the server 50, the image server 30 may be required, and when transmitting the identifier of the target image from the mobile terminal 10 to the server 50 The image server 30 may not be needed.

The video server 40 may store and manage a plurality of videos associated with each of the candidate images. If necessary, a plurality of videos associated with each of the candidate images in the video server 40 can be updated.

The video server 40 selects one video according to a set rule or randomly among a plurality of videos related to the target image extracted from the image server 30, and matches the selected video to the target image on the display of the mobile terminal 10. It can be transmitted to the mobile terminal 10 so that it can be played within the area. Here, the determined rule may be the selection probability of each of a plurality of videos related to the target image, as described above. Of course, the above set rules may mean selecting items one by one in order each time. Randomly described above may mean selecting randomly regardless of the established rules described above.

Of course, the video server 40 may also receive a target image (preferably, an identifier of the target image) from the mobile terminal 10.

If necessary, the video server 40 may not be present in the server 50. For example, even if the video is not stored on the server 50, it can be streamed through an external link such as a YouTube link.

In the above description, the image server 30 receives a target image from the mobile terminal 10, but may also receive a captured image from the mobile terminal 10, if necessary. Since the image server 30 extracts the target image from the captured image in the same manner as the target image extraction method in the portable terminal 10, the image server 30 may receive the captured image from the portable terminal 10. do.

FIG. 6 is a diagram of the internal configuration of the mobile terminal shown in FIG. 5, FIG. 7 is a diagram used to explain that the video for the same target image received in the mobile terminal is different depending on time, and FIG. 8 is a diagram of the mobile terminal. This is a diagram used to explain a case where a video is received from multiple locations by linking location, FIG. 9 is a diagram used to explain a case where a mobile terminal receives instructions from a server, and FIG. 10 is a diagram used to explain a case where a mobile terminal receives instructions from a server. FIG. This is a diagram used to explain a case where a terminal receives an extraction command from a user. Figure 11 is a diagram used to explain target image extraction when a captured image has a border pattern. Figures 12A and 12B are diagrams of a border pattern. This is a diagram showing another example. FIG. 13 is a diagram used to explain a case where the outer trajectory of the target image is not a square, and FIG. 14 is a diagram used to explain a case where the target image is attached to a curved object.

The portable terminal 10 may include a camera 11, a display 12, a location detection unit 13, a memory 14, a communication unit 15, and a processor 16.

The camera 11 can photograph any external object (eg, a photo, a product label, an advertising image of a bus stop shelter, a movie or concert poster, a book, etc.) and its surroundings. An image captured by the camera 11 may be displayed on the display 12.

The display 12 can display an image captured by the camera 11.

A touch screen may be installed on the front of the display 12. Accordingly, the display 12 can detect the user touch input and provide it to the processor 16.

The location sensor 13 can sense the location of the mobile terminal 10.

The memory 14 may store a plurality of candidate images. Here, the plurality of candidate images are images compared to the target image and can be referred to as recognition data.

Memory 14 may include fixed data blocks and temporary data blocks.

Data that needs to be fixed and stored, such as photos and videos, can be stored in the fixed data block. For example, since something like a book will be viewed for a long time, candidate images associated with a physical object such as a book can be stored in a fixed data block.

Candidate images related to location information for each region may be stored in the temporary data block. For example, when the mobile terminal 10 is located in a specific area and the app is running, candidate images related to the location information of the specific area may be downloaded and stored in a temporary data block. Accordingly, if the processor 16 determines that the current location of the mobile terminal 10 deviates from the location of the location information associated with the candidate images stored in the temporary data block, the processor 16 selects the candidate image associated with the location information stored in the temporary data block. can be deleted.

Additionally, in the case of temporary data blocks, old data may be deleted according to a first-in-first-out (FIFO) method.

The memory 14 can store a program consisting of instructions for performing a series of operations performed by the portable terminal 10.

The communication unit 15 may perform communication with the server 50.

The processor 16 can control the overall operation of the portable terminal 10.

The processor 16 may refer to a device that can perform a series of operations or decisions for the operation of the mobile terminal 10. For example, the processor 16 may be a central processing unit (CPU), a micro controller unit (MCU), a graphical processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). It is not limited to this.

First, the control operations of the processor 16 when candidate images are stored in the memory 14 will be described.

The processor 16 may extract a target image from a captured image captured by the camera 11 and extract a target image corresponding to the target image from candidate images. If the target image has been extracted in this way, it can be said that the target image has been recognized. The processor 16 transmits the extracted target image (preferably, an identifier of the target image) to the server 50 through the communication unit 15.

The processor 16 may receive a video related to the target image extracted from the server 50 through the communication unit 15. In other words, the video at this time refers to one video selected according to a set rule or at random among a plurality of videos related to the extracted target image. Here, the determined rule may be the selection probability of each of a plurality of videos related to the target image, as described above. Of course, the above set rules may mean selecting items one by one in order each time. Randomly described above may mean selecting randomly regardless of the established rules described above. In addition, the video is transmitted from the server 50 to be playable within the area of the target image on the display 12 of the mobile terminal 10. Accordingly, the processor 16 can play the received video within the area of the target image on the display 12. For example, the processor 16 can play a video in augmented reality.

At this time, if the current video that the processor 16 receives through the communication unit 15 is a video received for the same target image, it is preferable that it is different from a video previously received for the same target image. For example, as shown in FIG. 7, if the processor 16 of the mobile terminal 10 transmits the target image A to the server 50 at a predetermined time (e.g., t) and receives the video Aa , This means that if the same target image (A) is transmitted to the server 50 at a later time (eg, t+1), a video (Ab) different from the previously received video is received. According to the description of FIG. 7 above, multiple videos for the same image (e.g., a movie poster) can be played sequentially according to a set rule or randomly, making it fun for the user.

Meanwhile, the processor 16 can receive video from various locations in conjunction with the location of the mobile terminal 10. For example, as shown in FIG. 8, user A's mobile terminal 10 receives predetermined event information from the server 50 (e.g., "in area 1 (location A) and area 2 (location B) in Seoul). Upon receiving the message (including the content “Please participate in the 00 soju drinking event”), User A checks the event information and then moves to area 1 (location A) and area 2 (location B) to participate in the event. First, when going to area 1 (location A), user A drinks 00 soju and then photographs the label of 00 soju and transmits the extracted target image and the current location information of the mobile terminal 10 to the server 50. . Then, the server 50 allows the video Aa, selected according to a set rule or at random, from among a plurality of videos associated with the received target image to be played within the area of the target image on the display of user A's mobile terminal 10. send. Accordingly, the mobile terminal 10 of user A in area 1 (location A) plays the received video Aa within the area of the target image on the display. Afterwards, user A moves to area 2 (location B), drinks 00 soju, photographs the label of 00 soju, and transmits the current location information of the mobile terminal 10 to the server 50 along with the extracted target image. do. Then, the mobile terminal 10 of user A in area 2 (location B) receives a video (Ab) selected according to a set rule or at random among a plurality of videos related to the target image from the server 50, and The video Ab is played within the area of the target image on the display of user A's portable terminal 10.

In this way, the processor 16 can collect different videos from various locations by linking locations according to event content.

Meanwhile, as described above, selecting and playing videos according to established rules or randomly can also be used as a service in drinking games. For example, when a user (hereinafter referred to as “penalty performer”) who must perform a penalty is determined by playing a certain drinking game, the target image (preferably, of the target image) extracted by photographing the label of a liquor bottle is determined by the user. identifier) is transmitted to the server 50. Then, the server 50 transmits a video selected according to a set rule or randomly from among a plurality of videos associated with the received target image so that it can be played within the area of the target image on the display of the mobile terminal 10 of the penalty performer. . At this time, the selected video includes instructions for the person performing the corresponding penalty. Accordingly, the penalty performer's portable terminal 10 plays the received video within the area of the target image on the display. Accordingly, a video (eg, including the instruction “one shot”) as illustrated in FIG. 9 may be displayed within the area of the target image on the display of the mobile terminal 10. Of course, in Figure 9, only one shot is indicated, but a video of a cheerleader dancing can be displayed together with the instruction "one shot." Then, the penalty performer performs the penalty by watching the video on the display of the mobile terminal 10 (for example, including the instruction “one shot”).

Of course, examples of instructions other than one shot can be diverse, such as a love shot with the person on the left, a king game, and a one-shot if there is even a little bit of red in what you are wearing or wearing. Accordingly, the video received by the processor 16 through the communication unit 15 may include predetermined instructions for the user.

Alternatively, as described above, selecting and playing videos according to established rules or randomly can be used not only as a drinking game, but also when providing coupons. In this case, providing a coupon means providing a coupon selected according to a set rule or at random. Accordingly, the video received by the processor 16 through the communication unit 15 may include predetermined coupon content.

The above-mentioned drinking game service and random coupon service can be said to be part of a lottery event.

Meanwhile, in FIG. 6 , the processor 16 may operate to extract the target image 9 for the currently captured image upon receiving an extraction command (eg, touch input; see FIG. 10 ) from the user. Here, the captured image may mean an image displayed on the entire front of the display 12 (that is, including the target image 9 and other images). The target image 9 may refer to an image of an area in a captured image where an object to be recognized exists. For example, the target image 9 may refer to an image of an area that can be called a region of interest within a captured image.

If target images must be extracted for all images continuously captured by the camera 11, the load on the mobile terminal 10 will inevitably increase. However, as described above, if target image extraction for the currently captured image is performed only when an extraction command such as a touch input is received, unnecessary target image extraction can be prevented and the load on the mobile terminal 10 can be significantly reduced. there is.

Of course, if necessary, a specific code such as a QR code can be used instead of touch input. That is, when a user photographs the area where the QR code is located and the surrounding area in a photo, movie poster, etc., the QR code will be displayed on the display 12. Accordingly, when the processor 16 recognizes the QR code in the currently captured image, it can operate to extract the target image 9 for the currently captured image.

If the target image corresponding to the target image cannot be extracted from among the candidate images stored in the memory 14, the processor 16 may request the server 50 to extract the target image through the communication unit 15. there is. This is because if there is some degree of damage to the target image corresponding to the target image, it may not be extracted properly.

The processor 16 may extract a target image by detecting a predetermined border pattern. As illustrated in FIG. 11, the processor 16 searches for a predefined border pattern 8 in the captured image (i.e., the image displayed on the entire front surface of the display 12), and targets the area surrounded by the border pattern 8. The area of the image can be determined, and the image within the determined area can be extracted as the target image.

For example, the captured image displayed on the display 12 of FIG. 11 can be said to be an image that appears when a user captures a movie poster. The movie poster may have a border pattern (8) of a dot-and-dash rectangular shape pre-printed. Here, the border pattern 8 can be designated by the production company producing the movie poster or the client who requested the production of the movie poster.

In this way, the border pattern 8 can help make extraction of the target image faster. Compared to the case without the border pattern 8, when the border pattern 8 is present, the time required to find the target image in the captured image may be shorter.

In an embodiment of the present invention, the border pattern 8 is not limited to the rectangular shape of the dot-and-dash line illustrated in FIG. 11, but may also have a shape as illustrated in FIGS. 12A and 12B. Of course, if necessary, the border pattern 8 may be of a shape other than that shown in FIGS. 11, 12A, and 12B.

Meanwhile, the processor 16 selects a representative frame from among the frames of the video captured by the camera 11, associates the representative frame (i.e., representative image) with the video as a candidate image, and uploads it to the server 50. can do. If necessary, the processor 16 may not select a representative image from the captured video, but may select an image different from the video as the representative image. Accordingly, the processor 16 may insert a specific border pattern into a candidate image obtained by photographing a book, a product label, a photo, a movie poster, a concert poster, a company introduction, etc., according to the purpose. Additionally, the processor 16 can associate a candidate image with a specific border pattern inserted with a video containing a description thereof and upload it to the server 50.

Accordingly, candidate images stored in the memory 14 may be stored in association with one of a plurality of border patterns. For example, assuming there are three different border patterns, the first border pattern may be associated with (eg, inserted into) the first candidate image (eg, an image including the label of a liquor bottle). A second border pattern may be associated with (eg, inserted into) the second candidate image (eg, a movie poster). A third border pattern may be associated with (e.g., inserted into) a third candidate image (e.g., an image of a book cover). In this way, if candidate images are stored in association with one of a plurality of border patterns, the target image corresponding to the target image only needs to be extracted from among the candidate images associated with the specific border pattern, rather than compared with all candidate images. You can quickly extract the target image. In addition, users can easily obtain the videos they want to collect by shooting only the subjects with the desired border pattern.

Meanwhile, candidate images in the memory 14 may be stored in association with location information. The location information here may mean location information of a zone. For example, the first to third candidate images may be associated with location information indicating a first region, and the fourth to tenth candidate images may be associated with location information indicating a second region. If candidate images are stored in association with location information in this way, the processor 16 can extract a target image corresponding to the target image from among the candidate images associated with location information near the current location of the mobile terminal 10. For example, if the current location detected by the location detection unit 13 of the mobile terminal 10 corresponds to the location information of the first area, the processor 16 may detect the first to third locations associated with the location information of the first area. The target image corresponding to the target image is extracted from the candidate images. The reason for doing this is to more quickly find the target image corresponding to the target image. That is, if candidate images are stored in association with location information indicating a region, the processor 16 does not compare all candidate images, but only selects candidate images associated with location information near the current location of the mobile terminal 10. The target image corresponding to the target image can be extracted only by comparing . Because of this, the target image can be extracted more quickly.

Of course, as necessary, the processor 16 transmits the current location information of the mobile terminal 10 to the server 50, and receives candidate images associated with the location information near the current location from the server 50 to memory. You can also save it in (14). In this way, the storage capacity of the memory 14 can be minimized, and desired candidate images can be downloaded and used only when necessary.

On the other hand, if there are three museums in a specific area, and the current location of the mobile terminal 10 within the specific area corresponds to the location of the first museum, it would be desirable to compare only the candidate images assigned to the first museum. will be. If necessary, when the user selects the first museum among the three museums, only the candidate images assigned to the first museum can be downloaded from the server 50 and used.

Candidate images in the memory 14 may be stored in association with time information. In this case, the processor 16 may extract a target image corresponding to the target image from among candidate images related to time information to which the current time of the mobile terminal 10 belongs. For example, this is to allow videos related to target images to be played only during the time the museum is open. That is, if a video that can be played only during the time when the museum is open is stored, when the mobile terminal 10 approaches the museum or enters the museum during the time when the museum is open, the processor 16 generates a candidate image related to the time information. The target image corresponding to the target image can be extracted from among them. Accordingly, the processor 16 receives a video selected from the server 50 according to a set rule or randomly among a plurality of videos related to the target image extracted during the time the museum is open and displays the area of the target image on the display 12. You can play it within. For example, the processor 16 can play a video in augmented reality.

In FIG. 6, the processor 16 plays a video within the area of a square target image, but as illustrated in FIG. 13, the outer trace 17 of the target image may not be square. In Figure 13, the outer trace 17 of the target image is illustrated as an oval shape, but it may have a shape other than a square or oval. Even in this case, in order to play the video within the desired area, the processor 16 detects the outer trace 17 of the target image. And, the processor 16 can play the video within the detected oval-shaped outer trace 17. For example, the processor 16 can play a video in augmented reality.

The above oval-shaped outer trace detection can be used when the extracted target image corresponds to a part of a candidate image having a square outer trace.

Meanwhile, the processor 16 can extract the target image even when the target image is attached to a curved object. For example, as illustrated in FIG. 14, when the processor 16 determines that the target image is attached to a curved object 18, such as a liquor bottle, the processor 16 selects the target image (e.g., a label image attached to the curved object 18). After flattening (correcting) the curved surface information, the corresponding target image can be extracted. This is because extracting the target image in a flat state can be more accurate. Here, the curved surface information may be a numerical value calculated by calculating the curvature of the curved surface of the target image, or curved surface information that matches the curved surface of the target image among a plurality of previously stored curved surface information. Then, the processor 16 receives a video selected according to a set rule or randomly from the plurality of videos associated with the extracted target image from the server 50, curves the video based on the corresponding curved information, and then stores the video within the area of the target image. You can play. For example, the processor 16 can play a video in augmented reality.

On the other hand, the processor 16 receives candidate images associated with the physical object from the server 50 in response to a candidate image request command for a physical object such as a book, associates the physical object with the candidate images, and stores them in the memory 14. You can save it. Here, books may be math workbooks, cookbooks, knitting books, encyclopedias, etc., and physical objects may include museums, exhibition halls, etc. in addition to books. For example, the text of a book or an image (mathematics problem) can be downloaded as a candidate image, and the corresponding physical object and the downloaded candidate image can be associated and stored in the memory 14. In this case, if the mobile terminal 10 has photographed a book, the processor 16 extracts a target image from the captured image and extracts a target image corresponding to the target image from candidate images in the memory 14. Then, the processor 16 receives a video selected according to a set rule or randomly from among a plurality of videos (e.g., commentary) associated with the extracted target image from the server 50 and displays the video within the area of the target image on the display 12. You can play it on . For example, the processor 16 can play a video in augmented reality.

In FIG. 11 above, a movie poster was used as an example, but it can also be explained using an advertisement for a bus stop shelter (see FIG. 15) as an example.

In the advertisement for the bus stop shelter illustrated in FIG. 15, a border pattern 8 in the form of a square with dashed dots is assumed to be inserted (printed).

The user points the camera 11 of the mobile terminal 10 to the advertisement of the bus stop shelter and photographs the advertisement so that the border pattern 8 is visible on the display 12. Then, the captured image is displayed on the display 12 of the portable terminal 10. Here, the captured image is an image displayed on the entire front surface of the display 12 and may include the border pattern 8 and an image within the area surrounded by the border pattern 8. Of course, the captured image may include not only the border pattern 8 illustrated in FIG. 15 and images within the area surrounded by the border pattern 8, but also other images.

The processor 16 extracts a target image from the captured image. For example, in FIG. 15, the target image is an image within the area surrounded by the border pattern 8, so the processor 16 searches for the border pattern 8 in the captured image and targets the image within the area surrounded by the border pattern 8. Extracted as an image.

Then, the processor 16 extracts a target image corresponding to the target image from a plurality of candidate images stored in the memory 14.

In this way, if the target image is extracted from the captured image and the target image corresponding to the target image is extracted, it can be said that the captured image has been recognized.

Thereafter, the processor 16 transmits the extracted target image to the server 50 through the communication unit 15. Preferably, the identifier of the extracted target image may be transmitted to the server 50. Based on the identifier of the received target image, the server 50 selects a video according to a set rule or randomly from among a plurality of videos associated with the target image within the area of the target image on the display 12 of the mobile terminal 10. It is transmitted to the mobile terminal 10 so that it can be played.

Accordingly, the processor 16 reproduces the video received through the communication unit 15 within the area of the target image 9 on the display 12. For example, the processor 16 can play a video in augmented reality.

This time, the control operations of the processor 16 in the case where candidate images are not stored in the memory 14 of FIG. 6 will be described.

In FIG. 6, when candidate images are not stored in the memory 14, the processor 16 extracts a target image from the captured image captured by the camera 11 and sends the extracted target image to the server ( 50) and send it to. For example, the processor 16 looks for a predefined border pattern in a captured image (i.e., an image displayed on the entire front surface of the display 12), determines the area surrounded by the border pattern as the area of the target image, and determines the determined area. The image within can be extracted as a target image. Accordingly, the server 50 extracts a target image corresponding to the target image from a plurality of pre-stored candidate images. In addition, the server 50 allows the mobile terminal ( 10) will be sent to.

Because of this, the processor 16 plays the video received through the communication unit 15 within the area of the target image on the display 12. For example, the processor 16 can play a video in augmented reality.

Meanwhile, when candidate images are not stored in the memory 14, target image extraction by the processor 16 is performed on the currently captured image only when an extraction command such as a user's touch input or QR code recognition is received. Target image extraction may be performed. This can help reduce the load on the mobile terminal 10 by preventing unnecessary target image extraction.

In addition to the above, the processor 16 may extract a target image by detecting a predetermined border pattern (eg, a square border pattern). And, even when the outer trace of the target image is not a square, the processor 16 detects the outer trace of the target image as described above based on FIG. 13, and converts the video from the server 50 into the detected target image. It can be played within the outer trajectory of .

As described above, the processor 16 when candidate images are not stored in the memory 14 cannot extract the target image corresponding to the target image. It is only functionally different from (16). Other than that, it's pretty much the same.

The above explains the difference in functionality of the processor 16 based on whether candidate images are stored or not stored in the memory 14. If necessary, the processor 16 operates the two modes together, but may select one of the two modes depending on the situation and perform control operations accordingly. For example, the first mode is a mode activated when candidate images are not stored in the memory 14, and the second mode is a mode activated when candidate images are not stored in the memory 14.

FIG. 16 is a diagram of the internal configuration of the image server 30 shown in FIG. 5, FIG. 17 is a diagram used to explain a case in which candidate images are stored in association with a user account in the memory shown in FIG. 16, and FIG. 18 is a diagram used to explain a case in which candidate images are stored in association with location information in the memory shown in FIG. 16.

The image server 30 may include a communication unit 31, a memory 32, and a processor 33. The image server 30 may be an example of the first server described in the claims of the present invention.

The communication unit 31 can perform communication with the portable terminal 10 and the video server 40.

The memory 32 may store a plurality of candidate images. The mobile terminal 10 selects a representative frame from among the frames of the video captured by the camera 11, uses the representative frame (i.e., representative image) as a candidate image, associates it with the video, and uploads it to the server 50. You can. Of course, the representative image is not selected from the recorded video, but an image different from the video may be selected as the representative image. Accordingly, the plurality of candidate images stored in the memory 32 can be viewed as having been created and stored in the portable terminal 10.

The memory 32 may store at least some of the candidate images in association with a user account. For example, as illustrated in FIG. 17 , candidate images may be stored in the memory 32 in the form of a lookup table 32a in connection with a user account. Here, each user account may refer to the account of each mobile terminal user. This is because when comparing target images and candidate images, if you compare only the candidate image of the user account, you can more quickly extract the target image corresponding to the target image from the candidate images.

Meanwhile, candidate images stored in the memory 32 may be stored in association with location information. Here, location information may mean location information of a zone. Accordingly, as illustrated in FIG. 18, candidate images may be stored in the memory 32 in the form of a lookup table 32b in connection with local location information. For example, the first to fifth candidate images may be stored in association with location information indicating region 1.

Meanwhile, although not shown in the drawing, candidate images stored in the memory 32 may be stored in association with time information. For example, since a video related to a target image (i.e., a candidate image) can be played only during the times when the museum is open, association with time information may be necessary.

Although not shown in the drawing, at least some of the candidate images stored in the memory 32 may be stored in association with physical objects. Here, the physical object can be a book such as a math workbook, a cookbook, a knitting book, or an encyclopedia, and in addition to a book, it can be a museum, an exhibition hall, etc. For example, by using the text of a book or an image (a math problem) as a candidate image, the candidate image and the corresponding physical object (eg, a math workbook) may be associated with each other and stored in the memory 32.

In FIG. 16, the processor 33 can control the overall operation of the image server 30.

The processor 33 may refer to a device that can perform a series of operations or decisions for the operation of the image server 30. For example, the processor 33 may be a central processing unit (CPU), a micro controller unit (MCU), a graphical processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). It is not limited to this.

The processor 33 may receive the target image extracted from the captured image from the mobile terminal 10 through the communication unit 31.

The processor 33 may extract a target image corresponding to the target image from candidate images stored in the memory 32. In this way, if the target image corresponding to the target image is extracted, it can be said that the corresponding captured image has been recognized.

Of course, when receiving a target image, the processor 33 may also receive curved surface information of the object to which the target image is attached. When receiving the target image and the curved surface information of the object to which the target image is attached, the processor 33 may extract the target image after flattening the target image based on the received curved surface information.

Meanwhile, the processor 33 may extract a target image corresponding to the target image from candidate images associated with the user account of the mobile terminal received from the communication unit 31. In this way, by comparing with the target image only the candidate images of the user account of the mobile terminal, the target image corresponding to the target image can be extracted from the candidate images more quickly than comparing with all candidate images stored in the memory 32. there is.

Of course, the processor 33 can share and process candidate images that can be searched (compared). For example, 5 candidate images associated with user A's account are stored in memory 32, 10 candidate images associated with user B's account are stored in memory 32, and 10 candidate images associated with user C's account are stored in memory 32. Let us assume that seven candidate images are stored in the memory 32. User A may send a request to the image server 30 to share candidate images of his own account and candidate images associated with user B's account. Accordingly, the processor 33 shares the candidate images associated with user A's account and the candidate images associated with user B's account. Accordingly, the processor 33 may extract a target image corresponding to the target image from among candidate images associated with user A's account and candidate images associated with user B's account.

Meanwhile, the processor 33 extracts a target image corresponding to the target image from candidate images associated with location information near the current location of the mobile terminal 10 (i.e., candidate images stored in the memory 32). You can. For example, if the current location of the mobile terminal 10 corresponds to the location information of area 1, the processor 33 may extract the target image corresponding to the target image from candidate images associated with the location information of area 1. . In this way, if candidate images are stored in association with location information indicating an area, the processor 33 does not compare all candidate images, but only selects candidate images associated with location information near the current location of the mobile terminal 10. The target image corresponding to the target image can be extracted more quickly just by comparing .

Since candidate images may be stored in the memory 32 in association with time information, the processor 33 selects a target image corresponding to the target image from among the candidate images associated with the time information to which the current time of the mobile terminal 10 belongs. can be extracted.

Meanwhile, at least some of the candidate images may be stored in the memory 32 in association with physical objects (e.g., books, museums, exhibition halls, etc.), so when the processor 33 receives the target image from the mobile terminal 10 Information on the physical object on which the target image is displayed can also be received. In this way, if the target image and information on the physical object on which the target image is displayed are received, the processor 33 may extract a target image corresponding to the target image from candidate images associated with the physical object.

The processor 33 may provide the extracted target image to the video server 40. Preferably, the processor 33 will transmit the identifier of the extracted target image to the video server 40.

FIG. 19 is an internal configuration diagram of the video server 40 shown in FIG. 5, and FIG. 20 is a diagram used to explain a case where a video collection completion event signal is transmitted from the server to the mobile terminal.

The video server 40 may include a communication unit 41, a memory 42, and a processor 43. The video server 40 may be an example of a second server described in the claims of the present invention.

The communication unit 41 may perform communication with the mobile terminal 10 and the image server 30.

The memory 42 may store a plurality of videos associated with each candidate image.

The processor 43 can control the overall operation of the video server 40.

The processor 43 may refer to a device that can perform a series of operations or decisions for the operation of the video server 40. For example, the processor 43 may be a central processing unit (CPU), a micro controller unit (MCU), a graphical processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). It is not limited to this.

The processor 43 may receive the target image from the image server 30 through the communication unit 41. Preferably, processor 43 will receive an identifier of the target image.

The processor 43 enables the mobile terminal 10 to play a video selected according to a set rule or randomly from among a plurality of videos related to the target image within the area of the target image on the display of the mobile terminal 10. ) can be sent to. Here, the determined rule may be the selection probability of each of a plurality of videos related to the target image, as described above. Of course, the above set rules may mean selecting items one by one in order each time. Randomly described above may mean selecting randomly regardless of the established rules described above.

Meanwhile, the processor 43 may select a video selected for the same target image differently from a video previously selected for the same target image. A more detailed explanation of this is expected to be fully understood by those skilled in the art through the part described above with reference to FIG. 7.

Meanwhile, a video selected by the processor 43 according to a set rule or randomly from among a plurality of videos related to the target image may include predetermined instructions for the user. A more detailed explanation of this may be fully understood by those skilled in the art through the part described above with reference to FIG. 9.

Meanwhile, a video selected by the processor 43 according to a set rule or randomly from among a plurality of videos related to the target image may include coupon content. In other words, it can be seen that probability is incorporated into coupon provision.

Meanwhile, the processor 43 can record the transmission of the video transmitted to the mobile terminal 10 by associating it with the account of the user of the mobile terminal. Accordingly, the processor 43 may transmit a video collection completion event signal to the mobile terminal when all of the plurality of videos related to the target image are associated with the mobile terminal user's account and recorded for transmission. In this way, when a video collection completion event signal is received, a notification message as illustrated in FIG. 20 may be displayed on the display of the corresponding mobile terminal. Here, the notification message may include a text message notifying that all videos related to the target image have been collected and a data list in a table format listing all videos collected in association with the target image. In this way, since the video collection completion event signal can be transmitted to the mobile terminal 10, the user's desire to collect video can be maximized.

Meanwhile, the processor 43 may set a limit on the number of transmissions of videos selected for the same target image. Accordingly, the processor 43 does not transmit the video selected for the target image to the portable terminal 10 when the transmission number limit value for the corresponding user is exceeded.

Of course, if necessary, the above-described transmission number limit value may be a transmission number limit value within a predetermined period. For example, you can set a period such as once a day, five times a month, etc. and limit the number of transmissions within that period.

Meanwhile, the processor 43 may set the transmission time range of the video selected for the target image. Accordingly, the processor 43 does not transmit the video selected for the target image to the mobile terminal 10 if the current time does not fall within the transmission time range.

Meanwhile, the processor 43 may select a video from among a plurality of videos related to the target image according to the current location information of the mobile terminal 10. At this time, if necessary, the number of videos selected and transmitted in relation to the target image may be limited in relation to the location. For example, by allowing only one video to be transmitted from one location, the difficulty of video collection can be increased and the reward effect for video collection can be doubled.

Figure 21 is a flowchart for explaining a control method in a mobile terminal according to an embodiment of the present invention.

First, a plurality of candidate images are stored in the memory 14 of the mobile terminal 10 (S10). For example, the portable terminal 10 can capture a video and select a representative frame from among the frames of the captured video. The mobile terminal 10 can use the representative frame as a candidate image (i.e., representative image), associate it with the corresponding video, and upload it to the server 50. At this time, the processor 16 of the portable terminal 10 may store the generated candidate image in the memory 14 .

Next, the user photographs the desired subject (e.g., product label, menu board, bus stop shelter advertisement, movie poster, concert poster, book, company introduction, etc.) and its surroundings through the camera 11 of the mobile terminal 10. Take a picture (S12). Of course, you can only photograph the subject you want to photograph.

Next, the mobile terminal 10 extracts a target image from the captured image captured by the camera 11 (that is, the image displayed on the entire front of the display 12) (S14). The target image may refer to an image of an area in a captured image where an object to be recognized exists. For example, the target image may refer to an image of an area that can be called a region of interest within a captured image.

A predetermined border pattern may be inserted (printed) into the photographed object. If the user captures a subject with a border pattern using the camera 11, the processor 16 of the portable terminal 10 can find the border pattern in the captured image. Then, the processor 16 of the mobile terminal 10 may determine the area surrounded by the border pattern as the target image area and extract the image within the determined area as the target image. In this case, since the area that can be called the area of interest is marked with a border pattern, the processor 16 of the mobile terminal 10 can extract the target image more quickly through the border pattern. Compared to when there is no border pattern in the photographed object, if there is a border pattern, the time required to find the target image in the photographed image will be shorter.

Then, the processor 16 of the mobile terminal 10 extracts a target image corresponding to the target image from the plurality of candidate images stored in the memory 14 (S16).

In this way, when the mobile terminal 10 extracts the target image corresponding to the target image, the captured image is recognized, and thus the image recognition in the server 50 is performed instead. As a result, the server 50 does not need to perform image recognition, thereby reducing the load on the server 50.

The mobile terminal 10 transmits the extracted target image to the server 50, and preferably transmits the identifier of the extracted target image to the server 50 (S18).

Accordingly, the server 50 searches for a candidate image corresponding to the received target image among a plurality of candidate images stored in the internal memory 32. Then, the server 50 selects a video according to a set rule or randomly from among a plurality of videos associated with the found candidate image (i.e., target image) in the area of the target image 9 on the display 12 of the mobile terminal 10. It is transmitted to the mobile terminal 10 so that it can be played within.

Accordingly, the mobile terminal 10 receives the video from the server 50 (S20) and plays the received video within the area of the target image on the display 12 (S22). At this time, the video may be displayed in augmented reality.

Figure 22 is a flowchart for explaining a control method in a server according to an embodiment of the present invention.

First, the server 50 (preferably the image server 30) stores a plurality of candidate images in the internal memory 32 (S30), and the video server 50 (preferably the video server 40) stores the internal memory 32 (S30). A plurality of videos associated with each candidate image are stored in the memory 42 (S32).

The server 50 may receive a target image (preferably an identifier of the target image) from the mobile terminal 10, but hereinafter, it is assumed that the server 50 receives the target image.

Accordingly, the server 50 (preferably the image server 30) receives the target image extracted from the captured image from the mobile terminal 10 (S34). Of course, if necessary, the server 50 may receive a captured image from the mobile terminal 10 and extract a target image from the received captured image, but in this case, the load on the server 50 increases. It can be. Therefore, it may be desirable to receive the target image extracted from the captured image from the portable terminal 10 if possible.

Next, the server 50 (preferably the image server 30) extracts a target image corresponding to the target image from the plurality of candidate images stored in the memory 32 (S36).

Then, the server 50 (preferably, the video server 40) selects one video from the memory 42 according to a set rule or randomly among a plurality of videos associated with the extracted target image (S38).

Next, the server 50 (preferably the video server 40) transmits the selected video to the mobile terminal 10 so that it can be played within the area of the target image on the display 12 of the mobile terminal 10 (S40) ).

In this way, the mobile terminal 10 plays the received video within the area of the target image on the display 12. Here, the video can be displayed in augmented reality.

On the other hand, it is also possible to use a sticker camera instead of the mobile terminal described in the above-described embodiments. When a sticker camera captures a video, it can select a representative frame as a representative image among the frames of the captured video and print the representative image. And, the sticker camera associates the representative image with the video and uploads it to the server. Afterwards, when the printed representative image is photographed as needed, the video associated with the representative image can be received from the server and played back through the same operation as described above.

Additionally, the method according to the embodiment of the present invention described above can be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, computer-readable recording media can be distributed across computer systems connected to a network, so that computer-readable code can be stored and executed in a distributed manner. And, functional programs, codes, and code segments for implementing the method can be easily deduced by programmers in the technical field to which the present invention pertains.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications and variations will be possible to those skilled in the art without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

10: mobile terminal 11: camera
12: display 13: position detection unit
14, 32, 42: memory 15, 31, 41: communication unit
16, 33, 43: Processor 20: Network
30: Image server 40: Video server
50: server

Claims (19)

camera;
display;
a memory for storing candidate images; and
A target image is extracted from a captured image captured by the camera, a target image corresponding to the target image is extracted from the candidate images, and a plurality of videos related to the extracted target image are selected according to a set rule and received from the server. A processor that plays a video within an area of the target image on the display,
The processor,
As a limit on the number of transmissions of a video selected from among a plurality of videos associated with the target image for the same target image is set on the server, if the limit on the number of transmissions is exceeded for the user, the server will send the target image Not receiving the selected video,
Mobile terminal. According to clause 1,
The rules set above are:
The selection probability of each of a plurality of videos associated with the target image,
Mobile terminal. According to clause 1,
The video received for the same target image is different from the video previously received for the same target image.
Mobile terminal. According to clause 1,
The processor,
Playing a video selected from among a plurality of videos associated with the target image according to the current location information of the mobile terminal and received from a server within the area of the target image on the display,
Mobile terminal. According to clause 1,
The received video contains instructions for the user,
Mobile terminal. According to clause 1,
The received video contains coupon content,
Mobile terminal. a first server storing candidate images; and
A second server that stores a plurality of videos associated with each of the candidate images,
The second server selects a video according to a set rule among a plurality of videos related to the target image extracted from the candidate images, and plays the selected video so that it can be played within the area of the target image on the display of the mobile terminal. transmit to the mobile terminal,
The second server is,
For the same target image, a limit value is set for the number of transmissions of a video selected from a plurality of videos associated with the target image,
If the transmission count limit for the user is exceeded, the video selected for the target image is not transmitted to the mobile terminal.
system. According to clause 7,
The first server is,
Receiving a target image from a mobile terminal and extracting a target image corresponding to the target image from the candidate images,
system. According to clause 7,
The rules set above are:
The selection probability of each of a plurality of videos associated with the target image,
system. According to clause 7,
The second server is,
The video selected for the same target image is selected differently from the video previously selected for the same target image.
system. According to clause 7,
The selected video contains instructions for the user,
system. According to clause 7,
The selected video contains coupon content,
system. According to clause 7,
The second server is,
Recording the transmission of the video transmitted to the mobile terminal by associating it with the account of the mobile terminal user,
system. According to clause 13,
The second server is,
Transmitting a video collection completion event signal to the mobile terminal when all of the plurality of videos related to the target image are associated with the mobile terminal user's account and have a transmission record,
system. delete According to clause 7,
The transmission number limit value is a transmission number limit value within a predetermined period,
system. According to clause 7,
The second server is,
The transmission time range of the video selected for the target image is set,
If the current time does not fall within the transmission time range, the video selected for the target image is not transmitted to the mobile terminal.
system. According to clause 7,
The second server is,
Selecting a video from among a plurality of videos associated with the target image according to the current location information of the mobile terminal,
system. According to clause 18,
The number of videos selected and transmitted in relation to the target image is limited in relation to the location,
system

Images