WO2019151548A1

WO2019151548A1 - Method, system, and non-transitory computer-readable recording medium for providing information on basis of face recognition

Info

Publication number: WO2019151548A1
Application number: PCT/KR2018/001365
Authority: WO
Inventors: 홍석주
Original assignee: 라인플러스 주식회사
Priority date: 2018-02-01
Filing date: 2018-02-01
Publication date: 2019-08-08
Also published as: KR102500624B1; KR20200106026A

Abstract

An information providing method may comprise the steps of: configuring a user DB by matching, for each user, face information of the user and communication information related to the user with user information for identification of the user and storing the matched information in the user DB; receiving a camera image from an electronic device in response to an information provision request; and retrieving communication information and face information of the user, which are associated with the camera image, in the user DB by using the camera image, and when face information of a specific user in the retrieved face information is recognized from the camera image, transferring, to the electronic device, a face position of the recognized face information in the camera image and communication information of the specific user in the retrieved communication information.

Description

Method and system for providing information based on face recognition and non-transitory computer readable recording medium

The description below relates to techniques provided for identifying information.

Recently, due to the development of digital technology, the use of personal terminals has become common and various services using the same have been provided.

Examples of personal terminals include smart phones, tablets, wearable devices, notebook computers, etc. These terminals include data communication functions, voice communication functions, camera functions, multimedia player functions, and payment functions. It is equipped with various functions.

As an example of a data communication function, Korean Laid-Open Patent Publication No. 10-2003-0017043 (published March 03, 2003) discloses a memo based on an instant messenger, a P2P based file transfer, a text chat, a voice chat, a video. Disclosed are techniques for providing unified communications functions, including conversations, Internet telephony, short message transmission, email confirmation, and chat room participation.

By recognizing a face captured by the camera, communication information related to the recognized user may be provided through a camera screen.

An information providing method executed in a server, the server comprising at least one processor configured to execute computer readable instructions contained in a memory, wherein the information providing method is performed by the at least one processor for each user. Configuring the user DB by matching the face information of the user and the communication information related to the user with the user information for identifying the user in a user DB; Receiving, by the at least one processor, a camera image as a request for providing information from an electronic device; And searching, by the at least one processor, the face information and the communication information of the user associated with the camera image in the user DB by using the camera image, and the face information of a specific user among the found face information is stored in the camera. If the image is recognized in the image of the camera provides a method of providing information comprising the step of transmitting the communication information of the specific user of the face position of the recognized face information and the searched communication information to the electronic device.

According to an aspect of the present disclosure, the configuring may include storing communication information including information directly input by the user for communication or information input by another person by specifying the user and matching the user information. can do.

According to another aspect, the communication information stored in the user DB may include location information at the time when the information is input.

According to another aspect, when communication information stored in the user DB is input in the process of consuming a specific content, the communication information may include content reproduction information at the time when the information is input.

According to another aspect, the information providing method, based on the position information of the camera image by comparing the position information of the communication information stored in the user DB and the position information of the camera image by the at least one processor. The method may further include obtaining user information matched with communication information including location information within a predetermined range, wherein the searching comprises: face information and communication information matched with the obtained user information in the user DB. It may include the step of searching.

According to another aspect, the predetermined range may be determined in consideration of a position coordinate value of a camera of the electronic device and a direction value of the camera image with respect to the camera.

According to another aspect, the electronic device may track the face position of the recognized face information on the camera image and display communication information of the specific user corresponding to the face position.

An information providing method executed in an electronic device, wherein the electronic device includes at least one processor configured to execute computer readable instructions included in a memory, wherein the information providing method includes: information, by the at least one processor; Transmitting a camera image as a request for providing to a server; Receiving, by the at least one processor, a face location of a user recognized in the camera image and communication information related to the user from the server; And tracking, by the at least one processor, the face position on the camera image and displaying the communication information corresponding to the face position.

According to an aspect of the present disclosure, the transmitting may include transmitting location information including a position coordinate value of the camera of the electronic device and a direction value of the camera image with respect to the camera, together with the camera image to the server. Can be.

According to another aspect of the present invention, the communication information includes information directly input by the user for communication or information input by another person to specify the user, and includes location information at the time when the corresponding information is input, and the reception is performed. The step may include the face location of a specific user recognized in the camera image among the users having communication information including location information within a predetermined range from the server based on the location information of the camera image and associated with the specific user. And receiving communication information.

According to another aspect, when the communication information is input in the process of consuming a specific content, the communication information includes content reproduction information at the time when the information is input, the displaying step, the face location Correspondingly, displaying the content reproduction information together with the communication information.

A non-transitory computer readable recording medium having recorded thereon a program for executing the information providing method on a computer is provided.

An information providing system implemented in a server, comprising: a memory; And at least one processor coupled to the memory, the at least one processor configured to execute computer readable instructions included in the memory, wherein the at least one processor comprises, for each user, face information of the user and communication information associated with the user; A user DB configuration unit configured to configure the user DB by matching the user information for identifying the user and storing the same in a user DB; A camera image receiver configured to receive a camera image as a request for providing information from an electronic device; And searching for face information and communication information of a user associated with the camera image in the user DB by using the camera image, and if the face information of a specific user is recognized in the camera image, And an information retrieval unit for transmitting the communication information of the specific user to the electronic device among the face location of the recognized face information and the retrieved communication information.

An information providing system implemented in an electronic device, comprising: a memory; And at least one processor coupled to the memory and configured to execute computer readable instructions included in the memory, wherein the at least one processor is configured to transmit a camera image to a server as an information providing request. ; An information receiver configured to receive a face position of a user recognized in the camera image and communication information related to the user from the server; And an information display unit for tracking the face position on the camera image and displaying the communication information corresponding to the face position.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.

2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention.

3 is a block diagram illustrating an example of components that may be included in a processor of a server according to an embodiment of the present invention.

4 is a flowchart illustrating an example of a method that may be performed by a server according to an embodiment of the present invention.

5 is a block diagram illustrating an example of components that may be included in a processor of an electronic device according to an embodiment of the present invention.

6 is a flowchart illustrating an example of a method that an electronic device can perform according to an embodiment of the present invention.

Figure 7 shows an example of a user DB in an embodiment of the present invention.

8 to 10 illustrate an example of an information providing process based on face recognition according to an embodiment of the present invention.

11 to 12 illustrate another example of an information providing process based on face recognition according to an embodiment of the present invention.

13 to 15 illustrate another example of an information providing process based on face recognition according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to a technique for providing information based on face recognition.

Embodiments including those disclosed in detail herein may provide communication information related to a recognized user by recognizing a face captured by a camera through a camera screen, thereby improving efficiency, variety, convenience, cost, and the like. Significant advantages can be achieved in terms of.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 illustrates an example including a plurality of

electronic devices

110, 120, 130, and 140, a plurality of

servers

150 and 160, and a network 170. 1 is an example for describing the present invention, and the number of electronic devices or the number of servers is not limited as shown in FIG. 1.

The plurality of

electronic devices

110, 120, 130, and 140 may be fixed terminals or mobile terminals implemented as computer systems. Examples of the plurality of

electronic devices

110, 120, 130, and 140 include smart phones, mobile phones, navigation systems, computers, notebook computers, digital broadcasting terminals, personal digital assistants (PDAs), and portable multimedia players (PMPs). ), Tablet PCs, game consoles, wearable devices, wearable devices, Internet of things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, and the like. As an example, although FIG. 1 illustrates the shape of a smartphone as an example of the electronic device 110, in the embodiments of the present invention, the electronic device 110 may be substantially different from the network 170 using a wireless or wired communication scheme. It may mean one of various physical computer systems capable of communicating with the

electronic devices

120, 130, 140 and / or the

servers

150, 160.

The communication method is not limited and includes not only a communication method using a communication network (for example, a mobile communication network, a wired internet, a wireless internet, a broadcasting network, a satellite network, etc.) that the network 170 may include, but also a short range wireless communication between devices. Can be. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). And one or more of networks such as the Internet. The network 170 may also include any one or more of network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, trees, or hierarchical networks, but It is not limited.

Each of the

servers

150 and 160 communicates with the plurality of

electronic devices

110, 120, 130, and 140 through the network 170 to provide a command, code, file, content, service, or the like. It may be implemented in devices. For example, the server 150 may be a system that provides a first service to a plurality of

electronic devices

110, 120, 130, and 140 connected through the network 170, and the server 160 may also have a network ( It may be a system that provides a second service to the plurality of

electronic devices

110, 120, 130, and 140 connected through the 170. As a more specific example, the server 150 is a service (for example, an information providing service, etc.) for which the corresponding application is intended through an application as a computer program installed and driven in the plurality of

electronic devices

110, 120, 130, and 140. May be provided to the plurality of

electronic devices

110, 120, 130, and 140 as a first service. As another example, the server 160 may provide a service for distributing the file for installing and driving the above application to the plurality of

electronic devices

110, 120, 130, and 140 as a second service.

2 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention. 2 illustrates an internal configuration of the electronic device 110 and the server 150 as an example of the electronic device. In addition, the other

electronic devices

120, 130, 140, or the server 160 may also have the same or similar internal configuration as the above-described electronic device 110 or the server 150.

The electronic device 110 and the server 150 may include

memories

211 and 221,

processors

212 and 222,

communication modules

213 and 223, and input /

output interfaces

214 and 224. The

memories

211 and 221 are non-transitory computer readable recording media. The

memories

211 and 221 are non-transitory computer-readable recording media. The

memories

211 and 221 are non-transitory computer-readable recording media. Permanent mass storage devices. In this case, the non-volatile mass storage device such as a ROM, an SSD, a flash memory, a disk drive, or the like may be included in the electronic device 110 or the server 150 as a separate persistent storage device that is distinct from the

memories

211 and 221. In addition, the

memory

211, 221 includes an operating system and at least one program code (for example, a code installed in the electronic device 110, a browser running on the electronic device 110, or an application installed in the electronic device 110 to provide a specific service). Can be stored. These software components may be loaded from a computer readable recording medium separate from the

memories

211 and 221. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, memory card, and the like. In other embodiments, software components may be loaded into the

memory

211, 221 through a

communication module

213, 223 that is not a computer readable recording medium. For example, at least one program is a computer program that is installed by files provided by a file distribution system (for example, the server 160 described above) through the network 170 to distribute installation files of developers or applications. It may be loaded into the

memories

211 and 221 based on (for example, the above-described application).

Processors

212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the

processors

212, 222 by the

memory

211, 221 or the

communication modules

213, 223. For example, the

processors

212 and 222 may be configured to execute a command received according to a program code stored in a recording device such as the

memory

211 and 221.

The

communication modules

213 and 223 may provide a function for the electronic device 110 and the server 150 to communicate with each other through the network 170, and the electronic device 110 and / or the server 150 may be different from each other. It may provide a function for communicating with an electronic device (for example, the electronic device 120) or another server (for example, the server 160). For example, a request generated by the processor 212 of the electronic device 110 according to a program code stored in a recording device such as the memory 211 may be controlled by the server 150 through the network 170 under the control of the communication module 213. Can be delivered. Conversely, control signals, commands, contents, files, and the like provided according to the control of the processor 222 of the server 150 pass through the communication module 223 and the network 170 to the communication module 213 of the electronic device 110. ) May be received by the electronic device 110. For example, the control signal, command, content, file, etc. of the server 150 received through the communication module 213 may be transmitted to the processor 212 or the memory 211, and the content, file, etc. may be transferred to the electronic device ( 110 may be stored as a storage medium (permanent storage device described above) that may further include.

The input / output interface 214 may be a means for interfacing with the input / output device 215. For example, an input device may include a device such as a keyboard, a mouse, a microphone, a camera, and the like, and an output device may include a device such as a display, a speaker, a haptic feedback device, and the like. As another example, the input / output interface 214 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 215 may be configured as the electronic device 110 and one device. In addition, the input / output interface 224 of the server 150 may be a means for interfacing with an apparatus (not shown) for input or output that may be connected to or included in the server 150. As a more specific example, a service configured using data provided by the server 150 or the electronic device 120 when the processor 212 of the electronic device 110 processes a command of a computer program loaded in the memory 211. The screen or content may be displayed on the display through the input / output interface 214.

In addition, in other embodiments, the electronic device 110 and the server 150 may include more components than those of FIG. 2. However, it is not necessary to clearly show most of the prior art components. For example, the electronic device 110 may be implemented to include at least some of the above-described input and output devices 215 or other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, and the like. It may also include more. More specifically, when the electronic device 110 is a smartphone, an acceleration sensor or a gyro sensor, a camera module, various physical buttons, a button using a touch panel, an input / output port, a vibrator for vibration generally included in the smartphone Various components such as may be implemented to be further included in the electronic device 110.

Hereinafter, a specific embodiment of a method and system for providing communication information related to a user of a face recognized in a camera image based on face recognition will be described.

3 is a block diagram illustrating an example of a component that a processor of a server according to an embodiment of the present invention may include, and FIG. 4 is an example of a method that the server may perform according to an embodiment of the present invention. It is a flowchart showing.

The server 150 according to the present embodiment serves as a platform for providing an information providing service to a plurality of

electronic devices

110, 120, 130, and 140 that are clients. The server 150 may provide an information providing service in association with an application installed on the

electronic devices

110, 120, 130, and 140.

In order for the server 150 to perform the information providing method according to FIG. 4, the processor 222 of the server 150 is a component as shown in FIG. 3. The user DB component 310 and the camera image receiver ( 320, a user information acquisition unit 330, and an information retrieval unit 340. In some embodiments, the components of the processor 222 may be optionally included in or excluded from the processor 222. In addition, according to an embodiment, the components of the processor 222 may be separated or merged to represent the functions of the processor 222.

The processor 222 and the components of the processor 222 may control the server 150 to perform steps S410 to S450 included in the information providing method of FIG. 4. For example, the processor 222 and the components of the processor 222 may be implemented to execute instructions according to code of an operating system included in the memory 221 and code of at least one program.

Here, the components of the processor 222 may be representations of different functions of the processor 222 performed by the processor 222 according to an instruction provided by the program code stored in the server 150. . For example, the user DB configuration unit 310 may be used as a functional representation of the processor 222 controlling the server 150 in accordance with the above-described command such that the server 150 configures the user DB.

In operation S410, the processor 222 may read a command required from the memory 221 loaded with a command related to the control of the server 150. In this case, the read command may include a command for controlling the processor 222 to execute steps S420 to S450 to be described later.

In operation S420, the user DB configuration unit 310 matches user face information for each user with information for identifying the corresponding user (for example, device ID, etc .; hereinafter referred to as 'user information') and stores it in the user DB. User DB can be configured by doing so. The user DB configuration unit 310 may construct face information of each user as a DB for information recognition services for an information providing service based on face recognition. In addition, the user DB configuration unit 310 may match the communication information associated with the user for each user with the user information may be stored in the user DB. In this case, the communication information may be information registered directly by the user for communication, or may include information specified by the user by another person. In other words, the communication information may be information that the user intends to deliver for sharing with others or information that others want to convey to the user. For example, the communication information for the user A may mean a message input by the user A to deliver to another person or a message input by another person to the user A through various communication channels such as a messenger chat room. As another example, the communication information about the user A may mean a comment input by the user A with respect to the corresponding content while consuming content on the Internet such as news or video. The communication information may include location information (for example, location information such as location coordinate values of user A or another person) and time information at the time when the information is input or registered, and further, the comment of a specific content In this case, the content may further include content reproduction information (for example, scene information when the comment is input) when the corresponding comment is input.

In operation S430, the camera image receiver 320 may receive a camera image from the electronic device 110 as a request for providing information. The camera image may include location information, and the location information of the camera image may include location information of a camera of the electronic device 110 or location information of objects (persons or objects) included in the camera image. For example, the position information of the camera image may include a position coordinate value of the camera and a direction value of the image with respect to the camera (for example, an orientation in which the lens of the camera faces to capture the image by the camera). As another example, the position information of the camera image may include position coordinate values of an object included in the image captured by the camera closest to the camera or an object closest to the center of the captured image, and a direction value of the image with respect to the camera (eg, For example, the camera may include an orientation in which the lens of the camera faces to capture an image. The location information may be provided by the electronic device 110, or may be captured by the camera, the position value of the camera acquired from the electronic device 110, the focal length at the time of capturing the corresponding image, the image resolution, and the camera. It is also possible to calculate directly based on the collected images.

In operation S440, the user information acquisition unit 330 compares the location information of the camera image received in operation S430 with the location information of the communication information stored in the user DB, based on the location information of the camera image. User information with communication information whose information is within a certain range can be obtained. Some ranges include the position coordinate values of the camera image (e.g., the position coordinate values of the camera or the position coordinates of the object captured by the camera) and the orientation value of the camera image relative to the camera (e.g., the camera captures the image. In order to prevent the lens of the camera from facing each other). The user information acquisition unit 330 may identify which users exist at a location within a predetermined range from the camera or an object in the image in the direction of the image with respect to the camera through the user DB. In other words, the user information acquirer 330 may obtain a user list that transmits or receives communication information around a location corresponding to the camera image of the electronic device 110.

In step S450, the information retrieval unit 340 retrieves the user's face information and communication information matched with the user information obtained in step S440 in the user DB, and then receives the retrieved face information in step S430. If the at least one of the detected face information is recognized in the camera image, it is requested to provide information about the detected face information along with the position of the face in the camera image. The electronic device 110 may transmit the electronic device 110. The information retrieval unit 340 does not retrieve face information of all users on the user DB, but the position information is fixed from the camera image in consideration of the position coordinates of the camera image captured by the camera and the direction value of the camera image with respect to the camera. The face information may be searched only for users having communication information determined to be included in the range. The information retrieval unit 340 also searches for communication information in which the time information of the communication information is included within a predetermined time from the time when the camera image is received, in consideration of the time information when the camera image is received, and has such communication information. The face information may be searched only for users. The information retrieval unit 340 may provide a face position of a user existing at a location indicated by the camera image and communication information related to the user in response to the request for providing the information from the electronic device 110. As another example, the information retrieval unit 340 may search for face information and communication information matched with the user information acquired in operation S440 in the user DB and transmit the same to the electronic device 110, and the electronic device 110. Allow the face information to be recognized in the camera image. When two or more pieces of communication information are retrieved with respect to face information of a user associated with a camera image, the information retrieval unit 340 is located at a location indicated by the camera image in response to a request for providing information from the electronic device 110. All or part of the two or more communication information may be provided along with the face location of the. For example, when two or more pieces of communication information are retrieved with respect to face information of a user associated with a camera image, the information retrieval unit 340 is the most recently input of the user's communication information or a doctor who has the best response on SNS. The delivery information may be selected and provided as communication information related to the user.

Accordingly, the server 150 may provide a service for identifying which user has what communication information at a location captured by the camera of the electronic device 110 based on face recognition.

5 is a block diagram illustrating an example of a component that may be included in a processor of an electronic device according to an embodiment of the present invention, and FIG. 6 is a method that may be performed by an electronic device according to an embodiment of the present invention. A flow chart showing an example of the.

The electronic device 110 according to the present embodiment may be configured with a computer-based information providing system. For example, the information providing system may be implemented in the form of a program that operates independently, or may be implemented in an in-app form of a specific application so as to be able to operate on the specific application. Information service can be provided through interworking with 150).

An information providing system implemented in the electronic device 110 may perform an information providing method based on a command provided by an application installed in the electronic device 110. In order to perform the information providing method according to FIG. 6, the processor 212 of the electronic device 110 is a component, as shown in FIG. 5, a camera image transmitter 510, an information receiver 520, and information. The display unit 530 may be included. In some embodiments, components of the processor 212 may optionally be included or excluded in the processor 212. In addition, according to an embodiment, the components of the processor 212 may be separated or merged for representation of the functions of the processor 212.

The processor 212 and the components of the processor 212 may control the electronic device 110 to perform steps S610 to S640 included in the information providing method of FIG. 6. For example, the processor 212 and the components of the processor 212 may be implemented to execute instructions according to code of an operating system included in the memory 211 and code of at least one program.

Here, the components of the processor 212 are performed by the processor 212 according to a command provided by a program code stored in the electronic device 110 (for example, a command provided by an application driven in the electronic device 110). Which may be representations of different functions of the processor 212. For example, the camera image transfer unit 510 is used as a functional representation of the processor 212 controlling the electronic device 110 according to the above-described command so that the electronic device 110 transmits the camera image to the server 150. Can be.

In operation S610, the processor 212 may read a necessary command from the memory 211 loaded with a command related to the control of the electronic device 110. In this case, the read command may include a command for controlling the processor 212 to execute steps S620 to S640 to be described later.

In operation S620, the camera image transmitter 510 may transmit an image, that is, a camera image, input through the camera of the electronic device 110 to the server 150 as a request for providing information. Here, the camera image may include an image obtained by extracting only a predetermined object such as a face of a person from an image captured by the camera or an image captured by the camera. That is, the camera image transmitting unit 510 may transmit the image data captured by the camera of the electronic device 110 to the server 150 as a request for providing the information, and the predetermined image such as the face of a person is determined in the image captured by the camera. Data extracted only from the object may be transmitted to the server 150. The camera image may include location information, and the location information of the camera image may include location information of the camera of the electronic device 110 or location information of objects determined to be included in the camera image. For example, the position information of the camera image may be a position coordinate value of the electronic device 110 or a position coordinate value of the camera acquired by the electronic device 110 from GPS information, base station information, or the like. The lens may include opposing orientations. As another example, the position information of the camera image is the position coordinate value of the object closest to the camera among the objects included in the image captured by the camera or the object closest to the center of the captured image, and the lens of the camera to capture the image. May include opposite bearings.

In operation S630, the information receiver 520 may receive a face position and communication information of the user recognized in the camera image from the server 150. In other words, the information receiver 520 may receive face information of a user existing at a location indicated by the camera image and communication information related to the user in response to the request for providing the information from the server 150.

In operation S640, the information display unit 530 provides an AR (Augmented Reality) function that allows information to be checked through the camera. The information display unit 530 reflects the information received from the server 150 to the camera image and outputs the received image. Track the face position and display communication information of the corresponding user in response to the face position. In other words, the information display unit 530 may detect and display the communication information transmitted from the camera image captured by the camera or the received user through face recognition.

Accordingly, the electronic device 110 may show, as AR content on the camera image, which user has what communication information in the image captured by the camera based on face recognition.

7 shows an example of a user DB 700 in one embodiment of the present invention. In the user DB 700, face information 720 and communication information 730 of each user may be matched with the corresponding user information 710 and stored. The server 150 may receive the face image of the user for each user and store the face information 720 including the feature data extracted from the input face image in correspondence with the user information 710. As another example, the server 150 receives a face image of a user for each user, constructs a face DB in which feature data extracted from the input face image is stored together with each face ID, and creates a new pseudo for any user. When the delivery information is received, the face ID of the user may be stored as the face information 720 together with the new communication information 730. In addition, the server 150 communicates information related to the user through an interface screen interoperable with the server 150, for example, a chat room of an official account such as a fan meeting or a live broadcast, a comment registration screen such as SNS or news, etc. 730, that is, information that the user wants to share for sharing with others or information that another person wants to deliver to the user may be obtained and matched with the user information 710 and stored. In this case, the communication information 730 is the object 731 for communication, content 732 input for communication, location information 733 at the time when the information is input, and the information is input or transmitted. Time information 734 may be included. As shown in FIG. 7, as an example, the communication target 731 may access a content ID for identifying a live broadcast, SNS, news, and the like, a chat room ID in a messenger, an SNS, an online community, or the like. The location information 733 may include a URL that can be inputted, and the location information 733 includes a time point at which content input starts, a time point at which content input ends, a time point between start and end of content input, and contents input by a user operation. It may include location information of a predetermined specific time point, such as a transmission time point. As another example, the positional information 733 is a positional information array ({(positional coordinate value at the start of inputting content), (position of contents) representing the positional information from the time point at which content input starts to the end point as an array. A position coordinate value at the time point between the start and the end of the input), and (a position coordinate value at the time when the content input by the user's operation is transmitted)}. Although not shown, the communication information 730 for the specific content may further include content reproduction information at the time when the corresponding information is input. For example, the video may further include and store scene information (for example, a scene corresponding to 50 seconds after playback) when the communication information 730 is input.

8 shows a chat room 800 of an official account for receiving questions from participants in a fan meeting of a celebrity group. User_1 801, one of the fan meeting participants, inputs a question 803 by specifying User_3 802 belonging to a celebrity group through the chat room 800. In this case, when the question 803 is input, the electronic device 120 of the User_1 801 may transmit the communication information including the target to which the question 803 is input, the content input to the question 803, the current location, and the time. In this case, the server 150 may match the communication information received from the electronic device 120 with the user information User_3 specified in the question 803 and store it in the user DB 700. have.

FIG. 9 illustrates a camera image 900 in which another user or a relay electronic device 110 at a fan meeting site captures a celebrity group. The electronic device 110 transmits the camera image 900 to the server 150. The server 150 checks the user_3 received a question at the fan meeting site based on the information on the user DB 700, and then the camera image ( In operation 900, the facial location 910 of User_3 may be recognized, and the recognized facial location 910 and the corresponding question content may be transmitted to the electronic device 110. In this case, the electronic device 110 may track the face location 910 of User_3 in the camera image 900 and display the question content 920 received by User_3 in an area corresponding to the face location 910.

Therefore, the electronic device 110 may recognize the face of the user who received the question in the camera image of the fan meeting site through the interworking with the server 150 and display the corresponding question content.

As another example, when User_1 801 inputs a question 803 to User_3 802 belonging to a group through chat room 800, server 150 queries communication information received from electronic device 120. It may be stored in the user DB 700 by matching the user information User_1 inputted with the question 803 instead of the user information User_3 that has received 803. FIG. 10 illustrates a camera image 1000 in which another user or the relay electronic device 110 at the fan meeting site captures the fan meeting participants. Similarly, the electronic device 110 transmits the camera image 1000 to the server 150, and after this, the server 150 checks User_1 who inputs a question at the fan meeting site based on the information on the user DB 700. The face location 1010 of User_1 may be recognized in the camera image 1000, and the recognized face location 1010 and the corresponding question content may be transmitted to the electronic device 110. In this case, the electronic device 110 may track the face location 1010 of User_1 in the camera image 1000 and display the question content 1020 input by User_1 in an area corresponding to the face location 1010.

Accordingly, the electronic device 110 may recognize the face of the user who inputs the question in the camera image of the fan meeting site through the interworking with the server 150 and display the corresponding question content.

11 shows a chat room 1100 of an official account for a direct transaction. One user User_1 1101 moves to an appointment place for a transaction and executes location transmission 1102 through chat room 1100. In this case, when the location transmission 1102 is executed, the electronic device 120 of the User_1 1101 may transmit communication information including the location transmission contents, the current location, and the time to the server 150, and at this time, the server 150 The communication information transmitted from the electronic device 120 may be matched with the user information User_1 transmitting the location and stored in the user DB 700.

FIG. 12 illustrates a camera image 1200 captured by another user who has promised a direct transaction around an appointment place. The electronic device 110 transmits the camera image 1200 to the server 150. The server 150 checks User_1 transmitting the location around the place based on the information on the user DB 700, and then displays the camera image. In operation 1200, the face location 1210 of the user_1 is recognized, and the recognized face location 1210 is transmitted to the electronic device 110. In this case, the electronic device 110 may track the face location 1210 of User_1 in the camera image 1200 and display an indicator 1220 indicating User_1 in an area corresponding to the face location 1210.

Accordingly, the electronic device 110 may recognize the face of the user who promised the transaction in the camera image of the direct transaction appointment place through the linkage with the server 150 and show the corresponding location.

13 illustrates a content consumption screen 1300 including a comment function such as SNS or news, for example, a video playback screen. The user User_2 1301 views a video from the street and inputs a comment 1302 for the video. In this case, when the User_2 1301 inputs a comment 1302, the electronic device 120 of the User_2 1301 may display the target, the comment content, and the location, time, and scene information at the time when the comment 1302 is input. The communication information may be transmitted to the server 150. In this case, the server 150 matches the communication information received from the electronic device 120 with the user information (User_2) inputted in the comment 1302, and the user DB. Can be stored at 700.

14 illustrates a camera image 1400 in which another user captures a distance in the vicinity of where the user User_2 is located. The electronic device 110 of another user transmits the camera image 1400 to the server 150 together with the information about the communication target, and the server 150 at the corresponding distance based on the information on the user DB 700. After confirming User_2 inputting a comment on the communication target, the camera image 1400 recognizes the user's face position 1410 and transmits the recognized face position 1410 to the electronic device 110. In this case, the electronic device 110 may track the face location 1410 of User_2 in the camera image 1400 and display the comment content 1420 input by User_2. Then, the electronic device 110 will show the scene information 1430 at the time of inputting the comment along with the comment content 1420 input by User_2 in the camera image 1400 according to the information provided by the server 150. Can be.

As illustrated in FIG. 15, the electronic device 110 may separately include a user comment 1520 that does not recognize a user's face or does not have face information pre-registered among comments registered at a distance corresponding to the camera image 1400. It is also possible to show through region 1500 of.

Accordingly, the electronic device 110 may recognize the face of the user who inputs the comment at a peripheral distance from the camera image and display the corresponding comment content through the linkage with the server 150.

When a content owner or a general user consumes news or a video, the camera may be driven to check other users' reactions to the content through the camera image. Content owners can check the user's reaction by grouping the contents or grouping all the contents owned by them. Depending on the location of the point the camera is capturing or the direction the camera lens is facing, the response of another user located near the point the camera is capturing is displayed as the AR content of the camera image. New comments can be displayed and real time to see what reactions have been made in which scenes.

As described above, according to embodiments of the present invention, the communication information related to the user is searched through a DB stored with the location information to search for a user existing at a position corresponding to the camera image, and recognized by recognizing a face of the user found in the camera image. The location of the user's face and information delivery information related to the user may be shown through a camera image.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable PLU (programmable). It can be implemented using one or more general purpose or special purpose computers, such as logic units, microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium or device in order to be interpreted by or provided to the processing device or to provide instructions or data. have. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. In this case, the medium may be to continuously store a program executable by the computer, or to temporarily store for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a single or several hardware combined, not limited to a medium directly connected to any computer system, it may be distributed on the network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And ROM, RAM, flash memory, and the like, configured to store program instructions. In addition, examples of another medium may include a recording medium or a storage medium managed by an app store that distributes an application, a site that supplies or distributes various software, a server, or the like.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims

In the information providing method running on the server,

The server comprises at least one processor configured to execute computer readable instructions contained in a memory,

The information providing method,

Configuring, by the at least one processor, the user DB by storing face information of the user and communication information related to the user for each user with user information for identifying the user and storing the user information in the user DB;

Receiving, by the at least one processor, a camera image as a request for providing information from an electronic device; And

The face information and communication information of the user associated with the camera image are retrieved from the user DB by the at least one processor, and the face information of a specific user is searched for from the detected face information. Transmitting the communication information of the specific user to the electronic device among the face position of the recognized face information and the searched communication information in the camera image when recognized in

Information providing method comprising a.
The method of claim 1,

The configuring step,

Storing the communication information including information directly input by the user for communication or information input by others by specifying the user and matching the user information;

Including, the information providing method.
The method of claim 1,

The communication information stored in the user DB includes information on the position at the time when the corresponding information is input.
The method of claim 1,

When communication information stored in the user DB is input in the process of consuming a specific content, the communication information includes content reproduction information at the time when the information is input.
The method of claim 3,

The information providing method,

By the at least one processor, communication information including location information within a predetermined range based on the location information of the camera image by comparing the location information of the communication information stored in the user DB with the location information of the camera image Obtaining the matched user information;

The searching step,

Searching for face information and communication information matched with the obtained user information in the user DB;

Including, the information providing method.
The method of claim 5,

The predetermined range is determined in consideration of a position coordinate value of a camera of the electronic device and a direction value of the camera image with respect to the camera.
The method of claim 1,

And in the electronic device, a face position of the recognized face information is tracked on the camera image, and communication information of the specific user is displayed corresponding to the face position.
In the information providing method executed in an electronic device,

The electronic device comprises at least one processor configured to execute computer readable instructions contained in a memory,

The information providing method,

Transmitting, by the at least one processor, a camera image as a request for providing information to a server;

Receiving, by the at least one processor, a face location of a user recognized in the camera image and communication information related to the user from the server; And

Tracking, by the at least one processor, the face position on the camera image and displaying the communication information corresponding to the face position;

Information providing method comprising a.
The method of claim 8,

The delivering step,

Transmitting location information including a position coordinate value of a camera of the electronic device and a direction value of the camera image with respect to the camera, together with the camera image to the server.

Including, the information providing method.
The method of claim 9,

The communication information includes information directly input by the user for communication or information input by another person by specifying the user, and includes location information at the time when the corresponding information is input.

The receiving step,

Among the users having communication information including location information within a predetermined range from the server based on the location information of the camera image, the face position of the specific user recognized in the camera image and communication information related to the specific user are displayed. Receiving Step

Including, the information providing method.
The method of claim 8,

When the communication information is input in the process of consuming a specific content, the communication information includes content reproduction information at the time when the information is input,

The displaying step,

Displaying the content reproduction information together with the communication information corresponding to the face position

Including, the information providing method.
A non-transitory computer readable recording medium having recorded thereon a program for causing a computer to execute the method of providing information according to any one of claims 1 to 11.
In the information providing system implemented in the server,

Memory; And

At least one processor coupled with the memory and configured to execute computer readable instructions contained in the memory

Including,

The at least one processor,

A user DB configuration unit configured to configure the user DB by matching face information of the user and communication information related to the user with user information for identifying the user for each user and storing the same in the user DB;

A camera image receiver configured to receive a camera image as a request for providing information from an electronic device; And

Search for the face information and the communication information of the user associated with the camera image in the user DB by using the camera image, and if the face information of a specific user is recognized in the camera image among the searched face information, An information retrieval unit for transmitting the communication information of the specific user to the electronic device among the face position of the recognized face information and the searched communication information;

Information providing system comprising a.
The method of claim 13,

The user DB configuration unit,

And communication information including information directly input by the user for communication, or information input by others by specifying the user, is stored in matching with the user information.
The method of claim 13,

The communication information stored in the user DB includes information on the location of the time point at which the corresponding information is input.
The method of claim 13,

And when the communication information stored in the user DB is input in the process of consuming a specific content, the communication information includes content reproduction information at the time when the information is input.
The method of claim 15,

The at least one processor,

Comparing the location information of the communication information stored in the user DB with the location information of the camera image to obtain user information matched with the communication information including location information within a predetermined range based on the location information of the camera image; User information acquisition unit

More,

The information search unit,

And searching for face information and communication information matched with the obtained user information in the user DB.
The method of claim 17,

The predetermined range is determined in consideration of a position coordinate value of a camera of the electronic device and a direction value of the camera image with respect to the camera.
In the information providing system implemented in an electronic device,

Memory; And

At least one processor coupled with the memory and configured to execute computer readable instructions contained in the memory

Including,

The at least one processor,

A camera image transmitting unit which transmits a camera image as a request for providing information to a server;

An information receiver configured to receive a face position of a user recognized in the camera image and communication information related to the user from the server; And

An information display unit for tracking the face position on the camera image and displaying the communication information corresponding to the face position.

Information providing system comprising a.
The method of claim 19,

The communication information includes information directly input by the user for communication, information input by another person to specify the user, and location information at the time when the information is input,

The information receiver,

Receiving communication information related to the face position of a specific user recognized in the camera image among the users with communication information including location information within a predetermined range based on the location information of the camera image from the server Information providing system.