KR102034839B1 - Terminal and server providing a video call service - Google Patents

Abstract

Terminal and video call service providing a video call service that determines whether a user of a terminal providing a video call service is a limited user of the video call service using distribution information of facial feature points of a user limited in providing a video call service. Initiate a server that provides.

Description

Terminal and server providing video call service {TERMINAL AND SERVER PROVIDING A VIDEO CALL SERVICE}

A terminal for providing a video call service and a server for providing a video call service.

With the development of science and technology, most people have terminals such as smartphones and tablet PCs. Such a terminal may capture and play an image, communicate with another terminal, exchange information, or support a video call.

There are various kinds of video calling services based on video calling. For example, the random video chat service is a service that connects a terminal of a user who makes a request for using a random video chat service with a terminal of a user randomly selected among users using the random video chat service.

In order to provide various kinds of video call services including a random video chat service with good quality, a method of restricting the use of the video call service to a user who has intentionally cheated the other party using the video call service is required.

Terminal and video call service providing a video call service that determines whether a user of a terminal providing a video call service is a limited user of the video call service using distribution information of facial feature points of a user limited in providing a video call service. It is to provide a server that provides.

According to a first aspect, a server comprises: a communication interface for communicating with a terminal; storage; A processor; And a memory configured to store instructions executable by the processor, wherein the processor executes the instructions, thereby executing a user providing the video call service in the terminal. Receiving a captured video stream, extracts the facial feature points of the user of the terminal from the received video stream, the distribution information database of the facial feature points of the user limited to the provision of the video call service stored in the storage and the extracted terminal Based on the similarity between the distribution information of the facial feature points of the user, it may be determined whether the user of the terminal is limited to the provision of the video call service, and transmit the determination result to the terminal.

The server according to the second aspect comprises a communication interface for communicating with the terminal; storage; A processor; And a memory configured to store instructions executable by the processor, wherein the processor is further configured to execute the instructions, based on distribution information of facial feature points of a user whose provision of a video call service is limited. Generate and store a learning model for determining a user limited in the provision, and store in the storage, and receives an image stream photographing the user of the terminal as an application for providing a video call service is executed in the terminal, Extracting facial feature points of a user of the terminal from a video stream, and determining whether the user of the terminal is a limited user of the video call service using the learning model and the extracted facial feature points stored in the storage, and determining Send the result to the terminal Can.

The application stored in the computer-readable storage medium according to the third aspect is generated in the terminal based on the distribution information of the facial feature points of the user limited in the provision of the video call service, for determining the user limited in the provision of the video call service Receiving a learning model from a server; Acquiring a video stream photographing a user of the terminal as an application for providing a video call service is executed in the terminal; Extracting facial feature points of a user of the terminal from the obtained video stream; Determining whether a user of the terminal is limited to providing the video call server by using the learning model and the extracted facial feature points; And maintaining or limiting execution of an application that provides the video call service based on the determination result.

1 is a diagram illustrating an environment in which a video call service is provided.
2 is a block diagram illustrating a configuration of a terminal providing a video call service according to an exemplary embodiment.
3 is a block diagram illustrating a configuration of a server providing a video call service according to an exemplary embodiment.
FIG. 4 is a diagram illustrating an example in which a server providing a video call service is implemented by a plurality of distributed servers. Referring to FIG.
5 is a diagram for describing a process of determining whether a provision of a video call service is a limited user based on distribution information of facial feature points of a user, according to an exemplary embodiment.
6 and 7 illustrate that a server providing a video call service is restricted from providing a video call service by using a distribution information database of facial feature points of a user whose provision of a video call service is limited. A diagram for describing an embodiment of determining whether a user is a user.
FIG. 8 illustrates various methods of determining whether provision of a video call service is limited based on a similarity between a distribution information database of facial feature points of a user limited in providing a video call service and distribution information of facial feature points of a user of the terminal. It is a figure for following.
9 and 10 illustrate a user who is limited to providing a video call service by using a learning model for a server providing a video call service to determine a user who is limited in providing a video call service, according to another exemplary embodiment. It is a figure for explaining the Example which determines whether it is.
11 and 12 illustrate a method in which a terminal providing a video call service is limited in providing a video call service by using a learning model for determining a user whose provision of the video call service is limited. A diagram for describing an embodiment of determining whether a user is a user.
13 and 14 illustrate a method for providing a video call service by using a distribution information database of facial feature points of a user in which a terminal providing a video call service is limited in providing a video call service, according to another embodiment. FIG. Is a view for explaining an embodiment of determining whether a user is a limited user.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings. The embodiments described below may be embodied in various different forms. In order to more clearly describe the features of the embodiments, detailed descriptions of matters well known to those skilled in the art to which the following embodiments belong will be omitted.

On the other hand, when a certain configuration is "connected" with another configuration in the present specification, this includes not only 'directly connected', but also 'connected between the other configuration in the middle'. In addition, when one configuration "includes" another configuration, this means that, unless specifically stated otherwise, it may further include other configurations other than the other configuration.

In addition, terms including ordinal numbers such as 'first' or 'second' as used herein may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The embodiments of the present invention relate to a terminal and a server for providing a video call service, and thus detailed descriptions of matters well known to those skilled in the art to which the following embodiments belong will be omitted.

1 is a diagram illustrating an environment in which a video call service is provided.

The video call service refers to a service that allows a plurality of users to exchange their intentions by transmitting their video to the counterpart and receiving the counterpart's video using their respective terminals. Users using the video call service can exchange video and audio through their respective terminals, and can also send and receive text through a chat function. A user who wants to use the video call service may use the video call service with the other party by directly designating the other party or by designating the other party in a random or predetermined manner in a server providing the video call service.

Referring to FIG. 1, the terminal 100 is connected to a server 200 that provides a video call service through a communication network. The server 200 may store various programs or applications and data which help a plurality of users to use a video call service using their respective terminals 100. The server 200 may perform both short range communication and remote location communication. The server 200 may be connected to the plurality of terminals 100 through a communication network. The terminal 100 may be various types of user terminals that may be used to connect to the server 200. For example, the terminal 100 is a device capable of communicating with the server 200, and is a wearable device such as a smart watch, a mobile device such as a smart phone, a tablet PC, a laptop computer, or a stationary device such as a desktop computer. It may be a stationary device or the like. In addition, the terminal 100 may be a video call device that supports a video call, which can capture and play back images so that video calls can be made between users connected through a video call service.

2 is a block diagram illustrating a configuration of a terminal 100 according to an exemplary embodiment.

Referring to FIG. 2, the terminal 100 may include a memory 110, a processor 120, a user interface 130, a communication interface 140, and an audio and video input unit 150. Those skilled in the art will appreciate that other general purpose elements may be included in addition to the elements shown in FIG. 2.

The memory 110 may store software or a program. For example, the memory 110 may store an application, a program such as an application programming interface (API), and the like, and various kinds of data. The memory 110 may store instructions executable by the processor 120.

The processor 120 may execute instructions stored in the memory 110. The processor 120 may use various programs, contents, and data stored in the memory 110, or may store new programs, contents, and data in the memory 110.

The processor 120 may access the memory 110 and perform booting using an operating system stored in the memory 110. The processor 120 may perform various operations by using various programs, contents, data, and the like stored in the memory 110. For example, the processor 120 may display a predetermined screen on the display 136 using various programs, contents, and data stored in the memory 110. When a user manipulation of one region of the display 136 is performed, the processor 120 may perform a control operation corresponding to the manipulation of the user.

The processor 120 may include a graphic processing unit (GPU) specialized in graphic processing. When the booting of the terminal 100 is completed, the GPU displays a user interface screen on an area of the display 136. Specifically, the GPU may generate a screen on which a video call layout including various objects such as content, icons, and menus is displayed. The GPU may calculate attribute values such as coordinates, shapes, sizes, and colors in which the objects are to be displayed according to the video call layout of the screen. The GPU may generate screens of various layouts including objects based on the calculated attribute values. The screen generated by the GPU may be provided to the display 136 and displayed on each area of the display 136.

The processor 120 may include a video processor and an audio processor. The processor 120 may control the video processor and the audio processor to process the video data or the audio data included in the video stream received through the communication interface 140 or the video stream stored in the memory 110.

The user interface 130 may include an input unit 131 and an output unit 135.

The input unit 131 may receive various commands from a user. The input unit 131 may include at least one of a keypad 132, a touch panel 133, and a pen recognition panel 134.

The keypad 132 may include various types of keys, such as mechanical buttons and wheels, which are formed in various areas such as a front portion, a side portion, or a rear portion of the main body exterior of the terminal 100.

The touch panel 133 may detect a user's touch input and output a touch event value corresponding to the detected touch signal. When the touch panel 133 is combined with the display panel to form a touch screen, the touch screen may be implemented as various types of touch sensors such as capacitive, pressure sensitive, and piezoelectric.

The pen recognition panel 134 may detect a proximity input or touch input of the pen according to the operation of the user's touch pen (eg, a stylus pen) and output the detected pen proximity event or pen touch event. . The pen recognition panel 134 may be implemented by, for example, an electromagnetic radiation (EMR) method, and may detect a touch or a proximity input according to a change in the intensity of an electromagnetic field due to a proximity or a touch of a pen. The pen recognition panel 134 may include an electromagnetic induction coil sensor having a grid structure and an electronic signal processor for sequentially providing an AC signal having a predetermined frequency to each loop coil of the electromagnetic induction coil sensor.

The output unit 135 may include a display 136 and a speaker 137.

The display 136 may include a display panel and a controller to control the display panel. The display panel may be implemented in various ways such as a liquid crystal display (LCD), an organic light emitting diodes (OLED) display, an active-matrix organic light-emitting diode (AM-OLED), a plasma display panel (PDP), and the like. The display panel may be implemented to be flexible or wearable. The display 136 may be combined with the touch panel 133 of the input unit 131 and provided as a touch screen.

The speaker 137 may output sound based on the audio data. For example, the speaker 137 may output the user's voice according to the audio data included in the video stream.

The communication interface 140 may communicate with various types of external devices according to various types of communication methods. The communication interface 140 may include at least one of a Wi-Fi chip, a Bluetooth chip, a Near Field Communication (NFC) chip, and a wireless communication chip. The processor 120 may communicate with various external devices using the communication interface 140.

The Wi-Fi chip and the Bluetooth chip may communicate with each other by WiFi or Bluetooth. In the case of using a Wi-Fi chip or a Bluetooth chip, various connection information such as SSID and session key may be transmitted and received first, and then various communication information may be transmitted and received by using the same. NFC chip refers to a chip that operates in the NFC method using the 13.56MHz band of the various RF-ID frequency bands. Wireless communication chips are based on various communication standards such as Institute of Electrical and Electronics Engineers (IEEE), Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and 5th Generation (5G). Means a chip that performs communication.

The audio and video input unit 150 may include a microphone 151 and a camera 153. The microphone 151 may receive a user voice or other sound and convert it into audio data. The processor 120 may use the user's voice input through the microphone 151 in a video call, or convert the user's voice into audio data and store it in the memory 110. The camera 153 may capture a still image or a video under the control of the user. The camera 153 may be a camera module located at the front or the rear of the terminal 100. The processor 120 may generate a video stream for a video call by using the voice input through the microphone 151 and the image captured by the camera 153.

Meanwhile, the terminal 100 may operate in a motion control mode or a voice control mode. When operating in the motion control mode, the processor 120 may activate the camera 153 to photograph the user, track the motion change of the user, and perform a control operation corresponding thereto. When operating in the voice control mode, the processor 120 may analyze a user voice input through the microphone 151 and perform a control operation according to the analyzed user voice.

The names of the components of the terminal 100 described above may vary. In addition, the terminal 100 according to the present disclosure may be configured to include at least one of the above-described components, some components may be omitted or further include additional components. The terminal 100 may perform the following operations by using at least one of the above-described components.

3 is a block diagram illustrating a configuration of a server 200 that provides a video call service according to an exemplary embodiment.

Referring to FIG. 3, the server 200 includes a memory 210, a processor 220, a storage 230, and a communication interface 240. Those skilled in the art can recognize that the server 200 may further include other general purpose components in addition to the components illustrated in FIG. 3. Each component of the block diagram of FIG. 3 may be separated, added, or omitted depending on the implementation manner of the server 200. That is, one component may be subdivided into two or more components, two or more components may be combined into one component, and some components may be further added or removed according to an implementation manner.

The memory 210 may store instructions executable by the processor 220. The memory 210 may store software or a program.

The processor 220 may execute instructions stored in the memory 210. The processor 220 may control the overall operation of the server 200. The processor 220 may obtain information and a request received through the communication interface 240 and store the received information in the storage 230. In addition, the processor 220 may process the received information. For example, the processor 220 may generate information used for the video call service from the information received from the terminal 100 or perform processing to manage the received information and store the information in the storage 230. . In addition, the processor 220 provides a video call service to the terminal 100 through the communication interface 240 by using information stored in the storage 230 as a response to the request obtained from the terminal 100. Information can be sent.

The storage 230 may store various software and information necessary for the server 200 to provide a video call service. For example, the storage 230 may store programs, applications, and various data used for a video call service executed in the server 200.

The storage 230 may store and manage a database of personal information for each user using a video call service. The storage 230 may store various information such as user's personal information and virtual property information used for a video call service for each account for accessing the server 200. The personal information may include personal information such as nationality, name, gender, and interest of the user who uses the video call service. The virtual goods information may be a virtual goods for using a supplementary service provided by a user using a video call service for a fee. For example, in order to perform video call matching with a counterpart of a user's desired condition, the user must pay a predetermined virtual goods.

The communication interface 240 may communicate with an external device including the terminal 100. For example, the server 200 receives a video call service start request from the terminal 100, a request for setting information for establishing a video call service environment, and the like, and responds to the request of the terminal 100. Information related to the call service may be provided.

4 is a diagram for describing a server 200 that provides a video call service according to another exemplary embodiment in which a plurality of distributed servers are implemented. The contents described above with respect to the server 200 for providing a video call service may be applied even if omitted below.

Referring to FIG. 4, distributed servers providing a video call service may include a load balancing server 200-1 and function servers 200-3, 200-5, and 200-7 providing a video call service. have. The load balancing server 200-1 is a function server for providing a video call service when a request for a video call service of an external device such as the terminal 100 is provided (200-3, 200-5, 200-7). Determining any of the servers to connect to the terminal 100, or monitor the status of the function server (200-3, 200-5, 200-7) providing a video call service to select the optimal server 100 can be connected.

On the other hand, a service provider that provides a video call service may take action against a user who has cheated to provide a good quality video call service. An example of a cheating user may be a person who exposes a certain level or more, racist remarks or insults, or executes anti-social or anti-human content while using the video call service. A service provider providing a service for providing a video call service may restrict the provision of a video call service to a user who has cheated on behalf of other users. For example, the service provider may remove the cheating user from the user list that can use the video call service or store it in the user list that cannot use the video call service. The service provider may ignore or block the video call service start request of the user who cheated, terminate the application providing the executed video call service, or provide the video call service based on the history of cheating. By providing a message indicating that the limited can be informed that the further progress of the video call service is not possible. Hereinafter, the terminal 100 and the server 200 that determine whether the user of the terminal is limited in the provision of the video call service by using distribution information of the facial feature points of the user whose provision of the video call service is restricted due to cheating or the like. Explain.

5 is a diagram for describing a process of determining whether a provision of a video call service is a limited user based on distribution information of facial feature points of a user, according to an exemplary embodiment.

Referring to FIG. 5, a user may execute an application for providing a video call service in the user terminal 100. As the application is executed in the terminal 100, the terminal 100 may display information required for providing a video call service on a display of the terminal 100 and receive an input from a user. The terminal 100 may photograph the user's face in order to register or log in to the user for an application that provides a video call service. Alternatively, as the application is executed in the terminal 100, before the video call is made between the terminal 100 of the user and the terminal of the counterpart of the video call through the application, or during the video call, the terminal 100 faces the user's face. You can collect the images taken. If a user who is already restricted from using the video call service because of cheating attempts to register a new member, logs in with a different account, or tries to use the video call service using another user's terminal logged in with a different account, This is to block a user who has limited service provision from using the video call service.

The terminal 100 may obtain an image or an image stream of the user through the camera 132. The video stream may include at least one video. The terminal 100 may provide a guideline on the screen of the terminal 100 so that the angle and size of the face of the user of the terminal 100 are photographed according to a predetermined criterion. The reason for providing a guideline is to obtain a facial image of the maximum standardized angle and size. The terminal 100 may obtain an image stream in which the face of the user of the terminal 100 is photographed according to the provided guidelines.

The terminal 100 or the server 200 may extract facial feature points of the user from an image stream photographing the user based on a predetermined image processing algorithm. Here, the facial feature points may be obtained from a specific shape, pattern, color, brightness, or a combination thereof included in the facial image. For example, the terminal 100 or the server 200 captures the image of the user's face most front or the user's face most clearly based on the angle photographed from the image stream of the user. By selecting an image of the size closest to the size, the facial feature points are extracted from the eyes, nose, mouth, forehead, cheek, jaw and facial contours of the user's face of the terminal 100 from the selected image. can do.

The terminal 100 or the server 200 may determine whether the provision of the video call service is limited based on the distribution information of the facial feature points of the user.

If it is determined that the user is a user who is not limited to providing the video call service, the terminal 100 or the server 200 may maintain execution of an application for providing the video call service.

If it is determined that the user is a limited user of providing the video call service, the terminal 100 or the server 200 may limit the execution of the application for providing the video call service.

Hereinafter, various embodiments which are distinguished according to a subject and a method of determining whether the provision of a video call service is a limited user will be described with reference to FIGS. 6 to 14.

6 and 7 illustrate an example in which a server 200 providing a video call service uses a distribution information database of facial feature points of a user whose provision of a video call service is restricted, so that a user of the terminal 100 may display an image. FIG. 14 is a diagram for describing an embodiment of determining whether a provision of a call service is a limited user.

Referring to FIG. 6, the server 200 communicates with the terminal 100 to receive a video stream photographing a user of the terminal 100, and the user of the terminal 100 is limited in providing a video call service. The determination result of determining whether the user is a user may be transmitted to the terminal 100.

As the application for providing a video call service is executed in the terminal 100, the terminal 100 may photograph a user during a membership registration process, a login process, a video call service connection, or a video call service use process. When the terminal 100 satisfies a predetermined time or condition, the terminal 100 may photograph the user of the terminal 100 and transmit the captured image stream to the server 200.

The server 200 may store and manage information used to provide a video call service and personal information of a user who uses the video call service, for example, personal information such as nationality, name, gender, and interest of the user. . The server 200 may store information on a user who cheated while using the video call service. For example, the server 200 may store distribution information of facial feature points of a user who is restricted from providing a video call service by cheating. When the server 200 receives a video stream photographing the user from the terminal 100 and receives a request for determining whether the user included in the video stream is a user limited in providing a video call service, the server 200 determines the terminal after the determination. 100 can be sent.

Referring to FIG. 7, in block 710, the server 200 receives an image stream of a user of the terminal 100 from the terminal 100 as an application for providing a video call service is executed in the terminal 100. can do. For example, the server 200 may face the user's face photographed by the terminal 100 before the user registration process or login process for the application providing the video call service, or before the video call or during the video call. The included video stream can be received.

In block 720, the server 200 may extract facial feature points of the user of the terminal 100 from the received video stream.

In block 730, the server 200 provides a similarity between the distribution information database of the facial feature points of the user whose provision of the video call service stored in the storage 230 and the distribution information of the facial feature points of the user of the terminal 100 extracted from the video stream. Based on the determination, the user of the terminal 100 may determine whether the provision of the video call service is limited.

FIG. 8 illustrates various methods for determining whether provision of a video call service is limited based on a similarity between a distribution information database of facial feature points of a user limited in providing a video call service and distribution information of facial feature points of a user of the terminal 100. A diagram for describing the methods.

Referring to FIG. 8, there is an image in which user X and user Y are respectively photographed. The image of the user X may be an image of a user in which the provision of the video call service stored in the storage 230 of the server 200 providing the video call service is limited. The image of the user Y may be an image of the user of the terminal 100, photographed as an application for providing a video call service is executed in the terminal 100. In each image, facial feature points of a user are displayed, and distribution information of facial feature points can be obtained from the image.

Even if the same person has different facial feature points according to the angle, size, or shooting environment of the face in the captured image, it is preferable to compare images captured under the same shooting conditions. If there is a difference in facial angle, size, or shooting environment, the server 200 normalizes the video stream received from the terminal 100 and the video stream of the user limited in providing the video call service stored in the database. And use each normalized video stream. Based on the similarity between the distribution information of the facial feature points of the user limited in the provision of the video call service and the distribution information of the facial feature points of the user of the terminal 100, the server 200 is extracted from each normalized video stream. It may be determined whether the provision of the service is a limited user.

Referring to FIG. 8, a plurality of facial feature points may be detected in images of user X and user Y, respectively. In order to determine whether the user X and the user Y are the same person, facial feature points of each image may be used. For example, all of the feature points existing at positions corresponding to each other of the two images among the facial feature points of each image may be utilized, or some feature points that are less influenced by the user's facial expression change. In determining whether the users of the two images are the same person by using the facial feature points of the two images, the facial feature points of only one image may be further utilized.

Hereinafter, for convenience of description, three facial feature points among the facial feature points of the images of each of the user X and the user Y are taken as examples, and the distribution information database of the face feature points of the user X limited in the provision of the video call service and the user of the terminal 100 are described. Based on the similarity between the distribution information of the facial feature points of Y, a method of determining whether the provision of the video call service is limited user will be described. Referring to FIG. 8, there are facial feature points A, B, and C in an image of user X, and there are facial feature points A ', B', and C 'in an image of user Y. FIG.

According to the first method (1), the server 200 may recognize the facial feature points A, B, and C of the user X, and the facial feature points A of the user Y of the terminal 100 extracted from the video stream. The user Y may determine whether the user Y is limited in the provision of the video call service based on the similarity in which ', B' and C 'correspond to each other and considering the sum of the displacements between the corresponding facial feature points. For example, A and A 'are matched as a reference, the displacement between B and B' and the displacement between C and C 'are summed and less than the predetermined threshold H, or less than the predetermined threshold H. If they are the same, it can be determined that user X and user Y are the same person.

According to the second method (②), the server 200 sums up the displacement between the reference feature A and the peripheral feature points B and C and the video stream among the facial feature points A, B, and C of the user X who is restricted from providing the video call service. Based on the similarity between the sum of the displacements between the reference feature A 'and the peripheral feature points B' and C 'among the facial feature points A', B ', and C' of the user Y of the terminal 100 extracted from It may be determined whether the provision is limited to the user. For example, the displacements AB between A and B and the displacements AC between A and C are summed, the displacements A'B 'between A' and B 'and the displacements A'C' between A 'and C' If the similarity between the two is equal to or greater than a predetermined value, it may be determined that the user X and the user Y are the same person.

According to the third method (③), the server 200 determines the ratio of the displacement between the reference feature A and the peripheral feature points B and C and the video stream among the facial feature points A, B, and C of the user X who are limited in providing the video call service. Based on the similarity between the reference feature A 'and the ratio of the displacement between the peripheral feature points B' and C 'among the facial feature points A', B 'and C' of the user Y of the terminal 100 extracted from It may be determined whether the provision is limited to the user. For example, calculate the ratio of displacement AB between A and B and the displacement AC between A and C, and calculate the ratio of displacement A'B 'between A' and B 'and the displacement A'C' between A 'and C'. Thus, when the similarity between the two ratios is equal to or greater than a predetermined value, it may be determined that the user X and the user Y are the same person.

Meanwhile, the server 200 may generate a depth map of the face of the user of the terminal 100 from the received video stream and extract facial feature points from the generated depth map. The server 200 provides a similarity between the distribution information database of the facial feature points in the depth map of the user's face limited in providing the video call service and the distribution information of the facial feature points extracted from the depth map of the user's face in the terminal 100. Based on the determination, it may be determined whether the provision of the video call service is a limited user.

In addition, the server 200 may generate a 3D image of the face of the user of the terminal 100 from the received image stream, and extract the facial feature points in the generated 3D image. In order to generate a 3D image of a user's face, an image including at least one user's face may be used. Based on the similarity between the distribution information of the facial feature points in the 3D image of the user's face limited in providing the video call service and the distribution information of the facial feature points extracted from the 3D image of the user's face of the terminal 100 It may be determined whether the provision of the video call service is a limited user.

Referring back to FIG. 7, in block 740, the server 200 may transmit a determination result of determining whether the provision of the video call service is a limited user to the terminal 100.

9 and 10 illustrate that a server 200 providing a video call service uses a learning model for determining a user whose provision of the video call service is limited, so that the user of the terminal 100 makes a video call. FIG. 14 is a diagram for describing an embodiment of determining whether a service is limited to a user.

Referring to FIG. 9, the server 200 communicates with the terminal 100 to receive a video stream photographing the user of the terminal 100, and the user of the terminal 100 is limited in providing a video call service. The determination result of determining whether the user is a user may be transmitted to the terminal 100. In the above, detailed descriptions of the contents duplicated with those described in FIG. 6 will be omitted.

The terminal 100 may photograph the user of the terminal 100 and transmit the captured image stream to the server 200.

The server 200 generates a learning model for determining a user limited in the provision of the video call service based on the distribution information of the facial feature points of the user who has restricted the provision of the video call service by cheating, and stores the learning model in the storage 230. Can be.

The learning model may be a data recognition model used to detect a user who is limited to providing a video call service among users who use the video call service. The data recognition model may be a model based on an artificial neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), and a bidirectional recurrent deep neural network (BRDNN) may be used as the data recognition model, but is not limited thereto.

The learning model for determining a user limited in providing a video call service may be trained using distribution information of facial feature points of a user limited in providing a video call service. Or, the learning model for determining a user limited in providing a video call service is trained using information on distribution of facial feature points of a user who is limited in providing a video call service and information on cheating that is restricted in providing a video call service. It may be. The learning model for determining a user who is limited to providing a video call service may be a learning model related to the entire user who is limited to providing a video call service, or a learning model prepared for each kind of cheating, regardless of the kind of cheating. Using the learning model for the identification of the user limited in the provision of the video call service and the facial feature points of the user of the terminal 100 extracted from the video stream, it is possible to determine the probability that the user of the terminal 100 is the user limited in the provision of the video call service. It can be expressed numerically. For example, in order to obtain a probability that the user of the terminal 100 is limited in providing the video call service, the facial feature points of the user of the terminal 100 are applied to a learning model for determining a user limited in the provision of the video call service. can do. In order to determine whether the user of the terminal terminal 100 is limited to providing a video call service, a result of using the learning model and facial feature points of the user of the terminal 100 may be compared with a predetermined threshold. According to the comparison result, if the application result is more than a predetermined threshold, it may be determined that the user of the terminal 100 is a user limited to the provision of the video call service.

When the server 200 receives a video stream photographing the user from the terminal 100 and receives a request for determining whether the user is limited in the provision of the video call service, the server 200 determines the result of the determination after the determination using the learning model. ) Can be sent.

Referring to FIG. 10, in block 1010, the server 200 generates and stores a learning model for determining a user limited in provision of a video call service based on distribution information of facial feature points of a user limited in provision of a video call service. And 230.

In block 1020, as the application for providing a video call service is executed in the terminal 100, the server 200 may receive a video stream photographing the user of the terminal 100.

In block 1030, the server 200 may extract facial feature points of the user of the terminal 100 from the received video stream.

In block 1040, the server 200 may determine whether the provision of the video call service is limited by using the learning models stored in the storage 230 and the facial feature points of the user of the terminal 100 extracted from the video stream. Using facial feature points of the user of the terminal 100 providing the video call service extracted from the learning model and the video stream, the user of the terminal 100 may represent a probability that the user of the terminal 100 is limited in the provision of the video call service. . The result of using the learning model and the facial feature points of the user of the terminal 100 is compared with a predetermined threshold, and when the threshold value is greater than or equal to the predetermined threshold, it may be determined that the user of the terminal 100 is a user who is limited to providing a video call service. . On the contrary, if it is less than a predetermined threshold, it may be determined that the user of the terminal 100 is a user who is not limited to providing a video call service.

In block 1050, the server 200 may transmit a determination result of determining whether the provision of the video call service is a limited user to the terminal 100.

11 and 12 are views of a user of a terminal 100 using a learning model for determining whether a terminal 100 providing a video call service is limited in providing a video call service, according to another exemplary embodiment. FIG. 14 is a diagram for describing an embodiment of determining whether a provision of a call service is a limited user.

Referring to FIG. 11, the terminal 100 may receive a learning model for determining a user limited in providing a video call service from the server 200. The description of the learning model is as described with reference to FIG. 9. The terminal 100 may receive the updated learning model according to the update period from the server 200.

The terminal 100 may determine whether the user is a user who is limited to providing the video call service by using the user's facial feature points and a learning model for determining a user limited in the provision of the video call service. The terminal 100 may extract facial feature points from an eye part, a nose part, a mouth part, a forehead part, a cheek part, a chin part, and a facial contour of the user from the video stream. The terminal 100 determines whether the user is limited in the provision of the video call service by using distribution information of feature points of at least one of a learning model for determining a user limited in the provision of the video call service and a face part of the user. You can judge.

The terminal 100 may maintain or limit the execution of an application that provides a video call service based on the determination result using the learning model.

Referring to FIG. 12, in block 1210, the terminal 100 generates a learning model for determining a user limited in provision of a video call service, which is generated based on distribution information of facial feature points of a user limited in provision of a video call service. It may receive from the server 200. The terminal 100 may receive the learning model updated in the server 200 according to the update period from the server 200.

In block 1220, the terminal 100 may acquire an image stream of the user of the terminal 100 as an application for providing a video call service is executed in the terminal 100. For example, the terminal 100 may display a user's face photographed by the terminal 100 before a user registration process or a login process or a video call or a video call to an application that provides a video call service. The included video stream can be obtained. The terminal 100 may provide guidelines on the screen of the terminal 100 so that the angle and size of the face of the user of the terminal 100 are photographed at a predetermined level. The terminal 100 may obtain an image stream in which the face of the user of the terminal 100 is photographed according to the provided guidelines.

In block 1230, the terminal 100 may extract facial feature points of the user of the terminal 100 from the obtained image stream. The terminal 100 selects an image in which the face of the user of the terminal 100 is most photographed based on the photographed angle of the obtained image stream, and extracts facial feature points of the user of the terminal 100 from the selected image. can do. The terminal 100 may extract facial feature points from an eye part, a nose part, a mouth part, a forehead part, a cheek part, a chin part, and a facial contour of the user from an image stream photographing the user of the terminal 100.

In block 1240, the terminal 100 uses the learning model received from the server 200 and the facial feature points of the user of the terminal 100 extracted from the obtained video stream, so that the user of the terminal 100 is connected to the video call service. It can be determined whether the offer is limited to the user. Using facial feature points of the user of the terminal 100 extracted from the learning model and the video stream, the user of the terminal 100 may indicate numerically the probability that the user of the terminal 100 is limited in providing the video call service. The result of using the learning model and the facial feature points of the user of the terminal 100 is compared with a predetermined threshold, and when the threshold value is greater than or equal to the predetermined threshold, it may be determined that the user of the terminal 100 is a user who is limited to providing a video call service. . On the contrary, if it is less than a predetermined threshold, it may be determined that the user of the terminal 100 is a user who is not limited to providing a video call service.

In block 1250, the terminal 100 may maintain or limit the execution of an application that provides the video call service based on a determination result of determining whether the provision of the video call service is a limited user. When the terminal 100 determines that the provision of the video call service is limited, the terminal 100 terminates the application providing the executed video call service or informs that the provision of the video call service is restricted by the history of cheating. By providing a message, you can limit the execution of the application.

13 and 14 illustrate a terminal 100 that provides a video call service using a distribution information database of facial feature points of a user limited in providing a video call service. FIG. 14 is a diagram for describing an embodiment of determining whether provision of a video call service is a limited user.

Referring to FIG. 13, the terminal 100 may receive a distribution information database of facial feature points of a user limited in providing a video call service from the server 200. The terminal 100 may receive a distribution information database of facial feature points of a user limited in providing an updated video call service according to an update cycle from the server 200.

The terminal 100 determines the degree of similarity between the distribution information database of the facial feature points of the user whose provision of the video call service and the distribution information of the facial feature points of the user of the terminal 100 extracted from the video stream photographing the user of the terminal 100. On the basis of this, it may be determined whether the provision of the video call service is a limited user. The terminal 100 may maintain or limit the execution of an application that provides a video call service based on a determination result using a distribution information database of facial feature points of a user who is limited to providing a video call service.

Referring to FIG. 14, in block 1410, the terminal 100 may receive from the server 200 a database of distribution information of facial feature points of a user limited in providing a video call service.

In block 1420, the terminal 100 may acquire an image stream of the user of the terminal 100 as an application for providing a video call service is executed in the terminal 100. For example, the terminal 100 may display a user's face photographed by the terminal 100 before a user registration process or a login process or a video call or a video call to an application that provides a video call service. The included video stream can be obtained. The terminal 100 may provide a guideline on the screen of the terminal 100 so that the angle and size of the face of the user of the terminal 100 are photographed according to a predetermined criterion. The terminal 100 may obtain an image stream in which the face of the user of the terminal 100 is photographed according to the provided guidelines.

In block 1430, the terminal 100 may extract facial feature points of the user of the terminal 100 from the obtained image stream. The terminal 100 selects an image in which the face of the user of the terminal 100 is most photographed based on the photographed angle of the obtained image stream, and extracts facial feature points of the user of the terminal 100 from the selected image. can do. The terminal 100 may extract facial feature points from an eye part, a nose part, a mouth part, a forehead part, a cheek part, a chin part, and a facial contour of the user from an image stream photographing the user of the terminal 100.

In block 1440, the terminal 100 is based on the similarity between the distribution information database of the facial feature points of the user limited in providing the video call service and the distribution information of the facial feature points of the user of the terminal 100 extracted from the video stream. The user of 100) may determine whether the provision of the video call service is limited.

The terminal 100 may perform a normalization process with respect to a video stream of a user limited in providing a video call service and a video stream obtained from the terminal 100 stored in a database received from the server 200. The terminal 100 is based on the similarity between the distribution information of the facial feature points of the user limited in the provision of the video call service and the distribution information of the facial feature points of the user of the terminal 100 extracted from each normalized video stream. It may be determined whether the provision of the service is a limited user.

Based on the similarity between the distribution information database of the facial feature points of the user who is limited to providing the video call service and the distribution information of the facial feature points of the user of the terminal 100, various methods of determining whether the provision of the video call service is a limited user are as described above. Since the same methods as described with reference to FIG. 8 may be applied, detailed description thereof will be omitted.

In block 1450, the terminal 100 may maintain or limit the execution of an application that provides the video call service based on the determination result of determining whether the provision of the video call service is a limited user. When the terminal 100 determines that the provision of the video call service is limited, the terminal 100 terminates the application providing the executed video call service or informs that the provision of the video call service is restricted by the history of cheating. By providing a message, you can limit the execution of the application.

Embodiments of the above-described method for providing a video call service may be provided in the form of an application stored in a computer-readable storage medium for performing the method for providing a video call service in the terminal 100. In other words, embodiments of the method for providing the video call service described above may be implemented by an application or a computer program stored in a computer readable storage medium for executing the steps of the terminal 100 for providing the video call service. It may be provided in the form.

The above-described embodiments may be implemented in the form of a computer readable storage medium storing instructions or data executable by a computer or a processor. At least one of the instructions and data may be stored in the form of program code, and when executed by a processor, a predetermined program module may be generated to perform a predetermined operation. Such computer-readable storage media include read-only memory (ROM), random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD + Rs, CD-RWs, CD + RWs, DVD-ROMs , DVD-Rs, DVD + Rs, DVD-RWs, DVD + RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, Magnetic Tape, Floppy Disks, Magneto-optical Data Storage, Optical data storage devices, hard disks, solid-state disks (SSDs), and instructions or software, associated data, data files, and data structures; and instructions or instructions on a processor or computer for the processor or computer to execute instructions. It may be any device capable of providing software, associated data, data files, and data structures.

So far looked at the center of the embodiments. Those skilled in the art to which the disclosed embodiments belong will appreciate that the disclosed embodiments can be implemented in a modified form without departing from the essential characteristics. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the invention is set forth in the claims rather than the foregoing description of the embodiments, and all differences within the scope will be construed as being included in the scope of the invention.

Claims (15)

A communication interface for communicating with the terminal;
A storage storing a distribution information database of facial feature points of a user limited in providing a video call service;
A processor; And
A memory storing instructions executable by the processor;
The processor executes the instructions,
As the application for providing a video call service is executed in the terminal, the terminal receives a video stream photographing the user of the terminal from the terminal, extracts facial feature points of the user of the terminal from the received video stream, and the video. Distribution of facial feature points of a user with limited provision of a video call service, determined based on displacements between reference feature points and neighboring feature points among user's facial feature points with limited provision of a call service Based on the similarity between the information and the distribution information of the facial feature points of the user of the terminal, determining whether the user of the terminal is limited in the provision of the video call service, and transmitting the determination result to the terminal. The method of claim 1,
The processor,
The reference feature and the peripheral feature points among the facial feature points of the user of which the provision of the video call service is limited correspond to the reference feature points and the peripheral feature points of the user's facial feature points of the extracted terminal, and the displacement of the peripheral feature points corresponding to each other. Based on the similarity between sums, determining whether the user of the terminal is limited to providing the video call service. The method of claim 1,
The processor,
Based on the similarity between the sum of the displacements between the reference feature points and the peripheral feature points among the facial feature points of the user who is limited to providing the video call service and the sum of the displacements between the reference feature points and the peripheral feature points among the extracted facial feature points of the user of the terminal. To determine whether the user of the terminal is limited to the provision of the video call service. The method of claim 1,
The processor,
Based on the similarity between the ratio of the displacement between the reference feature point and the peripheral feature points among the facial feature points of the user who is limited to providing the video call service and the ratio of the displacement between the reference feature point and the peripheral feature points among the facial feature points of the extracted user To determine whether the user of the terminal is limited to the provision of the video call service. The method of claim 1,
The processor,
A depth map of a face of a user of the terminal is generated from the received video stream, facial feature points are extracted from the generated depth map, and a face of a user whose provision of the video call service is limited. Based on the similarity between the distribution information database of the facial feature points in the depth map and the distribution information of the facial feature points in the depth map of the extracted face of the user of the terminal, the user of the terminal is restricted from providing the video call service. Server to determine if it is. The method of claim 1,
The processor,
Generate a 3D image of the face of the user of the terminal from the received video stream, extract the facial feature points in the generated 3D image, 3D for the face of the user limited in the provision of the video call service Based on the similarity between the distribution information database of the facial feature points in the image and the distribution information of the facial feature points in the 3D image of the extracted face of the user of the terminal, the user of the terminal is restricted from providing the video call service. Server to determine if it is. The method of claim 1,
Performing a normalization process on the video stream and the received video stream of the user limited to the provision of the video call service stored in the database, and extracted from each normalized video stream, Based on the similarity between distribution information of facial feature points and distribution information of facial feature points of a user of the terminal, determining whether the user of the terminal is a limited user of the video call service. A communication interface for communicating with the terminal;
storage;
A processor; And
A memory storing instructions executable by the processor;
The processor executes the instructions,
Based on distribution information of facial feature points of a user who is limited to providing a video call service and information about misconduct that causes the provision of the video call service to be restricted, a learning model for determining a user who is limited to providing the video call service is provided. As the application is generated and stored in the storage and the video call service is executed in the terminal, the video stream photographing the user of the terminal is received, and the facial feature points of the user of the terminal are received from the received video stream. Extracting and applying the extracted facial feature points to the learning model stored in the storage to determine whether the user of the terminal is limited in providing the video call service, and transmits the determination result to the terminal. At the terminal,
Learning to determine a user limited in the provision of the video call service, generated based on distribution information of facial feature points of a user who is limited in the provision of the video call service, and information about misconduct that causes the provision of the video call service to be restricted Receiving a model from a server;
Acquiring a video stream photographing a user of the terminal as an application for providing a video call service is executed in the terminal;
Extracting facial feature points of a user of the terminal from the obtained video stream;
Applying the extracted facial feature points to the learning model to determine whether the provision of the video call service is limited by a history of a user cheating by the terminal; And
Based on the determination result, maintaining or limiting execution of an application that provides the video call service;
An application stored on the computer readable storage medium. The method of claim 9,
Acquiring an image stream photographing the user of the terminal,
The application stored in the computer-readable storage medium to obtain a video stream photographing the user of the terminal in the sign-up process or login process as the application for providing a video call service in the terminal is executed. The method of claim 9,
Acquiring an image stream photographing the user of the terminal,
Providing a guideline on a screen of the terminal so that the angle and size of the face of the user of the terminal are photographed according to a predetermined criterion; And
Acquiring an image stream of a face of the user of the terminal according to the provided guidelines;
And an application stored on the computer readable storage medium. The method of claim 9,
The extracting step,
A computer-readable storage medium for selecting an image in which the face of the user of the terminal is photographed in front of the acquired image stream, and extracting facial feature points of the user of the terminal from the selected image. Saved application. The method of claim 9,
The determining step,
By using the learning model and the extracted facial feature points, a result of numerically representing a probability that the provision of the video call service is a limited user is compared with a predetermined threshold value, and according to the comparison result, the user of the terminal An application stored on a computer readable storage medium for determining whether a service is provided to a limited user. The method of claim 9,
Maintaining or limiting the execution of the application,
As a result of the determination, when the user of the provision of the video call service is limited, the notification of the provision of the video call service is limited by the termination of the application providing the executed video call service or the history of the cheating. Provides a message to limit the execution of the application. The method of claim 9,
The receiving step,
An application stored on a computer-readable storage medium for receiving a training model updated on a periodic basis at the server.

Images