KR102501575B1 - Method and apparatus for transmitting a video text list to a user terminal using a plurality of neural networks by a server - Google Patents

Abstract

The present invention relates to a method and apparatus for transmitting a video text list to a user terminal by using a plurality of neural networks. The method of the present invention comprises the steps of: receiving a video from an external server; extracting information about the video; determining a plurality of first texts and a plurality of second texts; determining a plurality of third texts and a plurality of fourth texts; distinguishing between a first playback time of the video and a second playback time of the video; determining a video text list; and transmitting the video text list to a user terminal. According to the present invention, user convenience can be improved.

Description

Method and apparatus for transmitting a video text list to a user terminal using a plurality of neural networks

Embodiments of the present disclosure relate to a technology for transmitting a video text list to a user terminal, and to a technology for transmitting a video text list from a server to a user terminal using a plurality of neural networks.

On the other hand, due to the development of the Internet and the popularization of smart phones, watching videos in a mobile environment is greatly increasing, and learners watching videos for learning purposes are also increasing.

In the case of Korean subtitles provided by existing video platforms, the subtitles set by the user may be displayed in the video, but in the case of subtitles provided in the video by recognizing the audio of the video by the video platform itself, there is a problem that many typos occur. .

In addition, since there is no function to extract text displayed in the video and provide it to the user other than subtitles of the video, it may be difficult to create a table of contents or create learning materials using the text included in the video.

Accordingly, there is a need for a method and apparatus for recognizing text displayed in a video as well as subtitles displayed in a video using a neural network, automatically correcting typos in the recognized texts, and providing the information to the user.

Many typos occur in the case of subtitles that recognize the audio of the video and provide it to the video by the existing video platform itself, and there is no function to extract text displayed in the video and provide it to users other than the subtitle of the video.

Embodiments of the present disclosure provide a method and apparatus for extracting text displayed in a video in addition to subtitles of the video using a plurality of neural networks and transmitting a video text list in which typos in the texts are automatically corrected to a user terminal. can do.

Technical tasks to be achieved in the embodiments are not limited to those mentioned above, and other technical tasks not mentioned will be considered by those skilled in the art from various embodiments to be described below. can

A method in which a server transmits a video text list to a user terminal by using a plurality of neural networks according to an embodiment includes receiving a video from an external server, determining the file size of the video and the video text list from the video. Extracting information about the video including video playback time, frame size of the video, and color information about the video, and using a text recognition model using a first neural network based on the video and the information about the video determining a plurality of first texts included in a first area in a playback screen of the video and a plurality of second texts included in a second area in a playback screen of the video; Based on 2 texts, a plurality of third texts and a plurality of fourth texts are determined through a typo correction model using a second neural network, and the plurality of third texts correct typos included in the plurality of first texts. text, the plurality of fourth texts are texts in which typos included in the plurality of second texts are corrected, and the plurality of first texts displayed in the first area of the video playback screen are changed based on a time point To distinguish the second playback time of the video based on the first playback time of the video and the time when the plurality of second texts displayed in the second region of the video playback screen are changed, and the first playback of the video determining a video text list composed of the plurality of third texts matched to a viewpoint and the plurality of fourth texts matched to a second playback viewpoint of the video, and transmitting the video text list to a user terminal. can For example, when any one of the plurality of third texts and the plurality of fourth texts included in the video text list is selected by the user terminal, the video matching the selected text is reproduced from the playback time. Videos can be played.

For example, the first area may be an area where subtitles are located on the reproduction screen of the video. The second area may include an area other than the first area. The plurality of first texts and the plurality of second texts may be texts having a size greater than a minimum size. The minimum size may be determined based on the frame size of the video, the size of the subtitle, and color information of the video.

For example, the minimum size may be determined by the following equation.

In the above equation, T _min is the initial size, n _c is a value determined based on the distribution of RGB values included in the video, and n _d is a basic value for the distribution of RGB values included in the video. , wherein S _f is the frame size of the video, S _d is the average frame size, S _c is the size of the subtitle of the video, and T _r is the minimum size that the recognition model can recognize. It can be the size of the text.

For example, based on the fact that the number of the plurality of first texts is greater than or equal to a preset threshold value, the typo correction model is configured for the plurality of first texts and the plurality of second texts according to the plurality of first error types. Mistakes can be detected. Based on the number of the plurality of first texts being less than a preset threshold value, the typo correction model is configured to detect typos in the plurality of first texts and the plurality of second texts according to the plurality of second error types. can The plurality of first error types may include spaces errors, punctuation errors, numeral errors, and primary errors regarding spelling and grammar. The primary error may include a letter deletion error, a letter addition error, a letter replacement error, and a letter separation error. The plurality of second error types may include the plurality of first error types and secondary errors for spelling and grammar. The secondary error may include a typing error, a foreign word conversion error, a semantic error, and a neologism error.

For example, the text recognition model may be a convolutional recurrent neural network (CRNN) model. The first neural network may include a first input layer, one or more first hidden layers, and a first output layer. Data for learning each of the first neural networks composed of a plurality of videos and a plurality of correct answer texts is input to the first input layer of the first neural network, and the one or more first hidden layers and the first output layer are input. Pass through and output a first output vector, the first output vector is input to a first loss function layer connected to the first output layer, and the first loss function layer is applied to the first output vector and each training data The text recognition model generates a first loss value by using a first loss function that compares a first answer vector for , and the parameters of the first neural network are learned in a direction in which the first loss value decreases. It can be.

For example, the typo correction model may include a bidirectional long short-term memory (LSTM) model. The second neural network may include a second input layer, one or more second hidden layers, and a second output layer. Data for learning of each second neural network composed of a plurality of texts and a plurality of correct answer texts is input to the second input layer of the second neural network, and the one or more second hidden layers and the second output layer are input. Pass through and output a second output vector, the second output vector is input to a second loss function layer connected to the second output layer, and the second loss function layer is applied to the second output vector and each training data A second loss value is output using a second loss function that compares a second answer vector for , and the parameters of the second neural network are learned in a direction in which the second loss value becomes smaller. The typo correction model is generated. It can be.

Additionally, for example, the preset threshold value may be determined by the following equation.

In the above equation, n _th is the preset threshold value, n is the number of videos uploaded to the server, t _i is the playing time of the ith video, and p _v is the server per second The number of transactions that can be processed, n _c is the number of user terminals accessing the server, and n _d may be a default value for the preset threshold value.

Here, the playback time of the video may be in seconds. The p _v may be a transaction per second (TPS) unit. The floor function is a function that reflects the nearest rounding value. For example, the default value may be preset in the server.

Even if the number of texts displayed in the video is the same, the speed at which the server processes data may vary according to the playing time of the video. In particular, when the video playback time is long, the server may take more time to process the text data because the capacity of the video is relatively larger than when the video playback time is short. Therefore, by setting the preset threshold to a larger value as the playing time of the video is longer, the server detects a plurality of first error types with a relatively small number for a larger number of texts, thereby reducing the load on the server. can

For example, since the server can process a larger amount of data as the TPS is higher, by lowering the preset threshold value, the server has a relatively large number of second error types for a smaller number of texts. can be detected to improve the quality of error detection.

For example, as the number of user terminals accessing the server increases, the data to be processed by the server increases. Therefore, by setting the preset threshold value to a large value, the server can obtain a relatively small number of texts for a larger number of texts. A load on a server may be reduced by detecting a plurality of first error types.

According to embodiments, the server extracts subtitles and text other than subtitles displayed in a video through a text recognition model and transmits a video text list arranged in playback order to a user terminal, thereby providing the user terminal with text related to the video in addition to the subtitles. It is possible to increase user convenience by transmitting data according to playback time points.

In addition, the server can effectively extract text from the video by effectively determining the minimum size for the text recognition model to recognize text related to the video in consideration of the frame size of the video, subtitle size, and RGB values of the video.

According to embodiments, the server may automatically correct typos in text recognized through the text recognition model through the typo correction model, and in consideration of various factors affecting the data processing speed of the server, the typo correction model By changing the reference value for determining the error type to be processed, the quality of the server detecting typos in the text displayed in the video can be improved.

Effects obtainable from the embodiments are not limited to the effects mentioned above, and other effects not mentioned are clearly derived and understood by those skilled in the art based on the detailed description below. It can be.

BRIEF DESCRIPTION OF THE DRAWINGS Included as part of the detailed description to aid understanding of the embodiments, the accompanying drawings provide various embodiments and, together with the detailed description, describe technical features of the various embodiments.
1 is a diagram illustrating a configuration of an electronic device according to an exemplary embodiment.
2 is a diagram showing the configuration of a program according to an embodiment.
3 illustrates a method in which a server transmits a video text list to a user terminal according to an exemplary embodiment.
4 is a diagram of a text recognition model according to an embodiment.
5 is a diagram of a typo correction model according to an embodiment.
6 is a flowchart of a method for a server to transmit a video text list to a user terminal according to an exemplary embodiment.
7 is a block diagram illustrating a configuration of a server according to an exemplary embodiment.

The following embodiments combine elements and features of the embodiments in a predetermined form. Each component or feature may be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form not combined with other components or features. In addition, various embodiments may be configured by combining some components and/or features. The order of operations described in various embodiments may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment.

In the description of the drawings, procedures or steps that may obscure the gist of various embodiments are not described, and procedures or steps that can be understood by those skilled in the art are not described. did

Throughout the specification, when a part is said to "comprising" or "including" a certain element, it means that it may further include other elements, not excluding other elements, unless otherwise stated. do. In addition, terms such as “… unit”, “… unit”, and “module” described in the specification mean a unit that processes at least one function or operation, which is hardware or software or a combination of hardware and software. can be implemented as Also, “a or an”, “one”, “the” and like terms are used herein in the context of describing various embodiments (particularly in the context of the claims below). Unless otherwise indicated or clearly contradicted by context, both the singular and the plural can be used.

Hereinafter, embodiments according to various embodiments will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary embodiments of various embodiments, and is not intended to represent a single embodiment.

In addition, specific terms used in various embodiments are provided to help understanding of various embodiments, and the use of these specific terms may be changed into other forms without departing from the technical spirit of various embodiments. .

1 is a diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be. The electronic device 101 may also be referred to as a client, terminal, or peer.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model.

AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

The server 108 is connected to the electronic device 101 and can provide a service to the connected electronic device 101 . In addition, the server 108 may proceed with a membership sign-up procedure, store and manage various types of information of users subscribed as members, and provide various purchase and payment functions related to services. In addition, the server 108 may share execution data of service applications executed in each of the plurality of electronic devices 101 in real time so that the service can be shared among users. This server 108 may have the same configuration as a conventional web server or service server in terms of hardware. However, in terms of software, it may include a program module that is implemented through any language such as C, C++, Java, Python, Golang, or kotlin and performs various functions. In addition, the server 108 is generally connected to an unspecified number of clients and/or other servers through an open computer network such as the Internet, and receives requests from clients or other servers to perform tasks and derives and provides work results. It means a computer system and the computer software (server program) installed for it. In addition, the server 108, in addition to the above-described server program, a series of application programs operating on the server 108 and various databases (DB: Database, hereinafter referred to as It should be understood as a broad concept including DB"). Accordingly, the server 108 classifies member registration information and various information and data about games, stores them in a DB, and manages them. This DB may be implemented inside or outside the server 108 . In addition, the server 108 may be implemented using server programs that are provided in various ways according to operating systems such as Windows, Linux, UNIX, and Macintosh on general server hardware, As a representative example, a web service can be implemented using IIS (Internet Information Server) used in a Windows environment and CERN, NCSA, APPACH, TOMCAT, etc. used in a Unix environment. In addition, the server 108 may interoperate with an authentication system and a payment system for user authentication of services or payment for purchases related to services.

The first network 198 and the second network 199 are a connection structure capable of exchanging information between nodes such as terminals and servers, or a network connecting the server 108 and the electronic devices 101 and 104. means (Network). The first network 198 and the second network 199 include Internet, Local Area Network (LAN), Wireless Local Area Network (Wireless Local Area Network), Wide Area Network (WAN), Personal Area Network (PAN), and 3G , 4G, LTE, 5G, Wi-Fi, etc., but are not limited thereto. The first network 198 and the second network 199 may be closed first networks 198 and second networks 199 such as LAN and WAN, but are preferably open such as the Internet. The Internet includes protocols such as the TCP/IP protocol, TCP, and User Datagram Protocol (UDP), and various services that exist in the upper layer, such as HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), and DNS (Domain Name System). ), Simple Mail Transfer Protocol (SMTP), Simple Network Management Protocol (SNMP), Network File Service (NFS), and Network Information Service (NIS). ) structure.

A database may have a general data structure implemented in a storage space (hard disk or memory) of a computer system using a database management program (DBMS). The database may have a data storage form in which data can be freely searched for (extracted), deleted, edited, added, and the like. Databases are relational database management systems (RDBMS) such as Oracle, Informix, Sybase, and DB2, or object-oriented database management such as Gemston, Orion, and O2. It can be implemented according to the purpose of an embodiment of the present disclosure using a system (OODBMS) and XML Native Databases such as Excelon, Tamino, and Sekaiju, and its functions may have appropriate fields or elements to achieve.

2 is a diagram showing the configuration of a program according to an embodiment.

2 is a block diagram 200 illustrating a program 140 according to various embodiments. According to one embodiment, the program 140 includes an operating system 142, middleware 144, or an application 146 executable in the operating system 142 for controlling one or more resources of the electronic device 101. can include The operating system 142 may include, for example, Android™, iOS™, Windows™, Symbian™ or Tizen™. At least some of the programs 140 are, for example, preloaded in the electronic device 101 at the time of manufacture, or when used by a user, an external electronic device (eg, the electronic device 102 or 104), or a server ( 108)) can be downloaded or updated. All or part of program 140 may include a neural network.

The operating system 142 may control management (eg, allocation or reclamation) of one or more system resources (eg, process, memory, or power) of the electronic device 101 . Operating system 142 may additionally or alternatively include other hardware devices of electronic device 101 , such as input module 150 , sound output module 155 , display module 160 , audio module 170 . , sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or It may include one or more driver programs for driving the antenna module 197.

The middleware 144 may provide various functions to the application 146 so that the function or information provided from one or more resources of the electronic device 101 may be used by the application 146 . The middleware 144 includes, for example, the application manager 201, the window manager 203, the multimedia manager 205, the resource manager 207, the power manager 209, the database manager 211, and the package manager 213. ), connectivity manager 215, notification manager 217, location manager 219, graphics manager 221, security manager 223, call manager 225, or voice recognition manager 227. can

The application manager 201 may manage the life cycle of the application 146 , for example. The window manager 203 may manage one or more GUI resources used in a screen, for example. The multimedia manager 205 identifies, for example, one or more formats necessary for reproducing media files, and encodes or decodes a corresponding media file among the media files using a codec suitable for the selected format. can be done The resource manager 207 may manage a source code of the application 146 or a memory space of the memory 130 . The power manager 209 may manage, for example, the capacity, temperature, or power of the battery 189, and determine or provide related information necessary for the operation of the electronic device 101 by using corresponding information among them. . According to an embodiment, the power manager 209 may interoperate with a basic input/output system (BIOS) (not shown) of the electronic device 101 .

The database manager 211 may create, search, or change a database to be used by the application 146, for example. The package manager 213 may manage installation or update of applications distributed in the form of package files, for example. The connectivity manager 215 may manage, for example, a wireless connection or a direct connection between the electronic device 101 and an external electronic device. The notification manager 217 may provide a function for notifying a user of occurrence of a designated event (eg, an incoming call, message, or alarm), for example. The location manager 219 may manage location information of the electronic device 101, for example. The graphic manager 221 may manage, for example, one or more graphic effects to be provided to a user or a user interface related thereto.

Security manager 223 may provide system security or user authentication, for example. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101 . The voice recognition manager 227 transmits, for example, the user's voice data to the server 108, and at least partially based on the voice data, a command corresponding to a function to be performed in the electronic device 101; Alternatively, text data converted at least partially based on the voice data may be received from the server 108 . According to one embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to one embodiment, at least part of the middleware 144 may be included as part of the operating system 142 or may be implemented as separate software different from the operating system 142 .

The application 146 includes, for example, a home 251, a dialer 253, an SMS/MMS 255, an instant message (IM) 257, a browser 259, a camera 261, and an alarm 263. , Contacts (265), Voice Recognition (267), Email (269), Calendar (271), Media Player (273), Albums (275), Watch (277), Health (279) (e.g. exercise or blood sugar) measurement of biometric information) or environmental information 281 (eg, measurement of atmospheric pressure, humidity, or temperature information). According to an embodiment, the application 146 may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transmit designated information (eg, a call, message, or alarm) to an external electronic device, or a device management application configured to manage an external electronic device. there is. The notification relay application, for example, transmits notification information corresponding to a designated event (eg, mail reception) generated in another application (eg, the email application 269) of the electronic device 101 to an external electronic device. can Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide the notification information to the user of the electronic device 101 .

The device management application is, for example, a power source (eg, turn-on or turn-off) of an external electronic device that communicates with the electronic device 101 or some component thereof (eg, a display module or a camera module of the external electronic device). ) or functions (eg brightness, resolution, or focus). The device management application may additionally or alternatively support installation, deletion, or update of an application operating in an external electronic device.

Throughout this specification, a neural network, a neural network, and a network function may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may be generally referred to as “nodes”. These “nodes” may also be referred to as “neurons”. A neural network includes at least two or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more “links”.

In a neural network, two or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, the input node to output node relationship can be created around the link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, a node interconnecting the input node and the output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. An output node value may be determined based on the weight.

As described above, in a neural network, two or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values between the links, the two neural networks may be recognized as different from each other.

3 illustrates a method in which a server transmits a video text list to a user terminal according to an exemplary embodiment. One embodiment of FIG. 3 may be combined with various embodiments of the present disclosure.

Referring to FIG. 3 , in step S301, a server may receive a video from an external server.

The server may be a server that manages information collected from web sites related to learning content through web crawling and provides a service to a user terminal based on the collected information. Here, web crawling means an operation of extracting necessary information from web sites through various computer software technologies. The external server is a server that manages a video platform and may be a server that manages a website on which a plurality of videos are uploaded. For example, the server and external server may be server 108 of FIG. 1 .

In step S302, the server may extract information about the video, including file size of the video, playback time of the video, frame size of the video, and color information about the video, from the video.

Here, the frame size of a video can be expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. Color information for a video may include a value related to a color space. For example, the color space may be expressed in various ways such as red green blue (RGB), cyan, magenta, yellow, black (CMYK), YUV, YCbCr, and HSV. For example, the value related to the color space may include a value representing a matrix of coordinate values related to at least one color space for at least one pixel related to the video. For example, color information for a video may include RGB values for each pixel included in the video. For example, RGB may be a method expressed as coordinate values for three types of colors, red, green, and blue. For example, in YUV, color can be expressed using two axes, U and V, with a brightness component on the y-axis. Here, the U-axis may be a value obtained by subtracting the brightness component from blue, and the V-axis may be a value obtained by subtracting the brightness component from red. For example, YCbCr is a digitized color space of an analog signal, Y may be a luminance component, and Cb and Cr may be color difference components. For example, HSV may be a method that considers hue, saturation, and brightness information (eg, value).

In step S303, the server reproduces a plurality of first texts included in a first area in a playback screen of the video and the video through a text recognition model using a first neural network based on the video and information about the video. A plurality of second texts included in the second area of the screen may be determined.

The text recognition model may be a convolutional recurrent neural network (CRNN) model. The CRNN model is a neural network model designed to recognize time-series data from images. The CRNN model is a model that combines a CNN model and an RNN model to extract features from an image through the CNN model and predict letters for each sequence through the RNN model based on the extracted features.

For example, the first neural network may include a first input layer, one or more first hidden layers, and a first output layer. Data for learning each of the first neural networks composed of a plurality of videos and a plurality of correct answer texts is input to the first input layer of the first neural network, and the one or more first hidden layers and the first output layer are input. Pass through and output a first output vector, the first output vector is input to a first loss function layer connected to the first output layer, and the first loss function layer is applied to the first output vector and each training data The text recognition model generates a first loss value by using a first loss function that compares a first answer vector for , and the parameters of the first neural network are learned in a direction in which the first loss value decreases. It can be.

For example, the first area may be an area where subtitles are located on the reproduction screen of the video. The area where the subtitle is located may be an area from the lowermost end of the video to a position spaced apart by a specific width. For example, the specific width may be an average value of widths including subtitles at the bottom of the video. The first region may be set differently according to the frame size of the video. That is, the first text may include a subtitle of a video. The second area may include an area other than the first area. That is, the second text may be characters displayed in the video other than the subtitle of the video. For example, the second text may be a character displayed on a description part in a reproduction screen of a video, such as a presentation screen included in a reproduction screen of a video or a whiteboard. The plurality of first texts and the plurality of second texts may be texts having a size greater than a minimum size. The minimum size may be determined based on the frame size of the video, the size of the subtitle, and color information of the video. Here, the size of the caption may be determined by recognizing a region where the caption is located from the still screen of the moving picture, that is, a character displayed in the first region.

For example, the minimum size may be determined by Equation 1 below.

In Equation 1, T _min is the initial size, n _c is a value determined based on the distribution of RGB values included in the video, and n _d is the distribution of RGB values included in the video. It is a basic value, S _f is the frame size of the video, S _d is the average frame size, S _c is the size of the subtitle of the video, and T _r is the size of the video that the recognition model can recognize. It can be the minimum text size that exists.

Here, the value determined based on the distribution of RGB values included in the video may be a standard deviation of RGB values for the second region. A basic value for the distribution of RGB values included in the video is a reference value for determining the size of the n _c value, and may be a value preset in the server. The average frame size is a reference value for determining the frame size of the video, and may be preset in the server. The minimum text size that the recognition model can recognize may be different according to the performance of the server, and the minimum text size that the recognition model can recognize may be preset in the server. Here, the max function is a function that selects the largest value. That is, due to the max function, the minimum size may be determined as a value greater than or equal to the minimum text size that the recognition model can recognize.

For example, since various colors are included in the video as the value determined based on the distribution of RGB values included in the video increases, the minimum size of text recognized by the text recognition model may be determined to be small. In addition, the minimum size of text recognized by the text recognition model may be determined to be larger as the frame size of the video is larger than the average frame size and the size of the subtitle is larger.

Due to this, the server can recognize only the text related to the video among the text displayed on the video playback screen, and the minimum size for the text recognition model to recognize the text related to the video is determined by the frame size of the video, the size of the subtitle, and the video. It can be effectively determined by considering the RGB values.

In step S304, the server may determine a plurality of third texts and a plurality of fourth texts through a typo correction model using a second neural network based on the plurality of first texts and the plurality of second texts.

The plurality of third texts may be texts in which typos included in the plurality of first texts are corrected. The plurality of fourth texts may be texts in which typos included in the plurality of second texts are corrected.

The typo correction model may be a bidirectional long short-term memory (LSTM) model. In general, a recurrent neural network (RNN) can effectively model time-series information because hidden layer values for an existing input stored therein are considered in an output for a next input value. However, since RNN is a structure that depends on past observation values, problems such as vanishing gradient or exploding gradient may occur. A model to solve this problem is LSTM, and by replacing nodes inside the LSTM with memory cells, information can be accumulated or some of past information can be deleted, and the problem of the RNN can be supplemented. In addition, the bidirectional LSTM is an LSTM having bidirectionality, and may include a forward LSTM and a backward LSTM. For example, a linear function may be used as an activation function of a forward LSTM, and a sigmoid function may be used as an activation function of a backward LSTM.

The second neural network may include a second input layer, one or more second hidden layers, and a second output layer.

Data for learning of each second neural network composed of a plurality of texts and a plurality of correct answer texts is input to the second input layer of the second neural network, and the one or more second hidden layers and the second output layer are input. Pass through and output a second output vector, the second output vector is input to a second loss function layer connected to the second output layer, and the second loss function layer is applied to the second output vector and each training data A second loss value is output using a second loss function that compares a second answer vector for , and the parameters of the second neural network are learned in a direction in which the second loss value becomes smaller. The typo correction model is generated. It can be.

According to an embodiment, based on the fact that the number of the plurality of first texts is greater than or equal to a preset threshold value, the typo correction model determines the plurality of first texts and the plurality of second texts according to the plurality of first error types. can be detected for

Based on the fact that the number of the plurality of first texts is less than a preset threshold value, the typo correction model may detect the plurality of first texts and the plurality of second texts according to a plurality of second error types. .

The plurality of first error types may include a spacing error, a punctuation error, a numeral error, and a primary error regarding spelling and grammar.

Here, the error about spacing may be an error that violates rules of spacing in a specific language. Punctuation errors can be errors in missing or misplaced punctuation marks between sentences in a particular language. Errors about numerals can be errors about positive numerals indicating quantity and ordinal numerals indicating order. For example, when a specific language is Korean, an error on numeral may correspond to a case where 'eleven: fifty' is misspelled as 'eleven: fifty' or 'ten one: fifty'. Spelling and grammatical errors may be errors that violate the spelling and grammar of a particular language. For example, a specific language may include various languages such as Korean or English.

The primary error is an error for a specific language itself, and may include a letter deletion error, a letter addition error, a letter substitution error, and a letter separation error.

Here, the letter deletion error may be an error in which letters of a specific language are deleted. The character addition error may be an error in which characters of a specific language are added. A letter substitution error may be an error in which a word in a particular language is replaced with another word or the order of phonemes within a syllable is changed.

The plurality of second error types may include the plurality of first error types and secondary errors for spelling and grammar.

The second error may be an error for a foreign word or a coined word for a specific language. Foreign word conversion errors, semantic errors, and neologism errors may be included.

Here, the foreign word conversion error may be an error that occurs when converting a foreign word for a specific language. A semantic error may be an error when a foreign word for a specific language has an incorrect meaning in context. A neologism error may be an error in which a neologism for a specific language is incorrectly written or written.

For example, data for a plurality of first error types and data for a plurality of second error types may be pre-stored in the server through various methods such as crowdsourcing. For example, the data on the plurality of first error types may include text in which typos corresponding to the plurality of first error types occur and text in which typos corresponding to the plurality of first error types are corrected. Data on the plurality of second error types may include text in which typos corresponding to the plurality of second error types occur and text in which typos corresponding to the plurality of second error types are corrected.

In step S305, the server determines the first reproduction time of the video and the text displayed in the second region of the reproduction screen of the video based on the time when the plurality of first texts displayed in the first region of the reproduction screen of the video are changed. Based on the time point at which the plurality of second texts are changed, the second reproduction time point of the video may be distinguished.

In step S306, the server may determine a video text list composed of the plurality of third texts matched with the first reproduction time of the video and the plurality of fourth texts matched with the second reproduction time of the video.

For example, when 'first subtitle' is displayed in the first area of the video playback screen at 0:30, 'first subtitle' is displayed as 'second subtitle' in the first area of the video playback screen. The point in time when it is changed may be 0 minutes and 40 seconds. In this case, the plurality of first texts may include a 'first subtitle' and a 'second subtitle', and the 'first subtitle' may be divided into 0 minutes and 30 seconds, and the 'second subtitle' may be divided into 0 minutes and 40 seconds.

For example, if the 'first subtitle' is determined to be the 'first corrective subtitle' through the typo correction model, the 'first subtitle' is displayed in the first area of the video playback screen at 0:30. 1 correction caption' may be matched.

When the 'first phrase' is displayed in the second area of the video playback screen at 0:35, and the 'first phrase' is changed to the 'second phrase' in the second area of the video playback screen The time point may be 0 minutes and 45 seconds. In this case, the plurality of second texts may include a 'first phrase' and a 'second phrase', and the 'first phrase' may be divided into 0 minutes and 35 seconds, and the 'second phrase' may be divided into 0 minutes and 45 seconds.

For example, if the 'first phrase' is determined to be the 'first correctional phrase' through the typo correction model, the 'first phrase' is displayed in the second area of the video playback screen at 0:35. 1 correction phrase' can be matched.

A video text list composed of the plurality of third texts matched with the first reproduction time of the video and the plurality of fourth texts matched with the second reproduction time of the video may be determined.

For example, the video text list may have a form in which a plurality of third texts are sequentially arranged according to a first playback time of a video and a plurality of fourth texts are sequentially arranged according to a second playback time of a video.

Additionally, for example, the video text list may include first text before redaction among a plurality of third texts and second text before redaction among a plurality of fourth texts. In this case, the first text before correction may be separately displayed on the display module of the user terminal on the corrected portion of the plurality of third texts, and the second text before correction on the corrected portion among the plurality of fourth texts. It may be separately displayed on the display module of the user terminal.

In step S307, the server may transmit the video text list to the user terminal.

For example, when the user terminal requests a video text list for the video, the server may transmit the video text list for the video to the user terminal.

For example, the video text list may be displayed along with still images of the video through an interface set in the user terminal. Here, the still image of the video may include still images corresponding to the first playback time of the video and the second playback time of the video. An interface may be preset in the server, and a user terminal may receive the interface from the server and set the interface.

When any one of the plurality of third texts or the plurality of fourth texts included in the video text list is selected by the user terminal, the video is played from the playback time of the video matching the selected text. can

For example, when the user terminal selects the 'first correctional phrase' from the video text list displayed through the interface, the video may be played from 0:35 matched to the 'first correctional phrase'.

A video text list including a video text list composed of the plurality of third texts matched with the first reproduction time of the video and the plurality of fourth texts matched with the second reproduction time of the video may be determined.

For example, the video text list may have a form in which a plurality of third texts are sequentially arranged according to a first playback time of a video and a plurality of fourth texts are sequentially arranged according to a second playback time of a video.

Additionally, for example, the server may obtain log information of the user terminal, and based on the log information, the server may transmit to the user terminal the last playback time of the last video viewed by the user terminal.

4 is a diagram of a text recognition model according to an embodiment. The embodiment of FIG. 4 can be combined with various embodiments of the present disclosure.

Referring to FIG. 4 , the text recognition model may be a CRNN model 400. Here, the CRNN model 400 is composed of a CNN model and an RNN model, but may be jointly learned through one loss function.

In general, since the length of text data is different, it is difficult to expect an accurate recognition rate with a general algorithm, but the algorithm designed to solve this problem is the CRNN model.

The CRNN model may include a first neural network, and the first neural network may include a first input layer 410 , one or more first hidden layers 420 and a first output layer 430 . Also, one or more first hidden layers 420 may include a convolution layer, a recurrent layer, and a transcription layer.

Specifically, the server may extract a plurality of still screen images for each of the plurality of videos through data preprocessing, and generate an input vector obtained by adjusting the plurality of still screen images to have the same height.

The convolution layer may consist of convolution and max-pooling layers, excluding the fully connected layer, which is a layer used to classify images through a flattened matrix in the form of a one-dimensional array. there is. For example, the convolutional layer may extract a sequence of feature vectors from an input vector through a feature map. In this case, the sequence of feature vectors may be an input value to the recursive layer. Since the feature vectors are generated from left to right of the feature map, the ith feature vector may be a set of the ith column of all feature maps, and the width of each column may be fixed to a single pixel. That is, each feature vector included in the sequence of feature vectors may indicate a feature of a specific region of the still image. Here, a specific region may be referred to as a frame. In this case, the risk of overfitting may be reduced by adding a batch normalization layer, which is one of normalization techniques, to increase learning efficiency. Here, the batch normalization layer can perform a normalization operation using the mean and variance for each batch even if the data has various distributions for each batch unit in the learning process. A batch refers to a group of samples used when updating the weights of a model once.

The recursive layer includes one or more forward LSTM blocks and one or more reverse LSTM blocks, and each LSTM block includes a memory cell, an input gate, an erase gate, and an output gate. can include

For example, a memory cell is a node that outputs a result through an activation function, and the memory cell can perform a recursive operation of using a value output from a memory cell at a previous time as its input at a current time. there is. For example, when the current time point is t, a value output by a memory cell at the current time point t may be influenced by values of memory cells in the past. The memory cell may output a cell state (C _t ) value and a hidden state (h _t ) value. That is, the memory cell uses the cell state value (C _t-1 ) and the hidden state value (h _t-1 ) delivered by the memory cell at time t-1 as input values for calculating the cell state value and hidden state value at time t. can be used as

)가 활성화 함수인 레이어일 수 있다. 입력 게이트, 삭제 게이트 및 출력 게이트를 통해 셀 스테이트가 제어되고, 각 게이트와 입력에 따른 가중치들이 존재할 수 있다.For example, an input gate, a deletion gate, and an output gate may all include a sigmoid layer and indicate how much information input through the sigmoid layer is transmitted. For example, a sigmoid layer is a sigmoid function (

) may be a layer whose activation function is A cell state is controlled through an input gate, a deletion gate, and an output gate, and weights according to each gate and input may exist.

For example, a sequence of feature vectors from the convolution layer is input to an input gate, and the deletion gate is generated based on the sequence of feature vectors h _t-1 (hidden state at time t-1) and x _t (t A value between 0 and 1 can be transferred to C _t-1 based on the input value of the point in time). Here, a value between 0 and 1 is the amount of information that has gone through the deletion process. The closer to 0, the more information is deleted, and the closer to 1, the more intact information can be transmitted.

For example, an input gate may determine a value to be updated through a sigmoid layer, and a tanh layer may generate C _t vectors, which are new candidate values, and store them in a cell state. By updating the previous cell state C _t-1 , a new cell state C _t can be created. That is, for a cell state, information is deleted by multiplying f _t by the cell state, and a value obtained by scaling the updated value may be added. For example, tanh means a nonlinear activation function (hyperbolic tangent function).

For example, the output gate may determine a portion of a cell state to be output based on a sequence of feature vectors in the sigmoid layer, and the output gate converts the cell state determined in the sigmoid layer to -1 through a tanh layer. A value between 1 can be multiplied by the output value.

Through this process, the recursive layer back-propagates the value related to the sequence of feature vectors to the convolution layer, and updates the operation of changing the feature map of the convolution layer to the sequence of feature vectors, thereby giving feedback to the convolution layer. . Here, a value related to the sequence of feature vectors may be a differential value with respect to the sequence of feature vectors. In addition, the recursive layer may transmit prediction values for each frame of the sequence of feature vectors to the transcription layer.

Additionally, for example, an LSTM model in which the deletion gate and the input gate are combined may be used. In this case, since only the old information to which new information is added is deleted, the operation can be performed more quickly.

The transcription layer may convert prediction values for each frame generated in the recursive layer into a label sequence through a connectionist temporal classification (CTC) technique, and output the label sequence in an ordered form. The CTC technique is an algorithm used for learning sequence data in which only class labels are in order in the training data and the location of each class is unknown.

For example, the transcription layer may detect a label sequence having the highest probability according to a prediction value for each frame. The transcription layer may remove overlapping labels (characters) included in prediction values for each frame and match 1 to the removed prediction values. Also, the transcription layer may remove spaces included in prediction values for each frame. For example, in the case of the predicted value for each frame for "go-go-yang-ii-", "-" means blank, and the transcription layer is determined as "cat" excluding overlapping characters and blanks through the CTC technique. can That is, the transcription layer may calculate a loss based on the number of dimensions of a predicted value for each frame through the CTC technique and calculate a gradient accordingly.

5 is a diagram of a typo correction model according to an embodiment. The embodiment of FIG. 5 can be combined with various embodiments of the present disclosure.

Referring to FIG. 5 , the typo correction model 500 may include a bidirectional LSTM model.

The typo correction model 500 may include an embedding and encoder unit, a second neural network including a bidirectional LSTM, and an attention layer.

A vector input to the second neural network including the bidirectional LSTM may be a vector output by passing a plurality of texts through an embedding and encoder unit. For example, the embedding and encoder unit may include a plurality of token embedding layers, a plurality of position embedding layers, and an encoder. For example, the plurality of texts may include texts displayed on a playback screen of a plurality of videos.

The token embedding layer may convert input text preprocessed with a tokenizer for a plurality of texts into a token vector including tokens. For example, a token vector obtained by tokenizing a sentence into a plurality of morphemes may be generated by preprocessing a plurality of texts with a tokenizer through a token embedding layer. Here, the morpheme means the smallest unit of speech that cannot be further analyzed as an element having meaning.

In addition, by using a masked language model for the token vector, the encoder inserts the entire sentence through the given sequence into the model, then predicts and learns the word corresponding to the blank, and after learning the sentence, The corresponding sentence is displayed as a series of tokens, and the part to be blank in the token array can be set as a mask.

The position embedding layer may add a position embedding vector indicating a position of each morpheme to a token vector obtained by tokenizing one sentence into a plurality of morphemes. Here, the token vector may be converted into a Q vector representing a query, a K vector representing a key, and a V vector representing a value.

At this time, the server may determine a pair of word relationship through the position embedding layer. For example, to determine the closeness of each word, the server can use scaled dot-product attention. The server may determine a pair of word relationships by Equation 2 below based on scaled inner product attention.

In Equation 2, Q is a vector for the hidden state of the previous layer of the decoder, K is a vector for the output state of the encoder, V is a vector for the output state of the encoder, and d _k is the dimension of the K vector. can be That is, Q may be the token of the affected decoder, K may be the token of the influencing encoder, and V may be the token of the encoder to be multiplied by the weight for the influence.

That is, the server can determine a pair of matrices for words in a sentence by dividing the dot-product of Q and K using the cosine similarity by the square root of the dimension of the K vector and applying a softmax function. there is.

The second neural network may include a second input layer, one or more second hidden layers, and a second output layer. For example, the one or more second hidden layers include one or more forward LSTM blocks and one or more reverse LSTM blocks, and each LSTM block includes a memory cell, an input gate, and an erase gate , may include an output gate.

The server may input a pair of matrices of words in the sentence to the bidirectional LSTM, and input the vectors output through the bidirectional LSTM to the attention layer. At this time, the server can fine-tune the typo correction model by connecting hidden tokens through bidirectional LSTM.

The attention layer can input and output information from one sentence to the end of the sentence, and can process the first word to form a correlation with words farther away. Then, the output layer may determine a word having the highest probability based on any one of the plurality of first error types and the plurality of second error types as an output value through a softmax function.

That is, the server may learn a typo correction model including a bidirectional LSTM model based on a plurality of texts, and determine data in which errors are corrected for the plurality of data through the typo correction model.

6 is a flowchart of a method for a server to transmit a video text list to a user terminal according to an exemplary embodiment. The embodiment of FIG. 6 can be combined with various embodiments of the present disclosure.

Referring to FIG. 6 , in step S601, the server may receive a video from an external server and extract information about the video.

In step S602, the server may determine a minimum size for recognizing text displayed on the video playback screen. For example, the minimum size may be determined by Equation 1 described above.

In step S603, the server may recognize text displayed on the playback screen of the video through a text recognition model using a first neural network for the video based on the minimum size.

In step S604, the server may determine whether or not the recognized text is text included in a first area in the playback screen of the video.

For example, the first area in the playback screen of the video may be preset in the server. The first area in the playback screen of the video may be an average area of areas spaced apart by a specific width from the bottom of the video to include the subtitle.

In step S605, when the recognized text is text included in the first area of the video playback screen, the server may determine the recognized text as the first text.

In step S606, when the recognized text is text that is not included in the first area of the video playback screen, the server may determine the recognized text as second text.

In step S607, the server may determine whether the number of first texts is greater than or equal to a preset threshold value. For example, the predetermined threshold value may be determined based on a playback time of a video, a data processing speed of a server, and the number of user terminals accessing the server.

For example, the preset threshold value may be determined by Equation 3 below.

In Equation 3, n _th is the preset threshold value, n is the number of videos uploaded to the server, t _i is the playing time of the ith video, and p _v is the server The number of transactions that can be processed per second, n _c is the number of user terminals accessing the server, and n _d may be a default value for the preset threshold value.

Here, the playback time of the video may be in seconds. The p _v may be a transaction per second (TPS) unit. The floor function is a function that reflects the nearest rounding value. For example, the default value may be preset in the server.

Even if the number of texts displayed in the video is the same, the speed at which the server processes data may vary according to the playing time of the video. In particular, when the video playback time is long, the server may take more time to process the text data because the capacity of the video is relatively larger than when the video playback time is short. Therefore, by setting the preset threshold to a larger value as the playing time of the video is longer, the server detects a plurality of first error types with a relatively small number for a larger number of texts, thereby reducing the load on the server. can

For example, since the server can process a larger amount of data as the TPS is higher, by lowering the preset threshold value, the server has a relatively large number of second error types for a smaller number of texts. can be detected to improve the quality of error detection.

For example, as the number of user terminals accessing the server increases, the data to be processed by the server increases. Therefore, by setting the preset threshold value to a large value, the server can obtain a relatively small number of texts for a larger number of texts. A load on a server may be reduced by detecting a plurality of first error types.

Through this, by adjusting the preset threshold value in consideration of various factors affecting the data processing speed of the server, the quality of the server detecting typos in the text displayed in the video can be improved.

In step S608, when the number of first texts is greater than or equal to a preset threshold, the server may determine third text and fourth text according to a plurality of first error types. In step S609, when the number of first texts is less than a preset threshold value, the server may determine third text and fourth text according to a plurality of second error types. In step S610, the server may transmit the video text list to the user terminal.

7 is a block diagram illustrating a configuration of a server according to an exemplary embodiment. One embodiment of FIG. 7 may be combined with various embodiments of the present disclosure.

As shown in FIG. 7 , the server 700 may include a processor 710 , a communication unit 720 and a memory 730 . However, not all components shown in FIG. 7 are essential components of the server 700 . The server 700 may be implemented with more components than those shown in FIG. 7 , or the server 700 may be implemented with fewer components than those shown in FIG. 7 . For example, the server 700 according to some embodiments may further include a user input interface (not shown) and an output unit (not shown) in addition to the processor 710, the communication unit 720, and the memory 730. .

The processor 710 typically controls the overall operation of the server 700. The processor 710 may include one or more processors to control other elements included in the server 700 . For example, the processor 710 may generally control the communication unit 720 and the memory 730 by executing programs stored in the memory 730 . Also, the processor 710 may perform the functions of the server 700 described in FIGS. 3 to 6 by executing programs stored in the memory 730 .

The communication unit 720 may include one or more components that allow the server 700 to communicate with other devices (not shown) and servers (not shown). Another device (not shown) may be a computing device such as the server 700 or a sensing device, but is not limited thereto. The communication unit 720 may receive a user input from another electronic device or data stored in an external device from an external device through a network.

The memory 730 may store programs for processing and controlling the processor 710 . For example, the memory 730 may store information input to a server or information received from another device through a network. Also, the memory 730 may store data generated by the processor 710 . The memory 730 may store information input to or output from the server 700 .

The memory 730 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , an optical disk, and at least one type of storage medium.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (5)

A method for a server to transmit a video text list to a user terminal using a plurality of neural networks, the method comprising:
Receiving a video from an external server;
extracting information about the video, including file size of the video, playback time of the video, frame size of the video, and color information about the video, from the video;
A text recognition model using a first neural network based on the video and information about the video is applied to a plurality of first texts included in a first area in a playback screen of the video and a second area in a playback screen of the video through a text recognition model using a first neural network. determining a plurality of included second texts;
determining a plurality of third texts and a plurality of fourth texts based on the plurality of first texts and the plurality of second texts through a typo correction model using a second neural network;
The plurality of third texts are texts in which typos included in the plurality of first texts are corrected;
The plurality of fourth texts are texts in which typos included in the plurality of second texts are corrected;
Based on the time when the plurality of first texts displayed in the first area of the video playback screen are changed, the first playback time of the video and the plurality of second texts displayed in the second area of the video playback screen Dividing a second play point of the video based on the changed point of time;
determining a video text list composed of the plurality of third texts matched with a first reproduction time of the video and the plurality of fourth texts matched with a second reproduction time of the video; and
Transmitting the video text list to a user terminal; including,
When any one of the plurality of third texts or the plurality of fourth texts included in the video text list is selected by the user terminal, the video is played from the playback time of the video matching the selected text. ,
method.
According to claim 1,
The first area is an area where subtitles are located on the video playback screen;
The second area includes an area other than the first area,
The plurality of first texts and the plurality of second texts are texts having a size greater than a minimum size;
The minimum size is determined based on the frame size of the video, the size of the subtitle, and color information of the video.
method.
According to claim 2,
The minimum size is determined by the following equation,

In the above equation, T _min is the initial size, n _c is a value determined based on the distribution of RGB values included in the video, and n _d is a basic value for the distribution of RGB values included in the video. , wherein S _f is the frame size of the video, S _d is the average frame size, S _c is the size of the subtitle of the video, and T _r is the minimum size that the recognition model can recognize. the size of the text,
method.
According to claim 2,
Based on the fact that the number of the plurality of first texts is greater than or equal to a preset threshold, the typo correction model detects typos in the plurality of first texts and the plurality of second texts according to the plurality of first error types; ,
Based on that the number of the plurality of first texts is less than a preset threshold, the typo correction model detects typos in the plurality of first texts and the plurality of second texts according to the plurality of second error types; ,
The plurality of first error types include errors in spacing, errors in punctuation marks, errors in rhetoric, and primary errors in spelling and grammar,
The primary error includes a letter deletion error, a letter addition error, a letter substitution error, and a letter separation error,
The plurality of second error types include the plurality of first error types and secondary errors for spelling and grammar;
The secondary error includes a typing error, a foreign word conversion error, a semantic error, and a neologism error.
method.
According to claim 4,
The text recognition model is a convolutional recurrent neural network (CRNN) model,
The first neural network includes a first input layer, one or more first hidden layers and a first output layer;
Data for learning each of the first neural networks composed of a plurality of videos and a plurality of correct answer texts is input to the first input layer of the first neural network, and the one or more first hidden layers and the first output layer are input. Pass through and output a first output vector, the first output vector is input to a first loss function layer connected to the first output layer, and the first loss function layer is applied to the first output vector and each training data The text recognition model generates a first loss value by using a first loss function that compares a first answer vector for , and the parameters of the first neural network are learned in a direction in which the first loss value decreases. become,
A plurality of still screen images are extracted for each of the plurality of videos through data preprocessing for the plurality of videos,
An input vector generated by adjusting the plurality of still screen images to the same height is input to the first input layer;
The plurality of correct answer texts are text included in the plurality of still screen images,
The typo correction model includes a bidirectional long short-term memory (LSTM) model,
The second neural network includes a second input layer, one or more second hidden layers, and a second output layer;
Data for learning of each second neural network composed of a plurality of texts and a plurality of correct answer texts is input to the second input layer of the second neural network, and the one or more second hidden layers and the second output layer are input. Pass through and output a second output vector, the second output vector is input to a second loss function layer connected to the second output layer, and the second loss function layer is applied to the second output vector and each training data A second loss value is output using a second loss function that compares a second answer vector for , and the parameters of the second neural network are learned in a direction in which the second loss value becomes smaller. The typo correction model is generated. become,
The plurality of texts include the plurality of first texts and the plurality of second texts,
The plurality of correct answer texts include the plurality of third texts and the plurality of fourth texts,
method.

Images