KR102576358B1 - Learning data generating device for sign language translation and method of operation thereof - Google Patents

Abstract

A method of operating a learning data generating device that generates learning data for artificial intelligence-based sign language translation according to an embodiment of the present disclosure is to translate first text data composed of Korean sentences into second text data composed of sign language sentences. requesting, upon receiving the second text data, obtaining one or more semantic elements from the second text data, obtaining sign language image data for the first text data, the one or more semantic elements and generating learning data based on the sign language image data, and storing the learning data.

Description

Learning data generation device for sign language translation and its operation method {LEARNING DATA GENERATING DEVICE FOR SIGN LANGUAGE TRANSLATION AND METHOD OF OPERATION THEREOF}

The present disclosure relates to a learning data generating device and an operating method thereof, and more specifically, to a learning data generating device that generates learning data for sign language translation for artificial intelligence learning and an operating method thereof.

Sign language is the most representative method used when communicating to a deaf person and refers to a method of communicating through gestures. Sign language has its own grammar and expresses sentences by continuously performing gestures with a given meaning.

The grammar of sign language is different from the grammar of vocal language expressed through voice and text. Therefore, in order for hearing people to express themselves without discomfort in sign language, they must complete a long period of specialized training, so the majority of hearing people have difficulty communicating with deaf people using sign language. In order to solve these communication difficulties, technology that converts spoken language familiar to hearing people into sign language familiar to deaf people (hereinafter referred to as sign language translation technology) is essential, and recently, research has been conducted on sign language interpretation/translation technology using artificial intelligence. is actively underway.

Machine learning, or machine learning, is a field of artificial intelligence. It can be said to be a series of processes that train a computer with pre-prepared learning data and seek appropriate answers to new inputs based on the trained knowledge. At this time, if the learning data used to train the computer is given both a question (training input) and a correct answer (training output), it is said to be labeled.

On the other hand, when performing sign language translation (or interpretation, hereinafter the same) based on machine learning, the important thing in addition to feature extraction and learning algorithms is the collection of labeled learning data. The more labeled learning data is provided, the more important the learning becomes. It can be done more effectively. For this purpose, thousands to tens of thousands of labeled training data are needed, but since labeled training data is generally created manually, it is not easy to obtain a large amount of labeled training data. Therefore, for effective machine learning, a method for generating efficiently labeled learning data is needed.

The purpose of the present disclosure is to provide a learning data generation device and a method of operating the same that can efficiently generate the learning data necessary to secure an artificial intelligence model that can smoothly interpret/translate sign language.

A method of operating a learning data generating device that generates learning data for artificial intelligence-based sign language translation according to an embodiment of the present disclosure is to translate first text data composed of Korean sentences into second text data composed of sign language sentences. requesting, upon receiving the second text data, obtaining one or more semantic elements from the second text data, obtaining sign language image data for the first text data, the one or more semantic elements and generating learning data based on the sign language image data, and storing the learning data.

Depending on the embodiment, the step of obtaining the one or more semantic elements further includes separating the second text data into the one or more semantic elements that are the minimum semantic units of a sign language through morphological analysis of the second text data. It can be included.

Depending on the embodiment, the step of acquiring the sign language image data includes providing the first text data and the second text data to a photographing device including a camera, and receiving the sign language image data from the photographing device. It may further include.

Depending on the embodiment, the step of generating the learning data includes transmitting the one or more semantic elements and the sign language image data to a worker terminal, and the one or more semantic elements are transmitted from the worker terminal to the sign language image based on a timeline. It may further include receiving mapping information matched to data, and generating token information based on the mapping information.

According to an embodiment, the step of storing the learning data includes, when the accuracy of the token information is greater than or equal to the reference value, a learning database is created based on the learning data in which the input is the first text data and the correct answer is the sequence of the one or more semantic elements. A generating step may be further included.

A learning data generating device according to an embodiment of the present disclosure includes an interface configured to transmit and receive data with external devices, a sensor configured to acquire sign language image data, and a processor. The processor requests the sign language translation device among the external devices to translate first text data composed of Korean sentences into second text data composed of sign language sentences, and upon receiving the second text data, the second It is configured to acquire one or more semantic elements from text data, generate learning data based on the one or more semantic elements and the sign language image data, and store the learning data to create a database.

The learning data generation device and its operating method according to an embodiment of the present disclosure can efficiently generate labeled learning data necessary for learning an artificial intelligence-based sign language translation system. As a result, machine learning can be carried out effectively, and the usability of the sign language translation system can be increased by using the learning data generated through this.

Figure 1 is a block diagram showing a sign language learning system according to an embodiment of the present disclosure.
FIG. 2 is a block diagram showing the learning data generating device of FIG. 1.
FIG. 3 is a block diagram showing the processor of FIG. 2.
Figure 4 is a flowchart showing a method of operating a learning data generating device according to an embodiment of the present disclosure.
Figure 5 is a diagram for explaining step S150 of Figure 4.
FIGS. 6A and 6B are diagrams for explaining step S410 of FIG. 5.
Figure 7 is a block diagram showing a sign language translation system to which a learning data generation device according to an embodiment of the present disclosure is applied.

Hereinafter, embodiments of the present invention will be described clearly and in detail so that a person skilled in the art can easily practice the present invention. However, since the present invention can be implemented in various different forms within the scope described in the claims, the embodiments described below are merely examples, regardless of whether they are expressed or not. That is, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

Hereinafter, embodiments of the present invention will be described with reference to the attached drawings.

Figure 1 is a block diagram showing a sign language learning system according to an embodiment of the present disclosure. The sign language learning system 10 can generate labeled learning data used in machine learning for sign language translation. For example, the sign language learning system 10 may generate learning data through a data labeling task based on sign language sentences for Korean sentences and sign language images for Korean sentences. Referring to FIG. 1, the sign language learning system 10 may include a learning data generating device 100, a sign language translation device 200, a sign language image capturing device 300, and a worker terminal 400.

The learning data generating device 100 can receive Korean sentences for sign language translation. The Korean sentence may be a sentence specified or input through the UI by the user of the learning data generating device 100, and when mounted and operated in the broadcasting signal receiver of the learning data generating device 100, it is separated from the received broadcasting signal and decoded. It may be a sentence from a processed subtitle broadcast, or it may be a translation request sentence transmitted from another computer or server system that can be linked through a communication network.

The learning data generating device 100 is configured to control a series of operations for generating learning data using Korean sentences as input. For example, the learning data generating device 100 may request the sign language translation device 200 to translate a sign language sentence into a Korean sentence. The learning data generating device 100 may request the sign language image capturing device 300 to capture a sign language image for a Korean sentence. The learning data generating device 100 can control the labeling task by providing sign language sentences and sign language images to the worker terminal 400.

Depending on the embodiment, the learning data generating device 100 may be implemented as a computer, server, or cloud that can access a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. That is, the learning data generating device 100 can communicate with the sign language translation device 200, the sign language image capturing device 300, and the worker terminal 400 through a network. A detailed description of the learning data generating device 100 will be described later with reference to FIG. 2 .

Network (Network) The sign language learning system 10 can electrically connect the learning data generating device 100, the sign language translation device 200, the sign language image capture device 300, and the worker terminal 400. Network refers to a connection structure that allows information exchange between nodes such as a plurality of terminals and servers. Examples of such networks include a local area network (LAN) and a wide area network (WAN). Wide Area Network, Internet (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), and Wi-Fi. , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC ( It includes, but is not limited to, Near-Field Communication (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, and DMB (Digital Multimedia Broadcasting) network.

The sign language sentence translation device 200 is configured to receive a Korean sentence from the learning data generating device 100 and translate the Korean sentence into a sign language sentence. The sign language translation device 200 may perform a sign language translation task based on an input signal from a translation expert. In order to distinguish between Korean and signed language sentences, the Korean sentences may be referred to as first text data, and the signed language sentences may be referred to as second text data. For example, the sign language translation device 200 may receive first text data consisting of a Korean sentence and convert it into second text data consisting of a sign language sentence.

Because the grammatical system of sign language sentences is different from that of Korean sentences, translation of sign language sentences by experts is necessary. For example, if the sentence “I bought a strong house” is directly translated one-to-one, it can be expressed as “I + strong + house + bought”, which from the perspective of a deaf person who is not familiar with Korean sentences can be interpreted as “I am strong and (I am ) can be understood as “I bought a house.” Therefore, in order to properly convey the intention, changing the word order to “I + house + buy, house + strong” and adding descriptive expressions will allow for a more clear conveyance of meaning.

The sign language translation device 200 may provide second text data to the learning data generation device 100. The learning data generating device 100 may provide first text data and second text data to the sign language image capturing device 300.

The sign language image capturing device 300 may be configured to capture a sign language image of a Korean sentence. For example, the sign language image capturing device 300 may include a camera, and may generate sign language image data for the first text data through the camera. The sign language image may be captured by a sign language expert, and depending on the embodiment, the sign language expert may generate the sign language image by referring to the second text data. Depending on the embodiment, the sign language image capturing device 300 may be integrated into the learning data generating device 100.

The worker terminal 400 may include a mobile terminal of a worker performing a labeling task. The worker terminal 400 is a wireless communication device that ensures portability and mobility, and includes navigation, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), and PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone , may include all types of handheld-based wireless communication devices such as smartpads, tablet PCs, etc.

The worker terminal 400 may perform data labeling work through a web server or application provided by the learning data generating device 100. For example, the worker terminal 400 may generate a token by mapping semantic elements, which are the minimum semantic units of a sign language, to sign language image data based on the worker's input signal. The mapped token may be verified by the learning data generating device 100 and stored as labeled learning data.

Depending on the embodiment, the learning data generating device 100 may provide a labeling authoring tool to the worker terminal 400. The labeling authoring tool may include a user interface designed to efficiently perform the mapping task of sign language image data. Workers can use the labeling authoring tool through the worker terminal 400, and can perform labeling work more time-efficiently than existing authoring tools by generating tokens with a simple operation. A detailed description of the labeling authoring tool is provided below in FIG. 6A.

FIG. 2 is a block diagram showing the learning data generating device of FIG. 1. 1 and 2, the learning data generating device 100 may include an interface 110, a sensor 120, a database 130, a communication module 140, a memory 150, and a processor 170. You can.

The interface 110 may be configured to transmit and receive data or information with the sign language translation device 200, the sign language image capturing device 300, and the worker terminal 400. For example, the interface 110 may transmit first text data to the sign language translation device 200 and receive second text data. The interface 110 may transmit first text data and second text data to the sign language image capturing device 300 and receive sign language image data. The interface 110 may transmit morpheme information and sign language image data to the worker terminal 400 and receive token information.

The sensor 120 may include a camera and a motion recognition device to generate sign language image data. For example, a motion recognition device is worn by a deaf person. When a deaf person wears it on their hand and uses sign language, a number of motion recognition sensors are configured to recognize the movement of the hand using the built-in sensor, and are configured to recognize the corresponding motion. You can. The sensor 120 corresponds to the sign language image capturing device 300 of FIG. 1 and may be omitted depending on the embodiment. In this case, the external sign language image capturing device 300 can generate sign language image data.

The database 130 may be created by the processor 170. The database 130 may store labeled learning data. For example, the database 130 may store learning data in the form of a lookup table and provide the stored learning data when training an artificial intelligence model.

The database 130 may be comprised of non-volatile memory. Non-volatile memory refers to a storage medium that stores and maintains information even when power is not supplied, and can use the stored information again when power is supplied. Non-volatile memory is, for example, flash memory, hard disk, solid state drive (SSD), multimedia card micro type, card type memory (e.g. SD or It may include at least one of (XD memory, etc.), ROM (Read Only Memory, ROM), magnetic memory, magnetic disk, and optical disk.

The communication module 140 may communicate with other electronic devices or external devices under the control of the processor 170. The communication module 140 may connect to a network or communicate between devices through wireless or wired communication through a communication interface. Wireless communications include, for example, Wifi (wireless fidelity), BT (bluetooth), NFC (near field communication), GPS (global positioning system), or cellular communications (e.g., LTE, LTE-A, CDMA). , WCDMA, UMTS, WiBro, or GSM). Wired communication may include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS).

Memory 150 may store a program including one or more instructions. The memory 150 may store at least one of instructions, algorithms, data structures, program codes, and application programs that the processor 170 can read. Instructions, algorithms, data structures, and program codes stored in memory 150 may be implemented in, for example, programming or scripting languages such as C, C++, Java, assembler, etc.

For example, the memory 150 may be a flash memory type, RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), or EEPROM (Electrically Erasable Memory). It may include at least one type of hardware device among Programmable Read-Only Memory (Programmable Read-Only Memory) and PROM (Programmable Read-Only Memory).

The processor 170 may be electrically connected to the interface 110, sensor 120, database 130, communication module 140, and memory 150 to control each component. Depending on the embodiment, the processor 170 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). For example, the processor 170 may perform operations or data processing related to control and/or communication of at least one other component of the learning data generating device 100. The processing (or control) operation of the processor 170 is described in detail with reference to FIG. 3.

FIG. 3 is a block diagram showing the processor of FIG. 2. The processor 170 can control a number of hardware or software components connected to the processor 170 by, for example, running an operating system or application program, and can perform various data processing and calculations. The processor 170 may be implemented, for example, as a system on chip (SoC). Depending on the embodiment, the processor 170 may further include a graphic processing unit (GPU) and/or an image signal processor (ISP).

Referring to FIGS. 1 to 3 , the processor 170 may include a morpheme analysis unit 171, a token generation unit 172, and a verification unit 173.

The morpheme analysis unit 171 may be configured to extract semantic elements through morpheme analysis based on the second text data. Semantic elements can be composed of the minimum semantic unit of a sign language. For example, in the sentence “I bought a strong house,” if the sign language is translated as “I + house + buy, house + strong,” the four meanings are “me,” “house,” “buy,” and “strong.” Elements can be extracted. Information on these semantic elements can be named morpheme information. The morpheme analysis unit 171 may provide morpheme information to the worker terminal 400.

Depending on the embodiment, the morpheme analysis unit 171 may be designed to analyze the syntax of Korean sentences using a bidirectional LSTM (Long Short-Term Memory) type morpheme analyzer to understand the meaning and select the optimal word accordingly. there is. This morpheme analyzer can recognize the basic form of each morpheme that makes up a word and can remove postpositions that are unnecessary for sign language interpretation.

The token generator 172 may be configured to generate a token based on mapping information received from the worker terminal 400. For example, mapping information may include information in which one or more semantic elements are matched to sign language image data based on a timeline. This matching task can be performed through a labeling authoring tool. When the matching task of the worker terminal 400 is completed, the token generator 172 may generate token information including morpheme information and timeline information.

The verification unit 173 may be configured to verify the accuracy of token information. For example, the verification unit 173 may convert one or more semantic elements into test image data based on token information and compare the test image data with sign language image data. The verification unit 173 can determine the accuracy of token information by verifying whether the test image data and the sign language image data are similar using a similarity judgment algorithm. For example, if the similarity between the test image data and the sign language image data is above a certain value, the verification unit 173 may determine the accuracy of the token information to be above the reference value and complete the verification.

Figure 4 is a flowchart showing a method of operating a learning data generating device according to an embodiment of the present disclosure. Referring to FIG. 4, the operating method (S100) of the learning data generating apparatus 100 may include steps S110 to S160 to generate labeled learning data.

In step S110, the learning data generating device 100 may receive a Korean sentence. The Korean sentence may be a sentence specified or input through the UI by the user of the learning data generating device 100, and when mounted and operated in the broadcasting signal receiver of the learning data generating device 100, it is separated from the received broadcasting signal and decoded. It may be a sentence from a processed subtitle broadcast, or it may be a translation request sentence transmitted from another computer or server system that can be linked through a communication network. The learning data generating device 100 may store a Korean sentence as first text data in the memory 150.

In step S120, the learning data generating device 100 may request a sign language sentence for a Korean sentence. For example, the learning data generating device 100 may request a sign language sentence for a Korean sentence from the sign language translation device 200. The learning data generating device 100 may receive the translated sign language sentence as second text data and store it in the memory 150.

In step S130, the learning data generating device 100 may perform morphological analysis on the sign language sentence. For example, the learning data generating device 100 may extract one or more semantic elements from the second text data through morphological analysis. A semantic element can be defined as the minimum semantic unit of a sign language. A sign sentence may be composed of one or more semantic elements, and the learning data generating device 100 may separate the second text data into one or more semantic elements.

In step S140, the learning data generating device 100 may acquire a sign language image for a Korean sentence. For example, the learning data generating device 100 may request the sign language image capturing device 300 to capture a sign language image for a Korean sentence. In this case, the learning data generating device 100 may provide first text data to the sign language image capturing device 300, and, depending on the embodiment, may also provide second text data. The sign language image capturing device 300 may capture a sign language image based on the first text data and the second text data and provide the sign language image data to the learning data generating device 100. Depending on the embodiment, the learning data generating device 100 may acquire sign language image data through the internal sensor 120.

In step S150, the learning data generating device 100 may generate learning data. For example, the learning data generating device 100 may provide information about one or more semantic elements and sign language image data to the worker terminal 400, and receive mapping information on which the labeling task has been completed from the worker terminal 400. there is. Depending on the embodiment, the mapping information may include information in which one or more semantic elements are matched to sign language image data based on a timeline. The learning data generating device 100 may generate a token based on mapping information and generate learning data through verification of the token. A detailed description of this is provided later in FIG. 5 .

In step S160, the learning data generating device 100 may create a database. For example, the learning data generating device 100 may store verified learning data as labeled learning data in a database. That is, when the accuracy of token information is greater than the reference value, the learning data generating device 100 can create a database by storing labeled learning data in which the input is first text data and the correct answer is a sequence of one or more semantic elements.

Figure 5 is a diagram for explaining step S150 of Figure 4. Referring to FIGS. 4 and 5 , the learning data generating device 100 may generate learning data while communicating with the worker terminal 400.

In step S151, the learning data generating device 100 may transmit sign language image data and morpheme information to the worker terminal 400. The sign language image data may include a sign language image captured by the sign language image capturing device 300 or the sensor 120, and the morpheme information may include information about one or more semantic elements generated through morpheme analysis. .

Depending on the embodiment, the worker terminal 400 may perform a mapping or labeling task through sign language image data and morpheme information (S410). In this case, the learning data generating device 100 may support a labeling authoring tool for performing a mapping operation or a labeling operation, and the labeling authoring tool may be exposed to the worker terminal 400 through a UI.

For example, referring to FIG. 6A, the labeling authoring tool may include first to third screens S1, S2, and S3. The first screen (S1) may be a screen where a sign language image is played based on sign language image data. The second screen S2 may be a screen that displays information generated through a labeling authoring tool, such as task information, token information, and memos. For example, task information may include task name, file name, total time, person in charge, status, and last update date information. For example, token information may include word, congruent verb, and time (rearranged timeline) information. The third screen (S3) may be a screen that displays a timeline for the sign language image and semantic elements matched based on the timeline.

The worker can run the labeling authoring tool through the worker terminal 400. When the labeling authoring tool is executed, the worker terminal 400 plays a sign language image based on sign language image data on the first screen (S1) and lists one or more semantic elements based on morpheme information on the third screen (S3). You can do it. For example, the first to fourth semantic elements E1, E2, E3, and E4 may be listed on the third screen S3.

The first to fourth semantic elements (E1, E2, E3, E4) may be combined with time information to generate a token. The worker can add time information to the first to fourth semantic elements (E1, E2, E3, E4) through the worker terminal 400, and, for example, the first to fourth semantic elements (E1, E2, E3, E4) through an input device such as a mouse. 4 Semantic elements (E1, E2, E3, E4) can be matched to the playback time of the corresponding sign language video.

Referring to the third screen (S3), the worker terminal 400 matches the first to fourth semantic elements (E1, E2, E3, E4) to the sign language image in chronological order based on the worker's input signal. You can. For example, the first semantic element (E1) may match the first section (t11 to t12) of the signed language image, and the second semantic element (E2) may match the second section (t21 to t22) of the signed language image. The third semantic element (E3) may match the third section (t31 to t32) of the signed language image, and the fourth semantic element (E4) may match the fourth section (t41 to t42) of the signed language image. It can be. The operator can finely adjust the first to fourth sections, and as a result, the learning data generating device 100 can obtain timeline information matched to the sign language image.

Figure 6b is an example of the worker terminal 400 executing the labeling authoring tool according to Figure 6a. Referring to Figure 6b, the learning data generating device 100 generates a sign language image and multiple semantic elements (“return”, “Thing,” “there,” “but,” “person,” “come,” “stop,” “direct,” “send,” “possible,” “?”) can be obtained. When a worker executes a labeling authoring tool through the worker terminal 400, a sign language image is played, and a plurality of semantic elements can be automatically arranged over time (for example, arranged as a gloss-dominant item). The operator can generate token information combining morpheme information and time information by rearranging the timeline of a plurality of semantic elements. Depending on the embodiment, the worker terminal 400 may generate gloss non-dominant data and non-dominant data based on the worker's input signal.

In other words, the learning data generating device 100 can provide a labeling authoring tool through the worker terminal 400, and the worker can perform labeling work in a time-efficient manner by using the labeling authoring tool. Specifically, when the worker runs the labeling authoring tool, the sign language image and morpheme information are automatically placed, and the worker can create a token by using a mouse, etc. to place the morpheme information on the timeline to match the sign language image. As a result, the work time for token creation can be reduced by more than two times compared to before.

Referring again to FIG. 5 , in step S152, the learning data generating device 100 may receive mapping information from the worker terminal 400. Depending on the embodiment, the mapping information may include matching information in which the worker terminal 400 matches a sign language image and one or more semantic elements based on a timeline using the labeling authoring tool of FIG. 6A.

In step S153, the learning data generating device 100 may generate a token based on mapping information. Tokens may be the result of learning data in which information about semantic elements, time information, and information about sign language images are linked. Information about the generated token may be named token information.

In step S154, the learning data generating device 100 may verify the learning data by determining the accuracy of the token information. For example, the learning data generating device 100 may complete verification when the accuracy of the token information is greater than or equal to the reference value, and complete the learning data generation in step S155. The generated learning data may be labeled learning data. For example, the input may be first text data and the correct answer may be a sequence of one or more semantic elements.

Depending on the embodiment, the accuracy of token information may be determined by converting one or more semantic elements into test image data and comparing the test image data with sign language image data. For example, the learning data generating device 100 uses a similarity judgment algorithm to verify whether the test image data and the sign language image data are similar, and when the similarity between the test image data and the sign language image data is greater than a certain value, the token information Accuracy can be judged to be higher than the standard value.

Figure 7 is a block diagram showing a sign language translation system to which a learning data generation device according to an embodiment of the present disclosure is applied. Referring to FIG. 7, the sign language translation system 1000 includes a learning data generating device 1100, a sign language translator 1200, a microphone 1300, a sound to text (STT) conversion module 1400, and a display 1500. may include.

The learning data generating device 1100 described with reference to FIGS. 1 to 6B may be applied. Therefore, the learning data generating device 1100 can generate labeled learning data and provide it to the sign language translator 1200. The sign language translator 1200 may include a neural network 1250. For example, a deep learning neural network (DNN) may perform a sign language translation operation for a Korean sentence by training labeled learning data.

The microphone 1300 can convert the user's voice analog signal into a digital signal, and the STT conversion module 1400 can convert the digital signal into text data. For example, the STT conversion module 1400 may use a deep learning neural network that converts voice to sentence into STT on a sequence-to-sequence basis.

The sign language translator 1200 can translate text data into a sequence of semantic elements through the neural network 1250, fuse the sequence of semantic elements with an interactive sign language image, and provide it to the display 1500. The display 1500 can display an interactive sign language image to the user.

As described above, the learning data generation device 100 according to the present invention enables conversation between people who cannot use sign language and the deaf by generating labeled learning data, and can be applied to the sign language translation system 1000. In addition, the learning data generation device 100 can be used for kiosks (displaying sign language animation when hearing-impaired people use the kiosk), facility information centers, weather forecast information, video map services, sign language application services, etc., and furthermore, Korean-sign language translation. With the advancement of technology and sign language recognition technology, it can be expected that an artificial intelligence knowledge ecosystem will be established that can develop artificial intelligence algorithms and services in industrial fields such as IoT, sign language interpretation, and tourism.

The above-described details are specific embodiments for carrying out the present invention. The present invention will include not only the above-described embodiments, but also embodiments that can be simply changed or easily changed in design. In addition, the present invention will also include technologies that can be easily modified and implemented using the embodiments. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by the claims and equivalents of the present invention as well as the claims described later.

10: Sign language learning system
100: Learning data generation device
200: Sign language translation device
300: Sign language video recording device
400: worker terminal

Claims (6)

In the method of operating a learning data generation device that generates learning data for artificial intelligence-based sign language translation,
Receiving first text data consisting of Korean sentences based on a user input signal;
requesting a sign language translation device to translate the first text data into second text data composed of a sign language sentence formed in a grammatical system different from the Korean sentence;
Upon receiving the second text data from the sign language translation device, obtaining one or more semantic elements from the second text data;
Obtaining the sign language image data for the first text data by requesting the sign language image data for the first text data from a photographing device;
generating learning data based on the one or more semantic elements and the sign language image data; and
Including the step of storing the learning data,
The steps for generating the learning data are:
Transmitting the one or more semantic elements and the sign language image data to a worker terminal;
automatically arranging the one or more semantic elements and the sign language image data over time by providing a labeling authoring tool;
Receiving mapping information in which the one or more semantic elements are rearranged in the sign language image data based on a timeline from the worker terminal through the labeling authoring tool; and
A method of operating a learning data generating device further comprising generating token information based on the mapping information. According to claim 1,
The obtaining of the one or more semantic elements further includes the step of separating the second text data into the one or more semantic elements that are the minimum semantic units of a sign language through morphological analysis of the second text data. How the device works. According to claim 1,
The steps for acquiring the sign language image data are:
A method of operating a learning data generating device further comprising receiving the sign language image data captured based on the first text data and the second text data from a photographing device including a camera. delete According to claim 1,
The step of storing the learning data further includes the step of creating a learning database based on the learning data in which the input is the first text data and the correct answer is a sequence of the one or more semantic elements when the accuracy of the token information is greater than the reference value. A method of operating a learning data generation device including: An interface configured to transmit and receive data with external devices;
A sensor configured to acquire sign language image data; and
Includes a processor,
The processor:
Receive first text data consisting of Korean sentences based on the user input signal,
Requesting a sign language translation device among the external devices to translate the first text data into second text data consisting of a sign language sentence formed in a grammar system different from the Korean sentence,
Upon receiving the second text data from the sign language translation device, obtain one or more semantic elements from the second text data,
Requesting the sign language image data for the first text data from the sensor and receiving the sign language image data,
Generate learning data based on the one or more semantic elements and the sign language image data, and
It is configured to create a database by storing the learning data,
The processor:
Transmitting the one or more semantic elements and the sign language image data to the worker terminal,
By providing a labeling authoring tool, automatically arranging the one or more semantic elements and the sign language image data over time,
Receiving mapping information in which the one or more semantic elements are rearranged based on a timeline in the sign language image data from the worker terminal through the labeling authoring tool, and
A learning data generating device further configured to generate token information based on the mapping information.