KR20240048343A - Electronic apparatus and controlling method thereof - Google Patents

Abstract

An electronic device is disclosed. The electronic device is connected to a memory that stores a microphone, a communication interface, identification information for a plurality of translation support devices, an encoder corresponding to the first language, a decoder corresponding to the first language, and at least one command, and the memory is connected to the memory. It includes at least one processor that controls the electronic device, wherein the at least one processor identifies a translation support device located within a preset distance among the plurality of translation support devices by executing at least one command, and transmits the first signal through a microphone. Obtaining a user voice of a language, obtaining a first text of the first language corresponding to the user voice of the first language, inputting the first text into an encoder corresponding to the first language, and generating a first feature vector of the first text Obtain and transmit the first feature vector to the identified translation support device through a communication interface, and each of the plurality of translation support devices supports translation corresponding to each domain.

Description

Electronic device and its control method { ELECTRONIC APPARATUS AND CONTROLLING METHOD THEREOF }

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that performs domain-specific translation and a control method thereof.

The performance of a general domain translation model is lower than the commercialization level compared to a domain-specific translation model. Accordingly, domain-specific translation models are widely used.

However, the translation knowledge of a domain-specific translation model is optimized for that domain. Therefore, the translation support device must store a plurality of translation models specialized for each of the plurality of domains in order to support translation specialized for the plurality of domains.

An electronic device according to an embodiment of the present disclosure includes a microphone; communication interface; a memory storing identification information for a plurality of translation support devices, an encoder corresponding to a first language, a decoder corresponding to the first language, and at least one command; and at least one processor connected to the memory and controlling the electronic device, wherein the at least one processor supports translation support located within a preset distance among the plurality of translation support devices by executing the at least one command. Identify a device, obtain a user voice in the first language through the microphone, obtain first text in the first language corresponding to the user voice in the first language, and transmit the first text to the first language. Obtain a first feature vector of the first text by inputting it to a corresponding encoder, and transmit the first feature vector to the identified translation support device through the communication interface, and each of the plurality of translation support devices Supports translation corresponding to the domain.

A method of controlling an electronic device according to an embodiment of the present disclosure includes identifying a translation support device located within a preset distance among a plurality of translation support devices; Obtaining user speech in a first language; Obtaining first text in the first language corresponding to the user's voice in the first language; Inputting the first text into an encoder corresponding to the first language to obtain a first feature vector of the first text: and transmitting the first feature vector to the identified translation support device. And, each of the plurality of translation support devices supports translation corresponding to each domain.

In a non-transitory computer-readable recording medium including a program for executing a control method of an electronic device according to an embodiment of the present disclosure, the control method includes a translation located within a preset distance among a plurality of translation support devices. identifying a support device; Obtaining user speech in a first language; Obtaining first text in the first language corresponding to the user's voice in the first language; Inputting the first text into an encoder corresponding to the first language to obtain a first feature vector of the first text: and transmitting the first feature vector to the identified translation support device. And, each of the plurality of translation support devices supports translation corresponding to each domain.

1 is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure.
Figure 2 is a diagram for explaining the configuration of a translation support device according to an embodiment of the present disclosure.
FIG. 3 is a diagram for explaining the configuration of another electronic device according to an embodiment of the present disclosure.
Figure 4 is a diagram for explaining a learning method of an encoder and decoder according to an embodiment of the present disclosure.
FIG. 5 is a diagram for explaining the operation of an electronic device according to an embodiment of the present disclosure.

Since these embodiments can be modified in various ways and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted.

In addition, the following examples may be modified into various other forms, and the scope of the technical idea of the present disclosure is not limited to the following examples. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to completely convey the technical idea of the present disclosure to those skilled in the art.

The terms used in this disclosure are merely used to describe specific embodiments and are not intended to limit the scope of rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present disclosure, expressions such as “have,” “may have,” “includes,” or “may include” refer to the presence of the corresponding feature (e.g., component such as numerical value, function, operation, or part). , and does not rule out the existence of additional features.

In the present disclosure, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in the present disclosure can modify various components regardless of order and/or importance, and can refer to one component. It is only used to distinguish from other components and does not limit the components.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component).

On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

The expression “configured to” used in the present disclosure may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or determined) to” may not necessarily mean “specifically designed to” in terms of hardware.

Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

In an embodiment, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented with at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Meanwhile, various elements and areas in the drawing are schematically drawn. Accordingly, the technical idea of the present invention is not limited by the relative sizes or spacing drawn in the attached drawings.

Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that those skilled in the art can easily implement them.

FIG. 1 is a block diagram for explaining the configuration of an electronic device 100 according to an embodiment of the present disclosure.

Electronic device 100 may include memory 110, communication interface 120, user interface 130, microphone 140, speaker 150, display 160, sensor 170, and processor 180. You can. The electronic device 100 may omit some of the above components and may further include other components.

Additionally, the electronic device 100 may be implemented as a smartphone, but this is only an example, and may include a tablet PC, PC, server, smart TV, mobile phone, personal digital assistant (PDA), earphone, laptop, media player, It can be implemented in various forms such as e-book terminals, digital broadcasting terminals, navigation, kiosks, MP3 players, digital cameras, wearable devices, home appliances, and other mobile or non-mobile computing devices.

The memory 110 may store at least one instruction related to the electronic device 100. The memory 110 may store an operating system (O/S) for driving the electronic device 100. Additionally, the memory 110 may store various software programs or applications for operating the electronic device 100 according to various embodiments of the present disclosure. Additionally, the memory 110 may include a semiconductor memory such as flash memory or a magnetic storage medium such as a hard disk.

Specifically, the memory 110 may store various software modules for operating the electronic device 100 according to various embodiments of the present disclosure, and the processor 180 may execute various software modules stored in the memory 110. The operation of the electronic device 100 can be controlled. That is, the memory 110 is accessed by the processor 180, and data read/write/modify/delete/update, etc. can be performed by the processor 180.

Meanwhile, in the present disclosure, the term memory 110 refers to the memory 110, a ROM (not shown) within the processor 180, a RAM (not shown), or a memory card (not shown) mounted on the electronic device 100 (e.g. For example, it can be used to mean including micro SD card, memory stick).

In particular, the memory 110 may store information about an artificial intelligence model including a plurality of layers. Here, storing information about the artificial intelligence model means various information related to the operation of the artificial intelligence model, such as information about a plurality of layers included in the artificial intelligence model, parameters used in each of the plurality of layers (e.g. , filter coefficients, bias, etc.) may be stored.

Specifically, the memory 110 may store an encoder 111 corresponding to the first language and a decoder 112 corresponding to the first language. In this case, the encoder 111 corresponding to the first language and the first The decoder 112 corresponding to the language may be implemented as a neural network model. For example, the first language may be Korean.

The encoder 111 corresponding to the first language may be a model learned using the first text of the first language as input data and the feature vector in the vector space as output data. That is, the encoder 111 corresponding to the first language may be a neural network trained to output a feature vector of the text of the first language when the text of the first language is input.

The decoder 112 corresponding to the first language may be a model learned using a feature vector in a vector space as input data and a first text of the first language as output data. That is, the decoder 112 corresponding to the first language may be a neural network that has been trained to output text in the first language when a feature vector is input.

The communication interface 120 includes circuitry and is capable of communicating with external devices and servers. The communication interface 120 may communicate with an external device or server based on a wired or wireless communication method. In this case, the communication interface 120 may include a Bluetooth module (not shown), a Wi-Fi module (not shown), an infrared (IR) module, a local area network (LAN) module, an Ethernet module, etc. Here, each communication module may be implemented in the form of at least one hardware chip. In addition to the above-described communication methods, the wireless communication module includes zigbee, USB (Universal Serial Bus), MIPI CSI (Mobile Industry Processor Interface Camera Serial Interface), 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), and LTE (Long It may include at least one communication chip that performs communication according to various wireless communication standards such as Term Evolution), LTE Advanced (LTE-A), 4th Generation (4G), and 5th Generation (5G). However, this is only an example, and the communication interface 120 may use at least one communication module among various communication modules. The communication interface 120 can communicate with the translation support device 200 and other electronic devices 300 and transmit and receive data.

The user interface 130 is configured to receive user commands for controlling the electronic device 100. It can be implemented with devices such as buttons, touch pads, mice, and keyboards, or it can be implemented with a touch screen that can also perform display functions and operation input functions. Here, the button may be various types of buttons such as mechanical buttons, touch pads, wheels, etc. formed on any area of the exterior of the main body of the electronic device 100, such as the front, side, or back. The electronic device 100 may obtain various user inputs through the user interface 130.

The microphone 140 may be formed integrally with the electronic device 100, such as on the top, front, or side surfaces. The microphone 140 includes a microphone that collects user voice in analog form, an amplifier circuit that amplifies the collected user voice, an A/D conversion circuit that samples the amplified user voice and converts it into a digital signal, and noise components from the converted digital signal. It may include various configurations such as a filter circuit to remove . The microphone 140 can acquire user voice input.

The speaker 150 can output an audio signal. For example, the speaker 150 may output an analog signal by converting an audio signal into a physical vibration signal that can be audibly perceived by the user.

The display 160 may be implemented as a display including a self-emitting device or a display including a non-emitting device and a backlight. For example, Liquid Crystal Display (LCD), Organic Light Emitting Diodes (OLED) display, Light Emitting Diodes (LED), micro LED, Mini LED, Plasma Display Panel (PDP), and Quantum dot (QD) display. , QLED (Quantum dot light-emitting diodes), etc. can be implemented in various types of displays. The display 160 may also include a driving circuit and a backlight unit that may be implemented in the form of a-si TFT, low temperature poly silicon (LTPS) TFT, or organic TFT (OTFT).

The sensor 170 can detect the electronic device 100 and the environment surrounding the electronic device 100. For example, the sensor 170 may be a GPS sensor, gesture sensor, gyro sensor, barometric pressure sensor, magnetic sensor, acceleration sensor, grip sensor, proximity sensor, color sensor, IR (infrared) sensor, temperature sensor, humidity sensor, or illuminance sensor. It may include at least one of the sensors. For example, a GPS sensor may obtain location information of the electronic device 100.

The processor 180 may control the overall operation and functions of the electronic device 100. Specifically, the processor 180 is connected to the configuration of the electronic device 100 including the memory 110, and executes at least one instruction stored in the memory 110 as described above, thereby controlling the electronic device 100. You can control the overall movement.

Processor 180 may be implemented in various ways. For example, the processor 180 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor. Processor, DSP). Meanwhile, in the present disclosure, the term processor 180 may be used to include a central processing unit (CPU), a graphics processing unit (GPU), and a main processing unit (MPU).

Operations of the processor 180 for implementing various embodiments of the present disclosure may be implemented through a plurality of modules.

Specifically, data for a plurality of modules according to the present disclosure may be stored in the memory 110, and the processor 180 accesses the memory 110 to store data for the plurality of modules in the memory inside the processor 180. Alternatively, after loading into the buffer, various embodiments according to the present disclosure can be implemented using a plurality of modules.

However, at least one of the plurality of modules according to the present disclosure may be implemented as hardware and may be included in the processor 180 in the form of a system on chip.

Figure 2 is a diagram for explaining the configuration of a translation support device according to an embodiment of the present disclosure.

Specifically, the translation support device 200 may include a memory 210, a communication interface 220, a sensor 230, and a processor 240. The translation support device 200 may omit some of the above components and may further include other components. At this time, the translation support device 200 may further include at least a portion of the components included in the electronic device 100, and at this time, at least some of the operations of the components included in the translation support device 200 may be performed on the electronic device 100. It may be the same as the operation of the configuration included in (100).

In addition, the translation support device 200 may be implemented as a server, but this is only an example, and may be used in smartphones, tablet PCs, PCs, servers, smart TVs, mobile phones, personal digital assistants (PDAs), earphones, laptops, It can be implemented in various forms such as media players, e-book terminals, digital broadcasting terminals, navigation, kiosks, MP3 players, digital cameras, wearable devices, home appliances, and other mobile or non-mobile computing devices.

The memory 210 may store at least one instruction related to the translation support device 200. The memory 210 may store an operating system (O/S) for driving the translation support device 200. Additionally, the memory 210 may store various software programs or applications for the translation support device 200 to operate according to various embodiments of the present disclosure. Additionally, the memory 210 may include a semiconductor memory such as flash memory or a magnetic storage medium such as a hard disk.

Specifically, the memory 210 may store various software modules for the translation support device 200 to operate according to various embodiments of the present disclosure, and the processor 240 may execute various software modules stored in the memory 210. Thus, the operation of the translation support device 200 can be controlled. That is, the memory 210 is accessed by the processor 240, and data read/write/modify/delete/update, etc. can be performed by the processor 240.

Meanwhile, in the present disclosure, the term memory 210 refers to memory 210, ROM (not shown), RAM (not shown) in the processor 240, or a memory card (not shown) mounted in the translation support device 200 ( For example, it can be used to mean including micro SD card, memory stick).

In particular, the memory 210 may store information about an artificial intelligence model including a plurality of layers. Here, storing information about the artificial intelligence model means various information related to the operation of the artificial intelligence model, such as information about a plurality of layers included in the artificial intelligence model, parameters used in each of the plurality of layers (e.g. , filter coefficients, bias, etc.) may be stored.

Specifically, the memory 210 may store an encoder 211 corresponding to the first domain and a decoder 212 corresponding to the first domain.

At this time, the encoder 211 corresponding to the first domain uses a feature vector obtained from text in a specific language (e.g., the first language or the second language) as input data, and the feature vector corresponding to the specific language and the first domain. It may be a model learned using as output data. Here, the feature vector corresponding to the specific language and the first domain may be a feature vector that reflects the characteristics of the specific language and the characteristics of the first domain. That is, the encoder 211 corresponding to the first domain may be a neural network trained to output a feature vector reflecting the characteristics of the first language and the characteristics of the first domain when a feature vector obtained from a text of a specific language is input. there is.

That is, the decoder 212 corresponding to the first domain uses the first language output from the encoder 211 corresponding to the first domain and the feature vector corresponding to the first domain as input data, and It may be a model learned using feature vectors as output data. Here, the feature vector corresponding to the first domain may be a feature vector that excludes the characteristics of a specific language and reflects the characteristics of the first domain. That is, the decoder 212 corresponding to the first domain may be a neural network that has been trained to output a feature vector corresponding to the first domain when a specific language and a feature vector corresponding to the first domain are input.

The communication interface 220 includes circuitry and is configured to communicate with external devices and servers. The communication interface 220 may communicate with an external device or server based on a wired or wireless communication method. In this case, the communication interface 220 may include a Bluetooth module (not shown), a Wi-Fi module (not shown), an infrared (IR) module, a local area network (LAN) module, an Ethernet module, etc. Here, each communication module may be implemented in the form of at least one hardware chip. In addition to the above-described communication methods, the wireless communication module includes zigbee, USB (Universal Serial Bus), MIPI CSI (Mobile Industry Processor Interface Camera Serial Interface), 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), and LTE (Long It may include at least one communication chip that performs communication according to various wireless communication standards such as Term Evolution), LTE Advanced (LTE-A), 4th Generation (4G), and 5th Generation (5G). However, this is only an example, and the communication interface 220 may use at least one communication module among various communication modules. The communication interface 220 can communicate with the electronic device 100 and the external terminal device 300 and transmit and receive data.

The sensor 230 can detect the translation support device 200 and the environment surrounding the translation support device 200. For example, the sensor 230 may be a GPS sensor, gesture sensor, gyro sensor, barometric pressure sensor, magnetic sensor, acceleration sensor, grip sensor, proximity sensor, color sensor, IR (infrared) sensor, temperature sensor, humidity sensor, or illuminance sensor. It may include at least one of the sensors. For example, a GPS sensor can obtain location information of a translation support device.

The processor 240 may control the overall operation and functions of the translation support device 200. Specifically, the processor 240 is connected to the configuration of the translation support device 200 including the memory 210, and executes at least one command stored in the memory 210 as described above, thereby ) can be controlled overall.

Processor 240 may be implemented in various ways. For example, the processor 240 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor. Processor, DSP). Meanwhile, in the present disclosure, the term processor 240 may be used to include a central processing unit (CPU), a graphics processing unit (GPU), and a main processing unit (MPU).

Operations of the processor 240 for implementing various embodiments of the present disclosure may be implemented through a plurality of modules.

Specifically, data for a plurality of modules according to the present disclosure may be stored in the memory 210, and the processor 240 accesses the memory 210 to store data for the plurality of modules in the memory inside the processor 240. Alternatively, after loading into the buffer, various embodiments according to the present disclosure can be implemented using a plurality of modules.

However, at least one of the plurality of modules according to the present disclosure may be implemented as hardware and may be included in the processor 240 in the form of a system on chip.

FIG. 3 is a diagram for explaining the configuration of another electronic device 300 according to an embodiment of the present disclosure.

Other electronic devices 300 include memory 310, communication interface 320, user interface 330, microphone 340, speaker 350, display 360, sensor 370, and processor 380. can do. The electronic device 300 may omit some of the above components and may further include other components. At this time, at least some of the operations of the components included in the other electronic device 300 may be the same as the operations of the components included in the electronic device 100.

In particular, the memory 110 may store information about an artificial intelligence model including a plurality of layers. Here, storing information about the artificial intelligence model means various information related to the operation of the artificial intelligence model, such as information about a plurality of layers included in the artificial intelligence model, parameters used in each of the plurality of layers (e.g. , filter coefficients, bias, etc.) may be stored.

Specifically, the memory 310 may store an encoder 311 corresponding to the second language and a decoder 312 corresponding to the second language. In this case, the encoder 311 corresponding to the second language and the second The decoder 312 corresponding to the language may be implemented as a neural network model.

The encoder 311 corresponding to the second language may be a model learned using text of the second language as input data and feature vectors in a vector space as output data. That is, the encoder 311 corresponding to the second language may be a neural network trained to output feature vectors of the text in the second language when text in the second language is input.

The decoder 312 corresponding to the second language may be a model learned using feature vectors in a vector space as input data and text of the second language as output data. That is, the decoder 312 corresponding to the second language may be a neural network that has been trained to output text in the second language when a feature vector is input.

Additionally, the communication interface 320 can communicate with the electronic device 100 and the translation support device 200 and transmit and receive data.

Figure 4 shows an encoder 111 corresponding to the first language, a decoder 112 corresponding to the first language, an encoder 211 corresponding to the first domain, and an encoder corresponding to the first domain according to an embodiment of the present disclosure. This diagram is for explaining how the decoder 212, the encoder 311 corresponding to the second language, and the decoder 312 corresponding to the second language are learned.

As shown in FIG. 4A, each of the plurality of translation support devices may include an encoder and a decoder learned based on the domain-specific corpora 400, 410, and 420.

At this time, the first domain corpus 400 may include a text 401 in the first language and a text 402 in the second language. At this time, the second language text 402 may be a text in which the first language text 401 is translated into the second language to suit the first domain. Additionally, the text 401 in the first language may be a text in which the text 402 in the second language is translated into the first language to suit the first domain.

Likewise, the second domain corpus 410 may include text 411 in the first language and text 412 in the second language. At this time, the second language text 412 may be a text in which the first language text 411 is translated into the second language to suit the second domain. Additionally, the first language text 411 may be a text in which the second language text 412 is translated into the first language to suit the second domain.

Additionally, the third domain corpus 420 may include text 421 in the first language and text 422 in the second language. At this time, the second language text 422 may be a text in which the first language text 421 is translated into the second language to suit the third domain. Additionally, the first language text 421 may be a text in which the second language text 422 is translated into the first language to suit the third domain.

At this time, an encoder 111 corresponding to the first language, a decoder 112 corresponding to the first language, an encoder 211 corresponding to the first domain, a decoder 212 corresponding to the first domain, and a second language The encoder 311 corresponding to and the decoder 312 corresponding to the second language may be a neural network learned using the corpus 400 corresponding to the first domain as learning data.

At this time, the corpus corresponding to the first domain may include a text in the first language and a text in the second language, which is a text translated into a second language so that the text in the first language is suitable for the first domain.

Specifically, as shown in Figure 4b, an encoder 111 corresponding to the first language, an encoder 211 corresponding to the first domain, a decoder 212 corresponding to the first domain, and a decoder 212 corresponding to the second language. When the text of the first language included in the corpus corresponding to the first domain is input to the encoder 111 corresponding to the first language, the decoder 312 outputs the text from the decoder 312 corresponding to the second language. It may be learned so that the difference between the value and the text of the second language included in the corpus corresponding to the first domain is minimal.

At this time, the text 401 of the first language may be input to the encoder 111 corresponding to the first language. And, the value output by the encoder 111 corresponding to the first language may be input to the encoder 211 corresponding to the first domain. And, the value output by the encoder 211 corresponding to the first domain may be input to the decoder 212 corresponding to the first domain. And, the value output by the decoder 212 corresponding to the first domain may be input to the decoder 311 corresponding to the second language. And, the text 403 output by the decoder 311 corresponding to the second language can be obtained. At this time, the encoder 111 corresponding to the first language, the encoder 211 corresponding to the first domain, the decoder 212 corresponding to the first domain, and the decoder 312 corresponding to the second language, It may be a neural network trained to minimize the difference between the text 403 output by the decoder 311 corresponding to the language and the text 402 of the second language included in the corpus 400 corresponding to the first domain. there is.

Additionally, as shown in FIG. 4C, an encoder 311 corresponding to the second language, an encoder 211 corresponding to the first domain, a decoder 212 corresponding to the first domain, and a decoder corresponding to the first language (112) When the text 402 of the second language included in the corpus 400 corresponding to the first domain is input to the encoder 311 corresponding to the second language, the decoder ( The text 403 output by 112) can be learned to minimize the difference between the text of the second language included in the corpus 400 corresponding to the first domain.

At this time, text in the first language may be input to the encoder 311 corresponding to the second language. And, the value output by the encoder 311 corresponding to the second language may be input to the encoder 211 corresponding to the first domain. And, the value output by the encoder 211 corresponding to the first domain may be input to the decoder 212 corresponding to the first domain. And, the value output by the decoder 212 corresponding to the first domain may be input to the decoder 212 corresponding to the first language. And, the encoder 311 corresponding to the second language, the encoder 211 corresponding to the first domain, the decoder 212 corresponding to the first domain, and the decoder 112 corresponding to the first language are It may be a neural network that uses the difference between the value output by the corresponding decoder 212 and the text of the second language included in the training data as a loss function, and is trained so that the value of the loss function is minimized.

FIG. 5 is a diagram for explaining the operation of an electronic device according to an embodiment of the present disclosure.

The processor 180 may identify the translation support device 200 located within a preset distance from the electronic device 100 among the plurality of translation support devices 200, 201, and 202. At this time, the memory 110 may store identification information for a plurality of translation support devices 200, 201, and 202. Based on the identification information about the plurality of translation support devices stored in the memory 110, the processor 180 may identify a translation support device located within a preset distance from the electronic device 100 among the plurality of translation support devices.

Here, the processor 180 determines whether each of the plurality of translation support devices 200, 201, and 202 is located within a preset distance based on the signal connection strength with each of the plurality of translation support devices 200, 201, and 202. can be identified.

Alternatively, the processor 180 may identify whether it is located within a preset distance based on whether it is connected to each of the plurality of translation support devices 200, 201, and 202 by short-distance communication means. For example, when connected to the first translation support device 200 among the plurality of translation support devices 200, 201, and 202 through short-distance communication means, the processor 180 causes the first translation support device 200 to It can be identified by being located within a distance. Here, the short-distance communication means may mean a communication means using Bluetooth, Wi-Fi Direct, NFC, etc., but is not limited thereto, and the electronic device 100 and the translation support device 200 can perform direct communication. It can refer to various means of communication.

And, among the plurality of translation support devices 200, 201, and 202, when two or more translation support devices are located within a preset distance from the electronic device 100, the processor 180 can identify the closest translation support device. there is.

Meanwhile, each of the plurality of translation support devices 200, 201, and 202 may be a translation support device specialized for each domain. Specifically, the first translation support device 200 may be a translation support device specialized for the first domain. And, the second translation support device 201 may be a translation support device specialized for the second domain. And, the third translation support device 202 may be a translation support device specialized for a third domain.

At this time, the first translation support device 200 is a translation support device including an encoder 211 and a decoder 212 learned with a corpus corresponding to the first domain, and the second translation support device 201 is a second translation support device. A translation support device includes an encoder 211b and a decoder 212b learned with a corpus corresponding to a domain, and the third translation support device 202 includes an encoder 211c and a decoder learned with a corpus corresponding to a third domain. It may be a translation support device including.

Additionally, the translation support device 200 located within a preset distance from the electronic device 100 may be a translation support device specialized for a domain corresponding to the location where the electronic device 100 is located. For example, if the domain corresponding to the location where the electronic device 100 is located is the first domain, the processor 180 may identify the first translation support device 200 specialized for the first domain.

For example, if the location of the electronic device 100 is a shopping center, the identified translation support device 200 may be a translation support device specialized for the shopping domain. Alternatively, if the location of the electronic device 100 is an airport, the identified translation support device 200 may be a translation support device specialized for the airport domain. Alternatively, if the location of the electronic device 100 is a restaurant, the identified translation support device 200 may be a translation support device specialized for the restaurant domain.

Additionally, the processor 180 may identify the other electronic device 300 among at least one other electronic device.

At this time, the processor 180 may display a UI for selecting one of at least one other electronic device on the display 160. At this time, at least one other electronic device may be a device connected to the electronic device 100 using short-distance communication means.

Additionally, the processor 180 may obtain a user input for selecting one of a plurality of other electronic devices. At this time, the processor 180 may identify the selected device among the plurality of other electronic devices as the other electronic device 300.

Alternatively, the processor 180 may identify a device located in the direction pointed by the electronic device 100 among at least one other electronic device as the other electronic device 300.

Alternatively, the processor 180 may identify the device located closest to the electronic device 100 among at least one other electronic device as the other electronic device 300.

When the translation support device 200 and the other electronic device 300 are identified, the processor 180 transmits the identification information of the translation support device 200 to the other electronic device 300 and identifies the other electronic device 300. Information may be transmitted to the translation support device 200.

Based on the transmitted identification information, the electronic device 100, the translation support device 200, and another electronic device 300 may perform communication and transmit and receive data.

Additionally, the processor 180 may acquire the user's voice in the first language from the first user 10 through the microphone 140. When the user voice of the first language is acquired, the processor 180 may input the acquired user voice of the first language into a Speech to Text (STT) model to obtain the first text of the first language.

Then, the processor 180 may input the first text of the first language into the encoder 211 corresponding to the first language, and obtain a first feature vector corresponding to the first text of the first language. Subsequently, the electronic device 100 may transmit the first feature vector to the translation support device 200.

When the first feature vector is received, the translation support device 200 may obtain a feature vector reflecting the features of the first domain from the first feature vector.

Specifically, the translation support device 200 may obtain a second feature vector by inputting the first feature vector into the encoder 211 corresponding to the first domain.

Additionally, the translation support device 200 may obtain a third feature vector by inputting the second feature vector into the decoder 212 corresponding to the first domain.

When the third feature vector is obtained, the translation support device 200 may transmit the third feature vector to the other electronic device 300.

When the third feature vector is received, the other electronic device 300 may input the third feature vector into the decoder 312 corresponding to the second language to obtain the first text of the second language. At this time, the text in the second language may be a text in which the text in the first language has been translated into the second language.

When the first text in the second language is obtained, the other electronic device 300 may output the first text in the second language. At this time, the other electronic device 300 may display the first text in the second language on the display of the other electronic device 300 or output it through the speaker of the other electronic device 300.

When the first text in the second language is output, the second user 30 can utter the user's voice in the second language. At this time, another electronic device 300 may acquire the user's voice in the second language. Here, the user's voice in the second language may be a response to the user's voice in the first language.

When the user's voice in the second language is acquired, the other electronic device 300 converts the user's voice in the second language into a second text in the second language, and then converts the second text in the second language into a text corresponding to the first domain. It can be input to the encoder 311. At this time, the other electronic device 300 may input the second text of the second language into the encoder corresponding to the first domain and obtain the fourth feature vector.

When the fourth feature vector is acquired, the other electronic device 300 may transmit the fourth feature vector to the translation support device 200.

When the fourth feature vector is received, the translation support device 200 may obtain a feature vector reflecting the features of the first domain from the fourth feature vector.

At this time, the method for the translation support device 200 to obtain a feature vector reflecting the features of the first domain from the fourth feature vector is to obtain a feature vector reflecting the features of the domain from the first feature vector. The method of acquisition may be the same.

Specifically, the translation support device 200 may obtain the fifth feature vector by inputting the fourth feature vector into the encoder 211 corresponding to the first domain.

Additionally, the translation support device 200 may obtain a sixth feature vector by inputting the fifth feature vector into the decoder 212 corresponding to the first domain.

When the sixth feature vector is acquired, the translation support device 200 may transmit the sixth feature vector to the electronic device 100.

When the sixth feature vector is received, the electronic device 100 may input the sixth feature vector into the decoder 112 corresponding to the first language to obtain the second text of the first language. At this time, the second text in the first language may be a text in which the user's voice in the second language is translated into the first language.

When the text in the first language is obtained, the electronic device 100 can output the text in the first language. At this time, the electronic device 100 may display text in the first language through the display 160 or output text in the first language through the speaker 150.

Conventionally, servers stored a plurality of domain-specific encoders and decoders to perform domain-specific translation. At this time, the user terminal device transmitted information about the user's voice for translation to the server. Then, the server used the acquired information about the user's voice to determine the user's intention, identified the translation domain, and performed translation using the encoder and decoder corresponding to the identified domain. Accordingly, there was a problem that information about the user's voice was transmitted to the server and the user's personal information could be exposed.

Additionally, there was a problem in that the server had to store encoders and decoders corresponding to each of a plurality of domains in order to translate multiple domains. Additionally, if the server incorrectly identifies the translation domain in the process of identifying the user's intent and identifying the translation domain, there was a problem of degraded translation performance.

Additionally, in the related art, in order to translate a user's voice from a first language to a second language, there was a problem in that the user terminal device had to store an encoder and decoder corresponding to the first language and an encoder and decoder corresponding to the second language. In other words, there was a problem that the number of encoders and decoders stored in the user terminal device increased as much as the number of target languages for translation.

The present disclosure was created to solve the above-described conventional problem. The translation method according to an embodiment of the present disclosure is to identify the translation domain by storing only encoders and decoders corresponding to a specific domain in a plurality of translation support devices. This process can be omitted.

In addition, because each of the plurality of translation support devices stores an encoder and a decoder corresponding to a domain according to the location of each of the plurality of translation support devices, the electronic device 100 is stored in a domain according to the location of the electronic device 100. Since translation is performed while communicating with a translation support device that stores the corresponding encoder and decoder, the problem of incorrectly identifying the translation domain and deteriorating translation performance can be solved.

In addition, the electronic device 100 of the present disclosure can translate the user's voice in the first language into text in the second language using only the encoder 111 corresponding to the first language and the decoder 112 corresponding to the first language. You can. Accordingly, the electronic device 100 of the present disclosure can perform translation using fewer resources than a conventional user terminal device for translation.

In addition, the electronic device 100 of the present disclosure transmits only feature vectors obtained from the user's voice to the translation support device 200, which has the effect of protecting the user's personal information.

Functions related to artificial intelligence according to the present disclosure are operated through the processor 180 and memory 110 of the electronic device 100.

The processor 180 may be comprised of one or multiple processors 180. At this time, the one or more processors 180 may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a Neural Processing Unit (NPU), but are not limited to the example of the processor 180 described above. No.

The CPU is a general-purpose processor 180 that can perform not only general calculations but also artificial intelligence calculations, and can efficiently execute complex programs through a multi-layer cache structure. CPUs are advantageous for serial processing, which allows organic connection between previous and next calculation results through sequential calculations. The general-purpose processor 180 is not limited to the above-described example, except when specified as the above-described CPU.

GPU is a processor 180 for large-scale operations such as floating-point operations used in graphics processing, and can perform large-scale operations in parallel by integrating a large number of cores. In particular, GPUs may be more advantageous than CPUs in parallel processing methods such as convolution operations. Additionally, the GPU can be used as a co-processor 180 to supplement the functions of the CPU. The processor 180 for mass computation is not limited to the above-described example, except for the case where it is specified as the GPU.

The NPU is a processor 180 specialized in artificial intelligence calculations using an artificial neural network, and each layer that makes up the artificial neural network can be implemented with hardware (e.g., silicon). At this time, the NPU is designed specifically according to the company's requirements, so it has a lower degree of freedom than a CPU or GPU, but can efficiently process artificial intelligence calculations requested by the company. Meanwhile, as a processor 180 specialized in artificial intelligence calculations, the NPU can be implemented in various forms such as a TPU (Tensor Processing Unit), IPU (Intelligence Processing Unit), and VPU (Vision processing unit). The artificial intelligence processor 180 is not limited to the above-described example, except for the case specified as the above-described NPU.

Additionally, one or more processors 180 may be implemented as a System on Chip (SoC). At this time, in addition to one or more processors 180, the SoC may further include a memory 110 and a network interface such as a bus for data communication between the processor 180 and the memory 110.

If the SoC (System on Chip) included in the electronic device includes a plurality of processors 180, the electronic device uses some processors 180 of the plurality of processors 180 to perform artificial intelligence-related operations (e.g. , operations related to learning or inference of artificial intelligence models) can be performed. For example, the electronic device performs artificial intelligence-related operations using at least one of the plurality of processors 180, a GPU, NPU, VPU, TPU, or hardware accelerator specialized for artificial intelligence operations such as convolution operation, matrix multiplication operation, etc. It can be done. However, this is only an example, and it goes without saying that calculations related to artificial intelligence can be processed using the general-purpose processor 180, such as a CPU.

Additionally, the electronic device may perform calculations on functions related to artificial intelligence using multiple cores (eg, dual core, quad core, etc.) included in one processor 180. In particular, the electronic device can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel using the multi-cores included in the processor 180.

One or more processors 180 control input data to be processed according to predefined operation rules or artificial intelligence models stored in the memory 110. Predefined operation rules or artificial intelligence models are characterized by being created through learning.

Here, being created through learning means that a predefined operation rule or artificial intelligence model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server/system.

An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value, and the operation of the layer is performed using the operation result of the previous layer and at least one defined operation. Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), and Deep Neural Network (BRDNN). There are Q-Networks (Deep Q-Networks) and Transformer, and the neural network in this disclosure is not limited to the above-described examples except where specified.

A learning algorithm is a method of training a target device (eg, a robot) using a large number of learning data so that the target device can make decisions or make predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is specified. Except, it is not limited to the examples described above.

Meanwhile, the term “unit” or “module” used in the present disclosure includes a unit comprised of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. You can. A “part” or “module” may be an integrated part, a minimum unit that performs one or more functions, or a part thereof. For example, a module may be comprised of an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage media (e.g., a computer). The device is a device capable of calling instructions stored from a storage medium and operating according to the called instructions, and may include the electronic device 100 according to the disclosed embodiments. When the instruction is executed by a processor, the processor may perform the function corresponding to the instruction directly or using other components under the control of the processor. Instructions may contain code generated or executed by a compiler or interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium does not contain signals and is tangible, and does not distinguish whether the data is stored semi-permanently or temporarily in the storage medium.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed on a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or online through an application store (e.g. Play Store™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or created temporarily in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

Each component (e.g., module or program) according to various embodiments may be composed of a single or multiple entities, and some of the above-described sub-components may be omitted or other sub-components may be used. It may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity and perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added. It can be.

100: electronic device 110: memory
120: communication interface 130: user interface
140: Microphone 150: Speaker
160: display 170: sensor
180: processor

Claims (10)

In electronic devices,
mike;
communication interface;
a memory storing identification information for a plurality of translation support devices, an encoder corresponding to a first language, a decoder corresponding to the first language, and at least one command; and
At least one processor connected to the memory and controlling the electronic device,
The at least one processor executes the at least one instruction,
Identifying a translation support device located within a preset distance among the plurality of translation support devices,
Acquire the user's voice in the first language through the microphone,
Obtaining a first text in the first language corresponding to the user's voice in the first language,
Inputting the first text into an encoder corresponding to the first language to obtain a first feature vector of the first text,
transmitting the first feature vector to the identified translation support device through the communication interface,
Each of the plurality of translation support devices is an electronic device that supports translation corresponding to each domain. According to paragraph 1,
The at least one processor,
Receiving a second feature vector of a second text corresponding to a response voice in a second language to the user voice from the translation support device through the communication interface,
An electronic device that inputs the second feature vector into a decoder corresponding to the first language to obtain information about a third text in which the second text has been translated into the first language. According to paragraph 1,
The at least one processor,
An electronic device that identifies a device located within a critical distance from the electronic device among a plurality of translation support devices as a translation support device specialized for a domain corresponding to the user context. According to paragraph 1,
The at least one processor,
identify a user context based on the first text,
An electronic device that identifies a device corresponding to a user context among the plurality of translation support devices as the translation support device. According to paragraph 1,
The at least one processor,
Obtain user input for selecting one of at least one other electronic device identified using a short-range communication method,
An electronic device that receives a second feature vector for a response voice obtained from the selected other electronic device from the translation support device through the communication interface. According to clause 5,
The at least one processor,
An electronic device that transmits identification information about the selected external device to the translation support device. According to paragraph 1,
The electronic device is,
It further includes a display;
The at least one processor,
Controlling the display to display a UI that obtains a user input for selecting one of a plurality of external devices identified using a short-range communication method,
When a user input for selecting one of the plurality of external devices is obtained through the UI, identification information for the selected external device is transmitted to the translation support device,
An electronic device that receives a second feature vector for the response voice obtained from the selected external device. According to paragraph 1,
The at least one processor,
The translation support device is
An electronic device comprising an encoder corresponding to a first domain learned with domain data corresponding to the user context and a decoder corresponding to the first domain. In clause 7,
The translation support device is an electronic device that stores an encoder corresponding to the second language and a decoder corresponding to the second language. In a method of controlling an electronic device,
Identifying a translation support device located within a preset distance among a plurality of translation support devices;
Obtaining user speech in a first language;
Obtaining first text in the first language corresponding to the user's voice in the first language;
Inputting the first text into an encoder corresponding to the first language to obtain a first feature vector of the first text: and
Transmitting the first feature vector to the identified translation support device,
A control method wherein each of the plurality of translation support devices supports translation corresponding to each domain.

Images