WO2023197949A1

WO2023197949A1 - Chinese translation method and electronic device

Info

Publication number: WO2023197949A1
Application number: PCT/CN2023/086870
Authority: WO
Inventors: 谢雨晨; 常亚
Original assignee: 华为技术有限公司
Priority date: 2022-04-15
Filing date: 2023-04-07
Publication date: 2023-10-19
Also published as: CN116932706A

Abstract

Provided in the present application are a Chinese translation method and an electronic device. The method comprises: in response to an input of a user, an electronic device acquiring text information, wherein the text information comprises a keyword; the electronic device displaying a hand action corresponding to the text information; and the electronic device displaying a mouth action corresponding to the keyword. By means of the translation method and the electronic device provided in the present application, a mouth action compatible with the habit of a sign language user is added when text information is translated into sign language, thereby facilitating an improvement in the accuracy of language expressions when Chinese is translated into sign language, a reduction in misunderstanding of a translation result by the sign language user, an enhancement in exchange and communication with the sign language user, and an improvement in the application experience for users of an electronic device.

Description

Methods and electronic devices for Chinese translation

This application claims priority to the Chinese patent application filed with the China Patent Office on April 15, 2022, with the application number 202210396448.4 and the invention title "Method and Electronic Device for Chinese Translation", the entire content of which is incorporated into this application by reference. .

Technical field

The present application relates to the field of computers, specifically, to a Chinese translation method and electronic equipment.

Background technique

A sign language digital human can help sign language users understand language information through hand movements and/or oral movements.

When a digital signer makes hand movements and coordinates corresponding oral movements to express a certain sentence or a certain meaning, the oral movements sometimes do not serve the purpose of assisting the hand movements to help understanding, but may instead cause inaccuracies. Necessary misunderstanding. For example, when making hand movements in natural sign language and matching the oral movements of hearing-impaired people when speaking, since the hand movements and oral movements may not express the same word at the same time, this may cause misunderstandings.

Contents of the invention

This application provides a Chinese translation method that only adds oral movements to keywords when translating text information into sign language, which is beneficial to improving the accuracy of language expression when translating Chinese into sign language.

The first aspect provides a Chinese translation method, including: in response to user input, an electronic device obtains text information, the text information includes keywords; the electronic device displays the hand movements corresponding to the text information; the electronic device displays Oral movements corresponding to keywords.

In a possible implementation, the keyword is obtained by the electronic device based on the text information input by the user through one or more of the following methods: the content of the text information, the user's translation history information, or other users' identification of the text. Methods for determining keywords in information.

It should be noted that the keyword here may be one or more words contained in the text information or one or more words contained in the text information.

It should also be noted that the order of word expressions of sign language users when performing sign language movements may be different from the word order of natural spoken language. Here, the order of hand movements corresponding to the text information displayed on the electronic device may be based on the order of the sign language users. Habits are determined.

In order to only attach corresponding oral movements to keywords when typing sign language, this technical solution translates text information into sign language that better serves the expression habits of sign language users, which is conducive to improving the accuracy of the translation results of text information and is conducive to reducing the The probability of sign language users misunderstanding the translated sign language is conducive to enhancing mutual communication with sign language users.

Combined with the first aspect, in some implementations of the first aspect, the keyword is determined based on the language habits of the sign language user.

When typing in sign language, corresponding oral movements are attached to the keywords determined according to the language habits of the sign language users. This technical solution translates text information into sign language that better serves the expression habits of sign language users, which is conducive to improving the translation results of text information. The accuracy is conducive to reducing the probability of sign language users misunderstanding the translated sign language, and is conducive to enhancing mutual communication with sign language users.

With reference to the first aspect, in some implementations of the first aspect, the electronic device does not display oral movements corresponding to common words, the text information includes the common words, and the common words are different from the keywords.

In this technical solution, oral movements are not displayed for ordinary words that are not keywords, which is beneficial to reducing data transmission during the translation process, improving the efficiency of data transmission and processing during the translation process, and improving the application usage of electronic device users. experience.

With reference to the first aspect, in some implementations of the first aspect, the electronic device displays the mouth movement while displaying the hand movement corresponding to the keyword.

This technical solution displays the oral movements corresponding to the keywords while performing the hand movements corresponding to the keywords. The implementation of this technical scheme is conducive to ensuring the corresponding relationship between the hand movements and the oral movements, and is conducive to further improving the translation of text information. The accuracy of the results will help improve sign language users' understanding of the translated sign language.

Combined with the first aspect, in some implementations of the first aspect, the keyword is a proper noun.

The proper nouns may include one or more of the following words: names of people, places, institutions, works, and other proper nouns.

Attaching oral movements to proper nouns can help improve sign language users' understanding of difficult-to-understand proper nouns, and can help enhance mutual communication with sign language users.

In connection with the first aspect, in some implementations of the first aspect, before displaying the oral movements corresponding to the keywords, the electronic device displays the first vocabulary, the text information includes the first vocabulary, and the first vocabulary is recommended Words with oral movements attached; in response to the user's confirmation operation, the electronic device determines that the first word is a keyword.

This technical solution recommends keywords to electronic device users and adds mouth movements to the recommended keywords after the user confirms. The implementation of this technical solution is conducive to improving sign language learners' understanding of the use of sign language, is conducive to improving the application experience of electronic device users, and is conducive to improving the efficiency of sign language learners learning sign language.

In conjunction with the first aspect, in some implementations of the first aspect, before displaying the oral movements corresponding to the keywords, in response to the user's first input, the electronic device acquires a second vocabulary, the second vocabulary being additionally requested by the user. Vocabulary of oral movements; when the text information contains the second vocabulary, the electronic device determines the second vocabulary as the keyword; when the text information does not contain the second vocabulary, the electronic device displays update request information, The update request information is used to prompt that the text information does not contain the second vocabulary; in response to the user's second input, the electronic device obtains the updated second vocabulary.

This technical solution identifies words input by the user that request additional oral movements, and notifies the user of the recognition result of whether the text information contains the words that the user requests additional oral movements. The implementation of this technical solution is conducive to improving the efficiency of Chinese translation into sign language, improving the accuracy of text information translation, and improving the user's application experience.

In conjunction with the first aspect, in some implementations of the first aspect, the first vocabulary is determined based on the user's translation history, the translation history includes a second vocabulary input by the user, and the second vocabulary is a vocabulary for which the user requests additional oral movements. .

The second words included in the translation history can, to a certain extent, reflect the language habits and application usage habits of electronic device users. This technical solution recommends vocabulary for additional oral movements to the user based on the user's history of requesting additional oral movements. The implementation of this technical solution is conducive to determining the Chinese translation results according to the user's habits and is conducive to improving the efficiency of translation. The effect is conducive to improving the application experience of electronic device users.

Combined with the first aspect, in some implementations of the first aspect, the oral movement is determined according to the pronunciation mouth shape of the Chinese pinyin of the keyword.

In connection with the first aspect, in some implementations of the first aspect, the mixed shape value corresponding to the pronunciation mouth shape is stored in the mouth action database.

By establishing a mouth movement database, when the mouth movement needs to be displayed, the electronic device sends a request message to the server, and the server calls the required mouth movement data from the database. Compared with oral movement data obtained through deep learning and other solutions, it is conducive to simplifying the process of obtaining oral movement data, improving the efficiency of translation, and improving the application experience of electronic devices.

With reference to the first aspect, in some implementations of the first aspect, the hand movement includes a first hand movement and a second hand movement, the first hand movement precedes the second hand movement, and the electronic The device receives first hand action data from the server, and the first hand action data is used to display the first hand action; while displaying the first hand action, the electronic device receives second hand action data from the server, and the first hand action data is used to display the first hand action data. The second hand movement data is used to display the second hand movement.

It should be noted that the first hand movement or the second hand movement here may be a specific action, or may be an action picture contained in one or more frames of a specific action.

In this technical solution, the electronic device first receives the hand movement data that needs to be displayed first, and while displaying the hand movement that needs to be displayed first, it also receives the hand movement data that is displayed later. It uses fragmented transmission of hand movement data and displays it while The transmission solution is conducive to shortening the waiting time for data transmission and improving the user's application experience.

With reference to the first aspect, in some implementations of the first aspect, the oral action includes a first oral action and a second oral action, the first oral action precedes the second oral action, and the electronic device The first mouth movement data is received from the server, and the first mouth movement data is used to display the first mouth movement; while displaying the first mouth movement, the electronic device receives the second mouth movement data from the server, and the first mouth movement data is used to display the first mouth movement. The second oral movement data is used to display the second oral movement.

It should be noted that the first oral movement or the second oral movement here may be a specific action, or may be an action picture contained in one or more frames of a specific action.

In this technical solution, the electronic device first receives the oral action data that needs to be displayed first, and while displaying the oral action data that needs to be displayed first, it also receives the oral action data that is displayed later. It uses slices to transmit the oral action data and displays it while The transmission solution is conducive to shortening the waiting time for data transmission and improving the user's application experience.

In conjunction with the first aspect, in some implementations of the first aspect, before displaying the hand movement corresponding to the text information, the electronic device receives a response message from the server, the response message being used to indicate that the text information does not contain sensitive information.

Before translating text information into hand movements and/or mouth movements, text risk control checks are first performed on the text information. The implementation of this technical solution is conducive to filtering out bad text information and is conducive to improving the application experience of electronic device users.

In a second aspect, a Chinese translation method is provided, including: the server receives a translation request message, the translation request message includes text information, the text information includes keywords, the keywords are determined according to the language habits of the sign language user, and the translation request The message is used to request to obtain the hand movement data corresponding to the text information. The translation request message is also used to request to obtain the oral movement data corresponding to the keyword; the server determines whether to send the hand movement data and/or oral movement data based on the text information. action data.

Here, the hand movement data is used to display the hand movement corresponding to the text information, and the mouth movement data is used to display the mouth movement corresponding to the keyword.

In a possible implementation, the keyword is used by the electronic device according to the text information input by the user through the following One or more identifications in the formula are obtained: the content of the text information, the user's translation history information, or other methods for the user to determine the keywords in the text information.

In this technical solution, only oral movements are added to keywords determined according to user habits. The implementation of this technical solution is conducive to reducing the amount of data transmitted between the electronic device and the server when translating text information into sign language, and is conducive to improving the efficiency of electronic devices. The efficiency of text message translation.

Combined with the second aspect, in some implementations of the second aspect, the keyword is a proper noun.

Combined with the second aspect, in some implementations of the second aspect, the server determines whether the text information contains sensitive information; if the text information contains sensitive information, the server sends a first response message, and the first response message The response message is used to indicate that the text information contains sensitive information; if the text information does not contain sensitive information, the server sends a second response message, and the second response message includes hand movement data and/or mouth movement data.

Combined with the second aspect, in some implementations of the second aspect, the hand movement data includes first hand movement data and second hand movement data, and the first hand movement data is used to display the first hand movement data. action, the second hand action data is used to display the second hand action, the first hand action precedes the second hand action, and the server sends the second hand action after sending the first hand action data. action data.

In this technical solution, the server first sends the hand movement data that needs to be displayed first, and sends the hand movement data that needs to be displayed first while displaying the hand movement data that needs to be displayed first. It uses fragmented transmission of hand movement data and transmission while displaying. This solution will help shorten the waiting time for data transmission and improve the user's application experience.

Combined with the second aspect, in some implementations of the second aspect, the mouth action data includes first mouth action data and second mouth action data, and the first mouth action data is used to display the first mouth action data. action, the second mouth action data is used to display the second mouth action, the first mouth action is before the second mouth action, and the server sends the second mouth action after sending the first mouth action data action data.

In this technical solution, the server first sends the mouth movement data that needs to be displayed first, and sends the mouth movement data that needs to be displayed first while displaying the mouth movement data that needs to be displayed first. It adopts fragmented transmission of mouth movement data and transmission while displaying. This solution will help shorten the waiting time for data transmission and improve the user's application experience.

Combined with the second aspect, in some implementations of the second aspect, the server obtains the oral action data from an oral action database, where the oral action database contains mixed shape values corresponding to the pronunciation mouth shape of Chinese Pinyin.

By establishing a database for the mouth movement data, when the mouth movement needs to be displayed, the electronic device sends a request message to the server, and the server calls the required mouth movement data from the database. Compared with oral movement data obtained through deep learning and other solutions, it is conducive to simplifying the process of obtaining oral movement data, improving the efficiency of translation, and improving the application experience of electronic devices.

In a third aspect, an electronic device is provided, including a processor and a memory, the memory stores one or more A computer program, the one or more computer programs including instructions that, when executed by a processor, are configured to: in response to user input, obtain textual information, the textual information including keywords, the keywords being used according to sign language The language habits of the user are determined; the processor is also used to display the hand movements corresponding to the text information, and the processor is also used to display the oral movements corresponding to the keywords.

Combined with the third aspect, in some implementations of the third aspect, the keyword is determined based on the language habits of the sign language user.

Combined with the third aspect, in some implementations of the third aspect, the processor is also configured to not display oral movements corresponding to common words, the text information includes the common words, and the common words are different from the keywords.

Combined with the third aspect, in some implementations of the third aspect, the processor is specifically configured to display the mouth movement while displaying the hand movement corresponding to the keyword.

In conjunction with the third aspect, in some implementations of the third aspect, the processor is further configured to display a first vocabulary, the text information includes the first vocabulary, and the first vocabulary is a vocabulary that recommends additional oral movements; In response to the user's confirmation operation, the processor is also configured to determine that the first word is a keyword.

Combined with the third aspect, in some implementations of the third aspect, in response to the user's first input, the processor is configured to obtain a second vocabulary, the second vocabulary being a vocabulary for which the user requests additional oral movements; in the text information When the text information contains the second vocabulary, the processor is also used to determine that the second vocabulary is a keyword; when the text information does not contain the second vocabulary, the processor is also used to display update request information, and the update request message is The prompt text information does not include the second vocabulary; in response to the user's second input, the processor is also used to obtain the updated second vocabulary.

Combined with the third aspect, in some implementations of the third aspect, the hand movement includes a first hand movement and a second hand movement, the first hand movement precedes the second hand movement, and the processing The processor is also used to receive first hand movement data from the server, and the first hand movement data is used to display the first hand movement; while displaying the first hand movement, the processor is also used to receive a second hand movement data from the server. Hand movement data, the second hand movement data is used to display the second hand movement.

Combined with the third aspect, in some implementations of the third aspect, the oral action includes a first oral action and a second oral action, the first oral action precedes the second oral action, and the processor It is also used to receive the first oral movement data from the server, and the first oral movement data is used to display the first oral movement; while displaying the first oral movement, the processor is also used to receive the second oral movement data from the server. mouth movement data, and the second mouth movement data is used to display the second mouth movement.

Combined with the third aspect, in some implementations of the third aspect, the processor is further configured to receive a response message from the server, where the response message is used to indicate that the text information does not contain sensitive information.

In a fourth aspect, a server is provided, including a processor and a memory. The memory stores one or more computer programs. The one or more computer programs include instructions. When the instructions are executed by the processor, the processor is used to: Receive a translation request message. The translation message includes text information. The translation request message is used to request acquisition of hand movement data corresponding to the text information. The text information includes keywords. The keywords are determined according to the language habits of the sign language user. The translation request message is also used to request the oral movement data corresponding to the keyword; the processor is also used to determine whether to send hand movement data and/or mouth movement data based on the text information.

Combined with the fourth aspect, in some implementations of the fourth aspect, the processor is also used to determine whether the text information contains sensitive information; when the text information contains sensitive information, the processor is also used to send A first response message, the first response message is used to indicate that the text information contains sensitive information; when the text information does not contain sensitive information, the processor is also used to send a second response message, the second response message includes a handwritten message. facial movement data and/or oral action data.

In conjunction with the fourth aspect, in some implementations of the fourth aspect, the hand movement data includes first hand movement data and second hand movement data, and the first hand movement data is used to display the first hand movement data. action, the second hand action data is used to display the second hand action, the first hand action precedes the second hand action, and the processor is further configured to send the first hand action data after sending the Second hand movement data.

In conjunction with the fourth aspect, in some implementations of the fourth aspect, the mouth action data includes first mouth action data and second mouth action data, and the first mouth action data is used to display the first mouth action data. action, the second oral action data is used to display the second oral action, the first oral action precedes the second oral action, and the processor is further configured to send the first oral action data after sending the first oral action data. Second oral movement data.

In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processor is further configured to obtain the oral action data from an oral action database, where the oral action database contains mixed shapes corresponding to the mouth shapes of Chinese Pinyin pronunciations. numerical value.

In a fifth aspect, a Chinese translation device is provided, including an acquisition unit and a processing unit. The acquisition unit is used to acquire text information in response to user input. The text information includes keywords, and the keywords are based on the language habits of the sign language user. OK; the processing unit is used to display the hand movements corresponding to the text information; the processing unit is also used to display the mouth movements corresponding to the keywords.

Combined with the fifth aspect, in some implementations of the fifth aspect, the keyword is determined based on the language habits of the sign language user.

Combined with the fifth aspect, in some implementations of the fifth aspect, the processing unit is also configured to not display oral movements corresponding to common words, the text information includes the common words, and the common words are different from the keywords.

In conjunction with the fifth aspect, in some implementations of the fifth aspect, the processing unit is further configured to display the mouth movement while displaying the hand movement corresponding to the keyword.

Combined with the fifth aspect, in some implementations of the fifth aspect, the processing unit is also used to display a first vocabulary, the text information includes the first vocabulary, the first vocabulary is a vocabulary that recommends additional oral movements, and the response Based on the user's confirmation operation, the processing unit is also used to determine that the first vocabulary is a keyword.

In conjunction with the fifth aspect, in some implementations of the fifth aspect, the acquisition unit is further configured to acquire a second vocabulary in response to the user's first input, where the second vocabulary is a vocabulary for which the user requests additional oral movements; in When the text information contains the second vocabulary, the processing unit is also used to determine that the second vocabulary is a keyword; when the text information does not contain the second vocabulary, the processing unit is also used to display an update request message. The update request The message is used to prompt that the text information does not contain the second vocabulary; the acquisition unit is also used to obtain the updated second vocabulary in response to the user's second input.

In conjunction with the fifth aspect, in some implementations of the fifth aspect, the Chinese translation device further includes a communication unit, the hand movement includes a first hand movement and a second hand movement, and the first hand movement is in the Before the second hand movement, and before displaying the hand movement corresponding to the text information, the communication unit is used to receive the first hand movement data from the server, and the first hand movement data is used to display the first hand movement; The communication unit is further configured to receive second hand movement data from the server while displaying the first hand movement, and the second hand movement data is used to display the second hand movement.

In conjunction with the fifth aspect, in some implementations of the fifth aspect, the oral action includes a first oral action and a second oral action, the first oral action precedes the second oral action, and the communication unit It is also used to receive the first oral movement data from the server, and the first oral movement data is used to display the first oral movement; while displaying the first oral movement, the communication unit is also used to receive the second oral movement data from the server. mouth movement data, and the second mouth movement data is used to display the second mouth movement.

Combined with the fifth aspect, in some implementations of the fifth aspect, before displaying the hand movement corresponding to the text information, the communication unit is also used to receive a response message from the server, the response message is used to indicate that the text information does not contain sensitive information. information.

In a sixth aspect, a Chinese translation device is provided, including a communication unit and a processing unit. The communication unit is used to receive a translation request message. The translation request message includes text information. The translation request message is used to request to obtain the text corresponding to the text information. Hand movement data, the text information includes keywords, the keywords are determined according to the language habits of the sign language user, the translation request message is also used to request to obtain oral movement data corresponding to the keywords; the processing unit is used to obtain the oral movement data corresponding to the keywords; the processing unit is used to obtain the hand movement data according to the language habits of the sign language user. Information determines whether to send hand movement data and/or mouth movement data.

Combined with the sixth aspect, in some implementations of the sixth aspect, the processing unit is also used to determine whether the text information contains sensitive information; in the case where the text information contains sensitive information, the communication unit is also used to send A first response message, the first response message is used to indicate that the text information contains sensitive information; when the text information does not contain sensitive information, the communication unit is also used to send a second response message, the second response message includes a handheld message. facial movement data and/or oral movement data.

In conjunction with the sixth aspect, in some implementations of the sixth aspect, the hand movement data includes first hand movement data and second hand movement data, and the first hand movement data is used to display the first hand movement data. action, the second hand action data is used to display the second hand action, the first hand action precedes the second hand action, and the communication unit is also used to send the first hand action data after sending the Second hand movement data.

In conjunction with the sixth aspect, in some implementations of the sixth aspect, the mouth action data includes first mouth action data and second mouth action data, and the first mouth action data is used to display the first mouth action data. action, the second oral action data is used to display the second oral action, the first oral action precedes the second oral action, and the communication unit is further configured to send the first oral action data after sending the first oral action data. Second oral movement data.

In conjunction with the sixth aspect, in some implementations of the sixth aspect, the processing unit is further configured to obtain the oral action data from an oral action database, where the oral action database contains mixed shapes corresponding to the pronunciation mouth shapes of Chinese Pinyin. numerical value.

In a seventh aspect, a computer program product is provided. The computer program product includes computer program code. When the computer program code is run on a computer, the method in the first aspect or any possible implementation thereof is executed.

In an eighth aspect, a computer program product is provided. The computer program product includes computer program code. When the computer program code is run on a computer, the method in the second aspect or any possible implementation thereof is executed.

In a ninth aspect, a computer-readable storage medium is provided. Computer instructions are stored in the computer-readable medium. When the computer instructions are run on a computer, the method in the first aspect or any possible implementation thereof is executed.

In a tenth aspect, a computer-readable storage medium is provided. Computer instructions are stored in the computer-readable medium. When the computer instructions are run on a computer, the method in the second aspect or any possible implementation thereof is executed.

An eleventh aspect provides a chip, including a processor for reading instructions stored in a memory. When the processor executes the instructions, the chip implements the method in the first aspect or any possible implementation thereof. be executed.

In a twelfth aspect, a chip is provided, including a processor for reading instructions stored in a memory. When the processor executes the instruction, the chip implements the method in the second aspect or any possible implementation thereof. be executed.

Description of the drawings

FIG. 1 is a schematic diagram of the hardware architecture of an electronic device applicable to an embodiment of the present application.

FIG. 2 is a schematic diagram of an electronic device software architecture applicable to an embodiment of the present application.

Figure 3 is a schematic diagram of a Chinese translation method provided by an embodiment of the present application.

Figure 4 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 5 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 6 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 7 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 8 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 9 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 10 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 11 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 12 is a schematic diagram of another Chinese translation method provided by an embodiment of the present application.

Figure 13 is a schematic diagram of a Chinese translation device provided by an embodiment of the present application.

Figure 14 is a schematic diagram of another Chinese translation device provided by an embodiment of the present application.

Figure 15 is a schematic diagram of an electronic device provided by an embodiment of the present application.

Figure 16 is a schematic diagram of a server provided by an embodiment of the present application.

Detailed ways

The technical solutions in this application will be described below with reference to the accompanying drawings.

The terminology used in the following examples is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "said", "above", "the" and "the" are intended to also Expressions such as "one or more" are included unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one, two or more than two. The term "and/or" is used to describe the relationship between associated objects, indicating that there can be three relationships; for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, Where A and B can be singular or plural. The character "/" generally indicates that the related objects are in an "or" relationship.

Reference in this specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Therefore, the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in other embodiments", etc. appearing in different places in this specification are not necessarily References are made to the same embodiment, but rather to "one or more but not all embodiments" unless specifically stated otherwise. The terms “including,” “includes,” “having,” and variations thereof all mean “including but not limited to,” unless otherwise specifically emphasized.

The methods provided by the embodiments of this application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR)/virtual reality (VR) devices, notebook computers, ultra mobile personal computers (ultra -Mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA) and other electronic devices, the embodiments of this application do not place any restrictions on the specific types of electronic devices.

By way of example, FIG. 1 shows a schematic structural diagram of an electronic device 100 . The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) wait. Among them, different processing units can be independent devices or integrated in one or more processors.

The controller may be the nerve center and command center of the electronic device 100 . The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. Interfaces may include integrated circuit (inter-integrated circuit, I2C) interface, integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, pulse code modulation (pulse code modulation, PCM) interface, universal asynchronous receiver and transmitter (universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and /or universal serial bus (USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can separately couple the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to implement the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 can be coupled with the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to implement the function of answering calls through a Bluetooth headset.

The PCM interface can also be used for audio communications to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to achieve connection via a Bluetooth headset. Listen to the phone function. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100 . The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100 .

The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, display screen 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface that complies with the USB standard specification, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through them. This interface can also be used to connect other electronic devices, such as AR devices, etc.

It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only schematic illustrations and do not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The charging management module 140 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through the wireless charging coil of the electronic device 100 . While the charging management module 140 charges the battery 142, it can also provide power to the electronic device through the power management module 141.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, wireless communication module 160, etc. The power management module 141 can also be used to monitor battery capacity, battery cycle times, battery health status (leakage, impedance) and other parameters. In some other embodiments, the power management module 141 may also be provided in the processor 110 . In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.

The mobile communication module 150 can provide wireless communication including 2G/3G/4G/5G etc. applied on the electronic device 100. solution. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be disposed in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.

A modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (bluetooth, BT), and global navigation satellites. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi) -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, videos, etc. Display 194 includes a display panel. The display panel can use liquid crystal display (LCD) or organic light-emitting diode (OLED). Active matrix organic light emitting diode or active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, Quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

Camera 193 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other format image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can continuously learn by itself. Intelligent cognitive applications of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, etc.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. Such as saving music, videos, etc. files in external memory card.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The processor 110 executes instructions stored in the internal memory 121 to execute various functional applications and data processing of the electronic device 100 . The internal memory 121 may include a program storage area and a data storage area. Among them, the stored program area can store an operating system, at least one application program required for a function (such as a sound playback function, an image playback function, etc.). The storage data area may store data created during use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also called "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to hands-free calls.

Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or a voice message, the voice can be heard by bringing the receiver 170B close to the human ear.

Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with the human mouth and input the sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which in addition to collecting sound signals, may also implement a noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions, etc.

The headphone interface 170D is used to connect wired headphones. The headphone interface 170D may be a USB interface 130, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

The buttons 190 include a power button, a volume button, etc. Key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .

The motor 191 can generate vibration prompts. The motor 191 can be used for vibration prompts for incoming calls and can also be used for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. The motor 191 can also respond to different vibration feedback effects for touch operations in different areas of the display screen 194 . Different application scenarios (such as time reminders, receiving information, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

The indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, or may be used to indicate messages, missed calls, notifications, etc.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an embedded SIM (embedded-SIM, eSIM) card, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

It should be understood that the phone card in the embodiment of the present application includes but is not limited to SIM card, eSIM card, universal subscriber identity module (USIM), universal integrated circuit card (UICC), etc.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of this application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100 .

FIG. 2 is a software structure block diagram of the electronic device 100 according to the embodiment of the present application. The layered architecture divides the software into several layers, and each layer has clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system libraries, and kernel layer. The application layer can include a series of application packages.

As shown in Figure 2, the application package can include camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer can include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

A window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make this data accessible to applications. Said data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

The view system includes visual controls, such as controls that display text, controls that display pictures, etc. A view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide communication functions of the electronic device 100 . For example, call status management (including connected, hung up, etc.).

The resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc.

The notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows. For example, text information is prompted in the status bar, a beep sounds, the electronic device vibrates, the indicator light flashes, etc.

Android runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.

The core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.

The application layer and application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and application framework layer into binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

System libraries can include multiple functional modules. For example: surface manager (surface manager), media libraries (media libraries), 3D graphics processing libraries (for example: OpenGL ES), 2D graphics engines (for example: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing.

2D Graphics Engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

It should be understood that the technical solutions in the embodiments of this application can be used in Android, IOS, Hongmeng and other systems.

The hardware and software architecture of the electronic device suitable for the translation method provided by the present application has been introduced above with reference to Figures 1 and 2. The Chinese translation method provided by the embodiment of the present application will be described below with reference to Figures 3 to 16. Before formally introducing the embodiments of this application, some terms that may be used in the following embodiments are first introduced.

1. Chinese sign language (CSL): Chinese universal sign language, mainly used in mainland China.

2. Speech recognition (automatic speech recognition, ASR): It can also be called speech to text (speech to text, STT). Its goal is to use computers to automatically convert human speech content into corresponding text.

3. Optical character recognition (OCR): refers to the process of analyzing and recognizing image files of text data to obtain text and layout information.

4. Software development kit (SDK): refers to a collection of development tools used to create application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.

5. Blendshape: A technology that operates on the vertices of the three-dimensional model mesh to achieve a defined shape, which can be used to control the facial expressions of virtual characters.

6. Digital human: refers to the use of computer technology to digitize the human body structure, and a visible and controllable virtual human body form appears on the computer screen. The functional information of the human body is further attached to this human body form framework, and through virtual reality technology Through cross-fusion, this "digital human" will be able to imitate real people and make various reactions. If equipped with sound and force feedback devices, it can also provide an intuitive and natural real-time sense of sight, hearing, touch, etc.

7. Sign language (sign language, signed language, signing) is a language that does not use auditory-speech, but uses visual-gestural mode - using body movements and facial expressions to express and convey meaning.

8. Part of speech: refers to the characteristics of a word and is used to classify parts of speech. Modern Chinese words can be divided into two categories: content words and function words. Content words refer to those that can act alone as syntactic components or mostly as the main components of oranges. It has lexical and grammatical meanings. Includes nouns, verbs, adjectives, adverbs, numerals, quantifiers, pronouns and onomatopoeia. Function words cannot serve as syntactic components alone or mostly as auxiliary components of sentences. It has only grammatical meaning. Includes prepositions, conjunctions, particles and interjections.

Table 1 gives a classification method for parts of speech, in which proper nouns can include: names of people, place names, institutional groups, work titles and other proper nouns.

Table 1 Part-of-speech tags and their meanings

Figure 3 is a schematic diagram of a Chinese translation method provided by an embodiment of the present application. The following takes the process of an electronic device using App1 to translate text information into corresponding sign language as an example to introduce the Chinese translation method provided by an embodiment of the present application.

It should be noted that in the following embodiments, the user of App1 (ie, electronic device user or user) may be a hearing-impaired person or a hearing-impaired person.

Electronic device users can input data that needs to be translated through the input function control 304 of App1. The input function control 304 can be used to input one or more of the following data types to App1: text (for example, the content shown in 303), image , documents, audio, video, etc.

When the electronic device user inputs text, App1 can directly obtain the text information contained in the text input by the electronic device user. The text may be manually input by the electronic device user, or may be one or more texts provided by App1 (common sentences built into App1), and the electronic device user selects from the one or more texts.

When the terminal user inputs an image, App1 recognizes the text information contained in the image through OCR after receiving the image data. When the electronic device user inputs document data (such as example.txt, etc.), App1 parses the document after receiving the document data to obtain the text information contained in the document.

When the electronic device user inputs audio or video data, App1 recognizes the text information contained in the audio or video data through ASR and/or OCR after receiving the audio or video data. For example, when the video data received by App1 contains subtitles, App1 can identify the text information in the video through OCR. When the video data received by App1 contains audio data, App1 can identify the text information contained in the video data through ASR. When When the video data received by App1 contains both subtitles and audio data, App1 can simultaneously use ASR and OCR to identify the text information contained in the video, and perform mutual proofreading to improve the accuracy of text recognition.

When App1 obtains the text information shown in 303 "John went to the cinema this afternoon.", App1 can obtain the hand movement data corresponding to the text information based on the text information, and then use the hand movement data to drive the virtual character model , so that the virtual character model can display the hand movements corresponding to the text information.

In some embodiments, before App1 translates the obtained text into sign language, App1 will also identify the parts of speech of different words contained in the data input by the user to be translated. For proper nouns, App1 will also obtain the corresponding corresponding nouns. Oral movement data. When the avatar shows the hand movements of a proper noun, driven by the mouth movement data, the avatar will also show the mouth movements corresponding to the proper noun.

In some embodiments, when the electronic device user opens the interface shown in Figure 3, App1 displays prompt information 302 in response to the user's operation. The prompt information is used to prompt the electronic device user to use the method or step of App1.

For example, the prompt information can be used to prompt the electronic device user to input data to be translated into App1 through the input function control 304.

For example, the prompt information can also be used to prompt the user of the electronic device to input words that require additional oral movements.

Optionally, App1 can also display processing status information. For example, as shown at 301, App1 is currently executing action or the action the user is performing.

In the embodiment of this application, App1 uses OCR to recognize text information in images as an example to illustrate App1's processing of data to be translated that needs to recognize text information such as images and documents.

The electronic device user inputs a picture containing the text information "John went to the cinema this afternoon." to App1 through the input function control. After receiving the user's input image, App1 recognizes the text information in the image through OCR.

In some embodiments, App1 recognizes the text information correctly ("John went to the cinema this afternoon."), App1 displays the text recognition result confirmation prompt window (as shown in (a) in Figure 4), the user clicks "Confirm", and App1 Obtain the user's confirmation instruction and perform the next operation, which is the operation shown in (c) in Figure 4.

In other embodiments, App1 recognizes text information incorrectly ("John went to the cinema this morning."), App1 displays a text recognition result confirmation prompt window, and the user clicks "Modify" after confirming that App1 recognizes text information incorrectly, and App1 obtains the user The modification instruction displays the modification text recognition result prompt window as shown in Figure 4(b). The user clicks "Confirm" after inputting the correct text information ("John went to the cinema this afternoon."). In response to the user's Input, App1 obtains the modified text information and performs the next operation, which is the operation shown in (c) in Figure 4.

As shown in (c) in Figure 4, App1 can translate the text information into corresponding hand movements after obtaining the text information confirmed by the user.

In some embodiments, before App1 translates the confirmed text information into hand movements, it displays the operation prompt message "Please enter the keywords that require additional oral movements:", and the electronic device user uses the input function control according to the operation prompt message. Enter "John" into App1. In response to the user's input, App1 obtains the keyword "John" and obtains the hand movement data corresponding to the text information based on the text information. It also obtains the spoken word corresponding to the keyword "John". Hand movement data, so that App1 can use the obtained hand movement data and mouth movement data of keywords to drive the virtual character to display the corresponding hand movements and mouth movements.

In other embodiments, before App1 translates the confirmed text information into spoken language, it analyzes that the text information contains the proper noun "John", App1 uses the proper noun as a keyword that requires additional oral movements, and While obtaining the hand movement data corresponding to the text information, it also obtains the oral movement data corresponding to the proper noun, so that App1 can use the obtained hand movement data and the mouth movement data of the keywords to drive the virtual character to display the corresponding hands. movements and oral movements.

It should be noted that keywords can be words containing one or more Chinese characters.

Figure 5 shows a schematic diagram of another Chinese translation method provided by an embodiment of the present application. In the embodiment of the present application, the electronic device user inputs keywords that require additional oral movements, and App1 checks the keywords input by the user to reduce the probability of possible errors in the process of translating text information into hand movements.

As shown in (a) in Figure 5, the electronic device user inputs text information to be translated as “Please turn off the faucet after using up water.” In response to the user’s input, App1 displays the prompt message “Please enter keywords that require additional oral movements. :".

In some embodiments, the user of the electronic device inputs the keyword: "close, faucet" according to the above prompt information. In response to the user's input, App1 checks and determines that the text information contains the keyword entered by the user, and then performs the corresponding translation operation.

In other embodiments, the user of the electronic device inputs the keyword: "close, fire faucet" according to the above prompt information. In response to the user's input, App1 checks and determines that the text information contains the keyword "close" but does not contain the keyword "Fire faucet", then App1 displays the confirmation keyword prompt message as shown in Figure 5(b): ""Fire faucet" is not found, please confirm whether it is "faucet"?" According to this prompt message, the electronic device user confirms The entered keywords are incorrect and the keywords recognized by App1 are correct, and click "Confirm". In response to the user's confirmation operation, App1 updates the keywords that require additional mouth movements to: "close" and "faucet".

Or, when the user confirms that the entered keywords are incorrect and the keywords recognized by App1 are also incorrect, the user can click "Modify" to enter the correct keywords that require additional mouth movements.

In some embodiments, the user of the electronic device inputs the keyword: "faucet" according to the above prompt information. In response to the user's input, App1 checks and determines that the text information contains the keyword entered by the user, but cannot obtain the keyword. If the corresponding mouth movement data is obtained, App1 will issue a prompt message. The prompt message may be "Cannot find the mouth movement data corresponding to the "faucet" you entered. Manual service has been requested for you in the background, please wait.".

Optionally, App1 can establish a video connection with the artificial customer service for the user. After the connection is established, the artificial customer service can show the user the mouth movements of the above-mentioned unavailable keywords. Alternatively, the artificial customer service staff can supplement the oral action data of the above-mentioned unobtainable keywords in the background and call it to App1. After App1 obtains the oral action data, it will be displayed to the electronic device user.

In some embodiments, the electronic device user inputs the entire content of the text information to be translated according to the above prompt information. In response to the user's input, App1 detects that there are many keywords that require additional mouth movement data. App1 can issue a prompt message to remind the user: There are currently many keywords that require additional mouth movement data. You can re-enter the keywords that require additional mouth movement data. Action keywords.

In some embodiments, the user of the electronic device does not enter any keywords according to the above prompt information, and App1 detects that the keywords entered by the user are not obtained within the preset time period, then App1 can identify the content of the text information input by the user, Issue a prompt message containing keywords that recommend additional oral movements.

In one embodiment, the keywords for recommending additional oral actions can be determined based on the parts of speech of Chinese vocabulary as shown in Table 1, such as proper nouns, time words, etc.

In another embodiment, the recommended keywords for additional oral movements can also be determined based on user habits. For example, the user has used "John" as a keyword to append oral movements multiple times in the translation query history in App1. Then, when App1 obtains that the user input data to be translated also contains "John", "John" can be used as a keyword to recommend additional oral actions.

For the same example, when the user selects keywords for additional oral movements for the text information to be translated, and determines the subject and object in a sentence as keywords for additional oral movements multiple times, then when App1 is in the preset time When the keywords input by the user that require additional oral movements are not obtained, the subject and object in the text information input by the user to be translated can be determined as keywords that recommend additional oral movements.

In yet another embodiment, the keywords for recommending additional oral movements may be determined based on determination methods of other users. For example, for the same video, 80% of users identified "cinema" and "playground" as keywords that require additional oral movements. When the user inputs the same video and App1 does not When obtaining the keywords input by the user that require additional oral movements, App1 can use "cinema" and "playground" as keywords to recommend additional oral movements.

Optionally, when the keywords entered by the user that require additional mouth movement data only include "cinema", App1 can also send a prompt message to prompt the user whether to also add mouth movement data for "playground"? When the electronic device user determines to add mouth action data to "Amusement Park", in response to the user's operation, App1 will add "Cinema", "Amusement Park" As a keyword that requires additional oral movements.

As shown in (c) in Figure 5, when App1 obtains the keywords confirmed by the user, App1 will also display prompt information, which is used to prompt the updated keywords.

The above describes the text information input process of the Chinese translation method provided by the present application with reference to Figures 3 to 5. The display and use of the translation results corresponding to the Chinese translation method provided by the embodiment of the present application will be described below with reference to Figures 6 to 11. process.

After App1 obtains the hand movement data and mouth movement data corresponding to the text information based on the user's input, App1 displays the translation result interface as shown in Figure 6.

The translation result interface may include processing prompt information 630, which is used to prompt that the translation of the text information has been completed. Optionally, the prompt information is also used to prompt the electronic device user how to use the translation result.

The translation result interface may also include an overall display area 611, which is used to display the overall situation of the virtual character when signing. Optionally, when the user confirms that he or she needs to append oral movement data to one or more keywords, the overall display area is used to display the hand movements of the text information to be translated and the mouth of the keywords with the oral movement data attached. action.

The translation result interface may also include a hand movement display area 613, which is used to display the details of the hand movement of the text information input by the user to be translated. Optionally, the hand movement display area may include auxiliary lines and/or auxiliary text, and the auxiliary lines and/or auxiliary text are used to help the user understand sign language details such as finger movement trajectories.

The translation result interface may also include a mouth movement display area 612, which is used to display mouth movements for which the user requests additional mouth movement data words or recommends mouth movements for which additional mouth movement data words are recommended. Optionally, the oral movement display area may include auxiliary lines and/or auxiliary text, and the auxiliary lines and/or auxiliary text are used to help the user understand oral movement details such as mouth movement trajectories.

The translation result interface may also include a text status display area 614, which is used to display text corresponding to the currently displayed hand movements and/or mouth movements. Optionally, the text status display area also includes a pinyin annotation area, which is used to display the pinyin annotation of the text corresponding to the currently displayed hand movement and/or oral movement.

In some embodiments, the text status display area displays corresponding words in the order of hand movements.

It should be noted here that the order of hand movements may not be the same as the reading order from left to right for hearing-impaired people.

For example, the sign language expression order of "I didn't bring my mobile phone." is: mobile phone, me, bring, no. Therefore, if "mobile phone" is used as a keyword that requires additional oral movements, this sentence will be displayed through the text state The area can be displayed in the following form: "Mobile phone", "Me", "With", and "No" are displayed in order from left to right. For "mobile phone", you can highlight or bold the display.

In other embodiments, the text status display area displays text information in the order of natural spoken language, and highlights words corresponding to hand movements in the order of hand movements.

For example, the default color of the text in the text status display area is black, the text corresponding to the currently displayed hand movement is red, and the text corresponding to the currently displayed mouth movement is green and bold.

Similarly, for example, the sign language expression order of "I don't have a mobile phone." is: mobile phone, me, bring, no. Therefore, if "mobile phone" is used as a keyword that requires additional oral movements, this sentence will be passed through text. status display area It will be displayed in the following form: "Mobile" is displayed in bold green, "I" is displayed in red, "With" is displayed in red, and "Without" is displayed in red.

In some embodiments, the above-mentioned overall display area 611, mouth movement display area 612, hand movement display area 613, text status display area 614 and prompt information 630 constitute a translation result interface.

In other embodiments, the above-mentioned overall display area 611, mouth movement display area 612, hand movement display area 613 and text status display area 614 constitute the translation result display area 610 of the translation result interface, and the translation result display area 610 is Part of the translation results interface.

Optionally, the translation result display area 610 may also include prompt information 630 and an input area 620. The input area is used to display the text information to be translated that has been input by the user, the input prompt information issued by App1, and the additional information that the user has input. Keywords for oral movements, etc. Optionally, the electronic device user can also re-enter keywords that require additional oral action data in the input area. When the user re-enters keywords that require additional oral movement data in the input area, in response to the user's input, App1 obtains the oral movement data corresponding to the keywords re-entered by the user, and updates the overall display area in the translation result display area. Mouth movement display area, hand movement display area and text status display area.

For example, after the user has determined that "mobile phone" in the text message: "I did not bring my mobile phone." is a word that requires additional oral movements, the user re-enters "I" in the input area shown in Figure 6, and responds Based on the user's input, App1 determines "I" as a keyword that requires additional oral movements.

The basic composition of the translation result display interface is introduced above in conjunction with Figure 6. The following is a detailed introduction to the functions that each component of the translation structure display interface can have in conjunction with Figure 7.

The electronic device user triggers App1 to display the functional options of the translation result display area by clicking, double-clicking or long-pressing a blank space in the translation result display area.

The electronic device user triggers App1 to display and translate the function tabs in these areas by clicking, double-clicking or long-pressing the overall display area, the mouth movement display area, the hand movement display area, or the text status display area.

The above-mentioned function tabs may include one or more of the following functions: "View in full screen", "Play at double speed", "Insert into audio/video", "Hide", "Save" or "Share", etc.

When the electronic device user selects the "full screen view" function option, in response to the user's operation, App1 displays the entire display area, mouth movement display area, hand movement display area, or text status display area in full screen.

When the electronic device user selects the "double speed playback" function option, App1 displays a playback rate adjustment function window in response to the user's operation, and the user can select or input the playback rate that needs to be set in the playback rate adjustment function window. After obtaining the playback rate selected or input by the user, App1 plays the content contained in the overall display area, mouth movement display area, hand movement display area, or text status display area at the corresponding rate (slow or fast).

When the electronic device user selects the "insert into audio/video" function option, in response to the user's operation, App1 will display one or more of the overall display area, the mouth movement display area, the hand movement display area, or the text status display area. Insert into the corresponding audio or video. Optionally, after inserting any of the above areas into the audio file, App1 can save the modified audio file in the format of a video file.

When the electronic device user selects the "hide" function option, in response to the user's operation, App1 hides the overall display area, the mouth movement display area, the hand movement display area, or the text status display area. When the user clicks on the hidden area again, the function options corresponding to the area may include the "Show" function option. When the user selects the "Show" function option, App1 displays the hidden area in response to the user's operation.

It should be noted here that if the text information to be translated does not contain oral movements, or the user chooses not to attach oral movements to any keywords, the oral movement display area can be hidden by default.

When the electronic device user selects the "save" function option, App1 saves the data corresponding to the area selected by the user in response to the user's operation. Optionally, in response to the user's operation, App1 can also display a save prompt window. The save prompt window is used to prompt the user whether to save data corresponding to other related areas at the same time. The save prompt window is also used to obtain the user's instruction information. For example, when the user chooses to save data corresponding to other related areas at the same time, in response to the user's operation, App1 saves both the data corresponding to the user-selected area and the data corresponding to the related areas locally on the electronic device.

For example, when the user selects the "Save" function option in the overall display area, App1 displays a prompt message: "Do you want to save the data of the mouth movement display area, hand movement display area, and text status display area at the same time?" When the user selects When saving the mouth movement display area, App1 simultaneously saves data corresponding to the overall display area and the mouth movement display area in response to the user's selection.

When the electronic device selects the "share" function option, App1 displays the sharing function control in response to the user's operation, and the sharing function control includes one or more sharing channels. The end user can select one or more sharing channels. In response to the user's selection, App1 shares the data corresponding to the area selected by the user through one or more sharing channels selected by the user.

Optionally, when the electronic device selects the "share" function option, App1 can also display a sharing prompt window in response to the user's operation. The sharing prompt window is used to prompt the user whether to share data corresponding to other related areas at the same time. The sharing prompt The window is also used to obtain instructions from the user. For example, when the user chooses to share data corresponding to other related areas at the same time, in response to the user's operation, App1 uses the data corresponding to the area selected by the user and the data corresponding to the related areas as data to be shared.

For example, when the user selects the "Share" function option in the overall display area, App1 displays a prompt message: "Do you want to share the data of the mouth movement display area, hand movement display area, and text status display area at the same time?" When the user selects When sharing the mouth movement display area, in response to the user's selection, App1 simultaneously shares data corresponding to the overall display area and the mouth movement display area.

For the data corresponding to different display areas saved locally on the electronic device, the user of the electronic device can open it again for viewing, sharing, editing, etc.

Figure 8 shows the interface of the resource library. The resource library is used to classify, arrange and display data corresponding to different display areas saved locally on the electronic device according to certain rules. The above rules include classification rules and arrangement rules.

The classification rules may include any of the following rules: area (overall display area, mouth movement display area or hand movement display area, etc.), time (time saved locally on the electronic device, for example: today, yesterday, One week ago, etc.) or source (for example: from the current electronic device, from electronic devices of the same account, or from home electronic devices, etc.), etc.

Sorting rules can include any of the following rules: time (such as time from far to recent or from recent to far), text information contained in the data (for example: alphabetical order of text information) or additional oral action keys The order of words (the stroke order of the first word of the keyword).

The electronic device user can select the "classification method" function option 801 of the resource library to set different classification methods for the data stored locally. The electronic device user can also select the "arrangement" function option 802 of the resource library to set different arrangements for the data stored locally.

In some embodiments, the resource library also includes a search box 805 in which the user of the electronic device can enter words, Word, time, region, source and other content to quickly find the corresponding data.

In other embodiments, the resource library also includes a "Recycle Bin" function option 803, and the electronic device user can select the "Recycle Bin" function option to view the data that has been stored in the "Recycle Bin". The "Recycle Bin" is used to temporarily store data deleted by the user. Data that has not been restored by the user after a preset period of time or data that the user has confirmed deletion in the "Recycle Bin" will be removed from the storage medium of the electronic device by App1. Erase.

In some embodiments, the resource library also includes a "share" function option 804, which an electronic device user can select to share one or more data in the resource library.

When the electronic device user selects any data to open in the resource library, in response to the user's operation, the electronic device can display a playback interface as shown in Figure 9.

Similar to the translation result display area 610 shown in Figure 6, depending on the type of open data, the playback interface may include one of an overall display area, a mouth movement display area, a hand movement display area, and a text status display area. or more. In these areas, the option functions corresponding to the areas shown in Figure 6 can also be opened. For detailed triggering methods of the option functions and specific functions, please refer to the relevant descriptions in Figure 6. To avoid duplication, they are not included here. Again.

In some embodiments, the playback interface may include a playback function control 901. The playback function control may control the start and stop of data playback. The playback function control may also view the progress of the current data playback.

Optionally, the playback function control may also include a prompt control 902 with a keyword attached to the oral movement data. The electronic device user can directly view the oral movement of the keyword by selecting (for example, clicking) the prompt control.

In some embodiments, the playback interface may include a "share" function option 903, and the electronic device user may select the "share" function option to share one or more types of data being played in the playback interface.

The following describes the sharing process of translation data in detail with reference to Figure 10. It should be noted that the sharing process can be triggered through the sharing function as shown in Figure 6, or through the sharing function in the resource library interface as shown in Figure 8, or also It can be triggered through the sharing function of the playback interface in Figure 9, or it can also be triggered through other methods, and this application does not impose restrictions on this.

Figure 10 shows a sharing interface, which includes sharing selection prompt information 1001, a sharing data preview area 1002, and a sharing channel selection window 1003.

The sharing selection prompt information 1001 is used to prompt information about the currently selected data to be shared. The sharing selection prompt information may include the quantity of data to be shared. The sharing selection prompt information may also include the types included in the data to be shared.

For example, when the electronic device user selects data corresponding to three hand movement display areas, four mouth movement display areas, and four mouth movement display areas, the sharing selection prompt information may be displayed: 11 items have been selected, including: data corresponding to the hand movement display area (manual), data corresponding to the mouth movement display area (mouth movement), and data corresponding to the text status display area (text).

The shared data preview area 1002 is used to display data to be shared. For example, when the electronic device user chooses to share the data of the overall display area, the shared data preview area may display a certain frame of the shared overall display area for previewing the data of the overall display area.

Optionally, the shared data preview area may also include a function check box 1004. The electronic device user can select data to be shared or deselect data to be shared by clicking the function check box.

The sharing channel selection window 1003 is used to display one or more available sharing channels, and the sharing channel window is also used to obtain one or more sharing channels selected by the electronic device user. Illustratively, as shown in Figure 10, one or more of the above The sharing methods can include: Bluetooth sharing, uploading to cloud disk, or sending via email.

The translation method provided by the embodiment of the present application is introduced in detail with reference to Figures 3 to 10, taking App1 as an example. One or more functions of App1 described above can be turned on or off through the setting function options of App1. The following describes the setting function of App1 with reference to Figure 11.

This setting function option can include the "automatic keyword recognition and conversion" function option. Electronic device users can use this function option to turn on or off the keywords in the text, video, and audio data input by App1 during the input process. The keywords Refers to keywords that require additional mouth movement data.

The setting function option may also include a "keyword auto-correction" function option, through which the electronic device user can turn on or off App1 to prompt and/or make errors in the keywords entered by the user during the input process. Automatic correction.

The setting function options may also include a "translation acceleration function" function option. Electronic device users can use this function option to turn on a function that improves the efficiency of text information translation. Details of how to improve the efficiency of text information translation are provided in the following embodiments. introduce.

The setting function options may also include a "result display content" function option, through which electronic device users can select the content to be displayed on the translation results display interface. For example, if the electronic device user selects "hand movements" and "mouth movements" in this function option, then in the interface shown in Figure 6, the overall display area and the text status display area are not displayed by default, and the hand movements The display area and mouth movement display area are displayed by default.

This setting function option may also include a "resource library default classification method" function option, through which the electronic device user can select the default classification method in the resource library for different data that the user saves locally on the electronic device.

This setting function option may also include a "resource library default sorting method" function option, through which the electronic device user can select the default sorting method in the resource library for different data saved locally on the electronic device.

The above describes the Chinese translation method provided by the embodiment of the present application from the perspective of an electronic device user. The following describes the Chinese translation method provided by the embodiment of the present application and the implementation process within the electronic device with reference to FIG. 12 .

S1201. The electronic device obtains the text information to be translated.

The text information to be translated may be directly input to the electronic device by the user of the electronic device, or may be recognized by the electronic device based on data such as text, pictures, audio, or video input by the user. For a specific method of obtaining the text information to be translated, please refer to the relevant descriptions in Figures 3 to 5.

In some embodiments, the electronic device also obtains keywords that require additional mouth action data.

S1202. The electronic device sends a translation request to the server, and accordingly, the server receives the translation request.

This translation request is used to request the hand movement data corresponding to the text information to be translated. When the electronic device also obtains a keyword that requires additional oral movement data in S1201, the translation request is also used to request to obtain the oral movement data corresponding to the keyword.

In some embodiments, the translation request is used to request to obtain oral action data corresponding to a keyword that requires additional oral action data.

S1203. The server sends hand movement data and/or mouth movement data, and accordingly, the electronic device receives the hand movement data and/or mouth movement data.

The server determines to send the hand movement data and/or mouth movement data to the electronic device according to the content of the translation request message received in S1202.

Optionally, before sending the above hand movement data and/or mouth movement data to the electronic device, the server first The text information requested by the electronic device to be translated undergoes a text risk control check. The text risk control check is used to check whether the text information to be translated contains sensitive information, so as to filter out bad text information.

In some embodiments, the server directly sends the above-mentioned hand movement data and/or mouth movement data to the electronic device after determining that the text information to be translated passes the text risk control check.

In other embodiments, after the server determines that the text information to be translated passes the text risk control check, the server sends instruction information to the electronic device, and the instruction information is used to indicate that the text information to be translated passes the text risk control check. After receiving the instruction information, the electronic device sends a text-to-sign language request corresponding to the text that passed the text risk control check to the server. After receiving the text-to-sign language request, the server sends the above-mentioned hand movement data and/or mouth to the electronic device. action data.

In some embodiments, if the server determines that the text information to be translated does not pass the text risk control check, the server sends indication information to the electronic device, and the indication information is used to indicate that the text information to be translated does not pass the text risk control check.

The server can determine the hand movement data corresponding to the text information to be translated from the hand movement database, and send the hand movement data to the electronic device.

Similarly, the server can also determine the oral action data corresponding to the keyword from the oral action database, and send the oral action data to the electronic device.

In some embodiments, the server includes a part-of-speech tagging module and a verbal action database. The part-of-speech tagging module is used to tag each word in the text information received from the electronic device with a part-of-speech tag. The specific meaning of the part-of-speech tag is shown in Table 1. The mouth movement database is used to store mixed shape values corresponding to the mouth shapes of Chinese Pinyin. The mixed shape values can be used to display mouth movements corresponding to keywords.

Specifically, the video equipment first records a single pinyin mouth shape video of the model's face, such as the pinyin mouth shape "wu". After recording, the mixed shape value of each frame is saved into the mouth movement database.

Table 2 shows the Chinese pinyin of the mouth movements video that needs to be recorded during the creation of the oral movement database. The oral movements of different Chinese characters are determined based on their corresponding Chinese pinyin. By recording different mouth shape videos corresponding to Chinese Pinyin, the mouth shape videos are then converted into data that can drive the mouth movements of the virtual character. When obtaining keywords that require additional oral movement data, the server can call the mouth shape generation algorithm to obtain the data converted from the mouth shape video of the Chinese Pinyin pronunciation corresponding to the keyword, and send the data to the electronic device, so that the electronic device can The obtained data is used to drive the virtual character to make corresponding mouth movements.

Table 2 Chinese Pinyin

In some embodiments, the server sends the above-mentioned oral movement data and hand movement data together to the electronic device.

In other embodiments, the server sends hand movement data in different time frames in sequence according to the order of hand movements.

In some embodiments, the server sends oral action data in different time frames in sequence according to the order of the oral actions.

In some embodiments, the hand action data and oral action data have the same timestamp, and the server sends the hand action data and oral action data in different time frames in pieces according to the order of the hand movements or oral movements. data.

S1204, drive the virtual character.

The electronic device drives the virtual character model to display the hand movements and/or the mouth movements of the keywords corresponding to the text to be translated based on the hand movement data and/or mouth movement data received in S1203.

After obtaining the translation results corresponding to the text information, the electronic device user can save, share, edit and set the translation results. For the detailed execution process, please refer to the relevant descriptions in Figures 6 to 11. For the sake of brevity, they will not be repeated here.

Based on the same inventive concept, as shown in Figure 13, the embodiment of the present application also provides a Chinese translation device 1300. The Chinese translation device 1300 includes an acquisition unit 1310 and a processing unit 1320. The acquisition unit is used to acquire the data as shown in Figure 3 to Figure 13. For information input by the user of the electronic device in the embodiment shown in Figure 11, the processing unit is used to perform processing operations performed by the electronic device in the embodiment shown in Figure 3 to Figure 11, such as obtaining the corresponding hand according to the text information input by the user. Action data, etc.

Optionally, the Chinese translation device may also include a communication unit 1330, which is used to perform communication and data transmission operations with the server performed by the electronic device in the embodiments shown in Figures 3 to 11.

As shown in Figure 14, the embodiment of the present application also provides another Chinese translation device 1400. The Chinese translation device 1400 It includes a processing unit 1410 and a communication unit 1420. The processing unit is used to perform text risk control operation inspection on the text information sent by the electronic device to be translated. The communication unit is used to perform the embodiments shown in Figures 3 to 11. Communication and data transmission operations performed by servers and electronic devices.

Optionally, the Chinese translation device may also include a storage unit 1430, which is used to store one or more computer programs, hand movement data, oral movement data, etc.

As shown in Figure 15, the embodiment of the present application also provides an electronic device 1500. The electronic device includes a processor 1510 and a memory 1520. The processor is used to execute the steps performed by the electronic device in the embodiments shown in Figures 3 to 11. Processing operations, such as obtaining corresponding hand movement data based on text information input by the user, etc. One or more computer programs are stored in the memory. The one or more computer programs include instructions. When the instructions are processed by one or more When the processor is executed, any of the Chinese translation methods mentioned above will be executed.

As shown in Figure 16, the embodiment of the present application also provides a server 1600. The server includes a processor 1610 and a memory 1620. The processor is used to perform text risk control operations on text information to be translated sent by the electronic device. The The memory stores one or more computer programs, hand movement data, oral movement data, etc. The one or more computer programs include instructions. When the instructions are executed by one or more processors, any of the above A Chinese translation method is implemented.

Embodiments of the present application also provide a computer program product. The computer program product includes computer program code. When the computer program code is run on a computer, it causes the computer to implement the methods in the embodiments shown in FIGS. 3 to 12 .

Embodiments of the present application also provide a computer-readable storage medium. The computer-readable medium stores computer instructions. When the computer instructions are run on the computer, the computer implements the methods in the embodiments shown in Figures 3 to 12. .

An embodiment of the present application also provides a chip, including a processor for reading instructions stored in a memory. When the processor executes the instructions, the chip implements the embodiments shown in Figures 3 to 12. method.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each functional unit can be integrated into one processing unit. Each unit physically exists alone, or two or more units can be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application. should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A method of Chinese translation, which is characterized by including:

In response to the user's input, the electronic device obtains text information, the text information including keywords;

The electronic device displays the hand movements corresponding to the text information;

The electronic device displays the mouth movements corresponding to the keywords.
The method according to claim 1, characterized in that the keywords are determined according to the language habits of sign language users.
The method according to claim 1 or 2, characterized in that the method further includes: the electronic device does not display oral movements corresponding to common words, the text information includes the common words, and the common words are the same as The keywords are different.
The method according to any one of claims 1 to 3, characterized in that the electronic device displays mouth movements corresponding to keywords, including:

The electronic device displays the mouth movement while displaying the hand movement corresponding to the keyword.
The method according to any one of claims 1 to 4, characterized in that the keywords are proper nouns.
The method according to any one of claims 1 to 5, characterized in that before displaying the oral movements corresponding to the keywords, the method further includes:

The electronic device displays a first vocabulary, the text information includes the first vocabulary, and the first vocabulary is a vocabulary that recommends additional oral movements;

In response to the user's confirmation operation, the electronic device determines that the first vocabulary is the keyword.
The method according to any one of claims 1 to 6, characterized in that, before displaying the oral and hand movements corresponding to the keywords, the method further includes:

In response to the user's first input, the electronic device acquires a second vocabulary, the second vocabulary being a vocabulary for which the user requests additional oral movements;

When the text information includes the second vocabulary, the electronic device determines that the second vocabulary is the keyword;

When the text information does not include the second vocabulary, the electronic device displays update request information, and the update request information is used to prompt that the text information does not include the second vocabulary;

In response to the user's second input, the electronic device obtains the updated second vocabulary.
The method of claim 6, wherein the first vocabulary is determined based on the user's translation history, the translation history includes a second vocabulary input by the user, and the second vocabulary is the user's request for additional oral movements. vocabulary.
The method according to any one of claims 1 to 8, characterized in that the mouth movement is determined according to the pronunciation mouth shape of the Chinese pinyin of the keyword.
The method according to claim 9, characterized in that the mixed shape value corresponding to the pronunciation mouth shape is stored in an oral movement database.
The method according to any one of claims 1 to 10, wherein the hand movement includes a first hand movement and a second hand movement, and the first hand movement is performed on the second hand movement. Before partial action, the electronic device Display the hand movements corresponding to the text information, including:

The electronic device receives first hand movement data from the server, and the first hand movement data is used to display the first hand movement;

While displaying the first hand movement, the electronic device receives second hand movement data from the server, and the second hand movement data is used to display the second hand movement.
The method according to any one of claims 1 to 11, wherein the oral movement includes a first oral movement and a second oral movement, and the first oral movement is performed on the second oral movement. Before the oral movements, the electronic device displays the oral movements corresponding to the keywords, including:

The electronic device receives first mouth movement data from the server, and the first mouth movement data is used to display the first mouth movement;

While displaying the first mouth movement, the electronic device receives second mouth movement data from the server, and the second mouth movement data is used to display the second mouth movement.
The method according to any one of claims 1 to 12, characterized in that, before displaying the hand movement corresponding to the text information, the method further includes:

The electronic device receives a response message from the server, where the response message is used to indicate that the text information does not contain sensitive information.
An electronic device, characterized in that it includes a processor and a memory, the memory stores program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 13.
A Chinese translation device, characterized by comprising a module for implementing the method according to any one of claims 1 to 13.
A computer program product, characterized in that the computer program product includes computer program code, and when the computer program code is run on a computer, the method of any one of claims 1 to 13 is executed.
A computer-readable storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed by a computer, the method of any one of claims 1 to 13 is implemented.
A chip product, characterized in that it includes: a processor for reading instructions stored in a memory, and when the processor executes the instructions, the chip implements any one of claims 1 to 13. method described.