WO2018107489A1

WO2018107489A1 - Method and apparatus for assisting people who have hearing and speech impairments and electronic device

Info

Publication number: WO2018107489A1
Application number: PCT/CN2016/110475
Authority: WO
Inventors: 廉士国; 李强
Original assignee: 深圳前海达闼云端智能科技有限公司
Priority date: 2016-12-16
Filing date: 2016-12-16
Publication date: 2018-06-21
Also published as: CN107223277A

Abstract

A method and apparatus for assisting people who have hearing and speech impairments and an electronic device, aimed at assisting people who have hearing and speech impairments in sensing sound in a quick and easy manner. The method comprises: receiving sound (S11); identifying the sound, and converting the sound into a display signal according to an identification result (S12); and performing display under drive of the display signal (S13).

Description

Deaf-mute assistant method, device and electronic device

Technical field

The present invention relates to the field of smart device technologies, and in particular, to a deaf-mute assist method, device, and electronic device.

Background technique

Hearing is an important way for humans to perceive the world. Through hearing, human beings can realize the transmission and feedback of thoughts and feelings between people, and avoid dangerous situations in the environment.

According to the survey, the number of disabled people with hearing disabilities is the highest among the five disabled people, such as visual disability, disability, and disability. There are only about 20 million people with hearing disabilities in China, including many children under the age of seven. The hearing language disabled group has many obstacles in life due to the shortcomings of listening language ability. Therefore, the hearing language disabled group is in need of help. At present, common hearing aids for deaf-mute people include hearing aids or cochlear implants. These devices are helpful for many deaf people, but they also have certain limitations. On the one hand, the different degrees of disability have different requirements on the parameters of the hearing aid or the cochlear implant, and the user needs a complicated selection process when selecting the corresponding product. On the other hand, if the hearing is completely lost before the language and the adult, even if it is restored to hearing by means of cochlear implantation, it is not possible to understand the speech after restoring the hearing, but to conduct language training, and because of the missed In the best language learning period, the language training effect is often not ideal, which leads to great obstacles in communication. In summary, the deaf-mute auxiliary equipment in the prior art has certain limitations. How to assist the deaf-mute person to perceive the sound conveniently and quickly is still a problem that is continuously studied by those skilled in the art.

Summary of the invention

Embodiments of the present invention provide a deaf-mute person assisting method, apparatus, and electronic device, which are mainly used to assist a deaf-mute person to perceive sound conveniently and quickly.

In order to achieve the above object, embodiments of the present invention adopt the following technical solutions:

In a first aspect, a method for assisting a deaf person is provided, comprising:

Receiving sound

Identifying the sound and converting the sound into a display signal according to the recognition result;

Display is performed under the driving of the display signal.

In a second aspect, an AC assist device is provided, including:

a receiving unit, configured to receive a sound;

a converting unit, configured to identify the sound and convert the sound into a display signal according to the recognition result;

a display unit for displaying under the driving of the display signal.

In a third aspect, an electronic device is provided, comprising: a sound collection device, a display device, a memory, and a processor, a sound collection device, a display device, and a memory coupled to the processor; the memory for storing a computer execution code, The computer-executable code is for controlling the processor to perform the deaf-mute assist method of the first aspect.

According to a fourth aspect, a storage medium is provided, characterized in that the computer software instructions for storing the AC assist device of the second aspect are designed to perform the deaf-mute assist method described in the first aspect. code.

In a fifth aspect, a computer program product is provided that can be directly loaded into an internal memory of a computer and includes software code, and the computer program can be loaded and executed by a computer to implement the deaf-mute assist method according to the first aspect. .

The deaf-mute assisting method provided by the embodiment of the present invention first receives a sound, then recognizes the received sound line and converts the sound into a display signal according to the recognition result, and finally displays the display signal, which is implemented by the present invention. The deaf-mute assist method provided by the example can convert the received sound into a display signal and will display the signal The display is driven to convert the received auditory signal into a visual signal, so that the deaf-mute person can visually see the display content corresponding to the sound. Therefore, the deaf-mute assisting method provided by the embodiment of the present invention can be assisted. The dumb person perceives the sound. In addition, compared with the deaf-mute auxiliary device in the prior art, the deaf-mute assisting method provided by the embodiment of the present invention does not require a complicated selection process and does not require language training, and thus the present invention is compared to the prior art. The deaf-mute assist method provided can assist the deaf-mute person to perceive the sound conveniently and quickly.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any creative work.

1 is a flow chart of steps of a deaf-mute assist method according to an embodiment of the present invention;

2 is a second flowchart of steps of a deaf-mute assist method according to an embodiment of the present invention;

3 is a third flowchart of the steps of the deaf-mute assisting method provided by the embodiment of the present invention;

4 is a schematic diagram of a correspondence relationship between a sound orientation and a display position according to an embodiment of the present invention;

FIG. 5 is a fourth flowchart of steps of a deaf-mute assist method according to an embodiment of the present invention; FIG.

FIG. 6 is a schematic structural diagram of a deaf-mute auxiliary device according to an embodiment of the present invention; FIG.

FIG. 7 is a second schematic structural diagram of a deaf-mute auxiliary device according to an embodiment of the present invention; FIG.

FIG. 8 is a third schematic structural diagram of a deaf-mute auxiliary device according to an embodiment of the present invention; FIG.

FIG. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

detailed description

The term "and/or" in this context is merely an association describing the associated object, indicating that there may be three relationships, for example, A and / or B, which may indicate that A exists separately, and both A and B exist, respectively. B these three situations. In addition, the character "/" in this article generally indicates that the contextual object is an "or" relationship. Unless otherwise stated, "multiple" in this document refers to two or more.

It should be noted that, in the embodiments of the present invention, the words "exemplary" or "such as" are used to mean an example, an illustration, or a description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the invention should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the words "exemplary" or "such as" is intended to present the concepts in a particular manner.

It should be noted that, in the embodiments of the present invention, the meaning of "a plurality" means two or more unless otherwise stated.

It should be noted that, in the embodiment of the present invention, "(English: of)", "corresponding (relevant)" and "corresponding" may sometimes be mixed. It should be noted that When the difference is not emphasized, the meaning to be expressed is the same.

The technical solutions provided by the embodiments of the present invention are described below in conjunction with the accompanying drawings of the embodiments of the present invention. It is apparent that some of the embodiments of the invention are described, rather than all of the embodiments. It should be noted that some or all of the technical features of any of the various technical solutions provided below may be combined to form a new technical solution without conflict.

The basic principle of the technical solution provided by the embodiment of the present invention is: identifying the received sound, converting the received sound into a display signal, and displaying the content corresponding to the sound driven by the display signal, thereby making the hoarseness People correspond to sound by watching The visual information perceives the sound.

The execution body of the deaf-mute assisting method provided by the embodiment of the present invention may be a deaf-mute auxiliary device or an electronic device that can be used to execute the deaf-mute assisting method. The deaf-mute auxiliary device may be a combination of a central processing unit (CPU), a CPU and a memory in the electronic device, or may be another control unit or module in the electronic device.

Illustratively, the foregoing electronic device may be a mobile phone, augmented reality glasses (abbreviation: AR glasses), a personal computer (PC), a netbook, a personal digital assistant, which are assisted by the deaf-mute person by using the method provided by the embodiment of the present invention. (English: Personal Digital Assistant, PDA for short), server, etc., or the above electronic device may be a PC, a server, etc., which is installed with a software client or software system or software application that can assist the deaf person, and a specific hardware implementation environment. It can be in the form of a general-purpose computer, or an ASIC, an FPGA, or a programmable extension platform such as Tensilica's Xtensa platform.

Based on the above, an embodiment of the present invention provides a deaf-mute assisting method. Referring specifically to FIG. 1, the deaf-mute assisting method includes the following steps:

S11. Receive sound.

Specifically, the sound in the above embodiment may be a voice that is emitted when another person communicates with the user, a voice that is broadcasted, or the like; or a voice in the environment, such as a sound of a car whistle, a barking voice, The sound of thunder and so on.

In addition, in the above step S11, the sound can be received by a sound sensing device such as a microphone (English name: microphone, abbreviation: Mic) or a Mic array.

S12. Identify the sound and convert the sound into a display signal according to the recognition result.

Specifically, the process of recognizing the sound in the above embodiment and converting the sound into the display signal according to the recognition result may be completed inside the deaf-mute auxiliary device, or may be assisted by the remote service device.

When the process of recognizing the sound and converting the sound into the display signal according to the recognition result is completed inside the deaf-mute auxiliary device, step S12 can be specifically realized by the following steps: a. Identifying the sound by the internal sound processing device, b The sound is converted into a corresponding display signal according to the recognition result of the sound processing device.

When the process of recognizing the sound and converting the sound into the display signal according to the recognition result is completed by the remote service device, step S12 can be specifically implemented by: c. transmitting the sound to the remote server to make a sound with the end server. Identify and convert the sound into a display signal based on the recognition result. d. Receive the display signal sent by the remote server. Exemplarily, the remote service device can be a cloud server or the like.

Optionally, the converting the sound into the display signal in the step S12 may be: converting the sound into at least one of a display signal for displaying characters, a display signal for displaying the logo, and a display signal for displaying the dynamic screen. Kind.

Illustratively, when the received sound is a voice uttered by a person who communicates face-to-face with the user, the received sound can be converted into a display signal for displaying the text. For another example, when the received sound is a barking sound, the received sound can be converted into a display signal for displaying a logo such as a dog's cartoon drawing. For another example, when the received sound is a sound emitted when the car is walking, the received sound can be converted into a display signal for displaying a dynamic picture of the car walking. In addition, in the above embodiments, the received sound can be expressed more clearly by visual information by a combination of various ways. For example, when the received sound is a sound emitted when the car is walking, the received sound can be converted into a display signal for displaying a moving picture of the car and a car logo. Of course, it is also possible to convert the received sound into other types of display signals based on the above embodiments, but these are all reasonable modifications of the embodiments of the present invention, and therefore should all belong to the protection of the embodiments of the present invention. Within the scope.

S13. Displayed under the driving of the display signal.

The specific display manner of displaying the visual information may be selected based on the execution subject of the deaf-mute assisting method provided by the embodiment of the present invention. For example, when the execution body of the deaf-mute assisting method provided by the above embodiment is a mobile phone, displaying under the driving of the display signal may be performed by driving the screen of the mobile phone through the display signal, for example: When the execution body of the deaf-mute assisting method provided by the above embodiment is an AR glasses, displaying the visual information may be performed by projecting the display content on the lens of the AR glasses by driving the projection display device by the display signal.

The deaf-mute assisting method provided by the embodiment of the present invention first receives a sound, then recognizes the received sound line and converts the sound into a display signal according to the recognition result, and finally displays the display signal, which is implemented by the present invention. The deaf-mute assist method provided by the example can convert the received sound into a display signal and display it under the driving of the display signal, that is, the received auditory signal can be converted into a visual signal, thereby enabling the deaf person to pass the vision. The display content corresponding to the sound is seen, so the deaf-mute assisting method provided by the embodiment of the present invention can assist the deaf-mute person to perceive the sound. In addition, compared with the deaf-mute auxiliary device in the prior art, the deaf-mute assisting method provided by the embodiment of the present invention does not require a complicated selection process and does not require language training, and thus the present invention is compared to the prior art. The deaf-mute assist method provided can assist the deaf-mute person to perceive the sound conveniently and quickly.

Optionally, referring to FIG. 2, the sound is recognized in the above step S12, and the sound is converted into a display signal according to the recognition result, which can be specifically implemented by the following steps:

S121. Identify the type of sound.

In step S121, if it is determined that the sound is a speech sound by recognizing the type of the sound, step S122 is performed; and/or, if the sound is determined to be the ambient sound by recognizing the type of the sound, step S123 is executed. That is, step S122 and step S123 in the embodiment of the present invention may be performed either alternatively or alternatively.

It should be noted that the speaking voice in the embodiment of the present invention generally refers to a voice that is emitted by a human when talking, speaking, broadcasting news, and the like. In addition, in some cases, the voice may be received after being processed. For example, the voice of the speaker is amplified and output and received at the time of the lecture. Although such sounds are not spoken directly by humans, such sounds also belong to the voices of the embodiments of the present invention.

It should also be noted that the ambient sound in the embodiment of the present invention is other sounds different from the talking sound, and the received sound is divided into a talking sound and an ambient sound. Specifically, the ring The sound of the environment can be: the sound of the car whistling, the sound of the dog barking, the sound of thunder, the noise in the environment, and so on.

S122. Identify the content of the speech sound, and convert the speech sound into a display signal for driving the display text according to the content of the speech sound.

Optionally, the content of the voice recognition in the foregoing embodiment may be specifically implemented by: e. determining, by using a language type recognition technology, a language type of the received voice, for example, identifying the received voice as Chinese, English, French, etc. . f. Identify the spoken content according to the type of language in which the sound is received and the specific received voice. That is, when the received sound is a voice, the language type of the voice can be recognized first to identify the specific content of the voice.

Since the content of the speech is often complicated, it is difficult to clearly display the corresponding content by means of a logo, a dynamic picture, etc., so in the implementation of the present invention, when the sound is a voice, the sound is converted into a text according to the content of the voice. Thereby the content of the received speech is more clearly displayed.

S123. Identify a category of the ambient sound, and convert the ambient sound into a display signal for driving the display identifier according to the category of the ambient sound.

Exemplarily, the identifier in the above embodiment may specifically be: a cartoon drawing of a dog, a cartoon drawing of a car, a danger sign, a lightning sign, and the like.

Further, the deaf-mute assisting method provided by the above embodiment can assist the deaf-mute person to perceive the speaking sound and various sounds in the environment, but when the user receives the speaking sound in a noisy environment, the received speech may be Contains noise from the environment, which may cause inaccurate content recognition of the voice. The present invention provides a deaf-mute assisting method according to an embodiment of the present invention. Specifically, referring to FIG. 3, the deaf-mute assisting method provided by the embodiment of the present invention is provided on the basis of the deaf-mute assistant shown in FIG. It further includes:

S31. Obtain an image of a relative person.

Among them, the relative person is the person who makes the voice.

Specifically, an image of a relative person may be acquired by one or more of a monocular camera, a binocular camera, a depth camera, an image sensor, and the like. In the embodiment of the present invention, any image capturing device may be used to obtain an image of a relative person. In the embodiment of the present invention, the manner of acquiring the image of the opposite person is not limited, so that the image of the opposite person can be obtained. Moreover, the exemplary, relative person's image may be a dynamic picture when speaking to a person.

S32. Acquire a lip motion of the opposite person according to the image of the relative person.

In the above step S122, converting the speech sound into a display signal for driving the display character according to the content of the speech sound can be realized by the implementation provided in step S33.

S33. Convert the speech sound into a display signal for driving the display text according to the content of the speech sound and the lip motion of the opposite person.

In the above embodiment, by acquiring the image of the relative person, the image of the opposite person is recognized to obtain the lip motion of the opposite person, and then when the sound is the voice, the content of the voice is recognized, and according to the content of the voice and the relative person The lip action converts the content of the speech into a display signal corresponding to the text, and since the lip language recognition technology can interpret the relative words spoken by the relative person's lip motion, the accuracy of the conversion can be improved.

Further, the deaf-mute assisting method provided by the foregoing embodiment further includes:

Obtain the orientation of the sound;

The display in the above step S13 under the driving of the display signal may be specifically implemented by displaying the position of the sound under the driving of the display signal at a corresponding position of the display interface.

Exemplarily, referring to FIG. 4, when the sound is located at the rear F1 of the user, the display content 41 corresponding to the sound is below the display interface 40; when the sound is located at the front of the user F2, the sound is The corresponding display content 42 is displayed above the display interface 40; when the sound is located on the left side of the user F3, the sound and the sound The corresponding display content 43 is displayed on the left side of the display interface 40; when the sound is located on the right side F4 of the user, the display content 44 corresponding to the sound is displayed on the right side of the display interface 40.

According to the orientation of the sound, the display content is displayed on the corresponding position of the display interface, which can further enable the user to know the orientation of the sound, thereby helping the deaf-mute person to perceive the sound more comprehensively.

Deaf people can communicate and communicate with normal people through sign language, but when deaf people face people who do not understand their opponents, communication and communication will not be possible. The embodiment of the present invention further provides a deaf-mute assisting method. Specifically, referring to FIG. 5, the deaf-mute assisting method provided by the embodiment of the present invention includes:

S51. Detecting a user's hand motion.

Optionally, detecting the user's hand motion may specifically: acquiring a dynamic picture of the user by one or more of a monocular camera, a binocular camera, a depth camera, an image sensor, etc., and then acquiring the user according to the dynamic picture of the user. Hand movements. In addition, detecting the user's hand movement can also detect the motion parameters such as the acceleration and the rotation angle of the user's hand through the hand wearing device, and acquire the user's hand motion according to the motion parameter. Among them, the hand wearing device can be: a ring, a wristband, a data glove, and the like.

S52. Identify the user's hand motion and convert the user's hand motion into voice according to the recognition result.

Similarly, the process of recognizing the user's hand motion in the above embodiment and converting the user's hand motion into voice according to the recognition result may be completed inside the deaf-mute auxiliary device, or may be assisted by the remote service device.

When the process of recognizing the user's hand motion and converting the user's hand motion into voice according to the recognition result is completed inside the deaf-mute auxiliary device, step S52 can be specifically implemented by the following steps: A, assisted by the deaf person The image processing device inside the device recognizes the user's hand motion. B converts the user's hand motion into a corresponding voice according to the recognition result of the image processing apparatus.

When the process of recognizing the image and converting the image into the sound information according to the recognition result is completed by the remote service device, the step S52 may be specifically implemented by: sending the image to the remote server, so that the remote server is the user. The hand motion recognizes and converts the user's hand motion into voice based on the recognition result. D. Receive the voice sent by the remote server. Exemplarily, the remote service device can be a cloud server or the like.

S53. Broadcast the voice.

Specifically, the sign language content of the gesture expression can be converted into voice by voice synthesis technology, and the voice is broadcasted through a speaker (English name: Speaker).

In the above embodiment, the sign language content can be converted into a voice and broadcasted, so that the person who does not understand the sign language can learn the content expressed by the sign language deaf and dumb person through the broadcasted voice, thereby further assisting the deaf and mute person to communicate.

The device embodiments corresponding to the method embodiments provided above are provided in the following description of the embodiments of the present invention. It should be noted that the explanation of the related content in the following device embodiments can refer to the foregoing method embodiments.

In the case where the respective functional modules are divided by corresponding functions, FIG. 6 shows a possible structural diagram of the deaf-mute assisting device involved in the above embodiment. Referring to Figure 6, the deaf-mute aid includes:

a receiving unit 61, configured to receive a sound;

The converting unit 62 is configured to identify the sound and convert the sound into a display signal according to the recognition result;

The display unit 63 is configured to perform display under the driving of the display signal.

The deaf-mute auxiliary device provided by the embodiment of the present invention includes: a receiving unit, a converting unit, and a display unit, wherein the receiving unit is configured to receive a sound, and the converting unit is configured to identify the sound and convert the sound into a display signal according to the recognition result, The display unit is configured to display under the driving of the display signal, so the deaf-mute auxiliary device provided by the embodiment of the present invention can convert the received auditory signal into a visual signal, thereby making the deaf person By visually seeing the display content corresponding to the sound, the deaf-mute assisting device provided by the embodiment of the present invention can assist the deaf-mute person to perceive the sound. In addition, the deaf-mute auxiliary device provided by the embodiment of the present invention does not require a complicated selection process and does not need to perform language training, and thus is compared with the prior art embodiment of the present invention. The deaf-mute aids are provided to assist the deaf and mute to sense sound easily and quickly.

Optionally, the converting unit 62 is specifically configured to identify a type of the sound;

The converting unit 62 is specifically configured to recognize the content of the talking sound when the sound is a talking sound, convert the talking sound into a display signal for driving the displayed text according to the content of the talking sound; and/or identify the environment when the sound is an ambient sound The category of the sound, which converts the ambient sound into a display signal for driving the display identifier according to the category of the ambient sound.

Optionally, the receiving unit 61 is further configured to acquire an image of a relative person; wherein, the opposite person is a person who emits a voice;

The converting unit 62 is further configured to acquire a lip motion of the opposite person according to the image of the opposite person;

The converting unit 62 is specifically configured to convert the speaking sound into a display signal for driving the display text according to the content of the speaking sound and the lip motion of the opposite person.

Optionally, the receiving unit 61 is further configured to acquire an orientation of the sound;

The display unit 63 is further configured to perform display on a corresponding position of the display interface under the driving of the display signal according to the orientation of the sound.

Optionally, referring to FIG. 7, the converting unit 62 includes: a sending module 71 and a receiving module 72;

The sending module 71 is configured to send the sound to the remote server to identify the sound with the end server and convert the sound into a display signal according to the recognition result;

The receiving module 72 is configured to receive a display signal sent by the remote server.

Optionally, referring to FIG. 8, the deaf-mute assistant 600 further includes: voice broadcast Reporting unit 64;

The receiving unit 61 is further configured to detect a user's hand motion;

The identification unit 62 is further configured to identify a user's hand motion and convert the user's hand motion into voice according to the recognition result;

The voice broadcast unit 64 is configured to broadcast the voice.

That is, the receiving unit 61 is configured to implement the steps of receiving the sound, acquiring the image of the opposite person, and acquiring the orientation of the sound in the above-described deaf-mute assisting method; the identifying unit 62 is configured to implement the sound recognition in the above-described deaf-mute assisting method And converting the sound into a display signal according to the recognition result, recognizing the type of the sound, recognizing the content of the speech sound, converting the speech sound into a display signal for driving the display text, and identifying the environment sound according to the content of the speech sound, according to The category of the ambient sound converts the ambient sound into a display signal for driving the display mark, acquires the lip motion of the opposite person according to the image of the opposite person, converts the content of the voice according to the content of the voice and the lip motion of the opposite person into the text Corresponding display signal and a step of recognizing the user's hand motion and converting the user's hand motion into voice according to the recognition result; the sending module 71 is configured to implement the sound transmission to the remote server in the deaf-mute assist method The sending module 72 is configured to implement the receiving remote service in the above-mentioned deaf-mute assist method The step of displaying the display signal by the device; the display unit 63 is configured to implement the step of displaying the display under the driving of the display signal and displaying the identification on the corresponding position of the display interface according to the orientation of the ambient sound in the above-mentioned deaf-mute assisting method, the voice broadcast The unit 64 is configured to implement the step of broadcasting the voice in the above-described deaf-mute assist method.

It should be noted that all the related content of the steps involved in the foregoing method embodiments may be referred to the functional description of the corresponding functional modules, and details are not described herein again.

In hardware implementation, the receiving unit 61 may be one or more of a Mic, a Mic array, a camera, an image sensor, an ultrasonic detecting device, an infrared camera, and the like. The identification unit 62 may be a processor or a transceiver; the display unit 63 may be a display screen, a laser projection display device; the voice announcement unit 64 may be a speaker or the like. The programs corresponding to the actions performed by the above-mentioned deaf-mute auxiliary device can be stored in software. The deaf-mute auxiliary device is in the memory, so that the processor calls to perform the operations corresponding to the above respective units.

In the case of employing an integrated unit, FIG. 9 shows a possible structural diagram of an electronic device including the deaf-mute aid device involved in the above embodiment. The electronic device 900 includes a processor 91, a memory 92, a system bus 93, a communication interface 94, a sound collection device 95, and a display device 96.

The processor 91 may be a processor or a collective name of a plurality of processing elements. For example, the processor 91 can be a central processing unit (CPU). The processor 91 can also be other general purpose processors, digital signal processing (DSP), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or Other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like, can implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the present disclosure. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like. The processor 91 may also be a dedicated processor, which may include at least one of a baseband processing chip, a radio frequency processing chip, and the like. The processor can also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like. Further, the dedicated processor may also include a chip having other specialized processing functions of the device.

The memory 92 is used to store computer execution code, and the processor 91 is connected to the memory 92 through the system bus 93. When the electronic device is in operation, the processor 91 is configured to execute the computer execution code stored in the memory 92 to execute any of the embodiments provided by the embodiments of the present invention. A deaf-mute assisting method, for example, the processor 91 is configured to support the electronic device to perform step S12 shown in FIG. 1, steps S121, S122, and 123 shown in FIG. 2, and steps S32 and S33 shown in FIG. Step S52 shown in FIG. 5, and/or other processes for the techniques described herein, the specific deaf-mute assisting method may refer to the related descriptions above and in the drawings, and details are not described herein again.

System bus 93 can include a data bus, a power bus, a control bus, and a signal status bus. In the present embodiment, for the sake of clarity, various buses are shown in FIG. Means system bus 93.

Communication interface 94 may specifically be a transceiver on the device. The transceiver can be a wireless transceiver. For example, the wireless transceiver can be an antenna or the like of the device. The processor 91 communicates with other devices via the communication interface 94, for example, if the device is a module or component of the electronic device, the device is for data interaction with other modules in the electronic device.

The steps of the method described in connection with the present disclosure may be implemented in a hardware manner, or may be implemented by a processor executing software instructions. The embodiment of the present invention further provides a storage medium for storing computer software instructions for use in the electronic device shown in FIG. 9, which includes program code designed to execute the deaf-mute assist method provided by any of the above embodiments. The software instructions may be composed of corresponding software modules, and the software modules may be stored in a random access memory (English: random access memory, abbreviation: RAM), flash memory, read only memory (English: read only memory, abbreviation: ROM) , erasable programmable read-only memory (English: erasable programmable ROM, abbreviation: EPROM), electrically erasable programmable read-only memory (English: electrical EPROM, abbreviation: EEPROM), registers, hard disk, mobile hard disk, CD-ROM (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor to enable the processor to read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. Additionally, the ASIC can be located in a core network interface device. Of course, the processor and the storage medium may also exist as discrete components in the core network interface device.

The embodiment of the invention further provides a computer program product, which can be directly loaded into the internal memory of the computer and contains software code, and the computer program can be loaded and executed by the computer to implement the hoarseness provided by any of the above embodiments. Human assisted method.

Those skilled in the art will appreciate that in one or more examples described above, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored in a computer readable medium or transmitted as one or more instructions or code on a computer readable medium. Computer readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one location to another. A storage medium may be any available media that can be accessed by a general purpose or special purpose computer.

The above is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. All should be covered by the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

A deaf-mute assisting method, characterized in that it comprises:

Receiving sound

Identifying the sound and converting the sound into a display signal according to the recognition result;

Display is performed under the driving of the display signal.
The method according to claim 1, wherein said identifying said sound and converting said sound into a display signal according to the recognition result comprises:

Identifying the type of the sound;

When the sound is a talking sound, identifying the content of the speaking sound, converting the speaking sound into a display signal for driving the displayed text according to the content of the speaking sound; and/or when the sound is an environment In the case of sound, the category of the ambient sound is identified, and the ambient sound is converted into a display signal for driving the display identifier according to the category of the ambient sound.
The method of claim 2, wherein the method further comprises:

Obtaining an image of a relative person; wherein the relative person is a person who makes a voice;

Acquiring the lip motion of the opposite person according to the image of the opposite person;

Converting the voice into a display signal for driving the displayed text according to the content of the voice, including:

The speech sound is converted into a display signal for driving the display text according to the content of the speech sound and the lip motion of the opposite person.
The method of claim 1 further comprising:

Obtaining the orientation of the sound;

The displaying is performed under the driving of the display signal, including:

Displaying is performed at a corresponding position of the display interface under the driving of the display signal according to the orientation of the sound.
The method according to claim 1, wherein the recognizing the sound and converting the sound into a display signal according to the recognition result comprises:

Transmitting the sound to a remote server, so that the end server identifies the sound and converts the sound into a display signal according to the recognition result;

Receiving a display signal sent by the remote server.
The method of claim 1 further comprising:

Detecting the user's hand movements;

Identifying the user's hand motion and converting the user's hand motion into voice according to the recognition result;

The voice is broadcast.
A deaf-mute auxiliary device, comprising:

a receiving unit, configured to receive a sound;

a converting unit, configured to identify the sound and convert the sound into a display signal according to the recognition result;

a display unit for displaying under the driving of the display signal.
The device of claim 7 wherein:

The converting unit is specifically configured to identify the type of the sound;

The converting unit is specifically configured to: when the sound is a speaking sound, identify the content of the speaking sound, convert the speaking sound into a display signal for driving the displayed text according to the content of the speaking sound; and/or And when the sound is an ambient sound, identifying a category of the ambient sound, and converting the ambient sound into a display signal for driving the display identifier according to the category of the ambient sound.
The device according to claim 8, wherein the receiving unit is further configured to acquire an image of a relative person; wherein the relative person is a person who emits a voice;

The converting unit is further configured to acquire a lip motion of the opposite person according to the image of the opposite person;

The converting unit is specifically configured to convert the speaking sound into a display signal for driving the display text according to the content of the speaking sound and the lip motion of the opposite person.
The device of claim 7 wherein:

The receiving unit is further configured to acquire an orientation of the sound;

The display unit is further configured to display at a corresponding position of the display interface under the driving of the display signal according to the orientation of the sound.
The apparatus according to claim 7, wherein the converting unit comprises: a transmitting module and a receiving module;

The sending module is configured to send the sound to a remote server, so that the end server identifies the sound and converts the sound into a display signal according to the recognition result;

The receiving module is configured to receive a display signal sent by the remote server.
The device according to claim 7, wherein the device further comprises: a voice broadcast unit;

The receiving unit is further configured to detect a user's hand motion;

The identification unit is further configured to identify the user's hand motion and convert the user's hand motion into voice according to the recognition result;

The voice broadcast unit is configured to broadcast the voice.
An electronic device, comprising: a sound collection device, a display device, a memory and a processor, a sound collection device, a display device and a memory coupled to the processor; the memory for storing computer execution code, Computer The line code is for controlling the processor to perform the deaf-mute assist method of any of claims 1-6.
A storage medium for storing the computer software instructions for use in the communication assisting device of any one of claims 7 to 12, comprising the deaf-mute person according to any one of claims 1-6 The program code designed by the helper method.
A computer program product, which can be directly loaded into an internal memory of a computer and contains software code, and the computer program can be loaded and executed by a computer to implement the device according to any one of claims 1 to 6. Dumb assist method.