WO2022135254A1

WO2022135254A1 - Text editing method, electronic device and system

Info

Publication number: WO2022135254A1
Application number: PCT/CN2021/138655
Authority: WO
Inventors: 张丽; 唐吴全; 余平
Original assignee: 华为技术有限公司
Priority date: 2020-12-22
Filing date: 2021-12-16
Publication date: 2022-06-30
Also published as: CN114664306A

Abstract

A text editing method, an electronic device (100, 1600) and a system. The method comprises: a device A acquiring audio content (S1201); the device A sending first information to a device B (S1202), wherein the first information is the audio content, or the first information is first text content corresponding to the audio content; the device B displaying the first text content according to the first information (S1203); and the device B displaying second text content in response to an editing operation of a user on the first text content (S1204), wherein the second text content is text content obtained after the first text content is edited (S1205). The present invention is conducive to improving the editing efficiency of a user for text content.

Description

A method, electronic device and system for editing text

This application claims the priority of the Chinese patent application with the application number 202011529621.0 and the application title "A method, electronic device and system for editing text" filed with the China Patent Office on December 22, 2020, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the field of terminals, and more particularly, to a method, electronic device and system for editing text.

Background technique

Speech recognition (automatic speech recognition, ASR) technology has been widely used, which greatly improves the efficiency of people making minutes. However, due to certain conditions, ASR technology cannot be 100% accurate, and it is necessary to manually perform ASR results. It can only be used after manual editing. However, in general, ASR modules are stored on convenient mobile devices such as mobile phones, and there is no screen display and input method that is relatively friendly to human interaction, which makes the editing efficiency of ASR results low.

SUMMARY OF THE INVENTION

The present application provides a method, electronic device and system for editing text, which help to improve the editing efficiency of text editing by users.

In a first aspect, a system is provided, the system includes a first electronic device and a second electronic device, wherein the first electronic device is used to acquire audio content; the first electronic device is further configured to report to the second electronic device Send first information, where the first information is the audio content, or the first information is the first text content corresponding to the audio content; the second electronic device is configured to display the first text content according to the first information ; the second electronic device is further configured to display the second text content in response to the user's editing operation on the first text content, where the second text content is the edited text content of the first text content.

In this embodiment of the present application, the first electronic device can send the acquired text content corresponding to the audio content to the second electronic device, so that the text content can be displayed on the second electronic device, so that it is convenient for the user to display the text content on the second electronic device Edit the text content to help improve the efficiency of the user when editing the text content.

In some possible implementations, the first electronic device may store information of one or more electronic devices, and when the first electronic device acquires audio, the first electronic device may select a suitable one from the one or more electronic devices A second electronic device that performs text editing, thereby sending the first information to the second electronic device.

In some possible implementations, in response to receiving the first information, the second electronic device may open a first application, and display the first text content in the first application; the second electronic device may The input method of the second electronic device edits the first text content.

With reference to the first aspect, in some implementations of the first aspect, the second electronic device is further configured to send the second text content to the first electronic device.

In some possible implementations, the second electronic device is further configured to send the second text content to the first electronic device when the first operation of the user is detected.

In some possible implementations, the first operation is an operation in which the user clicks to save.

In this embodiment of the present application, the second electronic device may acquire the text content edited by the user and send the edited text content to the first electronic device, so that the first electronic device may save the edited text content. The edited text content can also be obtained by the first electronic device without any additional operation by the user.

With reference to the first aspect, in some implementations of the first aspect, the editing operation includes a format modification operation for the first text content, wherein the second electronic device is further configured to send the first electronic device the Format information of the second text content.

In this embodiment of the present application, when the user modifies the format of the text content, the second electronic device may also send the format information of the edited text content to the first electronic device, so that the first electronic device can restore the second electronic device according to the format information. User-edited text content on an electronic device.

With reference to the first aspect, in some implementations of the first aspect, the format information of the second text content includes the font color, font size, font background color, font inclination, font underline and the second text content of the second text content. One or more types of carriage return operations in the text content.

With reference to the first aspect, in some implementations of the first aspect, the first electronic device is further configured to: display the first electronic device according to the audio content before receiving the second text content sent by the second electronic device a text content; after receiving the second text content sent by the second electronic device, replace the first text content with the second text content.

In this embodiment of the present application, the first electronic device may display the corresponding first text content when acquiring the audio content, and after the first electronic device receives the user-edited second text content sent by the second electronic device, the first electronic device The electronic device can replace the previous first text content with the second text content, so that it is convenient for the user to watch the edited text content on the first electronic device and the second electronic device at the same time, which helps to improve the user's experience.

With reference to the first aspect, in some implementations of the first aspect, the first electronic device is further configured to send a query request, and the query request is used for the device that received the query request to determine whether it has a text editing function; in response to receiving the query request To the response sent by the second electronic device, the first information is sent to the second electronic device, where the response is used to indicate that the second electronic device has a text editing function.

In this embodiment of the present application, before sending the first information to the second electronic device, the first electronic device may inquire about a device with a text editing function. After determining that the second electronic device has a text editing function, the first electronic device can send the first information to the second electronic device, so as to facilitate the user to edit the text content on the second electronic device and help improve the user's ability to Efficiency when editing text.

With reference to the first aspect, in some implementations of the first aspect, the second electronic device is specifically configured to: in response to receiving the first information, prompt the user whether to perform text editing on the second electronic device; The user determines to perform a text editing operation on the second electronic device to display the first text content.

In this embodiment of the present application, when the second electronic device receives the first information, it may first prompt the user whether to agree to perform text editing on the second electronic device. If the second electronic device detects the operation agreed by the user, it can display the text content, which helps to avoid interference to the user, and the user can choose a device he thinks is suitable for text editing, which helps to improve the user's experience.

In some possible implementations, the first electronic device may also send request information to the second electronic device, where the request information is used to request the second electronic device to edit the first text content; the second electronic device may respond to The request information prompts the user whether to agree to edit the text content on the second electronic device; if the second electronic device detects that the user agrees to edit the text content on the second electronic device, the second electronic device can display the First text content.

With reference to the first aspect, in some implementations of the first aspect, the first information is the first text content, and the first electronic device is specifically configured to: in response to acquiring the audio content, convert the audio content into the first text content; sending the first text content to the second electronic device.

In this embodiment of the present application, after acquiring the audio content, the first electronic device can convert the audio content into text content, so as to send the text content to the second electronic device, and the second electronic device can display the corresponding text content, thereby facilitating the The user edits the text content.

With reference to the first aspect, in some implementations of the first aspect, the first information is the audio content, and the second electronic device is specifically configured to: in response to receiving the audio content, convert the audio content into the audio content a text content; display the first text content.

In the embodiment of the present application, the first electronic device may send the acquired audio content to the second electronic device, and the second electronic device may convert the audio content into text content and display the text content, thereby facilitating the user to edit the text content .

With reference to the first aspect, in some implementations of the first aspect, the account of the first electronic device is associated with the account of the second electronic device.

In a second aspect, a text editing method is provided, the method is applied to a first electronic device, the method includes: the first electronic device acquires audio content; the first electronic device sends first information to the second electronic device, The first information is the audio content, or the first information is the first text content corresponding to the audio content, so that the second electronic device displays the first text content according to the first information and detects the user's An editing operation for text content.

With reference to the second aspect, in some implementations of the second aspect, the method further includes: receiving, by the first electronic device, second text content sent by the second electronic device, where the second text content Text content after editing the first text content on the electronic device.

With reference to the second aspect, in some implementations of the second aspect, the method further includes: receiving, by the first electronic device, format information of the second text content sent by the second electronic device.

With reference to the second aspect, in some implementations of the second aspect, the format information of the second text content includes the font color, font size, font background color, font inclination, font underline and the second text content of the second text content. One or more types of carriage return operations in the text content.

With reference to the second aspect, in some implementations of the second aspect, the method further includes: before receiving the second text content sent by the second electronic device, the first electronic device displays the first electronic device according to the audio content a text content; after receiving the second text content sent by the second electronic device, the first electronic device replaces the first text content with the second text content.

With reference to the second aspect, in some implementations of the second aspect, the method includes: before sending the first information to the second electronic device, the first electronic device sends a query request, where the query request is used to receive The device of the query request determines whether it has a text editing function; in response to receiving a response sent by the second electronic device, the first electronic device sends the first information to the second electronic device, and the response is used to indicate the second electronic device. Electronic devices have text editing capabilities.

With reference to the second aspect, in some implementations of the second aspect, the first information is the first text content, and the method further includes: in response to acquiring the audio content, the first electronic device converts the audio content is the first text content; the first electronic device sends the first text content to the second electronic device.

With reference to the second aspect, in some implementations of the second aspect, the account of the first electronic device is associated with the account of the second electronic device.

In a third aspect, a text editing method is provided, the method is applied to a second electronic device, and the method includes: the second electronic device receives first information sent by the first electronic device, where the first information is the first information The audio content obtained by the electronic device, or the first information is the first text content corresponding to the audio content; the second electronic device displays the first text content according to the first information; In the content editing operation, the second electronic device displays the second text content, and the second text content is the edited text content of the first text content.

With reference to the third aspect, in some implementations of the third aspect, the method further includes: the second electronic device sending the second text content to the first electronic device.

With reference to the third aspect, in some implementations of the third aspect, the editing operation includes a format modification operation for the first text content, and the method further includes: the second electronic device sends the first electronic device the first electronic device. 2. The format information of the text content.

With reference to the third aspect, in some implementations of the third aspect, the format information of the second text content includes the font color, font size, font background color, font inclination, font underline and the second text content of the second text content. One or more types of carriage return operations in the text content.

With reference to the third aspect, in some implementations of the third aspect, the method further includes: before receiving the first information sent by the first electronic device, receiving, by the second electronic device, a query sent by the first electronic device request, the query requests the second electronic device to determine whether it has a text editing function; the second electronic device sends a response to the first electronic device, where the response is used to indicate that the second electronic device has a text editing function.

With reference to the third aspect, in some implementations of the third aspect, the second electronic device displays the first text content according to the first information, including: in response to receiving the first information, the second electronic device Prompting the user whether to perform text editing on the second electronic device; in response to the user determining that the text editing is performed on the second electronic device, the second electronic device displays the first text content.

With reference to the third aspect, in some implementations of the third aspect, the first information is the audio content, and before the second electronic device displays the first text content, the method further includes: in response to receiving the audio content , and convert the audio content into the first text content.

With reference to the third aspect, in some implementations of the third aspect, the account of the first electronic device is associated with the account of the second electronic device.

In a fourth aspect, an apparatus is provided, the apparatus comprising: an acquiring unit for acquiring audio content; a sending unit for transmitting first information to a second electronic device, where the first information is the audio content, or the The first information is the first text content corresponding to the audio content, so that the second electronic device displays the first text content according to the first information and detects a user's editing operation on the first text content.

In a fifth aspect, an apparatus is provided, the apparatus comprising: a receiving unit configured to receive first information sent by a first electronic device, where the first information is audio content acquired by the first electronic device, or the first information The information is the first text content corresponding to the audio content; the display unit is used to display the first text content according to the first information; the detection unit is used to detect the user's editing operation on the first text content; the display unit, It is also used to display the second text content, where the second text content is the edited text content of the first text content.

In a sixth aspect, an electronic device is provided, comprising: one or more processors; a memory; and one or more computer programs. Wherein, one or more computer programs are stored in the memory, the one or more computer programs comprising instructions. When the instructions are executed by the electronic device, the electronic device is caused to execute the method in any one of the possible implementations of the second aspect above.

In a seventh aspect, an electronic device is provided, comprising: one or more processors; a memory; and one or more computer programs. Wherein, one or more computer programs are stored in the memory, the one or more computer programs comprising instructions. When the instructions are executed by the electronic device, the electronic device is caused to execute the method in any one of the possible implementations of the third aspect above.

In an eighth aspect, there is provided a computer program product containing instructions, when the computer program product is run on a first electronic device, the electronic device causes the electronic device to perform the method described in the second aspect; or, when the computer program product is executed on a first electronic device The computer program product, when running on the second electronic device, causes the electronic device to perform the method of the third aspect above.

In a ninth aspect, a computer-readable storage medium is provided, comprising instructions, when the instructions are executed on a first electronic device, the electronic device causes the electronic device to perform the method described in the second aspect; or, when the instructions are executed When the instruction is executed on the second electronic device, the electronic device is caused to execute the method described in the third aspect.

In a tenth aspect, a chip is provided for executing an instruction, and when the chip is running, the chip executes the method described in the second aspect; or, the chip executes the method described in the third aspect.

Description of drawings

FIG. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

FIG. 2 is a block diagram of a software structure provided by an embodiment of the present application.

FIG. 3 is a set of graphical user interfaces provided by an embodiment of the present application.

FIG. 4 is another set of graphical user interfaces provided by an embodiment of the present application.

FIG. 5 is another set of graphical user interfaces provided by the embodiments of the present application.

FIG. 6 is another set of graphical user interfaces provided by an embodiment of the present application.

FIG. 7 is another set of graphical user interfaces provided by the embodiments of the present application.

FIG. 8 is another set of graphical user interfaces provided by the embodiments of the present application.

FIG. 9 is another set of graphical user interfaces provided by the embodiments of the present application.

FIG. 10 is another set of graphical user interfaces provided by the embodiments of the present application.

FIG. 11 is a schematic diagram of a system architecture provided by an embodiment of the present application.

FIG. 12 is a schematic flow chart of a method for editing text in accordance with an embodiment of the present application.

FIG. 13 is a schematic diagram of another system architecture provided by an embodiment of the present application.

FIG. 14 is a schematic structural diagram of an apparatus provided by an embodiment of the present application.

FIG. 15 is another schematic structural diagram of an apparatus provided by an embodiment of the present application.

FIG. 16 is another schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. Wherein, in the description of the embodiments of the present application, unless otherwise stated, “/” means or means, for example, A/B can mean A or B; “and/or” in this document is only a description of the associated object The association relationship of , indicates that there can be three kinds of relationships, for example, A and/or B, can indicate that A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" or "plurality" refers to two or more than two.

Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

The methods provided in the embodiments of the present application can be applied to mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR)/virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (ultra mobile personal computers) -mobile personal computer, UMPC), netbook, personal digital assistant (personal digital assistant, PDA) and other electronic devices, the embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

Exemplarily, FIG. 1 shows a schematic structural diagram of an electronic device 100 . The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and User identification (subscriber identification module, SIM) card interface 195 and so on. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby increasing the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may couple the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate with each other through the I2C bus interface, so as to realize the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled with the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through a Bluetooth headset.

The PCM interface can also be used for audio communications, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through a CSI interface, so as to realize the photographing function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to implement the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface that conforms to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through the wireless charging coil of the electronic device 100. While the charging management module 140 charges the battery 142 , it can also supply power to the electronic device through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives input from the battery 142 and/or the charging management module 140 and supplies power to the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110 . In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA) and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and then turn it into an electromagnetic wave for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the same device as at least part of the modules of the processor 110 .

The modem processor may include a modulator and a demodulator. Wherein, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and passed to the application processor. The application processor outputs sound signals through audio devices (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or videos through the display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110, and may be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellites System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2 .

In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the electronic device 100 may include one or N display screens 194 , where N is a positive integer greater than one.

The electronic device 100 may implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193 .

The camera 193 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy and so on.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos of various encoding formats, such as: Moving Picture Experts Group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process the input information and can continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing the instructions stored in the internal memory 121 . The internal memory 121 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data (such as audio data, phone book, etc.) created during the use of the electronic device 100 and the like. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playback, recording, etc.

The audio module 170 is used for converting digital audio information into analog audio signal output, and also for converting analog audio input into digital audio signal. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also referred to as "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or a voice message, the voice can be answered by placing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C through a human mouth, and input the sound signal into the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may further be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The earphone jack 170D is used to connect wired earphones. The earphone interface 170D can be the USB interface 130, or can be a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be provided on the display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, and the like. The capacitive pressure sensor may be comprised of at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example, when a touch operation whose intensity is less than the first pressure threshold acts on the short message application icon, the instruction for viewing the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

The gyro sensor 180B may be used to determine the motion attitude of the electronic device 100 . In some embodiments, the angular velocity of electronic device 100 about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device 100 shakes, calculates the distance that the lens module needs to compensate for according to the angle, and allows the lens to counteract the shake of the electronic device 100 through reverse motion to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenarios.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can detect the opening and closing of the flip holster using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Further, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, characteristics such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

Distance sensor 180F for measuring distance. The electronic device 100 can measure the distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light to the outside through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. Proximity light sensor 180G can also be used in holster mode, pocket mode automatically unlocks and locks the screen.

The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket, so as to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, accessing application locks, taking pictures with fingerprints, answering incoming calls with fingerprints, and the like.

The temperature sensor 180J is used to detect the temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

Touch sensor 180K, also called "touch panel". The touch sensor 180K may be disposed on the display screen 194 , and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the location where the display screen 194 is located.

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the pulse of the human body and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined with the bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibration bone block obtained by the bone conduction sensor 180M, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M, so as to realize the function of heart rate detection.

The keys 190 include a power-on key, a volume key, and the like. Keys 190 may be mechanical keys. It can also be a touch key. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .

Motor 191 can generate vibrating cues. The motor 191 can be used for vibrating alerts for incoming calls, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The motor 191 can also correspond to different vibration feedback effects for touch operations on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change of the power, and can also be used to indicate a message, a missed call, a notification, and the like.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be contacted and separated from the electronic device 100 by inserting into the SIM card interface 195 or pulling out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as call and data communication. In some embodiments, the electronic device 100 adopts an embedded SIM (embedded-SIM, eSIM) card, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

It should be understood that the telephone cards in the embodiments of the present application include but are not limited to SIM cards, eSIM cards, universal subscriber identity modules (USIM), universal integrated telephone cards (universal integrated circuit cards, UICC) and the like.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take an Android system with a layered architecture as an example to exemplarily describe the software structure of the electronic device 100 .

FIG. 2 is a block diagram of the software structure of the electronic device 100 according to the embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, and a kernel layer. The application layer can include a series of application packages.

As shown in Figure 2, the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and so on.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer may include window managers, content providers, view systems, telephony managers, resource managers, notification managers, and the like.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

Content providers are used to store and retrieve data and make these data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone book, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. View systems can be used to build applications. A display interface can consist of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide the communication function of the electronic device 100 . For example, the management of call status (including connecting, hanging up, etc.).

The resource manager provides various resources for the application, such as localization strings, icons, pictures, layout files, video files and so on.

The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages, and can disappear automatically after a brief pause without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also display notifications in the status bar at the top of the system in the form of graphs or scroll bar text, such as notifications of applications running in the background, and notifications on the screen in the form of dialog windows. For example, text information is prompted in the status bar, a prompt sound is issued, the electronic device vibrates, and the indicator light flashes.

Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one is the function functions that the java language needs to call, and the other is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, safety and exception management, and garbage collection.

A system library can include multiple functional modules. For example: surface manager (surface manager), media library (media library), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The Surface Manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, camera drivers, audio drivers, and sensor drivers.

It should be understood that the technical solutions in the embodiments of the present application can be used in systems such as Android, IOS, and Hongmeng.

FIG. 3 is a set of graphical user interfaces (graphical user interface, GUI) provided by the embodiments of the present application.

Referring to the GUI shown in (a) of FIG. 3 , the GUI is the desktop of the mobile phone. Among them, the desktop of the mobile phone includes icons of multiple applications, the multiple applications include clock, calendar, gallery, memo, file management, e-mail, music, calculator and so on. When the mobile phone detects that the user clicks the icon of the memo, the GUI as shown in (b) of FIG. 3 may be displayed.

Referring to the GUI shown in (b) of FIG. 3 , the GUI is the display interface of the memo application. A note recorded on September 1 is displayed on the display interface, and the content of the note is "Mom's birthday (September 28)". The display interface also includes a control 301 for creating a new note. When the mobile phone detects that the user clicks on the control 301, the mobile phone can display a GUI as shown in (c) of FIG. 3 .

Referring to the GUI shown in (c) of FIG. 3 , the GUI is another display interface of the memo application. Users can edit the title and body content of the note on this interface. The display interface also includes controls for list, text editing, inserting pictures, audio-to-text, and handwriting controls.

Referring to (d) in FIG. 3 , when the mobile phone detects that the user clicks the operation of the recording-to-text control 302, the mobile phone can send request information to the notebook computer, and the request information is used to request the notebook computer to correspond to the voice content acquired by the mobile phone. Edit for text content. In response to receiving the request information, the laptop computer can display a reminder box 303, wherein the reminder box 303 includes the prompt information "it is detected that the mobile phone is recording to text, whether to perform text editing on the laptop computer".

Referring to (e) of FIG. 3 , the notebook computer can start App1 in response to the user's operation of clicking on the control 304 . Where App1 is an application that can do text editing. In response to the user's operation of clicking on the control 302, the mobile phone can detect the voice content in the environment, and convert the detected voice content into text content. After the mobile phone converts the voice content into text content, the mobile phone can also send the converted text content to the laptop. Thereby, the display interface of the memo application of the mobile phone and the display interface of App1 of the notebook computer are simultaneously displayed with the text content "the popularity changes again... social mentality".

In one embodiment, the mobile phone may include a microphone and a speech recognition (automatic speech recognition, ASR) module, wherein the microphone is used to collect speech content in the environment, and the role of the ASR module is to convert the received speech content into text content.

In one embodiment, the text content on the display interface of App1 of the notebook computer and the display interface of the memo application of the mobile phone may be synchronized in real time. Exemplarily, the mobile phone can convert the voice content into text content at a certain time interval (for example, 5 seconds). In 0-5 seconds, the mobile phone converts the collected voice content into text content, "reflecting the demands of people's livelihood while the popularity changes and remains unchanged", so that the mobile phone can display the text content on the display interface of the memo, and the mobile phone can display the text content at the same time. The content is sent to the laptop, so that the laptop can display the text content on the display interface of App1; in 5-10 seconds, the mobile phone converts the collected voice content into text content. "Guangzhi Think Tank, we have just counted the eve of the 2020 National Two Sessions. ”, so that the mobile phone can display the text content on the display interface of the memo, and at the same time, the mobile phone can send the text content to the notebook computer, so that the notebook computer can display the text content on the display interface of App1.

In one embodiment, the laptop computer can display the cursor 306 after opening App1, wherein after the laptop computer receives the text content sent by the mobile phone, the laptop computer can display the cursor 306 at the back of the text content.

Referring to the GUI shown in (f) of Fig. 3, the mobile phone can continue to detect the voice content and convert the voice content into text content. The mobile phone can synchronize the text content to the App1 of the laptop in real time. At the same time, the notebook computer can edit the text content in App1 in response to the user's editing operation. For example, users can add content to the text content sent by the mobile phone (add "1," before "Hotness", and "2," before "Guangzhi Think Tank"). For another example, the user can modify the inaccurate content of the text (for example, "name" is modified to "public opinion", and "breakthrough" is modified to "top10"). For another example, the user can also modify the format of the text content (for example, the laptop detects the user's carriage return operation after the "reputation appeal").

As shown in (f) in FIG. 3 , in response to the notebook computer detecting that the user clicks the save control 307, the notebook computer can send the edited text content to the mobile phone, so that the mobile phone displays the notebook computer on the display interface of the memo. Edited text content.

In one embodiment, the mobile phone converts the voice content into text content after detecting the voice content in the environment, and sends the text content to the laptop computer, while the mobile phone may not display the text content. After the mobile phone receives the edited text content from the laptop computer, the mobile phone can display the edited text content on the display interface of the memo application.

In one embodiment, after detecting that the user moves the cursor to the position where editing is required, the notebook computer can edit the text content near the position. For example, when the current cursor 306 is behind "Social Mindset", when the laptop detects that the user has added a symbol "?" after "Social Mindset", the cursor 306 can move to the back of "Social Mindset?". At the same time, after the notebook computer receives other text content sent by the mobile phone, it can continue to display the received text content.

Referring to (g) in FIG. 3 , when the notebook computer continues to detect the user's editing operation on the text content, it can edit the text content received from the mobile phone. Exemplarily, when the notebook computer can detect the user's operation of changing "positive success" to "Zheng Chenggong"; for another example, the notebook computer can detect the user's operation of adding a symbol "." after "annual feature"; another example, The laptop can detect the user inserting the symbol "," in "Medical Education Housing Food Safety". As another example, a laptop can detect the user's press enter after "Social Mindset?".

As shown in (g) of FIG. 3 , when the notebook computer detects that the user clicks the save control 307, the notebook computer can send the edited text content to the mobile phone, so that the mobile phone displays the notebook computer on the display interface of the memo Edited text content.

Referring to the GUI shown in (h) of FIG. 3 , the GUI is another display interface of the memo application. When the mobile phone detects that the user clicks the control 308 to stop recording, the mobile phone can display a GUI as shown in (i) in FIG. 3 .

Referring to the GUI shown in (i) of FIG. 3 , the GUI is another display interface of the memo application. The user can continue to edit the text content edited by the laptop on the mobile phone (for example, the user can insert a picture into the text content). When the mobile phone detects that the user clicks on the control 309, the mobile phone can save the text content displayed on the current interface and display the GUI shown in (j) in FIG. 3 .

Referring to the GUI shown in (j) of FIG. 3 , the GUI is another display interface of the memo application. In response to detecting that the user clicks on the control 309, the mobile phone can update the title of the note to "1, hotspot". When the mobile phone detects that the user clicks on the control 310, the mobile phone can display the GUI as shown in (k) in Figure 3.

Referring to the GUI shown in (k) of FIG. 3 , the GUI is another display interface of the memo application. The mobile phone can display 2 notes included in the current memo application, and the newly added note content can be displayed at the top.

In one embodiment, if the laptop computer does not detect that the user clicks on the control 304 when the mobile phone starts to convert speech to text, the mobile phone can start to convert audio to text, and the laptop computer can continue to display the prompt box 301 . During the process of the mobile phone converting audio to text, if the notebook computer detects that the user clicks on the control 304, the notebook computer can send a response to the mobile phone, where the response is used to indicate that the notebook computer can perform text editing. After receiving the response, the mobile phone can send all the text content 1 that is voice-to-text before receiving the response to the laptop, and the laptop can open App1 and display the text content 1. Afterwards, if the mobile phone continues to detect the voice content input by the user, the mobile phone can continue to send the text content 2 corresponding to the voice content input by the user to the notebook computer. The notebook computer can append the text content 2 sent by the mobile phone to the back of the text content 1.

In the embodiment of the present application, when the mobile phone performs the recording-to-text operation, the mobile phone can notify the notebook computer that the recording-to-text function is being performed, so that the notebook computer can prompt the user whether to edit the text content on the notebook computer. When the user chooses to edit on the laptop computer, the laptop computer can display the text content sent by the mobile phone in real time, thereby facilitating the user to edit the text content and helping to improve the user experience.

FIG. 4 is another set of GUIs provided by the embodiments of the present application.

Referring to (a) in FIG. 4 , when the mobile phone detects that the user clicks the recording-to-text control, the mobile phone may display the interface shown in (b) in FIG. 4 . In this case, the mobile phone may send request information to the notebook computer, where the request information is used to request the notebook computer to edit the text content corresponding to the voice content obtained by the mobile phone.

Referring to (b) in FIG. 4 , in response to receiving the request information sent by the mobile phone, the notebook computer can start App1. In response to the mobile phone receiving the laptop computer's response to the request information, the mobile phone may display a reminder box 401, wherein the reminder box 401 includes prompt information "detecting the surrounding laptop computer, whether to send the text content to the laptop computer for editing". When the mobile phone detects that the user clicks the control 402, the mobile phone starts to detect the voice content in the environment.

Referring to (c) in FIG. 4 , after detecting the voice content in the environment, the mobile phone can start to convert the recording into text. The mobile phone can collect the voice content and convert the voice content into text content to "change the popularity...social mentality", and the mobile phone can also send the converted text content to the laptop, so that the laptop can display the text content through App1.

In one embodiment, when the notebook computer detects that the user edits the text content on the notebook computer and detects that the user clicks the save operation, the notebook computer can send the edited text content to the mobile phone, so that the mobile phone can The edited text content is displayed on the display interface of the memo. For specific processes, reference may be made to processes (f) to (g) in FIG. 3 , which are not repeated here for brevity.

In the embodiment of the present application, when the mobile phone performs the recording-to-text operation, if it is determined that there is a device (eg, a laptop) that is convenient for the user to edit text, the mobile phone can prompt the user whether to perform text editing on the laptop. When the user chooses to edit on the laptop computer, the laptop computer can display the text content sent by the mobile phone in real time, thereby facilitating the user to edit the text content and helping to improve the user experience.

FIG. 5 is another set of GUIs provided by the embodiments of the present application.

Referring to the GUI shown in (a) of FIG. 5 , the GUI is the desktop of the mobile phone. Wherein, the desktop of the mobile phone includes icons of multiple applications, and the multiple applications include a voice recorder application and the like. When the mobile phone detects that the user clicks on the icon of the audio recorder application, the GUI as shown in (b) of FIG. 5 can be displayed.

Referring to the GUI shown in (b) of FIG. 5 , the GUI is the display interface of the recorder application. A recording file recorded on October 3 is displayed on the display interface, and the recording file is named "20201003_100832". The display interface also includes a control 501 for creating a new recording file.

Referring to (c) in Figure 5, when the mobile phone detects that the user clicks on the control 501, the mobile phone can send instruction information to the notebook computer, where the instruction information is used to indicate that the mobile phone is recording. In response to receiving the instruction information, the laptop computer can display a reminder box 502, wherein the reminder box 502 includes a prompt message "It is detected that the mobile phone is recording, do you want to convert the recording to text on the laptop computer?".

Referring to (d) of FIG. 5 , in response to the notebook computer detecting that the user clicks on the control 503, the notebook computer can launch App1. In response to the mobile phone detecting that the user clicks on the control 501, the mobile phone can display a recording interface, wherein the recording interface shows that 25 seconds have been recorded and the mobile phone is expected to record more than 10 hours.

When the mobile phone collects the voice content, it can also send the voice content to the laptop computer. After the laptop computer receives the voice content, it can convert the voice content into text content. The text content is displayed in App1.

In one embodiment, an ASR module may be included in the notebook computer, wherein the function of the ASR module is to convert the received voice content into text content.

As shown in (e) of FIG. 5 , the notebook computer can edit the text content in App1 in response to the user's editing operation. For example, users can add content to the text content sent by the mobile phone (add "1," before "Hotness", and "2," before "Guangzhi Think Tank"). For another example, the user can modify the inaccurate content of the text (for example, "name" is modified to "public opinion", and "breakthrough" is modified to "top10"). For another example, the user can also modify the format of the text content (for example, the laptop detects the user's carriage return operation after the "reputation appeal").

In one embodiment, when the notebook computer detects that the user clicks the save control, it can send the edited text content to the mobile phone, so that the mobile phone can save the edited text content in a certain application layer sequence (eg, memo).

In the embodiment of the present application, when the mobile phone detects that the user starts recording, the mobile phone can send instruction information to the surrounding notebook computers, so that the notebook computer prompts the user whether the recording needs to be converted into text on the notebook computer. In this way, it is convenient for the user to convert the voice content collected by the mobile phone to the text content on the laptop computer, and at the same time, the convenience of editing by the laptop computer is utilized, which helps to improve the user experience.

FIG. 6 is another set of GUIs provided by the embodiments of the present application.

Referring to (a) of FIG. 6 , the mobile phone displays a caller ID interface, and the caller ID interface shows that the mobile phone number "XXX-XXXXXXXX" from Shenzhen, Guangdong is making a call. In response to the mobile phone receiving the incoming call, the mobile phone sends indication information to the surrounding notebook computers, where the indication information is used to instruct the mobile phone to detect the user's incoming call. In response to receiving the instruction information, the laptop computer can display a reminder box 601, wherein the reminder box 601 includes the prompt information "The mobile phone has received an incoming call, do you want to convert the content of the call to text on the laptop computer?".

Referring to (b) in FIG. 6 , in response to the notebook computer detecting that the user clicks on the control 602, the notebook computer can open App1 and send a response to the mobile phone, which is used to indicate that the notebook computer can be used as a text content editing device. In response to the mobile phone detecting the operation of the user accepting the incoming call, the mobile phone can establish a call connection to the terminal device. The mobile phone can receive the voice content of another user sent by the peer device. In response to receiving another user's voice content, the cell phone can send the voice content to the laptop.

In one embodiment, the mobile phone may also send indication information to the notebook computer when detecting the operation of the user accepting the incoming call, the indication information is used to indicate that the mobile phone is on a call, and request the notebook computer to edit the text content corresponding to the call content. . After receiving the instruction information, the laptop can prompt the user "The mobile phone is on a call, do you want to convert the content of the call to text on the laptop?". When the notebook computer detects that the user determines to use the notebook computer to convert the call content to text, the notebook computer can open App1 and receive the voice content of another user from the mobile phone. Thus, the notebook computer can convert the voice content into text content.

In one embodiment, after receiving the response, the mobile phone can convert the acquired voice content of another user into text content, so that the mobile phone can send the text content to the laptop.

Referring to (c) in FIG. 6 , in response to receiving the voice content from the mobile phone, the notebook computer can convert the voice content into text content "Hi, there is a meeting tomorrow morning...please attend on time", thereby displaying through App1 the text content.

In this embodiment of the present application, when the mobile phone detects an incoming call, if it is determined that there is a device (for example, a laptop computer) that is convenient for the user to edit text nearby, the mobile phone can indicate to the laptop computer that the mobile phone has detected an incoming call. When the user chooses to edit on the laptop, the laptop can convert the voice content obtained from the mobile phone into text content in real time and display it to the user, so as to facilitate the user to edit the text content and help improve the user experience.

FIG. 7 is another set of GUIs provided by the embodiments of the present application.

Referring to (a) in FIG. 7 , the mobile phone displays a video call interface of a social application, and the video call interface shows that another user is inviting the user to make a video call. In response to the mobile phone receiving the video call, the mobile phone sends indication information to the surrounding notebook computers, where the indication information is used to instruct the mobile phone to detect another user's video call request. In response to receiving the instruction information, the laptop computer may display a prompt box 701, wherein the prompt box 701 includes prompt information "The mobile phone has received a video call request, whether to convert the call content to text on the laptop computer?".

Referring to (b) of FIG. 7 , in response to the notebook computer detecting that the user clicks on the control 702, the notebook computer can open App1 and send a response to the mobile phone, which is used to indicate that the notebook computer can be used as a text content editing device. In response to the mobile phone detecting the user's operation of accepting the video call request, the mobile phone can establish a video call connection to the terminal device. The mobile phone can receive the voice content of another user sent by the peer device. In response to receiving another user's voice content, the cell phone can send the voice content to the laptop.

Referring to (c) of FIG. 7 , in response to receiving the voice content from the mobile phone, the notebook computer can convert the voice content into text content "Hi, there is a meeting tomorrow morning...please attend on time", thereby displaying through App1 the text content.

In this embodiment of the present application, when the mobile phone detects a video call, if it is determined that there is a device (for example, a laptop computer) that is convenient for the user to edit text nearby, the mobile phone can indicate to the laptop computer that the mobile phone detects an incoming call. When the user chooses to edit on the laptop, the laptop can convert the voice content obtained from the mobile phone into text content in real time and display it to the user, so as to facilitate the user to edit the text content and help improve the user experience.

It should be understood that when the mobile phone receives an incoming call and a video call request, the mobile phone indicates the surrounding laptops, so that the laptop prompts the user whether to convert the call content to text on the laptop. In this embodiment of the present application, the mobile phone may also instruct the laptop computer after receiving a voice call request from another user, so that the laptop computer prompts the user whether to convert the call content to text on the laptop computer.

FIG. 8 is another set of GUIs provided by the embodiments of the present application.

Referring to (a) of FIG. 8 , the mobile phone displays a caller ID interface, and the caller ID interface displays that the mobile phone number "XXX-XXXXXXXX" from Shenzhen, Guangdong is making a call. In response to the mobile phone receiving the incoming call, the mobile phone sends indication information to the surrounding notebook computers, where the indication information is used to instruct the mobile phone to detect the user's incoming call. In response to receiving the instruction information, the laptop computer can display a reminder box 801, wherein the reminder box 801 includes the prompt information "The mobile phone has received an incoming call, do you want to convert the content of the call to text on the laptop computer?".

Referring to (b) in FIG. 8 , when the notebook computer detects that the user does not click on the cancel control or the confirmation control 802 within a preset time period, the reminder box 801 can be hidden. In response to the mobile phone detecting the operation of the user accepting the incoming call, the mobile phone can establish a call connection to the terminal device.

Referring to (c) in FIG. 8 , in the process of whether the mobile phone is making a call, the user detects that the user clicks the operation of the recording control 803, and can send instruction information to the notebook computer, and the instruction information is used to indicate that the mobile phone is making a call, Request the laptop to edit the text content corresponding to the call content. In response to receiving the indication information, the notebook computer may display the reminder box 801 again. When the laptop computer detects that the user clicks on the control 802, the laptop computer can send a response to the mobile phone, where the response is used to instruct the laptop computer to edit the text content corresponding to the call content. In response to receiving the response, the cell phone can send the call content to the laptop.

Referring to (d) in FIG. 8 , in response to receiving the call content sent by the mobile phone, the ASR module of the notebook computer can convert the call content into text content “Hi, tomorrow morning… join” and display it in App1.

Referring to (e) in Figure 8, the laptop detects the user's editing operation, can edit the text content, and displays the edited text content in App1 ("Leader: Hi, tomorrow morning... see").

In the embodiment of the present application, when the mobile phone detects that the user clicks the recording operation during the process of answering the call, the mobile phone sends instruction information to the notebook computer, thereby instructing the notebook computer to edit the text content corresponding to the call content. When the user chooses to edit on the laptop, the laptop can convert the voice content obtained from the mobile phone into text content in real time and display it to the user, so as to facilitate the user to edit the text content and help improve the user experience.

The other two display modes on the notebook computer are described below with reference to FIG. 9 and FIG. 10 .

FIG. 9 is another set of GUIs provided by the embodiments of the present application.

As shown in (a) in Figure 9, the mobile phone can display the recording-to-text interface in the memo, and the mobile phone detects the voice content input by the user "reflecting the people's livelihood demands between the change of popularity and the constant change", and the mobile phone can convert the voice content to the voice content. Convert to text content and send that text content to the laptop. After receiving the text content, the notebook computer can display the text content to the user by means of subtitles. Exemplarily, the notebook computer can display the text content "reflecting the demands of people's livelihood between the change of popularity and the constant change" in the subtitle display area 901 .

As shown in (b) of Figure 9, when the mobile phone detects the voice content "We have just obtained statistics from the Guangzhi Think Tank" input by the user, the mobile phone can convert the voice content into text content and send the text content to the laptop. After receiving the text content, the notebook computer can display the text content to the user by means of subtitles. Exemplarily, the notebook computer can display the text content "We have just obtained statistics on Guangzhi Think Tank" in the subtitle display area 901 .

FIG. 10 is another set of GUIs provided by the embodiments of the present application.

As shown in (a) in Figure 10, the mobile phone can display the recording-to-text interface in the memo, and the mobile phone detects the voice content input by the user "reflecting the people's livelihood demands between the hotness and the constant change", and the mobile phone can convert the voice content to the voice content. Convert to text content and send that text content to the laptop. After receiving the text content, the notebook computer can display the text content to the user by means of subtitles, and at the same time, the notebook computer can also open App1 and display the text content in App1. Exemplarily, the laptop computer can display the text content "Reflecting the demands of people's livelihood between the change of popularity and the constant change" in the subtitle display area 1001, and simultaneously display the text content in App1. The user can edit the text content in App1.

As shown in (b) of Figure 10, when the mobile phone detects the voice content "We have just obtained statistics from the Guangzhi Think Tank" input by the user, the mobile phone can convert the voice content into text content and send the text content to the laptop. After the notebook computer receives the text content, it can display the text content to the user through subtitles. At the same time, the notebook computer can also display the text in App1 after "reflecting the demands of people's livelihood between the hotness and the constant change". content. Exemplarily, the notebook computer can display the text content "We have just obtained the statistics of Guangzhi Think Tank" in the subtitle display area 1001, and display in App1 "The demand for people's livelihood is reflected in the change of popularity and the constant change, we have just calculated the Guangzhi Think Tank. get".

The following describes the system architecture provided by the embodiments of the present application in combination with FIG. 11 . The system architecture includes device A and device B, where device A may be a device with a pickup function (eg, a mobile phone), and device B may be a device with an efficient editing function (eg, a laptop). Device A includes a pickup module 1110 , an ASR module 1120 , a transceiver control module 1130 and a replacement module 1140 ; device B includes a transceiver control module 1150 , a display unit 1160 and an editing control module 1170 .

The sound pickup module 1110 is used to acquire the voice content to be processed. In the embodiments of the present application, the manner in which the device A acquires the voice content is not specifically limited, and the acquisition manner may be various. For example, it can be real-time voice recorded from the surrounding environment of device A; or, it can also be the audio of the audio and video played by the user on the mobile phone; or, it can also be the audio (including remote and Near-end); it can also be the audio and video files in the mobile phone. The pickup module is responsible for converting the relevant audio into a specific audio format, such as a pulse code modulation (PCM) audio stream of a certain sampling rate, which is used as an input to the ASR module.

The ASR module 1120 may convert the speech content to textual content. By inputting a specific PCM audio stream to the ASR module 1120, the deep neural network acoustic model is trained to obtain the phoneme sequence with the highest probability, and then through the corresponding language model, the text series with the highest probability is obtained, completing the process from speech content to text. content conversion.

The transceiver control module 1130 is used to control information synchronization between device A and other devices. In this embodiment of the present application, the transceiver control module 1130 may be responsible for transparent transmission of information, and the transceiver control module 1130 is not involved in the logical processing of the transmission content. The transmission of information may be through a local area network, such as Bluetooth, Wi-Fi, or other transmission protocols (eg, the internet). Device A establishes a connection network link with other devices (eg, device B) through a network communication protocol, and then initiates a start transmission.

It should be understood that for the functional description of the transceiver control module 1150, reference may be made to the aforementioned transceiver control module 1130, which is not repeated here for brevity.

The display unit 1160 is used to prompt the user whether to start text editing on the device B. The transceiver control module 1150 receives the instruction sent by the transceiver control module 1130 for device A to acquire audio content (eg, voice content or audio files), or, the transceiver control module 1150 receives the instruction sent by the transceiver control module 1130 to perform text editing on device B. After the instruction is displayed, a prompt can be popped up in the form of a notification through the display unit 1160 . If the device B detects that the user determines to perform text editing through the device B, it can send a response to the transceiver control module 1130 through the transceiver control module 1150, and the response can be used to indicate that the device B can be used as a text editing device. After the transceiver control module 1130 of device A receives the response, device A starts to transmit the ASR result to device B in real time.

In one embodiment, after the device A detects the acquisition of audio content (eg, voice content or audio files), the transceiver control module 1130 may send a query request, where the query request is used to query surrounding devices with text editing functions.

Exemplarily, if device A and device B are devices under the same account, device A may store information such as device type, device name, and MAC address of device B. When device A detects the acquisition of voice content, device A can send BLE data packets to device B according to the MAC address of device B. The BLE data packet may include a PDU, and the query request may be carried in a service data field in the PDU, or may also be carried in a vendor-specific data field in the PDU. For example, the payload of the service data field may include multiple bits, wherein the multiple bits include scalable bits. Device A and Device B can agree on the content of a certain extensible bit. When a certain extensible bit is 1, device B can obtain device A and query whether device B has the text editing function.

For example, as shown in (c) of FIG. 5 , when the mobile phone detects that the user clicks on the control 501 (the control 501 is the control for starting recording), the mobile phone can send a BLE data packet, and the BLE data packet can carry the query ask.

In one embodiment, when device A detects that the user performs an operation of converting speech to text, device A can send a BLE data packet to device B according to the MAC address of device B.

For example, as shown in (d) of FIG. 3 , when the mobile phone detects that the user clicks on the control 302 (the control 302 is a control for the mobile phone to perform voice-to-text conversion), the mobile phone can send a BLE data packet, and the BLE data packet can carry the query request.

The transceiver control module of device B can call the interface for querying the text editing function (for example, the content provider interface) to send a request to one or more applications in the application layer, and the request is used to request the application to determine whether it has the text editing function; If an application program has a text editing function, the application program can send a response to the data synchronization module, and the response is used to indicate that app2 has logged into the account. Therefore, device B can determine that device B has installed and logged in app2.

After device B determines that it has the text editing function, device B can send a response to device A, and the response can be carried in a BLE data packet. The BLE data packet may include a PDU, and the query request may be carried in a service data field in the PDU, or may also be carried in a vendor-specific data field in the PDU. For example, the payload of the service data field may include multiple bits, wherein the multiple bits include scalable bits. Device A and Device B can agree on the content of a certain extensible bit. When a certain extensible bit is 1, device A can obtain the text clipping function of device B.

After device A determines that device B has the text editing function, device A can send the acquired text content and instruction information corresponding to the audio content to device B, where the instruction information is used to instruct device B to edit the text content.

Exemplarily, device A may send a BLE data packet to device B according to the MAC address of device B. The BLE data packet may include a PDU, and the text content and indication information may be carried in a service data field in the PDU, or may also be carried in a vendor-specific data field in the PDU. For example, the payload of the service data field may include multiple bits, wherein the multiple bits include scalable bits. Device A and Device B can agree on the content of a certain extensible bit. Device A can use encoding methods such as GBK, ISO8859-1 or Unicode (for example, UTF-8, UTF-16) to encode the text content output by the ASR module of device A, and carry the encoded information in an or on multiple scalable bits. Device A may also set an extensible bit to 1. After receiving the BLE data packet, the device B can obtain the text content and the indication information by decoding, so that the device B can display the text content according to the indication information.

Alternatively, after device A determines that device B has a text editing function, device A may send text content corresponding to the acquired audio content to device B.

Exemplarily, device A may send a BLE data packet to device B according to the MAC address of device B. The BLE data packet may include a PDU, and the text content may be carried in a service data field in the PDU, or may also be carried in a vendor-specific data field in the PDU. For example, the payload of the service data field may include multiple bits, wherein the multiple bits include scalable bits. Device A and Device B can agree on the content of a certain extensible bit. Device A can use encoding methods such as GBK, ISO8859-1 or Unicode (for example, UTF-8, UTF-16) to encode the text content output by the ASR module of device A, and carry the encoded information in an or on multiple scalable bits. After receiving the BLE data packet, device B can obtain the text content by decoding, so that device B can display the text content.

In one embodiment, device A and device B are devices under the same account, then device A can store information such as device type, device name, MAC address, and whether it has a text editing function of device B. Then, when device A acquires the audio content, device A can send the text content and indication information corresponding to the audio content to device B, where the indication information is used to instruct device B to edit the text content. In response to receiving the text content and the indication information, the device B may prompt the user whether to perform text editing on the device B through the display unit 1160 . In response to detecting that the user performs text editing on the device B, the device B can open an application that can be used for text editing, so as to display the text content obtained from the device A.

Alternatively, device A may send the text content corresponding to the audio content to device B. In response to receiving the text content, the device B can edit the text content on the device B, so that the device B can prompt the user whether to perform text editing on the device B through the display unit 1160 . In response to detecting that the user performs text editing on the device B, the device B can open an application that can be used for text editing, so as to display the text content obtained from the device A.

In this embodiment of the present application, the display unit 1160 can also be used to display the text content output by the ASR module (including the intermediate result and the final result); the display unit 1160 can also be used to display the text content edited by the user on the device B.

In one embodiment, after receiving the text content sent by the transceiver control module 1130, the transceiver control module 1150 of the device B can append the text content to the back of the previously displayed text content. Exemplarily, taking device B as a windows system as an example, after receiving the text content, the transceiver control module 1150 of device B can call the QT interface and perform the following steps: (1) select all the text content displayed by App1; ( 2) Move the cursor to the end of the text; (3) Insert the new text content sent by device A; (4) Save all the text content in the current App1.

The editing control module 1170 is used for saving, editing and displaying control of the received information.

In this embodiment of the present application, the results sent to device B through the ASR module 1120 may include two types:

The first type is an intermediate result. For example, before a sentence is finished, the text content determined by the ASR module 1120 is not finalized and can be used as an intermediate result. In order to reflect the real-time performance, the intermediate results are also displayed, but are not saved as the final results.

Exemplarily, the ASR module of device A detects that every time the user speaks a word, it sends a corresponding word to device B, so that text content can be displayed on device A and device B synchronously. For example, the user says "I am XX", when device A detects that the user has said "I", the ASR module of device A can send the text content ("I" to device B after determining the corresponding text "I" ), so that device B can display the text content "I". When device A detects that the user has said "yes", the ASR module of device A can send the text content ("yes") to device B after determining the corresponding text "yes", so that device B can Append the text "Yes" after "I" until "I am XX" is displayed.

Exemplarily, the ASR module of device A detects that every time the user speaks a phrase, it sends a corresponding phrase to device B, so that text content can be displayed on device A and device B synchronously. For example, the user says "no hardship can stop us from moving forward", when device A detects that the user has said "any", the ASR module of device A can send the corresponding word "any" to device B after determining the corresponding word "any" The text content ("any"), so that device B can display the text content "any". When device A detects that the user says "difficult", the ASR module of device A can send the text content ("difficult") to device B after determining the corresponding word "difficult", so that device B can Append the text "difficulty" after any" until it shows "no hardship can stop us from moving forward".

In one embodiment, the content displayed by device A can be synchronized with the content displayed on device B, and device A may correct the text content previously converted by the ASR module when judging that the user has finished speaking. Exemplarily, when the user says "I want to eat noodles today", the ASR module of device A detects that every time the user speaks a word or a phrase, it sends the corresponding text content to device B. When device A detects that the user says " When "thinking", it is possible that the ASR module determines that the corresponding text is "xiang", then device A will send the corresponding text content ("xiang") to device B, and device B can display the text content "xiang". After judging that the user has finished saying this sentence, device A can determine that there is an error in the conversion of "Xiang" in the previously converted text content "Today I'm eating noodles", then device A can automatically correct the "Xiang" to "Xiang". I want to", thereby updating the text content spoken by the user (from "Today I want to eat noodles" to "Today I want to eat noodles"). Device A can send the updated text content to device B, so that device B can also update "Xiang" in the previously displayed text content "Today I want to eat noodles" to "Want", thereby displaying the updated text content "Today I want to eat noodles".

The second is that after a sentence is spoken, the recognition result of the sentence determined by the ASR module 1120 will not be changed back in the future, and such a result will replace the previous intermediate result and display it as the saved result.

Exemplarily, the ASR module of device A can convert each word or phrase spoken by the user into text content, but device A may not send text content to device B before judging that the user has not finished a sentence. Instead, when device A determines that the user has finished speaking a sentence and corrects the previously converted text content, it sends the text content corresponding to the sentence to device B.

In one embodiment, the ASR module of device A can convert each word or phrase spoken by the user into text content and send it to device B. Device B can display the text content sent by device A in the form of subtitles, but the device B may not display the text content in app1. When device A judges that the user has finished speaking a sentence (and corrects the previously converted text content), device A can send the text content corresponding to the sentence and indication information to device B, and the indication information instructs device B to display in app1 The text content corresponding to the sentence. After receiving the text content and the instruction information, device B can display the text content corresponding to the sentence in app1.

If device A detects that the user has edited the text content on device B during the process of synchronizing the output result of the ASR module in real time, the editing control module 1170 will also re-save the edited result and display it, and the subsequent ASR The result is appended on top of that.

The editing control module 1170 is further configured to send the text content edited by the user to the sending and receiving control module 1150 , so that the sending and receiving control module 1150 sends the edited text content to the sending and receiving control module 1130 .

The replacement module 1140 is configured to replace the original displayed text content with the edited text content after receiving the edited text content sent by the transceiver control module 1130 .

In one embodiment, when the recording ends or the audio acquisition ends, the device A may indicate the end of the recording or the audio acquisition to the device B. After the user finishes editing on device B, the editing result can be synchronized to device A at one time, and the entire synchronization process is completed.

FIG. 12 shows a schematic flowchart of a text editing method 1200 according to an embodiment of the present application. The method 1200 can be implemented by the above-mentioned device A and device B, and the method 1200 includes:

S1201, device A acquires audio content.

Exemplarily, as shown in (d) of FIG. 3 , when the mobile phone detects that the user clicks on the control 302 , the mobile phone starts to collect the voice content in the environment.

Exemplarily, as shown in (c) of FIG. 5 , when the mobile phone detects an operation of the user clicking on the control 501, the mobile phone still collects the voice content in the environment.

Exemplarily, as shown in (b) of FIG. 6 , when the mobile phone detects the user's operation of accepting the incoming call, the mobile phone can acquire the voice content of another user.

Exemplarily, as shown in (b) of FIG. 7 , when the mobile phone detects that the user accepts the operation of the video call request, the mobile phone can acquire the voice content of another user.

S1202, device A sends first information to device B according to the audio content.

In one embodiment, the first information is text content corresponding to the audio content.

It should be understood that in this embodiment of the present application, after acquiring the voice content, device A may first convert the voice content into text content through the ASR module, so as to send the text content to device B. For the process of converting the voice content into text content by the device A, reference may be made to the descriptions in the foregoing embodiments, which are not repeated here for brevity.

In one embodiment, before device A sends the voice content to device B, the method further includes:

Device A sends a query request to request the device that received the query request to determine whether it has a text editing function;

In response to receiving the request information sent by device A, device B sends a response to device A, where the response is used to indicate that device B has a text editing function;

In response to receiving the response, device A sends device B the first information.

In this embodiment of the present application, device A broadcasts the query request to surrounding devices. Alternatively, device A may store device information of device B (eg, device A and device B are devices under the same account, or device A and device B are devices under different accounts in the same family group).

It should be understood that, for the process of sending the query request by the device A, reference may be made to the descriptions in the foregoing embodiments, which are not repeated here for brevity.

In one embodiment, in response to receiving the query request, device B may prompt the user whether to perform text editing on device B; in response to the user determining to perform text editing on device B, device B sends the response to device A.

Exemplarily, as shown in (d) in FIG. 3 , when the laptop computer detects the query request sent by the mobile phone, the laptop computer can display a prompt box 303, which includes the prompt message “It is detected that the mobile phone is recording and transferring. text, whether text editing is done on a laptop". In response to the user clicking on control 304, the laptop can send the response to the phone.

In one embodiment, after receiving the response sent by device B, device A may send request information to device A, where the request information is used to request device B to edit the text content output by device A; Request device B to edit the text content corresponding to the audio content.

Exemplarily, as shown in (d) in FIG. 3 , when the mobile phone detects that the user clicks on the control 302, the mobile phone can send a query request to the surrounding devices, and the query request is used to query whether the surrounding devices have a text editing function; In response to receiving the response sent by the laptop, the mobile phone can determine that the laptop has text editing capabilities. The mobile phone can continue to send request information to the notebook computer, and the request information is used to edit the text content output by the mobile phone. In response to receiving the request information from the cell phone, the laptop computer may display a prompt box 303 . In response to the user clicking on the control 304, the notebook computer can send a response to the mobile phone, the response being used to indicate that the notebook computer can edit the text content. In response to receiving the response, the cell phone can send the textual content to the laptop.

Exemplarily, as shown in (c) of FIG. 5 , when the mobile phone detects the operation of the user clicking on the control 501, it sends a query request to the surrounding devices; in response to receiving the response sent by the notebook computer, the mobile phone can determine that the notebook computer has text Edit function. The mobile phone can send request information to the laptop computer, where the request information is used to request the laptop computer to edit the text content corresponding to the audio content acquired by the mobile phone. In response to receiving the request information, the laptop computer may display a prompt box 502, wherein the prompt box 502 includes prompt information "it is detected that the mobile phone is recording, whether to convert the recording to text on the laptop computer". In response to the user clicking the control 503, the notebook computer can send a response to the mobile phone, where the response is used to instruct the notebook computer to edit the text content corresponding to the recording. In response to receiving the response, the cell phone can send the textual content to the laptop.

It should also be understood that device B can send the response information to device A through a BLE data packet, and the specific sending process can refer to the above-mentioned process of device A sending a BLE data packet to device B, which is not repeated here for brevity.

Exemplarily, the request information may be carried in a user datagram protocol (user datagram protocol, UDP) data packet. The UPD packet includes the data portion of the IP datagram. The data portion of an IP datagram may include extensible bits. Device A and Device B can agree on the content of a certain extensible bit. When a certain extensible bit is 1, device B can know that device A requests to edit the text content.

The UDP data packet can also carry the IP address and port number of device A (including source port number and destination port number, where the source port number refers to the port number used by device A to send data, and the destination port number refers to the port number that device A receives The port used by the data), the IP address and port number of device A can be carried in the UDP header of the data part of the IP datagram. In response to receiving the UDP data packet, device B can establish a transmission control protocol (transmission control protocol, TCP) connection with device A.

It should be understood that after device B and device A establish a TCP connection, device B can send the response to device A through the TCP connection.

Exemplarily, if the UDP data packet carries the IP address and destination port number of device A. Device B can establish a TCP connection with device A through the IP address and destination port number. After that, device A can send the text content input by the ASR module of device A to device B through the TCP connection.

In one embodiment, the method 1200 further includes: when the device A converts the voice content into text content, the device A displays the text content.

Exemplarily, as shown in (e) of FIG. 3 , after detecting the voice content in the environment, the mobile phone can convert the voice content into text content, so as to display the text content through the display interface of the memo application.

S1203: Device B displays the text content corresponding to the audio content according to the first information.

Exemplarily, as shown in (e) of FIG. 3 , after receiving the text content sent by the mobile phone, the notebook computer can display the text content through App1.

S1204, an operation of editing the text content by the user is detected, and the device B can display the edited text content.

Exemplarily, as shown in (f) and (g) of FIG. 3 , after the notebook computer detects an operation of editing the text content by the user, the notebook computer may display the edited text content.

In one embodiment, the method 1200 further includes: device B sending the edited text content to device A.

In one embodiment, device B detects the user's first operation, and sends the edited text content to device A.

Exemplarily, as shown in (f) of FIG. 3 , when the notebook computer detects that the user clicks on the control 307, the notebook computer can send the edited text content to the mobile phone.

Exemplarily, as shown in (g) of FIG. 3 , when the notebook computer detects that the user clicks on the control 307, the notebook computer can send the edited text content to the mobile phone.

In one embodiment, after receiving the edited text content, device A may use the edited text content to edit the previously displayed text content. Exemplarily, as shown in (e) and (f) of FIG. 3 , when the notebook computer detects that the user clicks the save control 307, the notebook computer may send the edited text content to the mobile phone. The phone can replace the previously displayed text content with the text content edited by the user.

In this embodiment of the present application, when device B detects an editing operation by the user on the text content sent by device A, it can edit the text content accordingly. Exemplarily, as shown in (g) of FIG. 3 , when the notebook computer detects that the user has edited the text content, it can edit the text content received from the mobile phone. Exemplarily, when the notebook computer can detect the user's operation of changing "positive success" to "Zheng Chenggong"; for another example, the notebook computer can detect the user's operation of adding a symbol "." after "annual feature"; another example, The laptop can detect the user inserting the symbol "," in "Medical Education Housing Food Safety". As another example, a laptop can detect the user's press enter after "Social Mindset?".

Device B can send the edited text content to device A through a BLE data packet or through a TCP connection. For the sending method, reference may be made to the description in the above embodiment, which is not repeated here for brevity.

In one embodiment, if device B detects that the user has edited the format of the text content, when device B sends the edited text content to device A, it can also indicate the format information of the text content to device A. Exemplarily, the format of the text content may include a line feed (or carriage return) operation between certain two words in the text content; or, there is a space between certain two words. Exemplarily, as shown in (g) of FIG. 3 , while sending the edited text content to the mobile phone, the notebook computer can also indicate to the mobile phone that there is a line break between "Q" and "2", and "?" There is a newline between "3".

In one embodiment, after detecting the user's operation of modifying the format of the text content, device B may send the edited text content and format information of the text content to device A. The implementation manner of the device B sending the edited text content to the device A may refer to the description in the foregoing embodiment, which is not repeated here for brevity. The following describes an implementation manner in which device B sends the format information of the edited text content to device A. Exemplarily, the text content format includes font size, font color, font inclination, font underline, font background color, and a carriage return operation after a certain text in this content.

Exemplarily, device A may send the format information of the edited text content to device B through a BLE data packet. This textual content may be carried in a service data field or a vendor specific data field in the PDU. For example, the payload of the service data field may include multiple bits, wherein the multiple bits include scalable bits. For a certain character (for example, text, character or symbol) in the edited text content, device A may agree with device B on the content of some extensible bits. Exemplarily, when a certain extensible bit is 000, the device A can know that the character is not oblique, there is no underscore, and there is no carriage return operation after the character. Exemplarily, when a certain extensible bit is 100, device A can learn that the character is oblique, has no underline, and there is no carriage return operation after the character. Exemplarily, when a certain extensible bit is 010, device A can know that the character is not oblique, has an underline, and there is no carriage return operation after the character. Exemplarily, when a certain extensible bit is 001, device A can know that the character is not slanted, has no underscore, and there is a carriage return operation after the character.

For a certain character (for example, text, character or symbol) in the edited text content, device A may agree with device B on the content of some extensible bits. Exemplarily, when a certain expandable bit is 000, device A can learn that the font color of the character is black. Exemplarily, when a certain extensible bit is 001, device A can learn that the font color of the character is gray. Exemplarily, when a certain expandable bit is 010, device A can know that the color of the character is blue. Exemplarily, when a certain expandable bit is 100, device A can learn that the font color of the character is blue.

It should be understood that the process of device B indicating the font background color of a certain character in the edited text content to device A may also refer to the description in the above embodiment.

For a certain character (for example, text, character or symbol) in the edited text content, device A may agree with device B on the content of some extensible bits. Exemplarily, when a certain expandable bit is 000, device A can learn that the font size of the character is 10. Exemplarily, when a certain extensible bit is 001, device A can learn that the font size of the character is 12. Exemplarily, when a certain extensible bit is 010, device A can learn that the font size of the character is 14. Exemplarily, when a certain expandable bit is 100, device A can learn that the font size of the character is 18.

It should be understood that the format of the text content is not specifically limited in the embodiment of the present application. After receiving the format information of the edited text content, the device B can to display the edited text content. The text content displayed on device B corresponds to the text content displayed on device A.

In one embodiment, the method 1200 further includes: device A replaces the original text content with the edited text content received from device B; in response to receiving the edited text content from device B, device A displays a second interface , the second interface includes the edited text content.

Exemplarily, as shown in (f) and (g) of FIG. 3 , the notebook computer sends the edited text content to the mobile phone in response to the operation of the user clicking the save control 307 . After receiving the edited text content, the mobile phone can replace the text content displayed on the original interface with the edited text content.

In one embodiment, when device B sends the edited text content to device A, it may also send the identification information of the edited text content to device A.

It should be understood that S1204 is an optional step, and device B may also not send the edited text content to device A. Instead, save the edited text content locally on device B.

FIG. 13 shows another system architecture provided by an embodiment of the present application. The system architecture includes device A and device B, where device A may be a device with a pickup function (eg, a mobile phone), and device B may be a device with an efficient editing function (eg, a laptop). Device A includes a pickup module 1310 and a transceiver control module 1320 ; device B includes a receive transceiver control module 1330 , an ASR module 1340 , a display module 1350 and an editing control module 1360 . Different from the system architecture shown in FIG. 11 , the ASR module can be included in device B. Device A can be responsible for acquiring voice content or audio files, and device A sends the acquired voice content or audio files to the transceiver control module 1330 of device B through the transceiver control module 1320, so that the ASR module 1340 of device B sends the voice content or audio files to the device B. Converted to text content, so that device B can display the converted text content to the user. Device B can also display the edited text content through the display module 1340 after detecting an editing operation of the text content by the user through the editing control module 1360 .

FIG. 14 shows a schematic block diagram of an apparatus 1400 provided by an embodiment of the present application. The apparatus 1400 can be set in the above-mentioned device A in FIG. 12 , and the apparatus 1400 includes: an acquiring unit 1410 for acquiring audio content; a sending unit 1420 for transmitting first information to the second electronic device, the first The information is the audio content, or the first information is the first text content corresponding to the audio content, so that the second electronic device displays the first text content according to the first information and detects the user's first text content. edit operation.

FIG. 15 shows a schematic block diagram of an apparatus 1200 provided by an embodiment of the present application. The apparatus 1500 can be set in the above-mentioned device B in FIG. 12 , and the apparatus 1500 includes: a receiving unit 1510 for receiving first information sent by a first electronic device, where the first information is acquired by the first electronic device audio content, or the first information is the first text content corresponding to the audio content; the display unit 1520 is used to display the first text content according to the first information; the detection unit 1530 is used to detect the user's An editing operation of text content; the display unit 1520 is further configured to display the second text content, where the second text content is the edited text content of the first text content.

FIG. 16 shows a schematic structural diagram of an electronic device 1600 provided by an embodiment of the present application. As shown in FIG. 16, the electronic device includes: one or more processors 1610, one or more memories 1620, and the one or more memories 1620 stores one or more computer programs, and the one or more computer programs include instruction. When the instruction is executed by the one or more processors 1610, the first electronic device or the second electronic device is caused to execute the technical solutions in the foregoing embodiments.

An embodiment of the present application provides a system including a device A and a device B, and the system is used to implement the technical solutions in the foregoing embodiments. The implementation principle and technical effect thereof are similar to the related embodiments of the above method, and are not repeated here.

The embodiments of the present application provide a computer program product, which enables the device A to execute the technical solutions in the foregoing embodiments when the computer program product runs on the device A (or the mobile phone in the foregoing embodiment). The implementation principle and technical effect thereof are similar to the related embodiments of the above method, and are not repeated here.

An embodiment of the present application provides a computer program product, which enables the device B to execute the technical solutions in the foregoing embodiments when the computer program product runs on the device B (or the notebook computer in the foregoing embodiment). The implementation principle and technical effect thereof are similar to the related embodiments of the above method, and are not repeated here.

An embodiment of the present application provides a readable storage medium, where the readable storage medium contains instructions, when the instructions are executed on device A (or, the mobile phone in the above embodiment), the device A is caused to execute the above embodiment technical solution. The implementation principle and technical effect thereof are similar, and are not repeated here.

An embodiment of the present application provides a readable storage medium, where the readable storage medium contains instructions, when the instructions are executed on a device B (or a notebook computer in the above embodiment), the device B is caused to perform the above implementation. example technical solution. The implementation principle and technical effect thereof are similar, and are not repeated here.

An embodiment of the present application provides a chip, which is used for executing an instruction, and when the chip is running, executes the technical solutions in the foregoing embodiments. The implementation principle and technical effect thereof are similar, and are not repeated here.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

A system, characterized in that the system includes a first electronic device and a second electronic device, wherein,

the first electronic device for acquiring audio content;

The first electronic device is further configured to send first information to the second electronic device, where the first information is the audio content, or the first information is the first text corresponding to the audio content content;

the second electronic device, configured to display the first text content according to the first information;

The second electronic device is further configured to display second text content in response to an editing operation of the first text content by the user, where the second text content is the text content after editing the first text content.
The system according to claim 1, wherein the second electronic device is further configured to send the second text content to the first electronic device.
The system according to claim 2, wherein the editing operation comprises a format modification operation of the first text content, wherein,

The second electronic device is further configured to send the format information of the second text content to the first electronic device.
The system according to claim 3, wherein the format information of the second text content includes font color, font size, font background color, font inclination, font underline and the second text content of the second text content. One or more types of carriage return operations in the text content.
The system according to any one of claims 2 to 4, wherein the first electronic device is further configured to:

Before receiving the second text content sent by the second electronic device, display the first text content according to the audio content;

After receiving the second text content sent by the second electronic device, the first text content is replaced with the second text content.
The system according to any one of claims 1 to 5, wherein the first electronic device is further configured to:

sending a query request, where the query request is used for the device receiving the query request to determine whether it has a text editing function;

In response to receiving a response sent by the second electronic device, the first information is sent to the second electronic device, where the response is used to indicate that the second electronic device has a text editing function.
The system according to any one of claims 1 to 6, wherein the second electronic device is specifically used for:

In response to receiving the first information, prompting the user whether to perform text editing on the second electronic device;

The first text content is displayed in response to a user determining an operation to perform text editing on the second electronic device.
The system according to any one of claims 1 to 7, wherein the first information is the first text content, and the first electronic device is specifically used for:

In response to acquiring the audio content, converting the audio content into the first text content;

The first text content is sent to the second electronic device.
The system according to any one of claims 1 to 7, wherein the first information is the audio content, and the second electronic device is specifically used for:

In response to receiving the audio content, converting the audio content to the first textual content;

The first text content is displayed.
The system according to any one of claims 1 to 9, wherein the account of the first electronic device is associated with the account of the second electronic device.
A method for editing text, wherein the method is applied to a first electronic device, and the method includes:

the first electronic device acquires audio content;

The first electronic device sends first information to the second electronic device, where the first information is the audio content, or the first information is the first text content corresponding to the audio content, so that the The second electronic device displays the first text content according to the first information and detects an editing operation of the first text content by the user.
The method according to claim 11, wherein the method further comprises:

The first electronic device receives second text content sent by the second electronic device, where the second text content is text content after the user has edited the first text content on the second electronic device.
The method of claim 12, wherein the method further comprises:

The first electronic device receives the format information of the second text content sent by the second electronic device.
The system according to claim 13, wherein the format information of the second text content includes font color, font size, font background color, font inclination, font underline and the second text content of the second text content. One or more types of carriage return operations in the text content.
The method according to any one of claims 12 to 14, wherein the method further comprises:

Before receiving the second text content sent by the second electronic device, the first electronic device displays the first text content according to the audio content;

After receiving the second text content sent by the second electronic device, the first electronic device replaces the first text content with the second text content.
The method according to any one of claims 11 to 15, wherein the method comprises:

Before sending the first information to the second electronic device, the first electronic device sends a query request, where the query request is used for the device receiving the query request to determine whether it has a text editing function;

In response to receiving a response sent by the second electronic device, the first electronic device sends the first information to the second electronic device, where the response is used to indicate that the second electronic device has a text editing function .
The method according to any one of claims 11 to 16, wherein the first information is the first text content, and the method further comprises:

In response to acquiring the audio content, the first electronic device converts the audio content into the first text content;

The first electronic device sends the first text content to the second electronic device.
The method according to any one of claims 11 to 17, wherein the account of the first electronic device is associated with the account of the second electronic device.
A method for editing text, wherein the method is applied to a second electronic device, and the method includes:

The second electronic device receives first information sent by the first electronic device, where the first information is the audio content acquired by the first electronic device, or the first information is the first information corresponding to the audio content text content;

The second electronic device displays the first text content according to the first information;

In response to a user's editing operation on the first text content, the second electronic device displays second text content, where the second text content is the text content after editing the first text content.
The method of claim 19, wherein the method further comprises:

The second electronic device sends the second text content to the first electronic device.
The method according to claim 20, wherein the editing operation comprises a format modification operation of the first text content, and the method further comprises:

The second electronic device sends the format information of the second text content to the first electronic device.
The method according to claim 21, wherein the format information of the second text content comprises font color, font size, font background color, font inclination, font underline and the second text content of the second text content. One or more types of carriage return operations in the text content.
The method according to any one of claims 19 to 22, wherein the method further comprises:

Before receiving the first information sent by the first electronic device, the second electronic device receives a query request sent by the first electronic device, the query requesting the second electronic device to determine whether there is a text editor Function;

The second electronic device sends a response to the first electronic device, where the response is used to indicate that the second electronic device has a text editing function.
The method according to any one of claims 19 to 23, wherein the second electronic device displays the first text content according to the first information, comprising:

In response to receiving the first information, the second electronic device prompts the user whether to perform text editing on the second electronic device;

The second electronic device displays the first text content in response to the user determining an operation to perform text editing on the second electronic device.
The method according to any one of claims 19 to 24, wherein the first information is the audio content, and before the second electronic device displays the first text content, the method further comprises: :

In response to receiving the audio content, converting the audio content to the first textual content.
The method according to any one of claims 19 to 25, wherein the account of the first electronic device is associated with the account of the second electronic device.
An electronic device, comprising:

one or more processors;

one or more memories;

The one or more memories store one or more computer programs comprising instructions that, when executed by the one or more processors, cause the electronic device to perform as The method of any one of claims 11 to 18.
An electronic device, comprising:

one or more processors;

one or more memories;

The one or more memories store one or more computer programs comprising instructions that, when executed by the one or more processors, cause the electronic device to perform as The method of any one of claims 19 to 26.
A computer-readable storage medium, characterized in that it includes computer instructions, which, when the computer instructions are executed on an electronic device, cause the electronic device to perform the method according to any one of claims 11 to 18; or ,

The computer instructions, when executed on an electronic device, cause the electronic device to perform the method of any of claims 19-26.