CN114900641A

CN114900641A - Data storage and backtracking method, device and system

Info

Publication number: CN114900641A
Application number: CN202210475926.0A
Authority: CN
Inventors: 王玮; 陶维琦; 苏文畅; 汪载舟
Original assignee: Anhui Tingjian Technology Co ltd
Current assignee: Anhui Tingjian Technology Co ltd
Priority date: 2022-04-29
Filing date: 2022-04-29
Publication date: 2022-08-12

Abstract

The invention provides a data storage and backtracking method, device and system, wherein the data storage method comprises the following steps: determining the starting time of voice recording; responding to the operation of a user on the electronic whiteboard, determining the operation, and determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of voice recording, wherein the relative execution time is relative time taking the start time of voice recording as a starting point; and storing the operation and the relative execution time of the operation into the whiteboard data, wherein the whiteboard data and the voice data obtained by voice recording correspond to the same event. The method, the device and the system realize the time correlation of the voice data and the whiteboard data, so that the voice data and the whiteboard data can be synchronously displayed without manual alignment of a user in the subsequent backtracking process.

Description

Data storage and backtracking method, device and system

Technical Field

The invention relates to the technical field of computers, in particular to a data storage and backtracking method, device and system.

Background

The intelligent office brings great convenience to work and study of people. Taking an intelligent conference as an example, voice transcription can record voice in the conference in real time and form characters, and the electronic whiteboard can record written explanation in the conference so as to facilitate a user to trace back conference contents after the conference is finished.

However, at present, for the writing explanation recorded on the electronic whiteboard, the backtracking form shows the final writing state of the whiteboard, i.e. shows the image after the writing explanation is completed. The user can not go back to the dynamic process of writing and explaining based on the electronic whiteboard in the conference process; in addition, because pronunciation transcription APP (application) and whiteboard APP realization that pronunciation transcription and whiteboard need mutual independence usually, lack the linkage between pronunciation transcription APP and the whiteboard APP, lead to the user to need operate two APPs respectively when going back to a meeting, complex operation, user experience is poor.

Disclosure of Invention

The invention provides a data storage and backtracking method, device and system, which are used for solving the defect that linkage is lacked between a voice transcription APP and an electronic whiteboard APP in the prior art.

The invention provides a data storage method, which comprises the following steps:

determining the starting time of voice recording;

responding to an operation of a user on an electronic whiteboard, determining the operation, and determining a relative execution time of the operation based on an absolute execution time of the operation and a start time of the voice recording, wherein the relative execution time is a relative time with the start time of the voice recording as a starting point;

and storing the operation and the relative execution time of the operation into whiteboard data, wherein the whiteboard data and the voice data obtained by voice recording correspond to the same event.

The data storage method provided by the invention further comprises the following steps:

determining an interruption duration based on the pause time and the restart time of the voice recording;

the determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of the voice recording comprises:

determining the relative execution time of the operation based on the absolute execution time of the operation, the start time of the voice recording and the interruption duration when the absolute execution time of the operation is after the restart time.

The invention also provides a data backtracking method, which comprises the following steps:

acquiring voice data and whiteboard data corresponding to the same event, wherein the whiteboard data comprises user operation on an electronic whiteboard and relative execution time of the operation, and the relative execution time is relative time taking the starting time of voice recording as a starting point;

determining an operation corresponding to the current display progress from the whiteboard data based on the current display progress of the voice data;

and analyzing the operation corresponding to the current display progress to obtain and display the operation image corresponding to the current display progress.

According to the data backtracking method provided by the invention, the operation corresponding to the current display progress is determined from the whiteboard data based on the current display progress of the voice data, and the operation comprises the following steps:

and acquiring the operation with the relative execution time before the current display progress from the whiteboard data as the operation corresponding to the current display progress.

According to a data backtracking method provided by the present invention, the obtaining, from the whiteboard data, an operation of the relative execution time before the current display progress as an operation corresponding to the current display progress includes:

acquiring the operation of which the relative execution time is before the current display progress and after the last display progress from the whiteboard data, and taking the operation as the operation corresponding to the current display progress;

the last display progress is a display progress before the current display progress.

According to the data backtracking method provided by the invention, the analyzing the operation corresponding to the current display progress to obtain and display the operation image corresponding to the current display progress comprises the following steps:

analyzing the operation corresponding to the current display progress based on the graphic data in the whiteboard data to obtain the graphic data corresponding to the current display progress;

dividing the graphic data corresponding to the current display progress into a plurality of graphic data sets;

rendering the plurality of graphic data sets based on the threads respectively corresponding to the plurality of graphic data sets, and obtaining and displaying the operation image corresponding to the current display progress.

According to the data backtracking method provided by the invention, the current display progress of the voice data comprises the display progress of voice and/or the display progress of the transcription text of the voice.

The data backtracking method provided by the invention further comprises the following steps:

responding to a first playing operation of a user, and displaying a first playing window;

and/or responding to a second playing operation of the user based on the first playing window to display a second playing window;

the displaying the operation image corresponding to the current display progress comprises the following steps:

and displaying the operation image corresponding to the current display progress in the first playing window or the second playing window.

The present invention also provides a data storage device comprising:

the starting determining unit is used for determining the starting time of voice recording;

the recording unit is used for responding to the operation of a user on the electronic whiteboard, determining the operation and determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of the voice recording, wherein the relative execution time is relative time taking the start time of the voice recording as a starting point;

and the storage unit is used for storing the operation and the relative execution time of the operation into whiteboard data, and the whiteboard data and the voice data obtained by voice recording correspond to the same event.

The present invention also provides a data backtracking apparatus, including:

the electronic whiteboard system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring voice data and whiteboard data corresponding to the same event, the whiteboard data comprises operation of a user for the electronic whiteboard and relative execution time of the operation, and the relative execution time is relative time taking the starting time of voice recording as a starting point;

the operation determining unit is used for determining the operation corresponding to the current display progress from the whiteboard data based on the current display progress of the voice data;

and the display unit is used for analyzing the operation corresponding to the current display progress to obtain and display the operation image corresponding to the current display progress.

The invention also provides a data processing system which comprises a voice recording device, a data storage device and a data backtracking device, wherein the voice recording device is used for synchronizing the voice recording starting time to the data storage device.

The invention also provides an electronic device, which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the program, the data storage method or the data backtracking method can be realized.

The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data storage method or a data backtracking method as described in any of the above.

The present invention also provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the data storage method or the data backtracking method according to any one of the above.

According to the data storage and backtracking method, device and system, the whiteboard data with the time axis aligned with the voice data is generated by taking the starting time of voice recording as the relative execution time of the operation determined by the starting point, so that the association of the voice data and the whiteboard data in time is realized, and the voice data and the whiteboard data can be synchronously displayed without manual alignment of a user in the subsequent backtracking process; and moreover, the dynamic process of writing and explaining based on the electronic whiteboard is recorded in the form of whiteboard data, so that the calculation burden is effectively reduced while the dynamic display is realized, and the response efficiency of data storage and backtracking is improved.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart of a data storage method provided by the present invention;

fig. 2 is a schematic diagram of an electronic whiteboard main interface provided by the present invention;

FIG. 3 is a schematic diagram of a caption mode interface for voice transcription provided by the present invention;

FIG. 4 is a schematic diagram of a main interface for voice transcription provided by the present invention;

FIG. 5 is a flowchart illustrating a data backtracking method according to the present invention;

FIG. 6 is a diagram illustrating step 520 of the data backtracking method according to the present invention;

FIG. 7 is a schematic diagram of a data backtracking interface provided by the present invention;

FIG. 8 is a schematic structural diagram of a data storage device provided by the present invention;

FIG. 9 is a schematic structural diagram of a data backtracking apparatus according to the present invention;

fig. 10 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Under the intelligent conference scene, pronunciation transcription APP and whiteboard APP are independent function usually, lack the linkage between the two, lead to the user to need operate two APPs respectively when backtracking to a meeting, constantly compare the progress bar of two APP showpings, just can reach the purpose that both show in step. Moreover, since the electronic whiteboard APP records an image after the completion of the writing and explanation, the user cannot obtain a dynamic writing process in the conference during backtracking, which results in poor user experience.

Therefore, the invention provides a data storage method which is used for storing data generated in an intelligent conference scene so as to realize linkage of voice recording and writing recording in the intelligent conference. In addition, the invention can also be applied to scenes such as intelligent classrooms and the like which need to record voice and write synchronously. For convenience of explanation, the following is developed by taking an intelligent conference scenario as an example.

Fig. 1 is a schematic flow chart of a data storage method provided by the present invention, and as shown in fig. 1, the method may be applied to an electronic device installed with two APPs, namely, a voice transcription APP and an electronic whiteboard APP, and may also be applied to an electronic device installed with only the electronic whiteboard APP and capable of communicating with the electronic device installed with the voice transcription APP, which is not specifically limited in this embodiment of the present invention.

The method comprises the following steps:

step 110, determining the start time of voice recording.

Specifically, the start of the conference may be generally identified as the start of the voice recording, and accordingly, the start time of the voice recording may be regarded as the start time of the conference. In addition, considering that the voice data obtained by voice recording takes the start time of voice recording as the beginning of the time axis, in order to ensure that the voice data obtained by voice recording and the whiteboard data recorded by the electronic whiteboard can be linked synchronously, the start time of voice recording can be determined firstly, and the start time of voice recording is applied to the time axis of the whiteboard data recorded by the electronic whiteboard.

Here, the start time of the voice recording may be a start time generated by the voice transcription APP in the electronic device in response to a user's start operation for the voice recording. Under the condition that the voice transcription APP and the electronic whiteboard APP are installed on the same electronic device, the start time of voice recording may be directly transmitted to the electronic whiteboard APP by the voice transcription APP, for example, the start time may be transmitted by aid (android Interface Definition language), and under the condition that the voice transcription APP and the electronic whiteboard APP are respectively installed on two electronic devices, the start time of voice recording needs to be transmitted to the electronic device where the electronic whiteboard APP is located by the electronic device where the voice transcription APP is located.

And 120, responding to the operation of the user on the electronic whiteboard, determining the operation, and determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of the voice recording, wherein the relative execution time is relative time with the start time of the voice recording as a starting point.

Specifically, in the intelligent conference process, a user can operate the electronic whiteboard, where the operation may be writing, erasing, panning and zooming, circling, and the like. The electronic device installed with the electronic whiteboard APP can detect an operation of a user on the electronic whiteboard, and respond to the operation, where the response includes not only displaying a graph corresponding to the operation on the electronic whiteboard, but also recording related information of the operation.

Here, the information related to the operation may include the operation itself, and the relative execution time of the operation.

Wherein the operation itself may comprise the type of operation;

the relative execution time of the operation may be a relative execution time of 31:00-31:05 determined by starting from a start time of the voice recording, recording a relative time of start and end of the operation, for example, a start time of 10:01:00 of the voice recording, and a time of start and end of the operation being 10:32:00, 10:32:05, respectively. The relative execution time may also be a relative time when the recorded operation ends, for example, the start time of the voice recording is 10:01:00, and the time when the operation ends is 10:32:05, so that the determined relative execution time of the operation is 31: 05.

And step 130, storing the operation and the relative execution time of the operation into whiteboard data, wherein the whiteboard data and the voice data obtained by voice recording correspond to the same event.

Specifically, the whiteboard data may be stored in step 120 in response to the operation determined by the user for the operation of the electronic whiteboard and the relative execution time of the operation, because of the record of the relative execution time of the operation in the whiteboard data

The relative execution time of the operation determined based on the starting time of the voice recording enables the operation of the electronic whiteboard recording to be synchronous with the time axis of the voice recording, so that the voice data obtained based on the voice recording and the whiteboard data obtained based on the electronic whiteboard recording both take the starting time of the voice recording as the starting point on the time axis, and the display progress is synchronous. The voice data and the whiteboard data correspond to the same event, for example, in an intelligent conference scene, the voice data and the whiteboard data correspond to the same conference, and for example, in an intelligent classroom scene, the voice data and the whiteboard data correspond to the same lesson. The voice data may include a voice obtained by recording the voice, and may also include a transcription text corresponding to the voice, which is not specifically limited in this embodiment of the present invention.

In addition, the related information of each operation of the user in the intelligent conference process is recorded in the whiteboard data, namely the dynamic process of the writing explanation is recorded in the whiteboard data, and the image after the writing explanation is finished is not recorded in a stereographic manner any more, so that the process of the user operation can be dynamically reflected in backtracking, and the user embodiment is improved. Meanwhile, compared with the operation of screen recording of the electronic whiteboard in the intelligent conference process, the recording of whiteboard data meets the requirement of recording a dynamic process, and meanwhile, the data volume needing to be stored is greatly reduced. Particularly, under the condition that the meeting time may reach 2 hours and 3 hours, the operation on the electronic whiteboard in the meeting may be infrequent, but a long video file of 2 hours and 3 hours is generated by directly recording the screen, and huge operation burden is brought to the electronic equipment for installing the electronic whiteboard APP no matter in the storage stage or the later backtracking stage. In the embodiment of the invention, the whiteboard data obtained by recording the relevant information of the operation only needs to record tens or hundreds of pieces of relevant information of the operation in the conference for 2 or 3 hours, so that the operation burden of the electronic equipment for installing the electronic whiteboard APP is greatly reduced, and the response efficiency of data storage and data backtracking is improved.

According to the method provided by the embodiment of the invention, the whiteboard data with the time axis aligned with the voice data is generated by taking the starting time of voice recording as the relative execution time of the operation determined by the starting point, so that the association of the voice data and the whiteboard data in time is realized, and therefore, the voice data and the whiteboard data can be synchronously displayed without manual alignment of a user in the subsequent backtracking process; and moreover, the dynamic process of writing and explaining based on the electronic whiteboard is recorded in the form of whiteboard data, so that the calculation burden is effectively reduced while the dynamic display is realized, and the response efficiency of data storage and backtracking is improved.

Based on the above embodiment, step 120 further includes:

responding to the operation of a user on the electronic whiteboard, and determining the graphic data corresponding to the operation;

step 130 further comprises:

and storing the graphic data corresponding to the operation into the whiteboard data.

Specifically, there is a correspondence between the operation and the graphics data, and the operation itself may include, in addition to the type of the operation, an identifier of the graphics data to which the operation corresponds.

The graphic data corresponding to the operation may be a handwritten track, a picture, characters obtained by OCR (Optical Character Recognition), and the like, and the handwritten track may be specifically represented by a track point position, a brush color, a brush size, a brush type, a brush tip handwriting type, and the like.

Accordingly, after the graphic data corresponding to the operation is determined, the graphic data corresponding to the operation, together with the operation itself and the relative execution time of the operation, may be stored in the whiteboard data.

Based on any of the above embodiments, the data storage method further includes:

correspondingly, in step 120, the determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of the voice recording includes:

and determining the relative execution time of the operation based on the absolute execution time of the operation, the start time of the voice recording and the interruption duration under the condition that the absolute execution time of the operation is after the restart time.

Specifically, during the intelligent conference, there may be a case of conference interruption, such as a midfieldbreak, or a participant temporarily receiving an unrelated call. Usually, when a conference is interrupted, a user operates a voice transcription APP to pause recording, and the time of temporary recording is the pause time of voice recording. And after the midfield rest or other conditions are processed, the middle section of the conference is finished, at the moment, the user can operate the voice transcription APP to restart and record, and the recording restart time is the restart time of voice recording.

Since the voice transcription APP stops voice recording during the conference interruption, i.e. the period from the recording pause to the recording restart, and the recorded voice after the restart starts from the pause time, for example, the voice recording is paused when the recording time reaches 50 minutes, and is restarted after 5 minutes, and the recording is still started from the 50 th minute.

Considering that the conference interruption directly affects the time axis of the voice data, it is necessary to synchronize the pause time and the restart time of the conference interruption to the electronic whiteboard APP so that the time axis of the whiteboard data can still be consistent with the time axis of the voice data.

Similar with the initial time that pronunciation were recorded, under the condition that pronunciation transcription APP and whiteboard APP installed at same electronic equipment, the pause time and the restart time that pronunciation were recorded can be directly transmitted to whiteboard APP by pronunciation transcription APP, and under the condition that pronunciation transcription APP and whiteboard APP installed respectively at two electronic equipment, the pause time and the restart time that pronunciation were recorded then need be transmitted to whiteboard APP at the electronic equipment by the electronic equipment that pronunciation transcription APP belonged.

When it is known that there is an interruption in voice recording, when recording the relative execution time of an operation on an electronic whiteboard, it is also necessary to consider whether the operation occurs before the interruption or after the interruption, and it may be specifically determined whether the absolute execution time of the operation is earlier than the pause time or later than the restart time:

if the absolute execution time occurs before the pause time, the operation occurs before the interruption, and a difference value between the absolute execution time of the operation and the start time of the voice recording is taken as a relative execution time;

if the absolute execution time occurs after the restart time, the operation occurs after the interruption, and it is necessary to subtract the interruption time length from the difference between the absolute execution time of the operation and the start time of the voice recording to obtain the relative execution time on the basis of obtaining the difference between the absolute execution time of the operation and the start time of the voice recording. For example, the starting time of voice recording is 10:01:00, one interruption occurs in the midway, the pause time and the restart time are 11:00:00 and 11:10:00 respectively, and the interruption time is 10 minutes; assume that the operations start and end at times 11:12:00, 11:12:05, respectively, and the relative execution time of the operations thus determined is 61:00-61: 05.

According to the method provided by the embodiment of the invention, after the voice recording is interrupted, the relative execution time of the operation is determined based on the interruption duration of the voice recording, so that the time axes of the whiteboard data and the voice data can be aligned, and the reliability of synchronous display of the voice data and the whiteboard data is ensured.

Based on any of the above embodiments, the whiteboard data may include operations, relative execution times of the operations, graphic data, and may further include a whiteboard background. The whiteboard data can be compressed by adopting json files and pictures to form compressed files and stored. Further, the whiteboard data, when stored, may be stored in pages, that is, image data and operations for each page, relative execution time of the operations, and the like are stored separately.

The whiteboard background can include drawing backgrounds of each display page of the electronic whiteboard, the types of the backgrounds can include a built-in background and other backgrounds, and the whiteboard background can be stored according to the following forms:

{"bg":{"p":"file:///android_asset/bg_three.webp"},"bgt":1,"id":"p_pr oductJOJ_700"}

where "bg" represents a background, "p" represents an address where the background is stored, "bgt" represents a background type, and "id" represents a background identification.

The graphic data in the whiteboard data may record drawn data, such as handwritten dots, pictures, and characters obtained by OCR, and the graphic data of the handwritten dots may be stored in the following form:

{"avalible":1,"id":"g_34_productNFY937","pc":"#FFFFFF","ps":10,"pt":1,"type":1,"vhdt":2,"wps":[[181992,128151,0],[181007,129844,319384],[180164,131473,319393],[179097,133543,319401],[177914,135929,319411],[176730,138480,319419],[175582,141189,319429],[174398,144043,319437],[173261,147030,319446],[172160,150068,319455],[171082,153114,319464],[170121,156060,319473],[169265,158829,319482],[168574,161340,319491],[168058,163529,319500],[167707,165342,319509],[167472,166693,319518],[167355,167609,319527],[167343,168064,319536]]}

wherein, the 'avalible' represents whether the graph is available, the 'id' represents the graph identification, the 'pc' represents the brush color, the 'ps' represents the brush size, the 'pt' represents the brush type, the 'type' represents the graph type, the 'vhdt' represents the brush type, and the 'wps' represents the handwriting point coordinate and the acquisition time.

In addition, the graphic data of the picture may include an absolute address of a file currently inserted into the picture, a matrix of the picture, and the like; the graphic data of the characters obtained by OCR can comprise the characters after OCR recognition, the positions of the characters, a character matrix and the like.

In the whiteboard data, the information of the operation itself and the relative start time may be stored in the following form:

{"et":0,"gid":"g_34_productNFY937","id":"p_productJOJ_700_a_36_productTDN_936","isUndone":0,"status":2,"type":0}

where "et" represents the relative execution time, "gid" represents the identifier of the operation-associated graphics data, "id" represents the operation identifier, "isUndone" represents whether the operation is revocable, "status" represents the action state, and "type" represents the action type.

Based on any one of the above embodiments, the data storage method may include the following steps:

fig. 2 is a schematic diagram of a main interface of an electronic whiteboard provided by the present invention, as shown in fig. 2, a user can click a voice translation button of a toolbar at the bottom of the whiteboard under an interface of the electronic whiteboard to enter a caption mode of voice transcription, fig. 3 is a schematic diagram of a caption mode interface of voice transcription provided by the present invention, as shown in fig. 3, a caption bar in the caption mode interface can display a text obtained by transcription after voice recording and a translation corresponding to the text in real time, and a button of a minimization and window mode is arranged on the caption bar, the minimization can be returned to the main interface of the electronic whiteboard by clicking the window mode, and the main interface of voice transcription can be entered by clicking the window mode. In addition, in the subtitle mode interface, the user can still operate on the electronic whiteboard, for example, writing "1234".

In addition, the user may click the window mode to enter the voice transcription main interface shown in fig. 4, a key for suspending recording and ending recording is provided in the voice transcription main interface, and a recording duration, for example, "00: 01: 07", is displayed below the key.

Under the interface of the electronic whiteboard, a user can click a button of the voice translation to start voice recording, and the electronic equipment records the voice recording as the starting time of the voice recording. And then, the user performs operations such as writing, inserting pictures, translating, zooming, circle selection amplification, circle selection movement, circle selection deletion and the like on the electronic whiteboard, and the electronic whiteboard can use the starting time of voice recording as a starting point to determine the relative execution time of operation execution, so that operation recording is performed. In the window mode of voice transcription, a user can click a key for finishing recording to finish a conference, and whiteboard data can be formed by the operation of recording the electronic whiteboard and stored in a conference record together with voice data formed by voice recording and transcription.

Based on any of the above embodiments, on the basis of completing data storage, backtracking can be performed based on the stored whiteboard data and voice data. Fig. 5 is a schematic flow chart of a data backtracking method provided by the present invention, and as shown in fig. 5, the method includes:

step 510, acquiring voice data and whiteboard data corresponding to the same event, where the whiteboard data includes an operation of a user on the electronic whiteboard and a relative execution time of the operation, and the relative execution time is a relative time with a start time of voice recording as a starting point.

Specifically, the voice data and the whiteboard data herein are data for the same event, for example, voice data and whiteboard data recorded for the same conference. Since the relative execution time of each operation in the whiteboard data is a relative time starting from the start time of the voice recording during the data storage, the time axis of the voice data and the time axis of the whiteboard data are aligned.

Step 520, determining an operation corresponding to the current display progress from the whiteboard data based on the current display progress of the voice data.

Specifically, when the same event is traced back by applying the voice data and the whiteboard data, the user usually performs an operation on the voice transcription APP, for example, clicking a play key of the voice data to perform voice playback, or dragging a play progress bar of the voice data to adjust a display progress of the voice data.

Therefore, in the backtracking process, in order to ensure the synchronism of the voice data and the whiteboard data display, the current display progress of the voice data needs to be determined first. Here, the current display progress of the voice data may be a time stamp of the currently played voice, or a time stamp of the voice corresponding to the currently displayed transcribed text, for example, 00:15: 02. The current display progress of the voice data can be synchronized to the electronic whiteboard APP by the voice transcription APP every preset time, and the preset time here can be 1s, 0.5s and the like.

After the current display progress of the voice data is determined, an operation corresponding to the current display progress can be determined from the operations included in the whiteboard data based on the current display progress. Here, the operation corresponding to the current display progress may be an operation occurring before the current display progress, and specifically, the relative execution time of each operation stored in the whiteboard data may be compared with the current display progress, so as to obtain an operation whose relative execution time is before the current display progress.

Here, the relative execution time of each operation stored in the whiteboard data can be directly compared with the current display progress, and no additional time alignment operation is needed, because the relative execution time and the current display progress are relative times taking the start time of voice recording as a start point, that is, the start points of timing of the two are consistent, thus, the operation corresponding to the current display progress can be quickly positioned from the whiteboard data, and the timeliness of synchronous display of the voice data and the whiteboard data is improved.

And step 530, analyzing the operation corresponding to the current display progress to obtain and display an operation image corresponding to the current display progress.

Specifically, after the operation corresponding to the current display progress is obtained, the operation can be analyzed, so that the picture corresponding to the operation corresponding to the current display progress is positioned from the whiteboard data, and accordingly, the operation image corresponding to the current display progress can be obtained and displayed, and therefore synchronous backtracking of voice and an electronic whiteboard display picture at the same moment for the same event is achieved.

According to the method provided by the embodiment of the invention, under the condition that the time axes of the voice data and the whiteboard data are aligned, the operation corresponding to the current display progress of the voice data is directly determined from the whiteboard data, and the synchronous backtracking of the voice and the electronic whiteboard display picture can be realized without manual alignment of a user.

Based on any of the above embodiments, step 520 includes:

Specifically, the respective operations in the whiteboard data may be sequentially arranged according to the relative execution times of the respective operations to form the form of the operation sequence shown in fig. 6. Under the condition that the current display progress is determined, the relative execution time of each operation in the operation sequence can be compared with the current display progress one by one according to the sequence of the operation sequence, so that each operation with the relative execution time before the current display progress is obtained and serves as the operation corresponding to the current display progress. For example, in fig. 6, the operation corresponding to the currently-shown progress is operation i in the operation sequence, and the operation arranged before operation i.

Here, the operation whose relative execution time is before the current display progress, that is, the operation whose execution time is completed before the current display progress, may also be performed on the electronic whiteboard according to the image change of the operation on the electronic whiteboard in the current display progress, and thus this part of the operation may be used as the operation corresponding to the current display progress.

Alternatively, the relative execution time referred to herein may be a relative time at which the execution of the operation starting from the start time of the voice recording is completed.

Based on any of the above embodiments, step 520 includes:

Specifically, when the operation corresponding to the current display progress is determined, if only starting from the current display progress, all the operations with relative execution time before the current display progress are taken as the operations corresponding to the current display progress, which may bring huge calculation amount to subsequent operation analysis and image rendering links, and affect the response speed of synchronous backtracking.

For the situation, when the operation corresponding to the current display progress is determined, the current display progress can be referred to, the last display progress of the current display progress can be referred to, the operation with the relative execution time before the current display progress and after the last display progress is selected from the whiteboard data, and the operation is used as the operation corresponding to the current display progress, namely, under the current display progress, the operation needs to be analyzed and then image rendering is performed.

Compared with the operation that all the operations of which the relative execution time is before the current display progress are taken as the operations corresponding to the current display progress, the operation corresponding to the current display progress determined based on the previous display progress and the current display progress in the embodiment of the invention does not include the operation that analysis and image rendering are already executed when synchronous backtracking is performed under the previous display progress, and only includes the operation that is executed between the previous display progress and the current display progress. For example, assuming that the relative execution time of the operation i-1 in fig. 6 is before the previous display progress, the operation i-1 has already performed the operation parsing and the image rendering when performing the synchronous backtracking for the previous display progress, and thus when performing the synchronous backtracking for the current display progress, the operation i-1 does not need to be repeatedly performed on the operation i-1, and thus the operation i-1 may not be taken as the operation corresponding to the current display progress.

According to the method provided by the embodiment of the invention, the compression of the selection interval of the operation corresponding to the current display progress greatly reduces the calculation amount required by the operation analysis and the image rendering aiming at the current display progress, and is beneficial to improving the response speed of the synchronous backtracking.

Based on any of the above embodiments, step 530 includes:

Specifically, the whiteboard data stores the graphic data corresponding to each operation, and the operation corresponding to the current display progress can be analyzed under the condition that the operation corresponding to the current display progress is determined, so that the identifier of the graphic data corresponding to the operation corresponding to the current display progress is obtained, and then the graphic data corresponding to the identifier is obtained from the whiteboard data, that is, the graphic data corresponding to the current display progress is obtained.

For example, the operation corresponding to the current display progress includes a writing operation, an identifier of the graphic data corresponding to the writing operation can be obtained by analyzing the writing operation, and then the data of the graphic corresponding to the identifier is found from the whiteboard data based on the identifier, that is, the writing data, which may include a brush color, a brush size, a brush type, a brush tip handwriting type, and a handwriting point coordinate.

After the graphic data corresponding to the current display progress is obtained, image rendering needs to be performed based on the graphic data, and under the condition that the number of operations is large and the amount of the graphic data corresponding to the operations is large, a large amount of time and computing resources are consumed for image rendering.

In order to improve the image rendering efficiency, after the graph data of each operation corresponding to the current display progress is determined, the graph data corresponding to the current display progress may be divided, so as to obtain a plurality of graph data sets. For example, in the case of graphic data whose current presentation progress corresponds to 6 operations, 6 groups of graphic data may be divided into 3 graphic data sets. Here, the number of the graphic data sets obtained by the division may be set in advance, and for example, the number of the graphic data sets may be set to match the number of the parallel threads set at the maximum, or may be determined according to the size of the data amount of the graphic data, and for example, the number of the sets divided may be large when the data amount is large, and the number of the sets divided may be small when the data amount is small.

After the graphics data set division is completed, the corresponding thread can be applied to each graphics data set for image rendering. Here, after determining the number of the divided graphics data sets, a corresponding number of threads may be started, and each thread independently processes one graphics data set, or the divided graphics data sets may be allocated to the corresponding threads for processing according to the number of threads started in advance, which is not particularly limited in the embodiment of the present invention.

According to the method provided by the embodiment of the invention, the image rendering time is effectively shortened and the response speed of synchronous backtracking is further improved by multithread parallel processing of the graphic data set.

Based on any of the above embodiments, the current display progress of the voice data includes a display progress of a voice and/or a display progress of a transcribed text of the voice.

Specifically, the voice data may include recorded voice, and transcribed text obtained by transcribing the voice. When the voice data is traced back, only the voice may be played, only the transcription text may be displayed, or the text being played in the transcription text may be highlighted while the voice is played.

Correspondingly, the current display progress of the voice data may be the display progress of the voice, that is, the specific time of the voice playing, or the display progress of the transcribed text, that is, the specific time of the part of the transcribed text in the display window corresponding to the voice, or the specific time of the highlighted part of the transcribed text in the voice playing process, which is not specifically limited in the embodiment of the present invention.

Based on any of the above embodiments, the data backtracking method further includes:

correspondingly, in step 530, the displaying the operation image corresponding to the current display progress includes:

Specifically, in the process of data backtracking, the user can realize the control of the size of the playing window when backtracking the electronic whiteboard through the playing operation, here, the first playing operation is the operation of starting the backtracking of the electronic whiteboard, and the first playing operation specifically can be that the user clicks the playing key of the whiteboard thumbnail on the main interface of the voice transcription APP, so as to start the display of the first playing window. The first play window corresponding to the first play operation may be a small window in the form of a thumbnail.

In addition, in order to make the user watch the backtracking content of the electronic whiteboard more clearly, an amplification key can be arranged on the first playing window, that is, the user can amplify the playing window through the second playing operation based on the first playing window. Here, the second playing operation may be that the user clicks an enlargement button in the first playing window, thereby starting presentation of the second playing window. The second play window corresponding to the second play operation may be a large window in an enlarged form. In the embodiment of the present invention, the second playing window may be a playing window larger than the first playing window.

Correspondingly, in step 530, the operation image corresponding to the current display progress may be displayed in the current playing window, where the current playing window may be the first playing window or the second playing window, and the size of the playing window in the backtracking process may be adjusted in real time along with the playing operation of the user.

Based on any of the above embodiments, the data backtracking method may include the following steps:

the user can click the conference recording button on the main interface of the voice transcription APP, and therefore the user can enter a conference recording playing interface. In the conference record playing interface, a user can click a whiteboard thumbnail playing key to enter a data backtracking interface. Fig. 7 is a schematic diagram of a data backtracking interface provided by the present invention, as shown in fig. 7, a whiteboard and a transcribed text are displayed in the interface, and the current display progress is displayed in a form of a progress bar, and a user can change the current display progress by dragging the progress bar, and correspondingly adjust the progress including the voice playing progress, the progress of highlighting the transcribed text, and an image displayed on the electronic whiteboard according to the current display progress.

In the process, the electronic whiteboard APP can detect the current display progress at regular time, the operation corresponding to the current display progress is determined from the whiteboard data according to the current display progress, the graphic data corresponding to the current display progress is determined through the analysis operation, and the dynamic reappearance of the whiteboard writing explanation process following the current display progress is achieved through image rendering.

Based on any of the above embodiments, fig. 8 is a schematic structural diagram of a data storage device provided by the present invention, and as shown in fig. 8, the data storage device includes:

a start determining unit 810 for determining a start time of voice recording;

a recording unit 820, configured to determine an operation performed by a user on an electronic whiteboard in response to the operation, and determine a relative execution time of the operation based on an absolute execution time of the operation and a start time of the voice recording, where the relative execution time is a relative time starting from the start time of the voice recording;

the storage unit 830 is configured to store the operation and the relative execution time of the operation into whiteboard data, where the whiteboard data corresponds to the same event as the voice data obtained by recording the voice.

According to the device provided by the embodiment of the invention, the whiteboard data with the time axis aligned with the voice data is generated by taking the starting time of voice recording as the relative execution time of the operation determined by the starting point, so that the association of the voice data and the whiteboard data in time is realized, and the voice data and the whiteboard data can be synchronously displayed without manual alignment of a user in the subsequent backtracking process; and moreover, the dynamic process of writing and explaining based on the electronic whiteboard is recorded in the form of whiteboard data, so that the calculation burden is effectively reduced while the dynamic display is realized, and the response efficiency of data storage and backtracking is improved.

Based on any embodiment above, the apparatus further comprises:

the interruption determining unit is used for determining interruption duration based on the pause time and the restart time of voice recording;

the recording unit is specifically configured to:

Based on any of the above embodiments, fig. 9 is a schematic structural diagram of a data backtracking apparatus provided by the present invention, and as shown in fig. 9, the data backtracking apparatus includes:

an obtaining unit 910, configured to obtain voice data and whiteboard data corresponding to a same event, where the whiteboard data includes an operation of a user on an electronic whiteboard and a relative execution time of the operation, and the relative execution time is a relative time with a start time of voice recording as a starting point;

an operation determining unit 920, configured to determine, based on the current display progress of the voice data, an operation corresponding to the current display progress from the whiteboard data;

the display unit 930 is configured to analyze the operation corresponding to the current display progress, and obtain and display an operation image corresponding to the current display progress.

According to the device provided by the embodiment of the invention, under the condition that the time axes of the voice data and the whiteboard data are aligned, the operation corresponding to the current display progress of the voice data is directly determined from the whiteboard data, and the synchronous backtracking of the voice and the electronic whiteboard display picture can be realized without manual alignment of a user.

Based on any of the above embodiments, the operation determination unit is configured to:

Based on any one of the above embodiments, the display unit is configured to:

rendering the plurality of graphic data sets based on the threads respectively corresponding to the plurality of graphic data sets to obtain and display the operation image corresponding to the current display progress.

Based on any of the above embodiments, the data backtracking apparatus further includes a play control unit, configured to:

the display unit is specifically configured to:

Based on any of the above embodiments, a data processing system includes a voice recording device, a data storage device, and a data backtracking device, where the voice recording device is configured to synchronize a start time of voice recording to the data storage device.

Specifically, in the data processing system, the voice recording device may be a voice transcription APP, or an electronic device installed with the voice transcription APP, such as a mobile phone, a computer, or a recording pen. The data storage device may be an electronic whiteboard APP, or an electronic device installed with the electronic whiteboard APP, such as a mobile phone, a computer, a smart television, and the like. The data backtracking device can be an electronic device provided with a voice transcription APP and an electronic whiteboard APP, and can be a mobile phone, a computer, a smart television and the like.

It should be noted that the voice recording device, the data storage device and the data backtracking device may be installed on the same electronic device, or may be installed on different electronic devices.

In addition, in order to realize linkage of voice recording and data storage of the electronic whiteboard, the voice recording device can be communicated with the data storage device, so that the voice recording starting time is transmitted to the data storage device, the data storage device can generate whiteboard data with a time axis aligned with the voice data according to the relative execution time of the operation determined by taking the voice recording starting time as a starting point, and therefore correlation of the voice data and the whiteboard data in time is realized, and in the subsequent backtracking process, the voice data and the whiteboard data can be synchronously displayed without manual alignment of a user.

Fig. 10 illustrates a physical structure diagram of an electronic device, and as shown in fig. 10, the electronic device may include: a processor (processor)1010, a communication Interface (Communications Interface)1020, a memory (memory)1030, and a communication bus 1040, wherein the processor 1010, the communication Interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. Processor 1010 may invoke logic instructions in memory 1030 to perform a data storage method that includes determining a start time for a voice recording; responding to the operation of a user on the electronic whiteboard, determining the operation, and determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of the voice recording, wherein the relative execution time is relative time taking the start time of the voice recording as a starting point; and storing the operation and the relative execution time of the operation into whiteboard data, wherein the whiteboard data and the voice data obtained by voice recording correspond to the same event.

A data backtracking method may also be performed, the method comprising: acquiring voice data and whiteboard data corresponding to the same event, wherein the whiteboard data comprises user operation on an electronic whiteboard and relative execution time of the operation, and the relative execution time is relative time taking the starting time of voice recording as a starting point; determining an operation corresponding to the current display progress from the whiteboard data based on the current display progress of the voice data; and analyzing the operation corresponding to the current display progress to obtain and display the operation image corresponding to the current display progress.

Furthermore, the logic instructions in the memory 1030 can be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product comprising a computer program storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of performing a data storage method comprising determining a start time of a voice recording; responding to the operation of a user on the electronic whiteboard, determining the operation, and determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of the voice recording, wherein the relative execution time is relative time taking the start time of the voice recording as a starting point; and storing the operation and the relative execution time of the operation into whiteboard data, wherein the whiteboard data and the voice data obtained by voice recording correspond to the same event.

In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program that when executed by a processor performs a data storage method, the method including determining a start time of a voice recording; responding to an operation of a user on an electronic whiteboard, determining the operation, and determining a relative execution time of the operation based on an absolute execution time of the operation and a start time of the voice recording, wherein the relative execution time is a relative time with the start time of the voice recording as a starting point; and storing the operation and the relative execution time of the operation into whiteboard data, wherein the whiteboard data and the voice data obtained by recording the voice correspond to the same event.

A data backtracking method may also be performed, the method comprising: acquiring voice data and whiteboard data corresponding to the same event, wherein the whiteboard data comprises the operation of a user on the electronic whiteboard and the relative execution time of the operation, and the relative execution time is relative time taking the starting time of voice recording as a starting point; determining an operation corresponding to the current display progress from the whiteboard data based on the current display progress of the voice data; and analyzing the operation corresponding to the current display progress to obtain and display the operation image corresponding to the current display progress.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A method of storing data, comprising:

determining the starting time of voice recording;

responding to the operation of a user on the electronic whiteboard, determining the operation, and determining the relative execution time of the operation based on the absolute execution time of the operation and the start time of the voice recording, wherein the relative execution time is relative time taking the start time of the voice recording as a starting point;

2. The data storage method of claim 1, further comprising:

3. A data backtracking method, comprising:

4. The data backtracking method according to claim 3, wherein the determining, based on the current display progress of the voice data, an operation corresponding to the current display progress from the whiteboard data includes:

5. The data backtracking method according to claim 4, wherein the obtaining, from the whiteboard data, the operation of which the relative execution time is before the current display progress as the operation corresponding to the current display progress comprises:

6. The data backtracking method according to any one of claims 3 to 5, wherein the analyzing the operation corresponding to the current display progress to obtain and display an operation image corresponding to the current display progress includes:

7. The data backtracking method according to any one of claims 3 to 5, wherein the current display progress of the voice data comprises a display progress of a voice and/or a display progress of a transcribed text of the voice.

8. The data backtracking method according to any one of claims 3 to 5, further comprising:

9. A data storage device, comprising:

and the storage unit is used for storing the operation and the relative execution time of the operation into the whiteboard data, and the whiteboard data and the voice data obtained by recording the voice correspond to the same event.

10. A data backtracking apparatus, comprising:

11. A data processing system comprising voice recording means and data storage means as claimed in claim 9, data backtracking means as claimed in claim 10, said voice recording means being arranged to synchronise the start time of voice recording to said data storage means.