CN115623263A

CN115623263A - Synchronous access display method and device for operation images and parameters

Info

Publication number: CN115623263A
Application number: CN202211203784.9A
Authority: CN
Inventors: 请求不公布姓名; 朱祥
Original assignee: Shanghai Microport Medbot Group Co Ltd
Current assignee: Shanghai Microport Medbot Group Co Ltd
Priority date: 2022-09-29
Filing date: 2022-09-29
Publication date: 2023-01-17

Abstract

The invention provides a method and a device for synchronously accessing and displaying surgical images and parameters, which comprises the following steps: collecting a multi-source video file, and storing according to a video frame timestamp of the multi-source video file; collecting robot operating parameters, and storing according to parameter timestamps of the robot operating parameters; responding to the playing operation of a user on the multi-source video file, and establishing a playing time axis according to the video frame timestamp and the parameter timestamp; synchronously playing and splicing the multi-source video files according to a playing time axis and the video frame time stamp, and determining spliced videos; drawing a parameter curve according to the playing time axis and the parameter timestamp; and playing the spliced video and synchronously displaying the real-time state of the parameter curve according to the playing time axis. And independently storing the multi-source video, and recording the video frame timestamp to realize accurate synchronous storage of the multi-source video. And synchronously drawing a core parameter curve and playing the associated multi-source video by acquiring the data of the robot motion control algorithm and the timestamp of each parameter.

Description

Synchronous access display method and device for operation images and parameters

Technical Field

The invention relates to the technical field of video data processing, in particular to a method and a device for synchronously accessing and displaying an operation image and parameters.

Background

This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

The existing video recording mode of the surgical robot generally comprises the steps that each scene records an independent video, a plurality of scenes are recorded in one surgery and can be recorded into a plurality of videos, and each video is manually and independently played for operation duplication; the recording of a plurality of network camera videos of a monitoring system on the market is realized by splicing a multi-source video code stream into a picture and then storing the spliced picture; typically the granularity of video storage for recording is large video files.

When a technician teaches or performs a repeated operation, videos of a plurality of operation scenes such as a doctor control end, an operating room, an endoscope and the like need to be opened at the same time, the multi-source video synchronization time of two scenes is difficult to find accurately, the operation is inconvenient, certain specialty is needed, and time and labor are wasted; the core parameter change of the motion control algorithm of the surgical robot and the master operation picture are not synchronously displayed, so that the problem of abnormal operation or postoperative abnormality is inconvenient to quickly position and solve.

Therefore, how to provide a new solution, which can solve the above technical problems, is a technical problem to be solved in the art.

Disclosure of Invention

The embodiment of the invention provides a synchronous access display method of operation images and parameters, which is used for independently storing multi-source videos and simultaneously recording the timestamp of each video frame, thereby realizing the accurate synchronous storage of the videos with different multi-source frame rates and different resolutions and the storage of high-definition videos. The method comprises the following steps of synchronously drawing a core parameter curve and playing a related multi-source video by collecting data of a robot motion control algorithm and time stamps of each parameter, wherein the method comprises the following steps:

acquiring a multi-source video file, and storing according to a video frame timestamp of the multi-source video file;

collecting robot operating parameters, and storing according to parameter timestamps of the robot operating parameters;

responding to the playing operation of a user on the multi-source video file, and establishing a playing time axis according to the video frame timestamp and the parameter timestamp;

synchronously playing and splicing the multi-source video files according to a playing time axis and the video frame time stamp, and determining spliced videos;

drawing a parameter curve according to the playing time axis and the parameter timestamp;

and playing the spliced video and synchronously displaying the real-time state of the parameter curve according to the playing time axis.

The embodiment of the present invention further provides a device for synchronously accessing and displaying surgical images and parameters, including:

the multi-source video file acquisition and storage module is used for acquiring the multi-source video file and storing the multi-source video file according to the video frame timestamp of the multi-source video file;

the robot running parameter acquisition and storage module is used for acquiring the robot running parameters and storing the robot running parameters according to the parameter timestamps of the robot running parameters;

the playing time axis establishing module is used for responding to the playing operation of a user on the multi-source video file and establishing a playing time axis according to the video frame timestamp and the parameter timestamp;

the spliced video determining module is used for synchronously playing and splicing the multi-source video file according to the playing time axis and the video frame time stamp and determining a spliced video;

the parameter curve drawing module is used for drawing a parameter curve according to the playing time axis and the parameter timestamp;

and the synchronous playing module is used for playing the spliced video and synchronously displaying the real-time state of the parameter curve according to the playing time axis.

The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when the computer program is executed by a processor, the synchronous access display method of the operation image and the operation parameters is realized.

The embodiment of the invention provides a method and a device for synchronously accessing and displaying surgical images and parameters, wherein the method comprises the following steps: acquiring a multi-source video file, and storing according to a video frame timestamp of the multi-source video file; collecting robot operating parameters, and storing according to parameter timestamps of the robot operating parameters; responding to the playing operation of a user on the multi-source video file, and establishing a playing time axis according to the video frame timestamp and the parameter timestamp; synchronously playing and splicing the multi-source video files according to a playing time axis and the video frame time stamp, and determining spliced videos; drawing a parameter curve according to a playing time axis and the parameter timestamp; and playing the spliced video and synchronously displaying the real-time state of the parameter curve according to the playing time axis. By accurately and synchronously playing the multi-source video files of a plurality of scenes, the problems of complexity and specialty of synchronously playing the multi-source video when technical personnel re-run a disk, high operation difficulty and difficult control of synchronization precision are solved; by drawing a parameter curve, the main hand operation picture, the endoscope picture and the core parameter change of the robot motion control algorithm are synchronously displayed, so that the method is beneficial to quickly positioning and solving the problems in the operation, and is convenient for analyzing and adjusting the parameters of the motion control algorithm after the operation; according to the invention, the multi-source video files of a plurality of operation scenes are accurately and synchronously played, so that the scenes when problems occur can be highly restored, the problems can be conveniently re-recorded by development, clinic and customer service, and the re-recording efficiency is greatly improved; by drawing a parameter curve, the real-time state change condition of the core parameter data of the robot motion control algorithm can be synchronously displayed while the video recorded in the operation is played, and the running condition of the robot can be judged according to the real-time state change condition of the parameter data, so that the operation of the robot is convenient to rerun; through the instant playback function, problems occurring in the operation can be conveniently positioned, fed back and solved in time, and the platform following efficiency is greatly improved; according to the invention, the multi-source videos are stored independently, and the timestamp of each video frame is recorded simultaneously, so that the accurate synchronous storage of the multi-source videos with different frame rates and different resolutions and the storage of the high-definition videos are realized. And synchronously drawing a core parameter curve and playing the associated multi-source video by acquiring the data of the robot motion control algorithm and the timestamp of each parameter.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:

fig. 1 is a schematic view illustrating a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 2 is a structural diagram of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 3 is a near-field multi-source video scene diagram of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 4 is a remote multi-source video scene diagram of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 5 is a data acquisition flowchart of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 6 is a configuration flowchart of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 7 is a flowchart illustrating a login cloud server of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 8 is a flowchart of a method for synchronously accessing and displaying surgical images and parameters for acquiring a multi-source video file according to an embodiment of the present invention.

Fig. 9 is a flow chart of a method for synchronously storing video frames for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 10 is a parameter acquisition flowchart of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 11 is a flowchart illustrating a synchronous parameter storage method for displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 12 is a flowchart illustrating synchronous reading and playing of a mosaic video and a parameter curve according to a method for synchronously accessing and displaying an operation image and a parameter according to an embodiment of the present invention.

Fig. 13 is a schematic view of a playing time axis of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 14 is a flow chart of multi-source video stitching of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 15 is a schematic diagram of horizontal YUV420P format stitching in a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 16 is a schematic diagram of longitudinal YUV 420P-format stitching in a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 17 is a flow chart of a synchronous parameter curve drawing method for displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 18 is a schematic view illustrating a display of a merged video and a parameter curve in a method for synchronously accessing and displaying an operation image and parameters according to an embodiment of the present invention.

Fig. 19 is a flowchart illustrating a local video image magnification processing method for a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 20 is a flow chart of a video local storage of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

FIG. 21 is a schematic diagram of a computer apparatus for executing a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Fig. 22 is a schematic view of a device for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.

Fig. 1 is a schematic diagram illustrating a method for displaying surgical images and parameters by synchronous access according to an embodiment of the present invention, and as shown in fig. 1, an embodiment of the present invention provides a method for displaying surgical images and parameters by synchronous access, which separately stores multi-source videos, and simultaneously records a timestamp of each video frame, thereby implementing accurate synchronous storage of videos with different frame rates and different resolutions from the multi-source videos, and storage of high-definition videos. The method comprises the following steps of synchronously drawing a core parameter curve and playing a related multi-source video by acquiring data of a robot motion control algorithm and time stamps of each parameter, wherein the method comprises the following steps:

step 101: acquiring a multi-source video file, and storing according to a video frame timestamp of the multi-source video file;

step 102: collecting robot operating parameters, and storing according to parameter timestamps of the robot operating parameters;

step 103: responding to the playing operation of a user on the multi-source video file, and establishing a playing time axis according to the video frame timestamp and the parameter timestamp;

step 104: synchronously playing and splicing the multi-source video files according to a playing time axis and the video frame time stamp, and determining spliced videos;

step 105: drawing a parameter curve according to a playing time axis and the parameter timestamp;

step 106: and playing the spliced video and synchronously displaying the real-time state of the parameter curve according to the playing time axis.

The embodiment of the invention provides a method for synchronously accessing and displaying surgical images and parameters, which comprises the following steps: acquiring a multi-source video file, and storing according to a video frame timestamp of the multi-source video file; collecting robot operating parameters, and storing according to parameter timestamps of the robot operating parameters; responding to the playing operation of a user on the multi-source video file, and establishing a playing time axis according to the video frame timestamp and the parameter timestamp; synchronously playing and splicing the multi-source video files according to a playing time axis and the video frame time stamps, and determining spliced videos; drawing a parameter curve according to a playing time axis and the parameter timestamp; and playing the spliced video and synchronously displaying the real-time state of the parameter curve according to the playing time axis. By accurately and synchronously playing the multi-source video files of a plurality of scenes, the problems of complexity and specialty of synchronously playing the multi-source video when technical personnel re-record, high operation difficulty and difficult control of synchronization precision are solved; by drawing a parameter curve, the main hand operation picture, the endoscope picture and the core parameter change of the robot motion control algorithm are synchronously displayed, so that the quick positioning and solution of the problems in the operation are facilitated, and the analysis and the optimization of the parameters of the motion control algorithm after the operation are facilitated; according to the invention, the multi-source video files of a plurality of operation scenes are accurately and synchronously played, so that the scenes when problems occur can be highly restored, the problems can be conveniently re-recorded by development, clinic and customer service, and the re-recording efficiency is greatly improved; by drawing a parameter curve, the real-time state change condition of the core parameter data of the robot motion control algorithm can be synchronously displayed while the video recorded in the operation is played, and the running condition of the robot can be judged according to the real-time state change condition of the parameter data, so that the operation of the robot is convenient to rerun; through the instant playback function, problems occurring in the operation can be conveniently positioned, fed back and solved in time, and the platform following efficiency is greatly improved; according to the invention, multi-source videos are stored independently, and the timestamp of each video frame is recorded at the same time, so that the accurate synchronous storage of the multi-source videos with different frame rates and different resolutions and the storage of high-definition videos are realized. And synchronously drawing a core parameter curve and playing the associated multi-source video by acquiring the data of the robot motion control algorithm and the timestamp of each parameter.

In the prior art, when videos in a gesture process are played back, videos collected by a plurality of video signal sources need to be opened at the same time, the video synchronization time of two scenes is difficult to find accurately, the operation is inconvenient, certain specialty is needed, and time and labor are wasted; the core parameter change of the motion control algorithm of the surgical robot and the master operation picture are not synchronously displayed, so that the problem of abnormal operation or postoperative abnormality is inconvenient to quickly position and solve. The playing of the multi-source video cannot realize the personalized layout playing of the dynamically selected video source, and the definition is not high generally; the recorded video can be viewed and played back only when the operation is finished, and the video at the previous moment cannot be viewed in time in the operation; the multi-source video synchronous playing is limited by the resolution of the display, the video picture is compressed, and the definition of a local picture cannot be improved at the limited resolution of the display.

When the method for synchronously accessing and displaying the surgical image and the surgical parameters provided by the embodiment of the invention is implemented, in one embodiment, the method comprises the following steps:

acquiring a multi-source video file consisting of two or more than two signal sources, and storing the multi-source video file according to a video frame timestamp of the multi-source video file;

drawing a parameter curve according to a playing time axis and the parameter timestamp;

Fig. 2 is an architecture diagram of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 2, the embodiment of the present invention provides a method for synchronously accessing and displaying surgical images and parameters, which needs to include a robot system end, a cloud server end, and a player end when applied;

the robot system end is a multi-source video data and parameter data generating end, video data are encoded by H264/H265 to generate multi-source video files and uploaded to a cloud server through an RTC (Real-Time Communication), and a plurality of robot operation parameter data are uploaded to the cloud server through a group package and comprise a video acquisition unit, a parameter acquisition unit and a configuration information setting unit; the configuration information setting unit is used for configuring the cloud server information, the operation information, the parameters and the acquired streaming media signal source.

The cloud server end is used for storing and managing video data and parameter data and comprises a login unit, a video synchronous storage server and a parameter synchronous storage server; the login unit is used for verifying the identities of the data uploading end and the data downloading end; the video synchronous storage server is used for accurately and synchronously storing multi-source video data and managing the video data; the parameter synchronous storage server is used for accurately and synchronously storing the parameter data and managing the parameter data.

The playing end is used for accurately and synchronously playing the multi-source video and drawing the parameter curve, and supports local storage and local amplification of the video; the system comprises a multi-source video synchronous playing unit, a parameter curve synchronous drawing unit and a video local amplifying unit; video data, parameter data and splicing layout which need to be cached are obtained according to layout analysis, and the played video source and parameters can be dynamically adjusted through a dynamic layout setting function.

By storing the multi-source videos individually in a small granule mode and recording the time stamp (video metadata file) of each video frame, the accurate synchronous storage of the videos with different multi-source frame rates and different resolutions and the storage of the high-definition videos are realized; the playing client caches the video frames and the metadata from the cloud server to the local, dynamically establishes a proper playing time axis by analyzing the metadata, synchronously splices and plays the multi-source video frames according to the playing time axis, and realizes accurate synchronous playing of the multi-source video file; the problems of complexity and specialty of multi-source video synchronous playing of technical personnel during repeated disk and teaching work, high operation difficulty and difficulty in controlling synchronous precision are solved. By storing the robot operating parameters and recording the time stamps (parameter metadata) of the parameters, the main hand operation picture, the endoscope picture and the core parameter change curve of the robot motion control algorithm are synchronously displayed through synchronous analysis, so that the method is beneficial to quick positioning and solving of problems in the operation, and is convenient for analysis and optimization of the parameters of the postoperative motion control algorithm.

The embodiment of the invention can be applied to both near-field multi-source video scenes and remote multi-source video scenes. The method supports remote distributed acquisition of streaming media signal sources of multiple devices, and is more suitable for operation video recording work of multiple remote robots.

Fig. 3 is a near-field multi-source video scene diagram of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, as shown in fig. 3, in the near-field multi-source video scene, the robots at the master and slave ends are in the same operating room, and the surgical equipment includes a doctor control end, a patient operating end device, an image trolley, and other auxiliary equipment in surgery, such as a sterile table, a ventilator, a detection device, etc. The image acquisition terminal is connected with the video output of the image trolley to acquire endoscope images, and is connected with the panoramic camera and the master camera, video data are pushed to the cloud server through an RTC (real time clock) protocol, and the cloud server stores the video data; the parameter acquisition terminal and the robot system are connected to acquire the robot operating parameters, and meanwhile, the parameter data are uploaded to the cloud server, and the cloud server stores the parameter data. And the playing terminal downloads the video data and the robot parameter data by connecting with the cloud server, and plays and draws the video data and the robot parameter data.

Fig. 4 is a remote multi-source video scene diagram of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, as shown in fig. 4, the remote multi-source video scene is similar to the above-mentioned near-field multi-source video scene, but the main differences include: the first end robot and the master-slave end robot are respectively arranged in a local operating room and a remote master end in different operating rooms, and the second end robot, the local operating room and the remote operating room are respectively provided with a parameter acquisition terminal and an image acquisition terminal.

Fig. 5 is a data acquisition flow chart of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 5, a system configuration flow needs to be executed first, a cloud server, a surgical robot, and acquired related information are configured, then a cloud server is logged in through identity authentication, a login failure flow is ended, the login is successful, surgical information (a surgical ID, a surgical name, hospital information, and the like) is read from the configuration information and uploaded to the cloud server, and multiple threads are started to respectively acquire parameter data and multi-source video data.

Fig. 6 is a configuration flowchart of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, as shown in fig. 6, when performing system configuration according to an embodiment of the present invention, a cloud server needs to be configured first: configuring an IP (Internet protocol) and a port number of a cloud server; then, configuring the operation information: including operation ID, operation name, hospital information, etc.; then, configuring the collected parameters: configuring video acquisition configuration information, configuring parameter acquisition configuration information, configuring a parameter list to be acquired, and generating a parameter description file according to the configuration; and finally, configuring a collection stream media signal source: the method is to configure video signals of a camera, an endoscope or a desktop and the like to be acquired.

Fig. 7 is a flowchart of a login cloud server of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 7, the login server process includes a cloud server side process and a terminal side process.

The cloud server side process comprises the following steps: and starting the server to monitor the appointed port to wait for the connection of the client, receiving an identity authentication request after the connection is established, authenticating login information by the server, and returning an authentication result to the terminal.

A terminal-side process, comprising: inputting login information, reading the ip and the port number of the cloud server from local configuration, connecting the cloud server by using the ip and the port of the cloud server, acquiring the login information from the input, sending a login instruction to the cloud server, and receiving a result returned by the cloud server.

Fig. 8 is a flowchart of a multi-source video file acquisition process of a method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention, and as shown in fig. 8, when the method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention is implemented, in an embodiment, the method further includes: setting video acquisition configuration information;

the acquiring multi-source video file comprises:

acquiring a plurality of target acquisition video signal sources according to video acquisition configuration information, starting multithreading to acquire multi-source video data of the plurality of target acquisition video signal sources, encoding the multi-source video data, and determining a multi-source video file; the target acquisition video signal source at least comprises: endoscope signal source, main hand camera signal source, panorama camera signal source.

In the embodiment, the video acquisition configuration information is already set, and further, the multi-source video file is acquired by reading the video acquisition configuration information, acquiring a plurality of video signal sources to be acquired, starting multithreading to start acquisition, and encoding the acquired video data by the encoding unit (H264/H265) to obtain the multi-source video file; and transmitting the encoded multi-source video file to a cloud server in real time through an RTC protocol.

In the prior art, the multi-source video code stream is spliced into a picture and then stored, and the storage granularity of the recorded video is a large video file; therefore, the playing of the multi-source video cannot realize the personalized layout playing of the dynamically selected video source, the definition is not high generally, the recorded video can be viewed and played back only after the operation is finished, and the video at the previous moment cannot be viewed in time in the operation.

Fig. 9 is a flow chart of a video frame synchronous storage method for a surgical image and parameter synchronous access display method according to an embodiment of the present invention, as shown in fig. 9, when the method for surgical image and parameter synchronous access display according to an embodiment of the present invention is specifically implemented, in an embodiment, storing a multi-source video file according to a video frame timestamp of the multi-source video file includes:

creating a video metafile and a video file; the video file is preset with a video data frame number threshold; the frame number threshold of the video data is an integral multiple of the frame rate of the video data;

analyzing the multi-source video file to determine video data;

writing the video data into a video file, and writing a video frame timestamp of the video data written into the video file into a video metafile;

judging whether the frame number of the video data written into the video file reaches a preset video data frame number threshold, if so, closing the current video file, creating a next video file and correspondingly creating a next video metafile, continuously writing the residual video data into the next video file until the video data is empty, and closing the finally written video file and the video metafile.

When the method for synchronously accessing and displaying surgical images and parameters provided by the embodiment of the present invention is implemented, in an embodiment, the method further includes: and if the frame number of the video data written into the video file does not reach the preset video data frame number threshold value, but the video data is empty, closing the current video file and the video metafile.

In the embodiment, a cloud server is used for realizing synchronous storage of a multi-source video file and a video frame timestamp, a video metafile and a video file are mainly stored, and the main process comprises the following steps:

(1) firstly, creating a metafile and a video file;

(2) the method comprises the steps of analyzing H264/H265 format video data after RTC data are received, writing the video data into a video file created in the front, and recording a timestamp when the video data are received into a video metafile;

(3) if the video data reception is finished, closing the opened video metafile and the opened video file, ending the whole process, otherwise, judging whether the video frame written in the current video file reaches the threshold value (integral multiple of the frame rate) of the frame number of the single video file with the preset value, if not, executing the process (2), otherwise, closing the last opened video file and the last opened video metafile, and then executing the process (1) and (2).

According to the embodiment of the invention, the multi-source video file is divided into the plurality of video files for small granular storage, so that the operation video of the local field can be played back in the video recording process, and the problem that the operation video cannot be played back in time is solved; through independent storage and accurate synchronous data of the video source, high-definition storage of the video is achieved, the problem that definition is sacrificed to multi-source videos is solved, and the problems that spliced videos cannot be dynamically selected and the video layout cannot be adjusted during playing are solved.

The embodiment of the invention realizes two points, namely, instant playback: the video can be quickly played back by carrying out small granular storage on each path of video stream, so that the intraoperative and postoperative quick playback is realized, and the waiting time before playing and after dragging is reduced. Secondly, accurate synchronous playing: by caching each video file individually and calculating the play time offset of each video frame through the video metadata of each video file, accurate synchronous play is realized. Moreover, each video source is an independent file, so that the definition cannot be reduced, and the playing software supports dynamic adjustment and layout synchronous splicing playing of one or more video sources dynamically selected for one operation.

Fig. 10 is a parameter acquisition flowchart of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 10, when a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention is specifically implemented, in an embodiment, the method further includes: setting parameter acquisition configuration information;

collecting robot operating parameters, including:

reading parameter names from the parameter acquisition configuration information, determining data types and data lengths according to the parameter names, and determining the parameter names, the data types and the data lengths as parameter description information;

and acquiring the running parameters of the robot according to the parameter description information and the set frequency.

In the embodiment, the parameter acquisition configuration information is already set, and further, when the operation parameters of the robot are acquired, the parameter names can be read from the parameter acquisition configuration information, and the parameter description information is uploaded, wherein the description information comprises the parameter names, the data types and the data lengths, and the parameter description information is determined and used for analyzing the parameters during playing; the robot system collects the data of the robot operation parameters through the ADS interface according to the fixed frequency, encapsulates all the collected robot operation parameter data into UDP messages, and sends the UDP messages to the cloud server through a UDP protocol.

Fig. 11 is a flow chart of a synchronous storage parameter of a method for synchronously accessing and displaying an operation image and a parameter according to an embodiment of the present invention, and as shown in fig. 11, when the method for synchronously accessing and displaying an operation image and a parameter according to an embodiment of the present invention is specifically implemented, in an embodiment, the storing a robot operating parameter according to a parameter timestamp of the robot operating parameter includes:

creating a parameter metafile, a parameter data file and a parameter description file;

writing the parameter description information into a parameter description file, receiving the robot operation parameters, and writing the robot operation parameters into a parameter data file;

and writing the parameter time stamp when the robot operation parameter is received into the parameter metafile.

In the embodiment, when the cloud server synchronously stores the parameters, the method comprises the steps of storing three types of data, namely parameter metadata, parameter description data and parameter data; the parameter metadata is stored in a metafile, a parameter timestamp when each piece of parameter data is received is recorded, the parameter description data records a parameter name, a data type and a data length, and the parameter file stores robot operation parameter data.

Fig. 12 is a flowchart illustrating a process of synchronously reading and playing a mosaic video and a parameter curve according to a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, as shown in fig. 12, the process of synchronously playing the mosaic video and the parameter curve is as follows:

(1) the user logs in the cloud server, and if the login fails and the error is reported, the process is ended; the successful login process continues;

(2) caching a parameter description file, acquiring a parameter list through the parameter description file (comprising parameter names, lengths and types and used for analyzing all parameters), selecting parameters to be drawn in the parameter list, and caching a corresponding parameter metafile;

(3) selecting a played video source, and caching a corresponding video metafile;

(4) analyzing the parameter metafile and the video metafile to establish a virtual playing time axis;

(5) caching the parameter data to a parameter queue;

(6) caching each path of video frame to each video queue;

(7) acquiring parameter data from the parameter cache queue, and synchronously drawing a parameter curve;

(8) analyzing the video frames from the cache video queue, splicing the multiple paths of video frames, and then rendering and splicing to obtain a spliced video;

(9) judging whether local storage is needed, if so, locally storing;

and (c) judging whether the playing is finished or not, if not, circularly executing (5) (6) (7) (8) (9), otherwise, clearing the cache file and finishing the playing.

In the playing process, the video can be played in a local amplification mode through the local amplification selection frame and the video area needing to be amplified in the frame, and whether the played multi-channel spliced video is stored locally or not can be selected in the playing process.

Fig. 13 is a schematic view of a playing time axis of a method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention, as shown in fig. 13, when a method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention is specifically implemented, in an embodiment, a playing time axis is established according to a video frame timestamp and a parameter timestamp in response to a user playing an multisource video file, and the method includes:

responding to the playing operation of the user on the multi-source video file, determining the video file selected to be played by the user, and caching the video metafile corresponding to the video file;

acquiring a parameter list according to the parameter description file, determining a target drawing parameter in the parameter list, and caching a parameter metafile corresponding to the target drawing parameter;

and establishing a playing time axis according to the video frame time stamp in the video metafile and the parameter time stamp in the parameter metafile.

Specifically, when the method for synchronously accessing and displaying surgical images and parameters provided by the embodiment of the present invention is implemented, in one embodiment, the establishing of the playing time axis according to the video frame time stamp in the video metafile and the parameter time stamp in the parameter metafile includes:

arranging video frame time stamps in the video metafile and parameter time stamps in the parameter metafile in parallel, and determining a plurality of metadata time axes; the metadata timeline, comprising: a video metadata time axis obtained by arranging the video frame time stamps along the time increment and/or a parameter metadata time axis obtained by arranging the parameter time stamps along the time increment; the interval between the video frame time stamps or the parameter time stamps of the metadata time axis is the sampling rate during acquisition;

the playing time axis is constructed by taking the earliest time stamp in the metadata time axes as a starting point, taking the latest time stamp in the metadata time axes as an end point and taking the sampling rate of the metadata time axis with the highest sampling rate as an interval to set the playing time stamps.

In an embodiment, the time axis of the metadata storage represents a time stamp list of video data or parameter data stored in the server, the interval of the time axis represents the sampling rate of acquisition, and the higher the density is, the higher the sampling rate is; the starting point of the playing time axis is the earliest time stamp in all metadata, and the end point is the latest time stamp in all metadata; the interval of the time axis is the interval of metadata with the highest sampling rate (the smaller the interval is, the more accurate the synchronization is, but the smaller the interval is, the performance of the cloud server is influenced).

Fig. 14 is a flowchart of a multi-source video stitching process of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 14, when a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention is specifically implemented, in an embodiment, according to a play time axis and a video frame timestamp, a multi-source video file is synchronously played and stitched, and a stitched video is determined, including:

caching a video file to a video queue;

extracting a plurality of paths of synchronous video frames of a target acquisition video signal source corresponding to a playing time stamp from a video queue according to the currently played playing time stamp of a playing time axis;

decoding the multi-channel synchronous video frames into a YUV format, splicing the multi-channel synchronous video frames in the YUV format according to a preset layout mode, rendering the spliced multi-channel synchronous video frames in the YUV format, and determining a spliced video.

In the embodiment, the synchronous playing and splicing of the multi-source video files mainly comprises the following steps: and (1) forward fetching the nearest video frame data from the queue according to the playing time stamp currently played by the playing time axis, and when the nearest video frame is not fetched from the video 1 queue and the

video

2,3 queue finds the nearest video frame, splicing the video 1 by continuously using the last video frame and the new video frame of the

video

2,3. And finding a synchronous video frame by analogy, decoding the video frame into a YUV format firstly when finding the synchronous video frame, and splicing the YUV data according to the set layout to form new YUV data serving as a spliced video.

In the foregoing preset layout manner, in the embodiment of the present invention, horizontal splicing may be adopted, as shown in fig. 15, in the YUV420P format horizontal splicing manner, (1) is a data format spliced to the YUV on the left, (2) is a data format spliced to the YUV on the right, and (3) is a data format after (1) and (2) horizontal splicing.

The above-mentioned predetermined layout manner, in the embodiment of the present invention, may adopt a longitudinal splicing manner, as shown in fig. 16,

in the YUV420P format transverse splicing mode, (1) is the data format spliced to the upper YUV, (2) is the data format spliced to the lower YUV, and (3) is the data format after longitudinal splicing of (1) and (2).

Fig. 17 is a flow chart of a parameter curve synchronously drawn by a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 17, when the method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention is implemented, in an embodiment, drawing a parameter curve according to a play time axis and a parameter timestamp includes:

caching the parameter data file to a parameter queue;

extracting parameter data corresponding to the playing time stamp from the parameter queue according to the playing time stamp currently played by the playing time axis;

analyzing the parameter data according to the parameter description information, and determining the value and the parameter name of the drawing parameter;

and taking the playing time stamps as a horizontal axis, taking the values of the drawing parameters as a vertical axis, establishing a two-dimensional coordinate axis, taking each playing time stamp and each value of the drawing parameters as a coordinate point, connecting the coordinate points front and back, and drawing to obtain a parameter curve.

In the embodiment, in the process of synchronously drawing the parameter curve: (1) and taking the nearest parameter data from the queue according to the playing time stamp currently played by the playing time shaft. (2) And analyzing corresponding parameters according to the data length and the data type according to bits. (3) The names of the parameters are read from the parameter description file for display. (4) The parameter curve is drawn by drawing a parameter coordinate point connecting line in a two-position coordinate axis, the X axis represents playing time stamps (moving backwards uniformly in the playing process), each time axis represented by the Y axis corresponds to the parameter size, each time axis (playing time stamp and parameter value) forms a coordinate point, and the front point and the rear point are connected to finish the curve drawing of the parameters.

Fig. 18 is a schematic view illustrating playing and displaying a spliced video and a parameter curve of a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 18, when a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention is implemented, in an embodiment, playing the spliced video and synchronously displaying a real-time state of the parameter curve according to a playing time axis includes:

and according to the playing time stamp of the current playing of the playing time axis, playing the spliced video corresponding to the playing time stamp of the current playing, and mapping the value of the drawing parameter corresponding to the playing time stamp of the current playing onto the parameter curve to synchronously display the real-time state of the parameter curve.

In the embodiment, a plurality of

video pictures

1, 2,3 and 4 are spliced to obtain spliced videos, the spliced videos are played and displayed, and further, a parameter curve window of a parameter curve is correspondingly played and displayed, the parameter curve window can be dragged, when a user selects the parameter curve window, the parameter curve window can move along with the mouse of the user, and when the user releases the mouse, the parameter curve window stays at the position when the mouse is released.

Fig. 19 is a flowchart illustrating a local video image magnification processing of a method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention, and as shown in fig. 19, when the method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention is implemented, in an embodiment, the method further includes:

setting a local amplification selection frame in an interface for playing the spliced video; the local amplification selection frame is a rectangular frame which is 100% transparent; the boundary of the local amplification selection frame is smaller than the interface for playing the spliced video;

a local amplification floating window is arranged outside an interface for playing the spliced video;

calculating image data needing to be amplified in the spliced video according to the coordinate point of the local amplification selection frame;

amplifying the image data to be amplified according to a preset amplification ratio by a bilinear interpolation algorithm to determine a locally amplified image;

and rendering and displaying the local amplification image in the local amplification floating window.

In an embodiment, referring to fig. 18 and 19, the video image local enlargement processing flow includes: calculating a coordinate point of the selected amplification area, calculating image data to be amplified according to the coordinate point, reading the proportion to be amplified, amplifying the spliced image through a bilinear interpolation algorithm, then creating a draggable floating window, rendering the amplified image in the floating window, and finishing the local amplification of the image.

Through the local amplification module, the problems that the resolution of a display is limited and the definition of a local picture cannot be increased in the process of compound disk or teaching are solved. The original picture before splicing is used for realizing image zooming through a bilinear interpolation algorithm, and the selected area is locally amplified, so that the definition of the amplified local area image is increased.

Fig. 20 is a flow chart of a video local storage of a method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention, and as shown in fig. 20, when the method for displaying surgical images and parameters in a synchronous access manner according to an embodiment of the present invention is implemented, in an embodiment, the method further includes:

creating a storage file, and writing file header information;

and performing video coding processing on the spliced video, writing the video into a storage file, writing file tail information after the spliced video is completely written into the storage file, closing the storage file, and determining a local storage video file.

In an embodiment, the video local storage process includes: storing the spliced YUV data into a queue, taking out the YUV data from the queue, starting multithreading (the number of threads can be determined according to the performance of a terminal) to encode the YUV data, writing the encoded video frame into a storage file, and finishing the local storage of the video.

The following briefly describes, in conjunction with a specific scenario, a method for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention:

the embodiment of the invention provides a synchronous access display system of operation images and parameters, which applies a synchronous access display method of the operation images and parameters, and comprises the following steps:

the system comprises a video acquisition unit, a parameter acquisition unit, a cloud server, a data presentation unit, a parameter curve synchronous drawing unit and a video local amplification unit; the above units are connected to each other through a network.

A video acquisition unit: the system is used for acquiring streaming media signal sources such as a camera, a video acquisition card and a desktop, sending the streaming media signal sources to a cloud server through a streaming media protocol, and uploading operation information including data such as an operation ID and an operation name.

A parameter acquisition unit: the robot parameter acquisition system is used for acquiring robot parameters and uploading the robot parameters to the cloud server.

The cloud server: for receiving video stream and machine parameter and synchronous storage (recording metadata), user management and video management.

A data presentation unit: the system is used for accurately and synchronously playing the operation videos, supports playing, fast forwarding, dragging and local area amplification, and supports dynamically selecting one or more videos for playing one operation to perform personalized dynamic layout and local storage.

A parameter curve synchronous drawing unit: the method is used for synchronously drawing the curve of the robot parameter, and is convenient to carry out mutual verification with the video.

A video local amplification unit: the method and the device are used for locally amplifying and playing the selected video area, and the key video information is conveniently analyzed.

Wherein, aforementioned video acquisition unit: the system comprises a login module, an image acquisition module, a video coding module and a data communication module; the login module is used for logging in a video server, so that the video uploading authority is obtained, the image acquisition module is used for acquiring a camera image at a doctor end, a camera image at a patient end and an endoscope image, the camera image is transmitted to the video server through the data communication module after being encoded and processed by the video encoding module, and the data communication module is used for communicating with the video server and comprises two functions: one for uploading operation information to the server and the other for uploading video data to the server.

Wherein, the parameter acquisition uploading unit: the system comprises a login module, a parameter acquisition module and a data communication module; the system comprises a module server for logging in to obtain the authority of uploading parameters, a parameter acquisition module for acquiring the data of the number of the robot parameters, and a data communication module for uploading the acquired number of the robot parameters to the server.

Wherein, the cloud server: the system comprises a communication module, a storage module, a video management module and a user management module; the communication module is used for communicating with the acquisition unit and the data presentation unit, the storage module receives the multi-channel video data sent in the step (1) and respectively and independently stores the multi-channel video data, meanwhile, a first frame timestamp (metadata) of each video file is recorded, the video management module manages uploading, accessing, updating and other operations of the video data and the metadata of the same operation field through an operation ID, the user management module is used for receiving a user login request, and different operation data authorities of users are granted according to different roles of the users, for example, an acquisition end user only has an uploading video authority, and a playing end user has an authority for accessing and changing the video data.

Wherein, the data presentation unit: the system comprises a login module, a data reading module, a video synchronous playing module, a core parameter curve synchronous drawing module and a local storage module; the login module is used for logging in the video server so as to obtain the access, updating and other permissions of the video data and the metadata, the data reading module is used for reading the video data and the metadata from the step (2) and caching the video data and the metadata into a local temporary file, the video synchronous playing module is used for reading the locally cached metadata and the video data to perform accurate synchronous dynamic layout playing and supporting fast forward and synchronous dragging, and the local storage module is used for locally storing the video with personalized dynamic layout.

Wherein, the parameter curve synchronous drawing unit: the system comprises a login module, a data reading module and a synchronous curve drawing module; the logging module is used for logging in the video server so as to obtain the access, updating and other authorities of the video data and the metadata, the data reading module is used for reading the parameter data and the time stamp of the parameter data from the step (2), and the parameter curve synchronous drawing module is used for drawing a robot motion control core parameter curve.

Wherein, the video local amplification unit: and the original image before splicing is zoomed through a bilinear interpolation algorithm, so that the local definition of the video is improved.

According to the embodiment of the invention, high-definition storage of the video is realized through independent storage and accurate synchronization information of the video sources, and the number and layout of the video sources are dynamically adjusted during playing, so that the operation process can be restored more conveniently and clearly. The multi-source video files of a plurality of operation scenes are accurately and synchronously played, so that the scenes when problems occur can be highly restored, development, clinic and customer service are facilitated to re-dial the problems, and the re-dial efficiency is greatly improved. Through the instant playback function, be convenient for in time fix a position, feed back and solve the problem that takes place in the art, greatly promote with a platform efficiency. Through the local storage function, offline copy and teaching are facilitated when the network environment is poor. The change of the robot motion control parameters and related video pictures, such as a master hand operation picture and an endoscope picture, can be displayed more intuitively through the core parameter curve synchronous drawing function; the method is convenient for engineers to review the disk and position, and the algorithm parameters are optimized. By the local amplification function, the definition of the amplified local area image is increased.

According to the embodiment of the invention, the playing time difference of the multi-channel videos is calculated according to the timestamp of the first frame by recording the timestamp (metadata) of the first frame of each channel of video, the multi-channel videos are accurately and synchronously spliced and played according to the playing time difference sequence, and the synchronization of the multi-channel videos can achieve frame level synchronization. The acquisition program supports distributed acquisition of streaming media signal sources of multiple devices in different places, and is more suitable for operation video recording work of remote robots in multiple places. The playing client acquires the recorded video list from the server, dynamically selects one or more video sources playing a surgery for personalized splicing, realizes accurate and synchronous playing of multi-source videos, and does not reduce definition. The playing end client supports local storage of the personalized spliced video, and brings great convenience to technical staff to rerun and teach in poor network environment.

Fig. 21 is a schematic diagram of a computer device for executing a method for synchronously displaying an operation image and a parameter according to the present invention, and as shown in fig. 21, an embodiment of the present invention further provides a computer device 2100, which includes a memory 2110, a processor 2120, and a computer program 2130 stored in the memory and executable on the processor, wherein the processor implements the method for synchronously displaying an operation image and a parameter when executing the computer program.

The embodiment of the invention also provides a computer program product, which comprises a computer program, and when the computer program is executed by a processor, the method for synchronously accessing and displaying the operation images and the parameters is realized.

The embodiment of the invention also provides a device for synchronously accessing and displaying the operation image and the parameters, which is described in the following embodiment. Because the principle of the device for solving the problems is similar to a method for synchronously accessing and displaying the operation images and the parameters, the implementation of the device can refer to the implementation of the method for synchronously accessing and displaying the operation images and the parameters, and repeated parts are not described again.

Fig. 22 is a schematic view of a device for synchronously accessing and displaying surgical images and parameters according to an embodiment of the present invention, and as shown in fig. 22, a device for synchronously accessing and displaying surgical images and parameters is further provided according to an embodiment of the present invention.

In one embodiment, the device for synchronously accessing and displaying surgical images and parameters according to the embodiments of the present invention comprises:

the multi-source video file acquisition and storage module 2201 is used for acquiring the multi-source video file and storing the multi-source video file according to the video frame time stamp of the multi-source video file;

a robot operating parameter acquisition and storage module 2202, configured to acquire and store robot operating parameters according to parameter timestamps of the robot operating parameters;

a playing time axis establishing module 2203, configured to respond to a playing operation of a user on a multi-source video file, and establish a playing time axis according to a video frame time stamp and a parameter time stamp;

a spliced video determining module 2204, configured to perform synchronous playing and splicing on the multi-source video file according to a playing time axis and a video frame timestamp, and determine a spliced video;

a parameter curve drawing module 2205, configured to draw a parameter curve according to the playing time axis and the parameter timestamp;

and a synchronous playing module 2206, configured to play the merged video according to a playing time axis and synchronously display a real-time state of the parameter curve.

When the device for synchronously accessing and displaying the surgical image and the parameters is implemented specifically, in one embodiment, the device further comprises a configuration setting module for setting video acquisition configuration information;

the multi-source video file acquisition and storage module is also used for:

acquiring a plurality of target acquisition video signal sources according to video acquisition configuration information, starting multithreading to acquire multi-source video data of the plurality of target acquisition video signal sources, encoding the multi-source video data, and determining a multi-source video file; the target acquisition video signal source comprises: endoscope signal source, master camera signal source, panoramic camera signal source.

When the device for synchronously accessing and displaying the surgical images and the parameters provided by the embodiment of the invention is implemented, in one embodiment, the multi-source video file acquisition and storage module is further configured to:

analyzing the multi-source video file to determine video data;

When the device for synchronously storing and displaying the operation images and the parameters provided by the embodiment of the invention is implemented specifically, in one embodiment, the multi-source video file acquisition and storage module is further configured to: and if the frame number of the video data written into the video file does not reach the preset video data frame number threshold value, but the video data is empty, closing the current video file and the video metafile.

When the device for synchronously accessing and displaying the operation image and the parameters is implemented, in one embodiment, the configuration setting module is further used for setting parameter acquisition configuration information;

the robot operating parameter acquisition and storage module is also used for:

When the device for synchronously accessing and displaying the surgical image and the parameters provided by the embodiment of the invention is implemented, in one embodiment, the robot operation parameter acquisition and storage module is further configured to:

In an embodiment of the present invention, when the device for synchronously accessing and displaying surgical images and parameters is implemented, the playing time axis establishing module is specifically configured to:

In an embodiment of the present invention, when the device for synchronously accessing and displaying surgical images and parameters is implemented, the playing time axis establishing module is further configured to:

arranging video frame time stamps in the video metafile and parameter time stamps in the parameter metafile in parallel, and determining a plurality of metadata time axes; the metadata timeline includes: a video metadata time axis obtained by arranging the video frame time stamps along the time increment and/or a parameter metadata time axis obtained by arranging the parameter time stamps along the time increment; the interval between the video frame time stamps or the parameter time stamps of the metadata time axis is the sampling rate during acquisition;

In an embodiment of the present invention, when the device for synchronously accessing and displaying surgical images and parameters is implemented, the merged video determining module is specifically configured to:

caching a video file to a video queue;

extracting a plurality of paths of synchronous video frames of a target acquisition video signal source corresponding to a playing time stamp from a video queue according to the currently played playing time stamp of a playing time shaft;

In one embodiment, the parameter curve drawing module is specifically configured to, when the device for synchronously accessing and displaying surgical images and parameters provided by the embodiment of the present invention is implemented:

caching the parameter data file to a parameter queue;

according to the playing time stamp of the current playing of the playing time shaft, extracting parameter data corresponding to the playing time stamp from the parameter queue;

In an embodiment of the present invention, when the device for synchronously accessing and displaying surgical images and parameters is implemented, the synchronous playing module is specifically configured to:

In one embodiment, the device for synchronously accessing and displaying surgical images and parameters according to the embodiments of the present invention further includes a local magnification module, configured to:

In an embodiment of the present invention, when the device for synchronously accessing and displaying surgical images and parameters is implemented, the device further includes a mosaic video storage module, configured to:

creating a storage file and writing file header information;

To sum up, the embodiment of the present invention provides a method and an apparatus for synchronously accessing and displaying surgical images and parameters, comprising: acquiring a multi-source video file, and storing according to a video frame timestamp of the multi-source video file; collecting robot operating parameters, and storing according to parameter timestamps of the robot operating parameters; responding to the playing operation of a user on the multi-source video file, and establishing a playing time axis according to the video frame timestamp and the parameter timestamp; synchronously playing and splicing the multi-source video files according to a playing time axis and the video frame time stamp, and determining spliced videos; drawing a parameter curve according to a playing time axis and the parameter timestamp; and playing the spliced video and synchronously displaying the real-time state of the parameter curve according to the playing time axis. By accurately and synchronously playing the multi-source video files of a plurality of scenes, the problems of complexity and specialty of synchronously playing the multi-source video when technical personnel re-run a disk, high operation difficulty and difficult control of synchronization precision are solved; by drawing a parameter curve, the main hand operation picture, the endoscope picture and the core parameter change of the robot motion control algorithm are synchronously displayed, so that the method is beneficial to quickly positioning and solving the problems in the operation, and is convenient for analyzing and adjusting the parameters of the motion control algorithm after the operation; according to the invention, the multi-source video files of a plurality of operation scenes are accurately and synchronously played, so that the scenes when problems occur can be highly restored, the problems can be conveniently re-recorded by development, clinic and customer service, and the re-recording efficiency is greatly improved; by drawing a parameter curve, the real-time state change condition of the core parameter data of the robot motion control algorithm can be synchronously displayed while the video recorded in the operation is played, and the running condition of the robot can be judged according to the real-time state change condition of the parameter data, so that the operation of the robot is convenient to rerun; through the instant playback function, problems occurring in the operation can be conveniently positioned, fed back and solved in time, and the platform following efficiency is greatly improved; according to the invention, the multi-source videos are stored independently, and the timestamp of each video frame is recorded simultaneously, so that the accurate synchronous storage of the multi-source videos with different frame rates and different resolutions and the storage of the high-definition videos are realized. And synchronously drawing a core parameter curve and playing the associated multi-source video by acquiring the data of the robot motion control algorithm and the timestamp of each parameter.

According to the technical scheme, the data acquisition, storage, use, processing and the like meet relevant regulations of national laws and regulations, and various types of data such as personal identity data, operation data, behavior data and the like related to individuals, clients, crowds and the like are authorized.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A synchronous access display method for operation images and parameters is characterized by comprising the following steps:

2. The method for synchronously accessing and displaying the operation images and the parameters as claimed in claim 1, further comprising: setting video acquisition configuration information;

the acquiring multi-source video file comprises:

according to the video acquisition configuration information, a plurality of target acquisition video signal sources are obtained, multi-thread acquisition of multi-source video data of the target acquisition video signal sources is started, the multi-source video data are encoded, and a multi-source video file is determined.

3. The method for synchronously accessing and displaying surgical images and parameters according to claim 1, wherein storing the multi-source video file according to the video frame time stamp of the multi-source video file comprises:

creating a video metafile and a video file; the video file is preset with a video data frame number threshold; the video data frame number threshold is an integral multiple of the video data frame rate;

analyzing the multi-source video file to determine video data;

4. The method for synchronously accessing and displaying the operation images and the parameters as claimed in claim 3, further comprising: and if the frame number of the video data written into the video file does not reach the preset video data frame number threshold value, but the video data is empty, closing the current video file and the video metafile.

5. The method for synchronously accessing and displaying the operation images and the parameters as claimed in claim 1, further comprising: setting parameter acquisition configuration information;

collecting robot operating parameters, including:

6. The method for synchronously accessing and displaying the surgical image and the parameters as claimed in claim 5, wherein the storing the robot operation parameters according to the parameter time stamp of the robot operation parameters comprises:

7. The method for synchronously accessing and displaying surgical images and parameters according to claim 1, wherein the step of establishing a playing time axis according to the video frame time stamp and the parameter time stamp in response to a user playing operation on the multi-source video file comprises the steps of:

8. The method as claimed in claim 1, wherein the step of establishing a playing time axis according to the video frame time stamp in the video metafile and the parameter time stamp in the parameter metafile comprises:

9. The method for synchronously accessing and displaying surgical images and parameters according to claim 1, wherein the step of synchronously playing and splicing the multi-source video files according to the playing time axis and the video frame time stamp to determine spliced videos comprises the following steps:

caching a video file to a video queue;

decoding the multi-channel synchronous video frames into a YUV format, splicing the multi-channel synchronous video frames in the YUV format according to a preset layout mode, rendering the spliced multi-channel synchronous video frames in the YUV format, and determining spliced video.

10. The method as claimed in claim 9, wherein the step of drawing a parameter curve according to the playing time axis and the parameter timestamp comprises:

caching the parameter data file to a parameter queue;

11. The method of claim 10, wherein the step of playing the merged video and synchronously displaying the real-time status of the parameter curve according to the playing time axis comprises:

12. The method for synchronously accessing and displaying the operation images and the parameters as claimed in claim 1, further comprising:

13. The method for synchronously accessing and displaying the operation images and the parameters as claimed in claim 1, further comprising:

creating a storage file and writing file header information;

14. A synchronous access display device for operation image and parameter, comprising:

the parameter curve drawing module is used for drawing a parameter curve according to the playing time shaft and the parameter timestamp;

15. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 13.