TWI755014B - Image synchronous processing system based on real-time web technology and method thereof - Google Patents

Abstract

An image synchronous processing system based on real-time web technology includes an image server, several user devices, and a real-time network server. The real-time web server connects the image server and the plurality of user devices. When at least one of the user devices marks at least one of labeled information on the first image received from the image server to display the second image information, in response to the operation of at least one of the user devices, the real-time network server transmits at least one tag information received from at least one of the user devices to remaining user devices of the user devices, and the remaining user devices generate second image information according to the at least one tag information which is received and the first image information. In addition, an image synchronous processing method is also disclosed.

Description

An image synchronization processing system and method based on real-time network technology

The present invention relates to an image synchronization processing system and method, and more particularly, to an image synchronization processing system and method based on real-time web technology.

With the rapid development of digital pathology imaging and the maturity of cloud technology, it has become one of the solutions to solve the lack of pathologists and uneven geographical distribution. When multiple doctors need to discuss a specific digital pathological image, they often use the method of face-to-face discussion or the method of online discussion through the Internet. However, in the process of online discussion, it is impossible for all participating physicians to annotate and edit specific digital pathological images at the same time. Therefore, how to solve the above problems encountered in the current online discussion process is one of the issues concerned by those skilled in the art.

The present invention provides an image synchronization processing system based on real-time network technology, comprising an image server, a plurality of user devices and a real-time network server. The image server is used for storing the first image information of the first image, and the plurality of user devices are respectively used for receiving the first image information from the image server. The real-time web server connects the image server and multiple user devices. When at least one of the plurality of user devices annotates at least one label information on the first image to display the second image information, the real-time network server responds to the operation of at least one of the plurality of user devices, from the plurality of user devices At least one of the user devices receives at least one annotation information, and transmits the received at least one annotation information to the remaining user devices of the plurality of user devices, wherein the remaining user devices of the plurality of user devices are used for according to The received at least one annotation information and the first image information generate second image information.

The present invention further provides a method for image synchronization processing based on real-time network technology, comprising the following steps: receiving first image information of a first image from an image server through a plurality of user devices; When at least one of the user devices marks at least one label information on the first image to display the second image information, in response to an operation of at least one of the plurality of user devices, the real-time network server is used to obtain information from the plurality of users. at least one of the devices receives at least one annotation information; transmits the received at least one annotation information to the remaining user devices of the plurality of user devices by the real-time network server; and The other user devices are used for generating second image information according to the received at least one annotation information and the first image information.

Based on the above, the system and method for image synchronization processing based on real-time network technology of the present invention can immediately synchronize all the annotation information of the same image on the user device used by each user to the data on all user devices. image.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings as follows.

FIG. 1 is a block diagram of an image synchronization processing system based on real-time web technology according to some exemplary embodiments of the present invention. Referring to FIG. 1, the image synchronization processing system 100 of this embodiment includes an image server 110, a plurality of user devices 120(1)-120(N), and a real-time network server 130, wherein N is any positive integer, There are no special restrictions. The image server 110 can store image information of a plurality of images. The user devices 120 ( 1 ) to 120 (N) can respectively receive image information corresponding to at least one of the plurality of images from the image server 110 . The real-time network server 130 may be connected between the image server 110 and the user devices 120(1)-120(N).

In some embodiments, the image server 110 is, for example, a server host or a network hard disk for storing and processing images. In a further embodiment, the image server 110 is, for example, an independent server host or a network hard disk with high security.

In some embodiments, the user devices 120(1)-120(N) are, for example, a personal computer (PC), a tablet computer, a mobile phone, a personal digital assistant (personal digital assistant) with a normal display or a touch-sensitive display. assistant, PDA) and other computing devices.

In some embodiments, the real-time web server 130 is, for example, a real-time web server for building real-time web applications.

In some embodiments, the image server 110, the plurality of user devices 120(1)-120(N), and the real-time web server 130 can all support hypertext transfer protocol (HTTP), file transfer protocol (file transfer protocol, FTP) or transmission control protocol and Internet protocol (transmission control protocol/internet protocol, TCP/IP) and other communication protocols. In some embodiments, the image server 110 may transmit the first image information to the user devices 120(1)-120(N) via the real-time web server 130 based on the HTTP protocol.

In some embodiments, the image synchronization processing system 100 may also include more than one image server (ie, including the image server 110 and other image servers other than the image server 110 ). In addition, these image servers may be various image servers such as a server for storing lesion images in a hospital or a server for storing teaching images in an educational institution.

FIG. 2 is a flowchart of an image synchronization processing method based on real-time network technology according to some exemplary embodiments of the present invention. The method of this embodiment is applicable to the image synchronization processing system 100 in FIG. 1, but is not limited thereto. Any image synchronization processing system that can apply the method shown in Figure 2 is within the scope of this case. For the sake of clarity, the following describes the detailed steps of the image synchronization processing method shown in FIG.

First, in step S201, the user devices 120(1)-120(N) respectively receive the first image information from the image server 110. In detail, before multiple users want to mark the specific first image, multiple users can send image request information to the image server 110 through the user devices 120(1)-120(N). to request the first image information of the first image. In this way, the image server 110 can obtain the first image information from a plurality of pre-stored image information according to the image request information, and transmit the first image information to the user devices 120(1)-120(N). Based on this, the user devices 120(1)-120(N) can display the first image according to the first image information, and a plurality of users can view the first image through the user devices 120(1)-120(N) respectively. an image.

Next, in step S203, when at least one label information is marked on the first image by at least one of the user devices 120(1)-120(N) to display the second image information, the user device 120 ( 1) An operation of at least one of the user devices 120(1) to 120(N), through the real-time web server 130 receiving at least one annotation information from at least one of the user devices 120(1) to 120(N). Specifically, when different user devices 120(1)-120(N) are operated to label at least one labeling information on the first image, the corresponding user device can transmit the corresponding labeling information to the real-time network server 130. In other words, once any user operates the corresponding user device to annotate the first image, the corresponding user device will immediately transmit all the corresponding annotation information to the real-time network server 130 .

For example, when M users use the user devices 120( 1 ) to 120(M) to mark M pieces of labeling information on the first image, respectively, to generate M second images, the user device 120( 1 )~120(M), the real-time network server 130 can receive M pieces of label information from the user devices 120(1)~120(M), where M is a positive integer not greater than N, and there is no special limit. In other words, M users can operate respective user devices to mark the first image, thereby generating M second images. Once the real-time web server 130 detects the above-mentioned operations, the real-time web server 130 can immediately receive M pieces of annotation information from the user devices 120(1)-120(M).

In some embodiments, after the first image is annotated, the user device that generates the annotation information may further display query information on its display to ask whether to share the annotation information. If the corresponding user inputs the consent to share the information, the user device that generates the annotation information will immediately transmit the annotation information corresponding to the annotation to the real-time web server 130 .

Next, in step S205, the real-time network server 130 transmits the received at least one annotation information to the remaining user devices in the user devices 120(1)-120(N). Specifically, after the real-time web server 130 receives at least one annotation information, the real-time web server 130 can immediately transmit the received at least one annotation information to the user devices 120(1)-120(N) the remaining user devices.

For example, continuing from the previous example, after the real-time web server 130 receives the M tagging information tagged by the M users from the user devices 120(1)-120(M), the real-time web server 130 may The annotation information marked by the first user among the M users is transmitted to the user devices 120(2)~120(M), and the annotation information marked by the second user is transmitted to the user device 120(1) , 120(3)~120(N). By analogy, the real-time web server 130 can use the same operation method for the marked information marked by the remaining users among the M users.

Finally, in step S207, the remaining user devices in the user devices 120(1)-120(N) are used to generate the second image information according to the received at least one annotation information and the first image information. Specifically, the remaining user devices in the user devices 120(1)-120(N) can label the received at least one label information on the first image corresponding to the first image information to generate the second image information.

In some embodiments, the at least one labeling information may further include at least one labeling area, at least one presentation message, or at least one time stamp (tag) and other related information that the user labels on the first image. The marked area may be an image area marked in the first image (for example, in the form of image coordinates), and the presented information may be various information marked by the user in the above marked area (for example, circled marks, drawings, etc.). line markings or text messages, etc.), and the time stamp may be the system time of the user device used by the user when the user marked the presentation of the message (that is, when any user marked the first image When a message is presented, the user device used by any user will generate a timestamp based on the system time of the user device).

In a further embodiment, the user devices 120(1)-120(N) may display the received at least one labeling area in the at least one labeling information received in the first image corresponding to the first image information. At least one presentation information in an annotation information is used to generate second image information.

For example, continuing the previous example, the first user can receive the second to Mth annotation information corresponding to the user devices 120(2)~120(M) through the user device 120(1), and the The 2 users can receive the first and second to M-th annotation information corresponding to the user devices 120(1), 120(3)-120(M), and the M+th label information through the user device 120(2). One user can receive the 1st to Mth tag information corresponding to the user devices 120(1)-120(M) through the user device 120(M+1). By analogy, other user devices can adopt the same operation method. In this way, all of the user devices 120(1)-120(N) can display the 1st to Mth annotation information in the 1st to Mth display areas corresponding to the 1st to Mth annotation information in the first image. The 1st to Mth second images are integrated to generate second image information of an integrated image.

In a further embodiment, the system 100 for image synchronization processing may further include at least one memory (not shown). At least one memory can be connected to the real-time network server 130 and store a plurality of databases. In addition, the real-time web server 130 can convert at least one marked area and at least one presentation message into text information, so as to store the text information and at least one time stamp in a plurality of databases. In some embodiments, the real-time web server 130 can convert the at least one markup area and the at least one presentation message into text information with structured text based on the JSON (javaScript object notation) format.

In a further embodiment, in response to the real-time web server 130 receiving image recording request information corresponding to the first image information from at least one of the user devices 120(1)-120(N), the real-time web server The device 130 may obtain the text information and the at least one time stamp from a plurality of databases to transmit the text information and the at least one time stamp to at least one of the user devices 120(1)-120(N). Thereby, at least one of the user devices 120(1)-120(N) can generate historical image information according to the text information, the at least one timestamp and the first image information.

For example, the M-th user can transmit the image recording request information corresponding to the second image information to the real-time web server 130 through the user device 120 (M), and the real-time web server 130 can obtain the data from a plurality of data The library obtains text information corresponding to the second image information and X time stamps arranged in sequence, so as to transmit the text information and X time stamps to the user device 120 (M), wherein X is a positive integer, and the value of X is The sum of the number of timestamps generated by the user devices 120(1)-120(N) for the first image information.

In this way, the user device 120(M) can convert the text information into at least one marked area and at least one presentation message, and extract the selected time after the Mth user selects a specific time stamp among the X time stamps Stamps the callout area and presentation message corresponding to all timestamps preceding this selected timestamp. Based on this, the user device 120(M) can display the selected time stamp and the selected time stamp on the first image in sequence according to the selected time stamp and all time stamps before the selected time stamp All the marked areas and presentation messages corresponding to all the time stamps of , and then generate historical image information (ie, play the selected time stamp and all the marked areas and presentation messages before the selected time stamp).

In a further embodiment, the system 100 for image synchronization processing may further include a training server (not shown), and the training server may store the lesion position determination model, and the training server may store data from at least one of the real-time network server 130 from at least one A plurality of databases stored in the memory receive text information, so as to use the text information to train the lesion position determination model, and the real-time network server 130 can receive the lesion position determination model from the training server, and use the image information of the patient to use The lesion location judgment model generates lesion location information.

In other words, when any user wants to determine the location of the lesion in the patient image (for example, the patient's MRI image, computed tomography image, or X-ray image, etc.), any user can use the user device 120 ( One of 1)˜120(N) transmits the image information of the patient image to the real-time web server 130 . Thereby, the real-time network server 130 can receive the lesion position determination model from the training server, and use the lesion position determination model to determine the lesion position in the patient image according to the image information of the patient to generate the lesion position information.

With the above-mentioned system 100 for image synchronization processing based on real-time network technology, all the annotation information of the same image on the user device used by each user can be immediately synchronized to all user devices 120(1)~ Image at 120(N). In this way, the system 100 for image synchronization processing can allow all users to annotate the same image at the same time, and all users can immediately see the annotation operations of all users.

FIG. 3 is a schematic diagram of an image synchronization processing system based on real-time network technology according to other exemplary embodiments of the present invention. Referring to FIG. 3, the image synchronization processing system 300 of this embodiment includes an image server 110, a plurality of user devices 120(1)-120(N), a real-time network server 130, a database 140, and a training server 150. The image server 110 can store image information of multiple images, and the image server 110 can also be more than one image server, which is not particularly limited). The user devices 120 ( 1 ) to 120 (N) can respectively receive image information corresponding to at least one of the plurality of images from the image server 110 . The real-time network server 130 can be connected to the image server 110, the user devices 120(1)-120(N), the database 140 and the training server 150, wherein N is any positive integer, and there is no special limitation .

First, a plurality of users can respectively receive the first image information of the first image from the image server 110 through the user devices 120(1)-120(N).

In some embodiments, the user devices 120(1)-120(N) may further include image viewer modules 1201(1)-1201(N). A plurality of users can request the first image information of the first image by transmitting image request information to the image server 110 through the user devices 120(1)-120(N). In this way, the image server 110 can obtain the first image information from a plurality of pre-stored image information according to the image request information, and transmit the first image information to the user devices 120(1)-120(N). Therefore, the image viewer modules 1201(1)~1201(N) can display the first image according to the first image information, and multiple users can use the image viewer modules 1201(1)~1201(N) respectively ) to view the first video.

Furthermore, when at least one of the user devices 120(1)-120(N) labels at least one label information on the first image to display the second image information, in response to the user devices 120(1)-120(N) ), the real-time web server 130 receives at least one annotation information from at least one of the user devices 120(1)-120(N).

In some embodiments, the user devices 120(1)-120(N) may further include real-time network modules 1202(1)-1202(N). When at least one user among the plurality of users uses at least one user device used by at least one user among the user devices 120(1) to 120(N) to label at least one labeling information on the first image information corresponding to the first image information When the images (for example, marked by various input devices or touch displays in the user devices 120(1)-120(N)) are used to generate the second image information corresponding to the at least one second image, the above at least one use The real-time network module of the user's device can transmit at least one annotation information to the real-time network server 130 . In other words, once any user marks the first image on the corresponding user device, the real-time network module of the corresponding user device will immediately transmit the annotation information corresponding to the mark to the real-time network server 130 .

Furthermore, the real-time network server 130 may transmit the received at least one annotation information to the remaining user devices in the user devices 120(1)-120(N).

In some embodiments, the real-time web server 130 may further include a web app module 1301 and a real-time message handler module 1302 . The instant message processing program module 1302 can receive the above at least one annotation information.

In some embodiments, the at least one label information may further include at least one network address information (eg, information of an internet protocol (IP) address), and the web application module 1301 may further include at least one network address information. At least one piece of network address information in an annotation information determines which of the user devices 120(1)-120(N) the at least one annotation information needs to transmit to to generate the purpose determination information. The instant message processing program module 1302 can transmit the received at least one annotation information to the remaining user devices in the user devices 120(1)-120(N) according to the purpose judgment information.

For example, when the real-time web server 130 receives the annotation information of the user devices 120(1), 120(2), the web application module 1301 can use the user devices 120(1), 120(2) The network address information in the annotation information determines that the annotation information of the user device 120(1) needs to be transmitted to the user devices 120(2)~120(N) and determines that the annotation information of the user device 120(2) needs to be transmitted It is transmitted to the user devices 120(1), 120(3)~120(N) to generate purpose judgment information. Thereby, the instant message processing program module 1302 can transmit the received annotation information of the user device 120(1) to the user devices 120(2)~120(N) according to the purpose judgment information, and can according to this purpose The determination information transmits the received annotation information of the user device 120(2) to the user devices 120(1), 120(3)-120(N).

In some embodiments, at least one of the plurality of users can further zoom the first image through at least one of the user devices 120(1)-120(N) to generate at least one zoom information for marking The at least one annotation information is in the zoomed first image, and then the at least one zoom information and the at least one annotation information are sent to the real-time network server 130 . Thus, the real-time message processing program module 1302 of the real-time web server 130 can ignore at least one scaling information.

Finally, the remaining user devices in the user devices 120(1)-120(N) are used to generate the second image information according to the received at least one annotation information and the first image information. In other words, the real-time network modules of the user devices 120(1)-120(N) that do not generate at least one annotation information can receive at least one annotation information, and the user devices 120(1)-120(N) ), the real-time network module of the user device that generates at least one annotation information can receive all annotation information that is not generated by itself in the at least one annotation information.

In some embodiments, the at least one markup information may further include at least one markup area and at least one presentation information. The image viewer modules 1201(1) to 1201(N) of the user devices 120(1) to 120(N) may receive at least one annotation information in the first image corresponding to the first image information. An annotation area displays at least one presentation information in the received at least one annotation information to generate the second image information. In this way, multiple users can watch the integrated image corresponding to the second image information at the same time.

In a further embodiment, the at least one annotation information may further include at least one time stamp. The web application module 1301 of the real-time web server 130 can convert at least one marked area and at least one presentation message into text information, and the real-time message processing module 1302 of the real-time web server 130 can convert the text information to the text information. At least one timestamp is stored in the database 140 . In some embodiments, the web application module 1301 can convert the at least one marked area and the at least one presentation message into text information with structured text based on the Json format.

In further embodiments, in response to the instant messaging program module 1302 of the real-time web server 130 receiving the first For image recording request information corresponding to image information, the web application module 1301 of the real-time web server 130 can obtain text information and at least one time stamp from the database 140, so as to transmit the text information and at least one time stamp to the user. A real-time network module of at least one of the user devices 120(1)-120(N). Thereby, the image viewer module of at least one of the user devices 120(1)-120(N) can generate historical image information according to the text information, the at least one timestamp and the first image information.

In some embodiments, the training server 150 can store the lesion location determination model, and the training server can receive text information from a plurality of databases stored in at least one memory via the real-time network server 130, so as to use the text information to train the lesions Location Judgment Model. The real-time message processing program module 1302 of the real-time web server 130 can receive the lesion location judgment model from the training server 150, and the web application module 1301 of the real-time web server 130 can use the image information of the patient to use The lesion location judgment model generates lesion location information.

To sum up, the system and method for image synchronization processing based on real-time network technology provided by the present invention can immediately synchronize all the annotation information of the same image on the user device used by each user to all users. image on the device. In this way, the user does not need to wait for other users to complete the labeling of the patient image before labeling the patient image. In addition, the user's annotation information on the patient image can be further converted into text information to facilitate training of the model for judging the location of the lesion in the patient image.

Although the present invention has been disclosed above by the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the scope of the appended patent application.

100, 300: Image synchronization processing system

110: Image server

120(1)~120(N): User device

130: Live web server

1201(1)~1201(N): Video viewer module

1202(1)~1202(N): Real-time network module

1301: Web Application Modules

1302: Instant Messaging Program Module

140:Database

150: Training Server

S201~S207: The flow of the image synchronization processing method

FIG. 1 is a block diagram of an image synchronization processing system based on real-time network technology according to some exemplary embodiments of the present invention. FIG. 2 is a flowchart of an image synchronization processing method based on real-time network technology according to some exemplary embodiments of the present invention. FIG. 3 is a schematic diagram of an image synchronization processing system based on real-time network technology according to other exemplary embodiments of the present invention.

100: Image synchronization processing system

110: Image server

120(1)~120(N): User device

130: Live web server

Claims (8)

An image synchronization processing system based on real-time web technology, comprising: an image server for storing first image information of a first image; a plurality of user devices for obtaining data from the image respectively The server receives the first image information; and a real-time network server connects the image server and the user devices, the real-time network server is used for: when at least one of the user devices is marked with at least one an annotation information is received from the at least one of the user devices in response to the operation of the at least one of the user devices when the first image is used to display the second image information, The at least one annotation information includes at least one annotation area and at least one presentation message, and the received at least one annotation information is transmitted to the rest of the user devices, wherein the user device of the user devices The other user devices are used for displaying the at least one presentation in the received at least one annotation information in the at least one annotation area in the received at least one annotation information in a first image corresponding to the first image information information to generate the second image information. The image synchronization processing system based on real-time network technology as claimed in claim 1, wherein the at least one annotation information further includes at least one timestamp, and the system further includes: at least one memory for storing a plurality of databases, where the real-time network The server converts the at least one marked area and the at least one presentation message into text information, so as to store the text information and the at least one timestamp in the databases. The real-time network technology-based image synchronization processing system as claimed in claim 2, wherein the real-time network server is further configured to: in response to receiving from the at least one of the user devices corresponding to the second image Image recording of information requires information, obtaining the textual information and the at least one time stamp from the databases to transmit the textual information and the at least one time stamp to the at least one of the user devices, wherein the The at least one of the user devices generates historical image information according to the text information, the at least one timestamp and the first image information. The image synchronization processing system based on real-time network technology as described in claim 2, further comprising: a training server connected to the real-time network server for storing a lesion location judgment model, and the training server further uses To: use the text information to train the lesion location judgment model, and use the lesion location judgment model to generate lesion location information according to image information of a patient. An image synchronization processing method based on real-time web technology, comprising: A plurality of user devices respectively receive first image information of a first image from an image server; when at least one label information is marked on the first image by at least one of the user devices to display the second image In the case of image information, in response to the operation of the at least one of the user devices, the real-time network server receives the at least one annotation information from the at least one of the user devices, wherein the at least one The annotation information includes at least one annotation area and at least one presentation message; the at least one annotation information received is transmitted to the remaining user devices of the user devices by the real-time network server; and by the use of the For the remaining user devices in the user device, the at least one annotation area in the received at least one annotation information in a first image corresponding to the first image information displays the at least one annotation area in the received annotation information. a presentation message to generate the second image information. The image synchronization processing method based on real-time network technology as claimed in claim 5, wherein the at least one marking information further comprises at least one timestamp, and further comprises: using the real-time network server to associate the at least one marking area with the The at least one presentation message is converted into text information to store the text information and the at least one time stamp in a plurality of databases in at least one memory. The image synchronization processing method based on real-time network technology as described in claim 6, further comprising: obtaining, by the real-time web server, the text from the databases in response to the real-time web server receiving image recording request information corresponding to the first image information from the at least one of the user devices information and the at least one time stamp to transmit the textual information and the at least one time stamp to the at least one of the user devices; and relying on the text by the at least one of the user devices The information, the at least one time stamp, and the first image information generate historical image information. The image synchronization processing method based on real-time network technology as described in claim 6, further comprising: using the text information to train a lesion location judgment model stored in the training server by a training server, and based on a patient The image information is used to generate the lesion location information using the lesion location judgment model.

Images