CN108347621B - Network live broadcast data processing method and system - Google Patents
Network live broadcast data processing method and system Download PDFInfo
- Publication number
- CN108347621B CN108347621B CN201810031400.7A CN201810031400A CN108347621B CN 108347621 B CN108347621 B CN 108347621B CN 201810031400 A CN201810031400 A CN 201810031400A CN 108347621 B CN108347621 B CN 108347621B
- Authority
- CN
- China
- Prior art keywords
- value
- video
- camera device
- preset
- central server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4408—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving video stream encryption, e.g. re-encrypting a decrypted video stream for redistribution in a home network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/84—Generation or processing of descriptive data, e.g. content descriptors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Abstract
The invention provides a method and a system for processing network live broadcast data, wherein the method comprises the following steps: the camera device obtains a video frame which meets the preset condition of not losing frames, converts the video frame into a video data stream, and performs DRM digital copyright encryption authenticity verification on the video data stream; when the video data stream passes the DRM digital copyright encryption authenticity verification, the camera device sends the state information of the internal processor to the central server, so that the central server matches the state information with the preset state information, and when the state information is successfully matched with the preset state information, the multimedia server in an idle state is inquired from the multimedia server cluster; the camera device receives the address of the multimedia server in the idle state sent by the central server and sends the video data stream to the multimedia server in the idle state, so that reasonable supervision on the network live broadcast platform is realized, and the safety of the network platform is ensured; meanwhile, the operation efficiency of the network live broadcast platform is effectively improved.
Description
Technical Field
The present invention relates to the field of multimedia network communication technologies, and in particular, to a method and a system for processing live broadcast data.
Background
In recent years, live webcasting is popular, but the data volume in the live webcasting process is huge, and meanwhile, malicious and illegal data sources are known to have a profit-making behavior in the live webcasting process, and the security, reliability and fluency of live webcasting become more and more important in the existing network society.
The current network platforms are numerous, users have certain blindness during selection, and the selection of the live broadcast platform based on the network environment directly determines the live broadcast effect for the users who shoot live broadcast and the users who watch live broadcast; meanwhile, the network live broadcast is developed rapidly, the scale of users is increased rapidly, incomplete statistics is carried out, various network live broadcast platforms in the current environment reach more than 150 families, the scale of users exceeds 2 hundred million, registered users of some large network live broadcast platforms are hundreds of millions of users in a month, the number of users in a room in a peak period can reach tens of thousands of people.
The network broadcast is rapidly developed, and meanwhile, the outstanding safety problem is brought. Therefore, it is necessary to reasonably limit and monitor the live content.
In view of the above, it is desirable to provide a live webcast data processing method, which is not only significant in effectively developing timeliness of current live webcasts, but also beneficial to creating a good network environment, and is convenient for users to display active contents on a live webcast platform.
Disclosure of Invention
The invention provides a method and a system for processing live network data, aiming at solving the problems that the live network environment is lack of supervision control and the data transmission efficiency of a network platform is low in the prior art.
In one aspect, the present invention provides a method for processing live webcast data, including:
s1, the camera device obtains the video frame which accords with the condition of no frame loss, converts the video frame into the video data stream, and verifies the DRM digital copyright encryption authenticity of the video data stream;
s2, when the video data stream passes the DRM digital copyright encryption authenticity verification, the camera device sends the state information of the internal processor to the central server, so that the central server matches the state information with the preset state information, and when the state information is successfully matched with the preset state information, the multimedia server in an idle state is inquired from the multimedia server cluster;
and S3, the camera device receives the address of the multimedia server in the idle state sent by the central server and sends the video data stream to the multimedia server in the idle state.
Preferably, the step S1 is preceded by: the camera device receives an initialization authentication request sent by the central server, so that the central server authenticates basic information of the camera device.
Preferably, the preset no-frame-loss condition in step S1 includes: and the time difference between the receiving time stamp of the video frame and the generating time stamp of the video frame is smaller than a preset time difference threshold value and/or the time difference between the receiving time stamp of the video frame and the generating time stamp of the video frame is smaller than a preset proportion of the time length of the video stream where the video frame is located.
Preferably, the step S1 further includes: and the camera device carries out digital encryption processing and authority control on the video data stream.
Preferably, the state information of the internal processor in step S2 includes: a voltage indicator in a motherboard chip of the internal processor, a temperature indicator of the motherboard chip, data load level information of the internal processor, and data throughput rate information of the internal processor.
Preferably, the step S2 of successfully matching the status information with the preset status information further includes:
the voltage index is not lower than a preset lowest voltage value and not higher than a preset highest voltage value; and/or the temperature index is not lower than a preset lowest temperature value and the temperature index is not higher than a preset highest temperature value; and/or the data load level value is not higher than a preset maximum data load level value; and/or the data throughput rate value is not lower than a preset minimum data throughput rate value.
Preferably, the step S2 of querying the multimedia server in an idle state from the multimedia server cluster further includes:
if the number of the types of the multimedia data processed in the multimedia server is not greater than a preset data number threshold value, determining that the multimedia server is a multimedia server in an idle state; and/or if the CPU running speed of the multimedia server does not exceed a preset speed threshold, determining that the multimedia server is in an idle state; and/or if the ratio of the unused storage space of the multimedia server to the used storage space is not higher than a preset ratio, determining that the multimedia server is a multimedia server in an idle state; and/or if the number of the connection using users of the multimedia server is not larger than a preset user number threshold value, determining that the multimedia server is in an idle state.
Preferably, before the step S3 of sending the video data stream to the multimedia server in the idle state, the method further includes:
and the camera device establishes communication connection with the multimedia server in the idle state.
Preferably, the step S3 of sending the video data stream to the multimedia server in the idle state further includes: and sending the video data stream to the multimedia server in the idle state by utilizing an RTMP protocol.
In one aspect, the present invention provides a network live broadcast data processing system, including a camera device, a central server and a multimedia server, wherein:
the camera device is used for acquiring a video frame meeting a preset condition of no frame loss, converting the video frame into a video data stream, performing DRM digital rights encryption authenticity verification on the video data stream, and sending the state information of the internal processor to the central server when the video data stream passes the DRM digital rights encryption authenticity verification;
the central server is used for matching the state information with preset state information, inquiring a multimedia server in an idle state from a multimedia server cluster when the state information is successfully matched with the preset state information, and sending the address of the multimedia server in the idle state to the camera device, so that the camera device sends the video data stream to the multimedia server in the idle state.
The invention provides a method and a system for processing live webcast data, which are used for authenticating the identity of a network platform before the live webcast of video data is initiated, and simultaneously, performing level inspection on the contents to be live webcast and live webcast, and reasonably supervising the live webcast platform; meanwhile, the multimedia server in an idle state is inquired, so that correct selection of the multimedia server is facilitated, and further the operation efficiency of the live network platform is improved.
Drawings
Fig. 1 is an overall flow diagram of a live webcast data processing method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an image pickup apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an overall structure of a live webcast data processing system according to an embodiment of the present invention.
Detailed Description
The following detailed description of embodiments of the present invention is provided in connection with the accompanying drawings and examples. The following examples are intended to illustrate the invention but are not intended to limit the scope of the invention.
Fig. 1 is a schematic overall flow chart of a live webcast data processing method according to an embodiment of the present invention, and as shown in fig. 1, the present invention provides a live webcast data processing method, including:
s1, the camera device obtains the video frame which accords with the condition of no frame loss, converts the video frame into the video data stream, and verifies the DRM digital copyright encryption authenticity of the video data stream;
s2, when the video data stream passes the DRM digital copyright encryption authenticity verification, the camera device sends the state information of the internal processor to the central server, so that the central server matches the state information with the preset state information, and when the state information is successfully matched with the preset state information, the multimedia server in an idle state is inquired from the multimedia server cluster;
and S3, the camera device receives the address of the multimedia server in the idle state sent by the central server and sends the video data stream to the multimedia server in the idle state.
Specifically, in the present embodiment, when network live broadcasting is performed, video frames are captured by the image pickup device. Fig. 2 is a schematic structural diagram of an image capturing apparatus according to an embodiment of the present invention, and as shown in fig. 2, in this embodiment, the image capturing apparatus mainly includes: the device comprises a protective film 1, a lens 2, a digital signal processor 3, an image sensor 4 and a core mainboard chip 5. The protective film 1 mainly plays a role in protecting the camera device from being damaged by water inflow or external sharp objects; the lens 2 is composed of several lenses, which can be plastic lenses or glass lenses; the digital signal processor, DSP for short, is mainly used for optimizing digital image signal parameters through a series of complex mathematical algorithm operations; a CCD image sensor, also called an image controller, specifically a semiconductor device, which is capable of converting an optical image into an electrical signal; the core mainboard chip comprises an interface for interacting with the server.
On the basis of the technical scheme, the camera device shoots video frames through the image controller, obtains the video frames meeting the preset condition of no frame loss from the shot video frames, processes the video frames meeting the preset condition of no frame loss to generate corresponding video data streams, and finally verifies the authenticity of the video data streams through DRM digital rights encryption. Wherein, after the camera device obtains the video data stream, the camera device applies for a random number to the internal processor, the internal processor generates a random fingerprint value and records the life cycle value, then the fingerprint value and the life cycle are transmitted to the central server, the central server abstracts the video data stream, the fingerprint value and the life cycle value to generate an abstract value X1, then a private key in the internal processor of the camera device is used for digitally signing the abstract value to generate a signature value Y which is transmitted to the central server through the network, the central server receives the video data stream, extracts the fingerprint value, the life cycle value and the abstract value, then the fingerprint value is compared with the fingerprint value generated in advance by the central server and judges the life cycle, if the fingerprint value does not exist or the random number is inconsistent, the video stream is considered to be a video stream which is not authentic, prompting that the authenticity verification of the video stream fails; if the fingerprint value exists and the life cycle is legal, the video data stream, the fingerprint value and the life cycle value are abstracted by adopting the same abstract algorithm to generate an abstract value X2, a public key in a pre-reserved internal processor of the camera device is used for carrying out signature verification on the abstract value X2 and the signature value Y, if the signature verification is successful, the verification of the video stream authenticity is passed, and if the signature verification is not successful, the verification of the video stream authenticity is failed.
Further, when the video data stream is authenticated by DRM digital rights encryption, the camera device sends the state information of the internal processor to the central server, the central server matches the state information of the internal processor with the preset state information in the central server after receiving the state information of the internal processor sent by the camera device, and when the state information of the internal processor is successfully matched with the preset state information in the central server, the central server inquires the multimedia server in an idle state from the multimedia server cluster and sends the inquired address of the multimedia server in the idle state to the camera device.
Furthermore, after receiving the address of the multimedia server in the idle state sent by the central server, the camera device sends the processed video data stream to the multimedia server in the idle state according to the address.
The invention provides a network live broadcast data processing method, before the network live broadcast of video data is initiated, the identity authentication is carried out on a network platform, meanwhile, the level inspection is carried out on the contents to be live broadcast and the contents which are live broadcast, and the network live broadcast platform is reasonably supervised; meanwhile, the multimedia server in an idle state is inquired, so that correct selection of the multimedia server is facilitated, and further the operation efficiency of the live network platform is improved.
Based on any one of the above embodiments, there is provided a live webcast data processing method, where before the step S1, the method further includes: the camera device receives an initialization authentication request sent by the central server, so that the central server authenticates the basic information of the camera device.
Specifically, when an image controller in the camera device raises a video network live broadcast request to a monitoring device of the central server, the monitoring device of the central server first needs to initiate an initialization authentication request to the camera device, authenticate basic information of the camera device, start a connection request processing module, monitor an 8080 port, judge a message type according to a first byte value, and return a confirmation message if the message type is a registration message; if a new process needs to be started, firstly starting a command service module, monitoring a 7070 port, waiting for a connection request, responding to the request for establishing connection after receiving the connection request, and then receiving the sent command and performing different processing on different commands; and for the video starting command, extracting the node ID number and the camera device ID number in the command, extracting the IP of the node from the configuration table according to the node ID number, then sending a video acquisition command to a video acquisition control module of the node and initiating a connection request to a central server.
The method for processing the live webcast data, provided by the invention, has the advantages that the identity authentication is carried out on the web platform before the video data webcast is initiated, the security of the webcast content is favorably ensured, and a good network environment is favorably built.
Based on any of the above embodiments, a method for processing live webcast data is provided, where in step S1, the preset no-frame-loss condition includes: and the time difference between the receiving time stamp of the video frame and the generating time stamp of the video frame is smaller than a preset time difference threshold value and/or the time difference between the receiving time stamp of the video frame and the generating time stamp of the video frame is smaller than a preset proportion of the time length of the video stream where the video frame is located.
Specifically, an image controller in the camera device needs to perform data stream separation and segmentation processing on image video data for subsequent work, the internal video data is subjected to segmentation processing according to video frames, and a preset marking clock conforming to each segment of a live stream in the camera device is established according to clock information and key frame information in the live stream; and finding out the key frame of the first segment from the separated video data, and taking the clock of the key frame of the first segment in the live stream as a first preset mark clock. If the live broadcast streams comprise a plurality of live broadcast streams, taking the largest clock in key frames of a first segment of the live broadcast streams as a first preset marking clock; according to the first preset marking clock C and the segmentation duration D, calculating a preset marking clock CS of a subsequent segment, and calculating a preset marking clock CS (n) ═ C + (n-1) × D of an nth segment; finding out the clock CIDR (n) of the minimum key frame between CS (n +1) and CS (n) in the separated video data as a preset mark clock of the nth segment; recombining and packaging the video data according to the preset mark clock of each section, and converting the video data into the section which accords with the live stream of the multimedia server; video data satisfying the condition CIDR (n +1) > C' > (CIDR (n)) is regarded as the same segment data. Performing permutation and combination and CS packaging according to the clock sequence of the segmented data to form a clock which is in accordance with the live stream, wherein C' is a clock of each complete video data packet; calculating each segment duration cd (n), cd (n) ═ CIDR (n +1) -CIDR (n);
on the basis of the technical scheme, the video frame of which the time difference between the timestamp of the video frame received by the last camera device and the timestamp of the video frame is smaller than a preset time difference threshold value is determined as a non-frame-loss video; and/or determining the video frame with the time difference between the timestamp of the video frame received by the last camera device and the timestamp of the video frame smaller than the preset proportion of the video stream time length of the video frame as the frame loss free video. For example, the preset no-frame-loss condition may be that the time difference between the receiving time stamp of the video frame and the time stamp of the video frame is less than 85% of the time length of each segment of the preset marking clock of the video stream where the video frame is located.
The invention provides a live network data processing method, which comprises the steps of determining a video frame of which the time difference between a timestamp of the video frame received by a last camera device and the timestamp of the video frame is less than a preset time difference threshold value as a non-frame-lost video; and/or the video frame of which the time difference between the timestamp of the video frame received by the last camera device and the timestamp of the video frame is smaller than the preset proportion of the time length of the video stream where the video frame is located is determined as a frame loss-free video, so that the effectiveness of the video content can be effectively ensured, and the video content can be conveniently and subsequently live broadcast.
Based on any one of the above embodiments, a live webcast data processing method is provided, where step S1 further includes: and the camera device carries out digital encryption processing and right control on the video data stream.
Specifically, the camera device can perform DRM digital copyright encryption authenticity verification on the video data stream and simultaneously perform playing control on the video data stream, hierarchically mark video content according to a preset legal user and illegal user list, process videos according to different safety factors, perform authority control on the output video data stream, perform level division on the video content if pictures which are not suitable for being watched by a specific age group are displayed, classify the videos into live videos watched by an appointed age group, perform access control on network users who watch the live videos in advance, and upload personal information and video head portrait real-time information when the users watch the network live. In addition, the starting time, the playing duration and the caching time of the video frame can be recorded, and the video data stream is divided into video data stream segments to be encrypted in a segmented mode; and/or partially or selectively encrypting the video stream based on metadata parameters of the video stream and the data stream bit rate; the image controller inside the camera device stores the encrypted license in the form of a data chain.
According to the method for processing the live webcast data, the camera device carries out digital encryption processing and authority control on the video data stream, so that the safety of the video data is ensured, reasonable supervision on a live webcast platform is realized, and a good network environment is created.
Based on any of the above embodiments, there is provided a live webcast data processing method, where the state information of the internal processor in step S2 includes: a voltage index in a motherboard chip of the internal processor, a temperature index of the motherboard chip, data load level information of the internal processor, and data throughput rate information of the internal processor.
Specifically, after the video data stream is subjected to the authentication of the DRM digital rights encryption authenticity, the camera device and the central server perform mutual authentication, the camera device performs initialization authentication on an internal processor, and the state information of a core main board chip of the processor is sent and reported to the central server. Specifically, the camera device transmits a voltage index in a main board chip of the internal processor to the central server; and/or transmitting the temperature index in the mainboard chip to a central server; and/or transmitting data load level information of the processor to the central server; and/or transmitting data throughput rate information of the processor to the central server. Wherein:
the voltage operation in the main board chip of the camera module internal processor directly determines the running performance of the system, the imaging chip of the digital image acquisition equipment is generally CCD and CMOS, the working principle of the imaging chip is to convert collected photons into electric charge by utilizing the photoelectric effect, the main difference of the two is that the modes of reading charge signals are different, but finally the charge signals are converted into analog voltage signals. The imaging device in this embodiment may also optionally employ a CMOS image sensor. The high integration of the CMOS image sensor reduces the complexity of the system, can integrate multiple functions on one chip as required, the single chip can finish all electrical functions of the CCD camera, reduces the manufacturing cost, and the reading and processing of the obtained image information become simple and rapid, so that a more flexible small-sized imaging system can be designed, and the CMOS image sensor has the advantages of compatibility with other CMOS circuits, random access of programming of local pixel images and the like;
the temperature in the main board chip of the camera device also affects the network shooting environment, and the poor network live broadcast effect is easily caused by the fact that the temperature is not controlled. When there is a temperature difference between parts of the camera measurement system, the heat conduction coefficients of different materials are different, and for precise camera measurement, not only the temperature, but also the speed and direction of temperature change, the gradient of temperature distribution, and the like, which affect the measurement result. The refractive index of the optical material constituting the lens also changes with temperature, thereby causing changes in the focal length and optical axis of the lens. If the originally calibrated camera parameters are also used in the measurement, measurement errors will be brought. Of course, such measurement errors are generally small and can be tolerated for general computer vision applications that do not aim at measurement, however, for camera measurement applications that aim at high-precision measurement, errors caused by ambient temperature changes must be taken into account, and in live webcasting, users often demand a better viewing effect, and therefore, temperature changes should also be controlled;
data load level measurement of an internal processor of the camera device adopts an MDS cluster load balancing mechanism, each execution node independently calculates respective residual load condition, and then applies for a proper task according to the residual calculation capacity; in this embodiment, a processor of the image pickup apparatus implements load balancing by using a centralized push method, calculates storage pressure and access pressure of each storage node by a Master, and then assigns a task to a storage node with the minimum pressure in a current cluster to execute the task. The reason for adopting the Master to calculate the load pressure is that the cluster is not a calculation type, and the Master has low calculation pressure and cannot cause overload of the Master;
the data throughput rate performance of the internal processor of the camera device also influences the live webcasting effect, live webcasting data has the characteristic of mass data, high-resolution and high-dynamic live webcasting content is high in spatial resolution, high in data rate, short in period and large in data volume, and the remote sensing data volume acquired by the live webcasting platform every day reaches TB level. Therefore, the data throughput rate of the internal processor of the camera device directly determines the quality of the live webcasting and monitoring effects.
In addition, in practical application, the camera device is provided with a two-dimensional code outside, and the central server can directly scan the two-dimensional code to obtain the state information.
According to the network live broadcast data processing method provided by the invention, the camera device sends the voltage index in the main board chip of the internal processor, the temperature index of the main board chip, the data load level information of the internal processor and the data throughput rate information of the internal processor to the central server, so that the interactive authentication of the camera device and the central server is realized, the reasonable supervision of a network live broadcast platform is realized, and the good network environment is favorably built.
Based on any of the above embodiments, a method for processing live webcast data is provided, where the successful matching between the state information and the preset state information in step S2 further includes:
the voltage index is not lower than a preset lowest voltage value and not higher than a preset highest voltage value; and/or the temperature index is not lower than a preset lowest temperature value and the temperature index is not higher than a preset highest temperature value; and/or the data load level value is not higher than a preset maximum data load level value; and/or the data throughput rate value is not lower than a preset minimum data throughput rate value.
Specifically, the central server receives the state information sent by the camera device, and if the state information is successfully matched with the state information preset in the central server, the central server queries the multimedia server in an idle state from the multimedia server cluster. And if the matching is not successful, the central server is disconnected with the camera device. Wherein, the successful matching of the state information and the state information preset inside the central server specifically comprises the following steps:
the voltage index in a mainboard chip of an internal processor of the camera device is not lower than the lowest voltage value preset in the central server and not higher than the highest voltage value preset in the central server; and/or the temperature index of the mainboard chip is not lower than the lowest temperature value preset in the central server and not higher than the highest temperature value preset in the central server; and/or the data load level value of the processor in the camera device is not higher than the highest data load level value preset in the central server; and/or the data throughput rate value of the internal processor is not lower than the lowest data throughput rate value preset in the central server. The concrete implementation is as follows:
because different rated voltages exist in different types of mainboard chips and the voltage environments of different types of single chip transceiver chips in the normal working environment are different, the rated voltage and the rated temperature of the single chip transceiver chips are stored for several main single chips in the central server, wherein for TRF9200, the rated voltage is 2.2V-3.6V, the rated temperature is-20 to +60 ℃, the rated voltage is 2.4C-5.0C for RF2545 RFMD, and the rated temperature is-40 to +85 ℃;
the central server internally stores the maximum reply time T1, sets an upper extreme value and a lower extreme value of the data load level for the internal processor of the camera device under the constraint of the requirement of the maximum reply time, and objectively predicts the data processing load level in the current camera through the data relationship of the upper and lower extreme values and the load information communication between the camera nodes;
the data throughput rate of the internal processor of the camera device is characterized in multiple ways inside the central server, and the lowest standard rate V1 is used for limiting the lowest configuration level of the internal processor of the camera device; the highest standard rate V2 is used for limiting the highest configuration level of the camera device internal processor, and the lowest effective rate V3 is used for limiting the lowest effective processing rate of the camera device internal processor for the video stream; the maximum effective rate V4 is used to limit the effective processing maximum rate of the video stream by the camera internal processor.
According to the network live broadcast data processing method provided by the invention, the central server matches the state information sent by the camera device with the state information preset in the central server, so that the interactive authentication of the camera device and the central server is realized, the reasonable supervision on a network live broadcast platform is further realized, and a good network environment is favorably built.
Based on any of the above embodiments, there is provided a method for processing live webcast data, where the querying a multimedia server in an idle state from a multimedia server cluster in step S2 further includes:
if the number of the types of the multimedia data processed in the multimedia server is not greater than a preset data number threshold value, the multimedia server is determined to be a multimedia server in an idle state; and/or if the CPU running speed of the multimedia server does not exceed a preset speed threshold, determining the multimedia server to be in an idle state; and/or if the ratio of the unused storage space to the used storage space of the multimedia server is not higher than a preset ratio, the multimedia server is considered to be a multimedia server in an idle state; and/or if the number of the connection using users of the multimedia server is not more than a preset user number threshold value, the multimedia server is considered to be the server in an idle state.
Specifically, the central server queries a server in an idle state from the multimedia server cluster, and sends an address of the multimedia server in the idle state to the image pickup device. Wherein querying the multimedia server in an idle state from the cluster of multimedia servers further comprises: if the number of the types of the multimedia data processed in a certain multimedia server is not larger than a preset data number threshold value, determining that the multimedia server is in an idle state; and/or if the running speed of a CPU in a certain multimedia server does not exceed a preset speed threshold, determining that the multimedia server is in an idle state; and/or if the ratio of the unused storage space to the used storage space in a certain multimedia server is not higher than the preset ratio, determining that the multimedia server is in an idle state; and/or if the number of the connection using users of a certain multimedia server is not larger than a preset user number threshold value, determining that the multimedia server is a server in an idle state.
In practical applications, the processing speed of the multimedia server is limited, the multimedia types are numerous, and different storage spaces are required to be established for storing multimedia file data for each different multimedia, so that a multimedia server storing the multimedia types with the number meeting specific requirements needs to be selected to perform communication operations with the camera device, for example, a number threshold of the types of the multimedia data is set, for example, the number of the types of the multimedia is 4, and the video data stream file in the MP4/AVI/3GP/MPEG format is allowed to establish a communication relationship with the multimedia server;
for a video live broadcast system, the bottleneck of hardware resources mainly focuses on the master frequency of a server CPU, the CPU time allocated by the server to each task thread, the memory of the server, and the throughput rate of the bus data. The multimedia server cluster comprises N proxy servers, each user has a CPU utilization rate and a running rate, the CPU usage space and the memory usage space of each cluster node are the sum of the CPU usage space and the memory usage space of all the users, the sizes of the CPU and the memory on the cluster nodes are not variable, each server cluster node has two indexes, one is the preset rate threshold value, the other is the preset ratio value, the upper limit value cannot be exceeded, and if any index is exceeded, the server cluster node is not considered as a multimedia server node in an idle state.
When the multimedia server receives the request of the camera device, a connection pool management system in the server detects the current preset user quantity threshold value according to the preset maximum connection number and the preset connection number. If the requirement of the number of the connected users in the request of the current camera device does not reach a preset user number threshold value in the server, the server is considered as an idle state server; if the number of connected users in the request of the camera device reaches or exceeds a preset user number threshold value in the server, the request of the camera device cannot be served immediately, and only can enter a sleep state through a synchronization mechanism. For some users of the camera device with a requirement on response time, in order to avoid long waiting time, a timing mechanism can be set, when a specified timeout time comes and the camera device request is not responded, the blocking is terminated, and an empty connection is returned.
The invention provides a network live broadcast data processing method, before the network live broadcast of video data is initiated, the identity authentication is carried out on a network platform, meanwhile, the level inspection is carried out on the contents to be live broadcast and the contents which are live broadcast, and the network live broadcast platform is reasonably supervised; meanwhile, the multimedia server in an idle state is inquired, so that correct selection of the multimedia server is facilitated, and further the operation efficiency of the live network platform is improved.
Based on any of the above embodiments, a method for processing live webcast data is provided, where before sending the video data stream to the multimedia server in the idle state in step S3, the method further includes: and the camera device establishes communication connection with the multimedia server in the idle state.
Specifically, after the camera device receives the address of the multimedia server in the idle state, the camera device establishes a data stream communication connection with the multimedia server, records live data source information, and establishes a network transmission channel. The concrete implementation is as follows:
the method comprises the following steps that a data sharing mechanism is established between an internal memory of the camera device and a multimedia server, and the content of a video frame stored in the camera device is in data synchronization with the multimedia server; recording interface information of a live broadcast signal source to which the camera device belongs, and capturing the live broadcast signal source information by the multimedia server for operations such as live broadcast, uploading, releasing, recording and broadcasting of network videos; the network interface inside the camera device establishes a bidirectional link with the network interface of the multimedia server, and the data inside the camera device directly establishes communication interaction with the VOD service of the multimedia server.
According to the method for processing the live webcast data, the camera device needs to establish communication connection with the multimedia server in the idle state before sending the video data stream to the multimedia server in the idle state, so that effective transmission of the video data stream is ensured, and further the running efficiency of a live webcast platform is improved.
Based on any of the above embodiments, providing a method for processing live webcast data, where the sending the video data stream to the multimedia server in the idle state in step S3 further includes: and sending the video data stream to the multimedia server in the idle state by utilizing an RTMP protocol.
Specifically, the video data stream is transmitted to the multimedia server by the RMTP protocol inside the camera device, which is specifically implemented as follows: the camera device initializes an RTMP data structure through a lib built-in library, then initiates a session connection message to the multimedia server, and starts to circularly send the message when the connection is successful, wherein a timestamp and the data size need to be set; video files in MP4 format in the camera device are distributed to the URL appointed by the multimedia server in the form of video data stream, and the URL of any RTMP can be distributed to any URL of any RTMP by modifying the file name and the URL of the RTMP.
The method comprises the following steps that after video information is collected by a camera device, the video information is converted into an analog video signal, and an internal controller of the camera device receives the analog video signal collected by the camera device, converts the analog video signal into a network video signal and stores a video record; the multimedia server is connected with the video publishing end of the monitoring video pushing end through the Internet; the central server is used for converting the streaming media format video in the multimedia server file conforming to the RTMP protocol into a fragment format video stream; the multimedia server is connected with the central server through the internet, generates a network unique identity UID corresponding to the video configuration information, and transmits the network unique identity UID to the central server. The central server records the unique network identity UID, stores the video configuration information in an internal database, and generates and records the URL address pointing to the video stream in the fragment format stored in the multimedia server. Based on the URL address, the multimedia server plays live broadcast or recorded broadcast content in the camera device.
According to the method for processing the live webcast data, the camera device sends the video data stream to the multimedia server in the idle state by utilizing the RTMP protocol, so that the safety of the video data stream transmission process can be effectively ensured, the safety of live webcast content can be ensured, and a good network environment can be created.
Fig. 3 is a schematic diagram of an overall structure of a live webcast data processing system according to an embodiment of the present invention, and as shown in fig. 3, the present invention provides a live webcast data processing system including a camera 31, a central server 32, and a multimedia server 33, where:
the camera device 31 is configured to acquire a video frame meeting a preset no-frame-loss condition, convert the video frame into a video data stream, perform DRM digital rights encryption authenticity verification on the video data stream, and send state information of the internal processor to the central server 32 when the video data stream passes the DRM digital rights encryption authenticity verification;
the central server 32 is configured to match the state information with preset state information, and when the state information is successfully matched with the preset state information, query a multimedia server 33 in an idle state from a multimedia server cluster, and send an address of the multimedia server in the idle state to the image pickup device 31, so that the image pickup device 31 sends the video data stream to the multimedia server 33 in the idle state.
Specifically, in this embodiment, the image capturing device 31 captures video frames through the image controller, acquires video frames meeting the preset no-frame-loss condition from the captured video frames, processes the video frames meeting the preset no-frame-loss condition to generate corresponding video data streams, and finally performs DRM digital rights encryption authenticity verification on the video data streams to verify the authenticity of the video data streams. Wherein, after the camera device 31 obtains the video data stream, the camera device 31 applies for a random number to the internal processor, the internal processor generates a random fingerprint value and records a life cycle value thereof, then the fingerprint value and the life cycle are transmitted to the central server 32, the central server 32 abstracts the video data stream, the fingerprint value and the life cycle value to generate an abstract value X1, then a private key in the internal processor of the camera device 31 is used for digitally signing the abstract value to generate a signature value Y which is transmitted to the central server 32 through the network, the central server 32 receives the video data stream, extracts the fingerprint value, the life cycle value and the abstract value thereof, then the fingerprint value is compared with the fingerprint value generated in advance by the central server 32 and judges the life cycle thereof, if the fingerprint value does not exist or the random number is not consistent, the video stream is considered to be possibly unreal, and the video stream authenticity verification is prompted to fail; if the fingerprint value exists and the life cycle is legal, the video data stream, the fingerprint value and the life cycle value are abstracted by adopting the same abstraction algorithm to generate an abstract value X2, and a public key in a pre-reserved internal processor of the camera device 31 is used for carrying out signature verification on the abstract value X2 and the signature value Y, if the signature verification is successful, the verification of the video stream authenticity is passed, otherwise, the verification of the video stream authenticity is failed.
Further, when the video data stream passes the verification of the authenticity of the DRM digital rights encryption, the camera device 31 sends the state information of the internal processor to the central server 32, the central server 32 matches the state information of the internal processor with the state information preset in the central server after receiving the state information of the internal processor sent by the camera device 31, and when the state information of the internal processor is successfully matched with the state information preset in the central server, the central server queries the multimedia server 33 in an idle state from the multimedia server cluster, and sends the queried address of the multimedia server 33 in the idle state to the camera device 31.
Further, the image pickup device 31 receives the address of the multimedia server 33 in the idle state transmitted from the center server 32, and transmits the video data stream processed as described above to the multimedia server 33 in the idle state in accordance with the address.
The network live broadcast data processing system carries out identity authentication on a network platform before the network live broadcast of the video data is initiated, and simultaneously carries out level inspection on the contents to be live broadcast and the contents already live broadcast so as to reasonably supervise the network live broadcast platform; meanwhile, the multimedia server in an idle state is inquired, so that correct selection of the multimedia server is facilitated, and further the operation efficiency of the live network platform is improved.
Finally, the method of the present application is only a preferred embodiment and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (2)
1. A method for processing network live broadcast data is characterized by comprising the following steps:
s1, the camera device obtains the video frame which accords with the condition of no frame loss, converts the video frame into the video data stream, and verifies the DRM digital copyright encryption authenticity of the video data stream;
s2, when the video data stream passes the DRM digital copyright encryption authenticity verification, the camera device sends the state information of the internal processor to the central server, so that the central server matches the state information with the preset state information, and when the state information is successfully matched with the preset state information, the multimedia server in an idle state is inquired from the multimedia server cluster;
s3, the camera device receives the address of the multimedia server in the idle state sent by the central server and sends the video data stream to the multimedia server in the idle state; step S1 is preceded by: the camera device receives an initialization authentication request sent by the central server, so that the central server authenticates basic information of the camera device; the preset no-frame-loss condition in step S1 includes: the time difference between the receiving time stamp of the video frame and the generating time stamp of the video frame is smaller than a preset time difference threshold value and/or the time difference between the receiving time stamp of the video frame and the generating time stamp of the video frame is smaller than a preset proportion of the time length of a video stream where the video frame is located; and/or determining the video frame of which the time difference between the timestamp of the video frame received by the last camera device and the timestamp of the video frame is less than a preset time difference threshold value as a non-frame-loss video; and/or identifying the video frame of which the time difference between the timestamp of the video frame received by the last camera device and the timestamp of the video frame is smaller than the preset proportion of the video stream time of the video frame as a non-frame-loss video; the step S1 further includes: the camera device carries out digital encryption processing and authority control on the video data stream;
the state information of the internal processor in step S2 includes: a voltage indicator in a motherboard chip of the internal processor, a temperature indicator of the motherboard chip, data load level information of the internal processor, and data throughput rate information of the internal processor;
the step S2 of successfully matching the status information with the preset status information further includes:
the voltage index is not lower than a preset lowest voltage value and not higher than a preset highest voltage value; and/or the temperature index is not lower than a preset lowest temperature value and the temperature index is not higher than a preset highest temperature value; and/or the data load level value is not higher than a preset highest data load level value; and/or the data throughput rate value is not lower than a preset lowest data throughput rate value;
the querying the multimedia server in an idle state from the multimedia server cluster in step S2 further includes:
if the number of the types of the multimedia data processed in the multimedia server is not greater than a preset data number threshold value, determining that the multimedia server is in an idle state; and/or if the CPU running speed of the multimedia server does not exceed a preset speed threshold, determining that the multimedia server is in an idle state; and/or if the ratio of the unused storage space of the multimedia server to the used storage space is not higher than a preset ratio, determining that the multimedia server is in an idle state; and/or if the number of the connection using users of the multimedia server is not larger than a preset user number threshold value, determining that the multimedia server is a server in an idle state;
before the step S3 of sending the video data stream to the multimedia server in the idle state, the method further includes:
the camera device establishes communication connection with the multimedia server in the idle state; the step S3 of sending the video data stream to the multimedia server in the idle state further includes: sending the video data stream to the multimedia server in an idle state by utilizing an RTMP protocol;
when an image controller in the camera device raises a video network live broadcast request to a monitoring device of a central server, the monitoring device of the central server firstly needs to initiate an initialization authentication request to the camera device, authenticates basic information of the camera device, starts a connection request processing module, monitors an 8080 port, judges the type of the message according to a first byte value, and returns a confirmation message if the message is a registration message; if a new process needs to be started, firstly, a command service module is started, a 7070 port is monitored, a connection request is waited, after the connection request is received, the connection establishment is responded, then the sent command is received, and different processing is carried out on different commands; for a video starting command, extracting a node ID number and a camera device ID number in the command, extracting the IP of the node from a configuration table according to the node ID number, then sending a video acquisition command to a video acquisition control module of the node and initiating a connection request to a central server;
after the camera device obtains the video data stream, the camera device applies for a random number to the internal processor, the internal processor generates a random fingerprint value and records a life cycle value of the random fingerprint value, then the fingerprint value and the life cycle are transmitted to the central server, the central server abstracts the video data stream, the fingerprint value and the life cycle value to generate an abstract value X1, then a private key in the internal processor of the camera device is used for digitally signing the abstract value to generate a signature value Y which is transmitted to the central server through a network, the central server receives the video data stream, extracts the fingerprint value, the life cycle value and the abstract value, then compares the fingerprint value with a fingerprint value generated in advance by the central server and judges the life cycle, if the fingerprint value does not exist or the random number is inconsistent, the video stream is considered to be an unreal video stream, prompting that the authenticity verification of the video stream fails; if the fingerprint value exists and the life cycle is legal, the video data stream, the fingerprint value and the life cycle value are abstracted by adopting the same abstract algorithm to generate an abstract value X2, a public key in a pre-reserved internal processor of the camera device is used for carrying out signature verification on the abstract value X2 and the signature value Y, if the signature verification is successful, the verification of the video stream authenticity is passed, otherwise, the verification of the video stream authenticity is failed.
2. A network live broadcast data processing system is characterized by comprising a camera device, a central server and a multimedia server, wherein:
the camera device is used for acquiring a video frame meeting a preset condition of no frame loss, converting the video frame into a video data stream, performing DRM digital rights encryption authenticity verification on the video data stream, and sending state information of the internal processor to the central server when the video data stream passes the DRM digital rights encryption authenticity verification; data load level measurement of an internal processor of the camera device adopts an MDS cluster load balancing mechanism, each execution node independently calculates respective residual load condition, and then applies for a proper task according to the residual calculation capacity; a processor of the camera device realizes load balancing by adopting a centralized push mode, a Master calculates the storage pressure and the access pressure of each storage node, and then assigns a task to the storage node with the minimum pressure in the current cluster to execute; the reason for adopting the Master to calculate the load pressure is that the cluster is not a calculation type, and the Master has low calculation pressure and cannot cause overload of the Master;
the central server is used for matching the state information with preset state information, inquiring a multimedia server in an idle state from a multimedia server cluster when the state information is successfully matched with the preset state information, and sending the address of the multimedia server in the idle state to the camera device so that the camera device sends the video data stream to the multimedia server in the idle state;
when an image controller in the camera device raises a video network live broadcast request to a monitoring device of a central server, the monitoring device of the central server firstly needs to initiate an initialization authentication request to the camera device, authenticates basic information of the camera device, starts a connection request processing module, monitors an 8080 port, judges the type of the message according to a first byte value, and returns a confirmation message if the message is a registration message; if a new process needs to be started, firstly, a command service module is started, a 7070 port is monitored, a connection request is waited, after the connection request is received, the connection establishment is responded, then the sent command is received, and different processing is carried out on different commands; for a video starting command, extracting a node ID number and a camera device ID number in the command, extracting the IP of the node from a configuration table according to the node ID number, then sending a video acquisition command to a video acquisition control module of the node and initiating a connection request to a central server;
after the camera device obtains the video data stream, the camera device applies for a random number to the internal processor, the internal processor generates a random fingerprint value and records a life cycle value of the random fingerprint value, then the fingerprint value and the life cycle are transmitted to the central server, the central server abstracts the video data stream, the fingerprint value and the life cycle value to generate an abstract value X1, then a private key in the internal processor of the camera device is used for digitally signing the abstract value to generate a signature value Y which is transmitted to the central server through a network, the central server receives the video data stream, extracts the fingerprint value, the life cycle value and the abstract value, then compares the fingerprint value with a fingerprint value generated in advance by the central server and judges the life cycle, if the fingerprint value does not exist or the random number is inconsistent, the video stream is considered to be an unreal video stream, prompting that the authenticity verification of the video stream fails; if the fingerprint value exists and the life cycle is legal, the video data stream, the fingerprint value and the life cycle value are abstracted by adopting the same abstract algorithm to generate an abstract value X2, a public key in a pre-reserved internal processor of the camera device is used for carrying out signature verification on the abstract value X2 and the signature value Y, if the signature verification is successful, the verification of the video stream authenticity is passed, otherwise, the verification of the video stream authenticity is failed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810031400.7A CN108347621B (en) | 2018-01-12 | 2018-01-12 | Network live broadcast data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810031400.7A CN108347621B (en) | 2018-01-12 | 2018-01-12 | Network live broadcast data processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108347621A CN108347621A (en) | 2018-07-31 |
| CN108347621B true CN108347621B (en) | 2020-05-22 |
Family
ID=62960720
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810031400.7A Expired - Fee Related CN108347621B (en) | 2018-01-12 | 2018-01-12 | Network live broadcast data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108347621B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109412886B (en) * | 2018-09-26 | 2020-05-05 | 北京文香信息技术有限公司 | Distributed recording and broadcasting system, communication detection method thereof and recording and broadcasting control method |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020175995A1 (en) * | 2001-05-26 | 2002-11-28 | Marc Sleeckx | Video surveillance system |
| US20040151315A1 (en) * | 2002-11-06 | 2004-08-05 | Kim Hee Jean | Streaming media security system and method |
| CN1964479A (en) * | 2006-11-24 | 2007-05-16 | 清华大学 | Method for realizing management of digital copyright based on stream media broadcast |
| CN105657334B (en) * | 2014-11-14 | 2018-12-18 | 中国移动通信集团公司 | A kind of method of transmission of video, video monitoring platform and video monitoring equipment |
| CN106209921A (en) * | 2016-09-26 | 2016-12-07 | 周世平 | A kind of IP Camera and application process thereof |
-
2018
- 2018-01-12 CN CN201810031400.7A patent/CN108347621B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN108347621A (en) | 2018-07-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021208952A1 (en) | Block chain-based image data recording, obtaining and verifying | |
| US9479845B2 (en) | System and method for auto content recognition | |
| US10275578B2 (en) | Continuous content identification of broadcast content | |
| US20160381436A1 (en) | System and method for auto content recognition | |
| JP4352653B2 (en) | Video content management system | |
| US20180249190A1 (en) | Method and apparatus for cloud storage and cloud download of multimedia data | |
| US7269601B2 (en) | Information delivery system, information delivery method, information delivery server, content delivery server and client terminal | |
| US20150138375A1 (en) | Method and apparatus in a motion video capturing system | |
| WO2014169828A1 (en) | Method, device, and system for playing surveillance video | |
| US11451866B2 (en) | Systems and methods for data processing, storage, and retrieval from a server | |
| CN112165596A (en) | Monitoring video data transmission method and device, terminal equipment and storage medium | |
| EP3659311A1 (en) | Data stream integrity | |
| CN108650527B (en) | EPG security monitoring method and system based on IPTV | |
| CN108347621B (en) | Network live broadcast data processing method and system | |
| CN108810475B (en) | Android video monitoring device based on Onvif standard and Sip protocol | |
| US11128678B2 (en) | Multi-platform digital rights management for placeshifting of multimedia content | |
| CN108965272A (en) | A kind of method of detection network camera weak password | |
| KR100996449B1 (en) | System and Method for Managing Supervisory Video Using IP Network | |
| US10712994B2 (en) | Image compressing method, image reconstructing method, image compressing device, image reconstructing device, image compressing program product, and image reconstructing program product | |
| Kang et al. | Privacy-preserving multimedia cloud computing via compressive sensing and sparse representation | |
| CN113014956B (en) | Video playing method and device | |
| CN114143616A (en) | Target video processing method and system, storage medium and electronic device | |
| CN106257461B (en) | Data storage method and device | |
| EP4016969A1 (en) | Methods for data processing, storage, and retrieval from a server | |
| Rainer et al. | Merge and forward: Self-organized inter-destination multimedia synchronization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20200522 Termination date: 20210112 |